D3L-SLAM: A Comprehensive Hybrid Simultaneous Location and Mapping System with Deep Keypoint, Deep Depth, Deep Pose, and Line Detection

Abstract

Robust localization and mapping are crucial for autonomous systems, but traditional handcrafted feature-based visual SLAM often struggles in challenging, textureless environments. Additionally, monocular SLAM lacks scale-aware depth perception, making accurate scene scale estimation difficult. To address these issues, we propose D3L-SLAM, a novel monocular SLAM system that integrates deep keypoints, deep depth estimates, deep pose priors, and a line detector. By leveraging deep keypoints, which are more resilient to lighting variations, our system improves the robustness of visual SLAM. We further enhance perception in low-texture areas by incorporating line features in the front-end and mitigate scale degradation with learned depth estimates. Additionally, point-line feature constraints optimize pose estimation and mapping through a tightly coupled point-line bundle adjustment (BA). The learned pose estimates refine the feature matching process during tracking, leading to more accurate localization and mapping. Experimental results on public and self-collected datasets show that D3L-SLAM significantly outperforms both traditional and learning-based visual SLAM methods in localization accuracy.

1. Introduction

Simultaneous localization and mapping (SLAM) is a foundational technology that plays a crucial role in the autonomous navigation of robots, underwater platforms [1], and self-driving vehicles [2]. It enables unmanned systems to accurately determine their position within an unknown environment while simultaneously constructing a detailed map of the surroundings in real-time. By integrating data from various sensors such as LiDAR, cameras, and inertial measurement units (IMUs), SLAM provides a comprehensive understanding of the environment, which is essential for safe and efficient operation. In robotics, visual SLAM is widely utilized due to its cost-effectiveness and versatility, aiding in tasks like industrial automation, exploration, and service delivery. Drones rely on SLAM for obstacle avoidance and mapping in GPS-denied environments, while autonomous vehicles depend on it to navigate and make decisions in complex traffic scenarios. Overall, visual SLAM is indispensable for the development of intelligent systems capable of operating autonomously in the physical world.

Visual SLAM can be broadly categorized into two approaches based on the type of visual features: handcrafted feature-based and direct methods. The feature-based approach is widely adopted due to its efficiency and precision. In this method, the SLAM front-end tracks keypoints in adjacent frames and optimizes positioning and mapping using multi-view geometry optimization techniques. However, handcrafted local features rely on low-level image information, such as pixel gradients, for keypoint detection and descriptor generation. These features are sensitive to changes in lighting conditions, leading to unstable tracking and potential SLAM failures. Furthermore, monocular SLAM lacks the ability to perceive the depth of keypoints, resulting in ambiguous scale estimation for mapping.

To address the limitations of handcrafted feature-based methods, some studies have incorporated deep learning techniques into visual SLAM [3,4,5,6,7,8,9,10,11,12,13,14,15]. These approaches, known as learning-based visual SLAM, can be divided into end-to-end learning methods and learning-feature-based SLAM methods. End-to-end learning approaches directly convert images into pose or depth estimates, achieving globally consistent scale estimation by training on large-scale datasets. However, these methods lack additional optimization modules and cannot correct accumulated localization errors. In contrast, learning-feature-based SLAM combines deep neural networks for visual feature extraction with traditional multi-view geometry models in the back-end, offering a hybrid approach of “geometric model + learning.” These methods leverage self-supervised local features for scene perception, exhibiting robustness in complex environments with significant lighting variations. However, they may struggle in texture-sparse scenes, where the front-end lacks global feature perception, and monocular keypoints do not provide depth information, leading to scale degradation. Additionally, feature matching in these methods often relies on a constant velocity model, which may not align with real-world motion, further limiting their robustness and performance.

In light of these challenges, we propose a novel hybrid visual SLAM system that combines end-to-end visual methods with learning-feature-based approaches. Our system, D3L-SLAM, integrates Deep keypoints, Deep depth estimation, Deep pose priors, and a Line detector to enhance the SLAM front-end’s performance in complex lighting conditions. By incorporating learned local features, trained in a self-supervised manner, into the tracking module, D3L-SLAM improves feature tracking in challenging environments. To ensure stable performance in texture-sparse scenarios, we introduce line features into the front-end. Additionally, we integrate self-supervised pose estimation as a motion model for adjacent frames, refining the matching of point–line features. To address the inherent scale inconsistency in monocular SLAM, we employ globally consistent depth estimation to transform monocular SLAM into a virtual stereo SLAM system, ensuring that pose estimation and mapping are scale-consistent. In the back end, we construct a bundle adjustment for learned point and line features to optimize pose estimation, and we develop an online binary bag-of-words (BoW) model to correct accumulated errors.

We validate our proposed D3L-SLAM on both the public benchmark, i.e., the KITTI dataset, and our self-collected dataset, comparing its performance against model-based benchmarks like ORB-SLAM3, LDSO, and VISO-M, as well as the learning-based benchmark LIFT-SLAM. The experimental results demonstrate that D3L-SLAM achieves a 39.42% improvement in translation accuracy compared to the traditional handcrafted feature-based ORB-SLAM3 and a 75.95% improvement compared to the learning-based LIFT-SLAM. On our self-collected dataset, D3L-SLAM outperforms ORB-SLAM3 by 43.36% in terms of localization accuracy.

In summary, our contributions are as follows:

• We propose D3L-SLAM, a hybrid monocular visual SLAM system that integrates deep keypoints, deep pose priors, deep depth estimation, and a line detector, enabling robust operation in complex, low-texture environments.
• To enhance SLAM performance in low-texture scenarios, we incorporate point–line feature constraints to optimize pose estimation and mapping through the construction of a tightly coupled point–line bundle adjustment (BA).
• To ensure scale-consistent pose and map estimates in monocular visual SLAM, we employ self-supervised depth estimation with RGB images to form a pseudo-RGBD sensor and integrate a virtual baseline to create a pseudo-stereo SLAM system.
• We conducted extensive experiments on both public and self-collected datasets, demonstrating that D3L-SLAM significantly outperforms representative traditional SLAM (e.g., ORB-SLAM3) and learning-based SLAM (e.g., LIFT-SLAM).

2. Related Works

2.1. Deep-Learning-Based Visual SLAM

The integration of deep learning neural networks into visual SLAM has emerged as a promising research direction for end-to-end pose estimation and depth perception. Learning-based SLAM methods can be categorized into supervised and self-supervised approaches based on their supervision types. For instance, DeepVO [5] represents a typical supervised odometry approach that uses convolutional neural networks (CNNs) for visual feature extraction and fully connected layers for pose regression. It follows the local mapping approach of traditional SLAM, optimizing visual features of image sequences using LSTM. DROID-SLAM [16] merges pose and depth estimation tasks, constructing a complete learned SLAM system by integrating a hidden state optimization model based on GRU [17] with traditional bundle adjustment models. NeuralBundler [18] employs both stereo DepthNet and PoseNet, along with monocular and stereo photometric loss functions, to build a scalable framework. It also utilizes a bag-of-words (BoW) model for loop closure detection and constructs a pose graph optimization model to correct positioning errors. DeepSLAM [9] features three modules—mapping net, tracking net, and pretrained loop net—dedicated to depth estimation, pose estimation, and loop closure, respectively.

In other approaches, [19,20] leverage the optical flow estimation network PWC-Net [21] to track keypoints between adjacent frames and construct an epipolar geometry model for relative pose estimation. [19] integrates DepthNet’s depth estimates to form a photometric loss, using triangulation to maintain depth scale consistency by constructing 3D spatial points corresponding to matching keypoints. The framework adopts a self-supervised training approach, initially constructing photometric loss with PWC-Net and then jointly optimizing DepthNet and PWC-Net parameters.

2.2. Hybrid Visual SLAM

Combining model-based and learning-based SLAM approaches to form a hybrid system leverages the strengths of both methods. GCN-SLAM [11] utilizes a deep learning feature extraction algorithm [22] to generate binary descriptors similar to ORB descriptors, which are directly applicable to SLAM. By deploying GCN on the front-end of the ORB-SLAM2 system and using a single-layer image pyramid for feature extraction, the algorithm improves efficiency. GCN-SLAM also trains a BoW model offline for loop node detection and closure, showing improved performance over ORB-SLAM2 in indoor scenes. DXSLAM [10] incorporates learning-based floating local features from [23] and constructs a loop node recognition module using the fast bag-of-words model FBow [24], which is faster than DBow2 and DBow3. DXSLAM further integrates NetVLAD [25] for global feature extraction and relocalization. It uses NetVLAD to identify candidate loop keyframes similar to the current frame’s global feature, then performs local feature extraction and loop node detection using a BoW model built with FBow. LIFT-SLAM [12] uses a learned LEFT extractor for feature extraction and constructs a loop closure module with an offline-trained BoW model. Despite this, experimental results indicate only modest improvements compared to traditional VSLAM methods. SC-Depth [14] proposes a pseudo RGBD VSLAM system, enhancing monocular SLAM by incorporating depth and pose estimation for improved perception and feature tracking. Ref. [26] combines self-supervised depth labels as a pseudo depth sensor and constructs a pseudo stereo SLAM with a virtual baseline to improve localization performance. AirVO [3] uses SuperPoint [27] for feature extraction and stereo images as input, enhancing perception in low-texture environments by also extracting line features. The framework employs Nvidia TensorRT for real-time inference, showing superior performance in low-light conditions compared to model-based visual VO or SLAM. However, AirVO lacks a loop closure module, which limits its ability to correct accumulated errors. SP-Loop [15] converts SuperPoint descriptors into 32-bit integer descriptors through binarization and trains an offline BoW model for loop closure. SL-SLAM [13] combines stereo and IMU data to form a robust feature extraction front-end for scenes with poor lighting. It uses SuperPoint with LightGlue [28] for feature extraction and matching, trains a BoW model based on SuperPoint features using DBow3, and performs loop correction with this BoW model. SL-SLAM demonstrates superior performance across various datasets compared to model-based ORB-SLAM3. Ref. [29] introduces depth estimates generated by the LiDAR depth completion network to form an RGB-L visual SLAM system, benefiting from the precise global absolute scale of LiDAR for accurate pose and depth estimation. D3VO [4] integrates learned pose and depth priors into the direct method DSO [30], enhancing the front end’s feature matching rate and serving as an initial value for depth filtering. However, D3VO lacks a loop closure module and is unable to perform error correction. These two works [31,32] also introduce line features in SLAM. Among them, PLDS-SLAM [31] uses semantic segmentation to identify dynamic objects in the scene, and when estimating pose and mapping, it removes the negative impact of dynamic objects.

A comparison of these methods with the proposed D3L-SLAM is summarized in

Table 1. A comparison of key visual SLAM modules of our proposed D3L-SLAM with other representative works.

Method	Deep Keypoint	Line Detector	Deep Depth	Deep Pose	Loopclosing
GCN-SLAM	✓
DX-SLAM	✓				✓
LIFT-SLAM	✓	✓			✓
SC-Depth			✓	✓	✓
Pseudo RGBD-SLAM			✓		✓
AirVO	✓	✓
SL-SLAM	✓	✓			✓
D3VO			✓	✓
PLDS-SLAM		✓			✓
RGBL-SLAM			✓		✓
D3L-SLAM (Ours)	✓	✓	✓	✓	✓

. The table highlights how D3L-SLAM integrates deep learning techniques more comprehensively.

3. Hybrid Visual SLAM with Deep Keypoints, Depths, Poses, and Line Features

3.1. Framework

Figure 1 illustrates our proposed D3L-SLAM, a novel hybrid SLAM framework that integrates learning-based keypoints, poses, depths, and line detectors. In the front end, we utilize a learning-based local feature extractor to identify keypoints from images. To enhance feature detection in low-texture regions, we incorporate a handcrafted line extractor. Additionally, to ensure that the detected features are scaled, we introduce depth estimates from a self-supervised depth estimation network. Although these depth estimates lack absolute scale, they maintain global consistency. We further transform monocular SLAM into virtual stereo SLAM by designing a virtual baseline. Leveraging these learned depths, D3L-SLAM can recognize relative object distances in the scene, mitigating scale drift caused by insufficient tracking features. In traditional model-based SLAM, the initial selection range for adjacent frame feature matching is determined by a uniform motion model. However, this model is often too idealized and does not reflect real-world scenarios, thereby limiting feature matching accuracy. To address this limitation, we introduce relative pose estimates derived from a self-supervised pose estimation network as a prior. These estimates, obtained through extensive data training with self-supervised methods based on photometric loss, are more accurate and robust compared to uniform motion models. Finally, in the back end, we design an online learning bag-of-words (BoW) module that uses binary descriptors for online training. This module converts float descriptors from selected keyframes into binary descriptors. The online learning-based BoW addresses the issue of low generalization in learning-based descriptors. The detected loop closures are then utilized to correct accumulated errors in pose estimation.

3.2. Integrating Line Features and Deep Keypoints in the Front End

3.2.1. Line Feature Detection

To enhance the stability and robustness of feature extraction, we detect line features alongside point features. Point features typically capture keypoints with significant gradient changes, while line features represent regions composed of sets of points with such changes. As a result, line features depict object edges in the image, providing stable and recognizable information from various perspectives. Introducing line features into the SLAM front-end tracking module significantly improves feature tracking stability, especially in scenes with drastic lighting variations.

In our feature tracking module, we employ the line segment detector (LSD) [33], a method for extracting line segment features from images. We then compute binary descriptors for the line features using the line band descriptor (LBD) [34] and track the line segments accordingly. As shown in Equation (1), our LSD feature extraction method first utilizes a two-dimensional kernel function to calculate the image gradient, which includes the horizontal and vertical components of the image coordinates $\left(\mathbf{G}\right(u),\mathbf{G}(v\left)\right)$. Here, $\mathbf{I}(u,v)$ represents the pixel value at the coordinate point $(u,v)$ in the image.

$$\begin{array}{cc}\mathbf{G}\left(u\right)\hfill & ={\displaystyle \frac{\mathbf{I}(u+1,v)+\mathbf{I}(u+1,v+1)-\mathbf{I}(u,v)-\mathbf{I}(u,v+1)}{2}}\hfill \\ \mathbf{G}\left(v\right)\hfill & ={\displaystyle \frac{\mathbf{I}(u,v+1)+\mathbf{I}(u+1,v+1)-\mathbf{I}(u,v)-\mathbf{I}(u+1,v)}{2}}\hfill \end{array}$$

(1)

We compute the gradient magnitude, $\nabla {\mathbf{I}}_{u,v}$, at point $(u,v)$ in the image, as well as the angle ${\mathit{\theta }}_{u,v}$ relative to the horizontal axis.

$$\begin{array}{cc}\nabla {\mathbf{I}}_{u,v}\hfill & =\sqrt{\mathbf{G}{\left(u\right)}^2+\mathbf{G}{\left(v\right)}^2}\hfill \\ {\mathit{\theta }}_{u,v}\hfill & =arctan\left[{\displaystyle \frac{{\mathbf{G}}_u}{{\mathbf{G}}_v}}\right]\hfill \end{array}$$

(2)

and then, LSD uses non-maximum suppression to enhance real edges and reduce noise. Next, it connects adjacent edge points to form line segments, possibly involving clustering and sorting. Finally, it refines these segments by removing points that do not fit a line segment and merging close segments to create more accurate and complete line representations.

Before conducting line feature tracking and matching, a brute force matching method is employed to identify corresponding line features based on the Hamming distance. Next, the 3D line features’ start and end points in world coordinates are projected onto the current frame using the initial pose estimation.

3.2.2. Self-Supervised Learning of Deep Keypoints

Traditional handcrafted keypoint methods, such as ORB [35] and SURF [36], detect corners with significant brightness changes in the image using image gradient operators and then use the local low-level features around the corner as descriptors. However, low-level image features are quite sensitive to lighting changes in environment. If the same scene is captured at different times, the same keypoint may have different descriptors due to varying lighting conditions. This low robustness introduces considerable uncertainty in SLAM feature matching, especially in scenarios with loop closures. In contrast, learning-based keypoint extraction methods transform images into descriptor and keypoint tensors end-to-end. These tensors are derived from the high-level features of the neural network. These high-level features are essentially global features of the image. Compared to the local features obtained by traditional methods, global features are more robust. Learning-based keypoint extraction methods employ supervised training, and the supervised signals come from the self-labeled method [27].

The backbone of our local feature extractor is a convolutional neural network (CNN). Similar to other learned local feature methods, it is divided into an encoder and a decoder. The encoder leverages the convolutional layers of VGG16, while the decoder consists of two parts: a keypoint position decoder and a descriptor decoder. The keypoint position decoder converts high-dimensional visual features from the encoder into a heatmap of the same size as the input image, where each pixel value represents the probability of a keypoint’s presence. A threshold is preset, and pixels in the heatmap that exceed this threshold are considered candidate keypoints. The descriptor decoder, on the other hand, transforms the high-dimensional visual features from the encoder into 256-dimensional descriptors. These descriptors are extracted at the candidate keypoint positions to serve as the output descriptors.

Our local feature extractor employs a self-supervised approach, using self-labeled training data to supervise the extraction process. Initially, we pretrain corner extractors on a synthetic geometric dataset. Then, we use the pretrained corner extractor to detect keypoints on the COCO dataset [37], treating these detected keypoints as annotations for the corresponding RGB images. We combine the RGB images with the self-annotated keypoints to form the training set for our learned local feature method.

The training loss is composed of two components: keypoint localization loss and descriptor loss. The keypoint localization loss measures the discrepancy between the predicted keypoint locations and the self-annotated positions. As shown in Equation (3), we train the keypoint localization loss using both the original image and its homography-warped version. We employ CrossEntropy loss as the metric for keypoint localization. This loss consists of the raw keypoint localization loss ${\mathcal{L}}_k$ and the homography-warped keypoint localization loss ${\mathcal{L}}_{wk}$. Here, $H_w$ and $W_w$ denote the height and width of the heatmap, with $H_w=H/8$ and $W_w=W/8$, while i represents the keypoint position index, and H and W denote the height and width of the input image. The variables $\mathbf{x}$ and $\tilde{\mathbf{x}}$ represent the keypoint positions and keypoint label positions, respectively.

$${\mathcal{L}}_{wk}={\mathcal{L}}_k={\displaystyle \frac{1}{H_w{W}_w}}\sum _{\begin{array}{c}i=1\end{array}}^{H_w{W}_w}{l}_{cross}\left({\mathbf{x}}_i;{\tilde{\mathbf{x}}}_i\right)$$

(3)

Unlike SuperPoint, we implement a sparse descriptor loss. This sparse descriptor loss is computed between the original image and its homography-warped version. It consists of two main components: the loss for descriptors matched through homography ${\mathcal{L}}_{d1}$ and the loss for those that are unmatched ${\mathcal{L}}_{d2}$. The matched descriptor loss, ${\mathcal{L}}_{d1}$, is defined in Equation (4), involving N pairs of aligned keypoint locations $({\mathbf{x}}_i,\widehat{{\mathbf{x}}_i})$ with indices i ranging from 0 to N. The unmatched descriptor loss, ${\mathcal{L}}_{d2}$, is outlined in Equation (5) and considers Z pairs of descriptors near the matched ones, with indices j ranging from 0 to Z, representing unmatched keypoint pairs $({\mathbf{x}}_i,{\mathbf{x}}_j^{\prime })$.

The overall descriptor loss, ${\mathcal{L}}_d$, described in Equation (6), incorporates a margin boundary b and is designed to minimize the feature discrepancy among corresponding descriptors while maximizing it among non-corresponding ones. The total training loss, ${\mathcal{L}}_{\mathrm{kd}}$, integrates the detector and descriptor losses, with a balancing factor $\mu$ that controls the influence of each component.

$${\mathcal{L}}_{d1}\left({\mathbf{x}}_i\right)={∥{\mathbf{d}}_I\left({\mathbf{x}}_i\right)-{\mathbf{d}}_{wI}\left(\widehat{{\mathbf{x}}_i}\right)∥}^2$$

(4)

$${\mathcal{L}}_{d2}\left({\mathbf{x}}_i\right)={\displaystyle \frac{1}{Z}}\sum _{j=1}^Z\left({∥{\mathbf{d}}_I\left({\mathbf{x}}_i\right)-{\mathbf{d}}_{wI}\left({\mathbf{x}}_j^{\prime }\right)∥}^2\right)$$

(5)

$${\mathcal{L}}_d={\displaystyle \frac{1}{N}}\sum _{i=1}^N\max (0,b+{\mathcal{L}}_{d1}-{\mathcal{L}}_{d2})$$

(6)

$${\mathcal{L}}_{\mathrm{kd}}=({\mathcal{L}}_k+{\mathcal{L}}_{wk})+\mu {\mathcal{L}}_d$$

(7)

3.3. Virtual Stereo Visual SLAM

Integrating self-supervised depth estimation as a pseudo depth sensor provides keypoints with depth perception, addressing the issue of scale degradation. We first introduce methods for self-supervised depth and pose estimation, then discuss the integration of the learned depth into SLAM, and finally, explain how to incorporate the learned pose into the SLAM framework.

3.3.1. Self-Supervised Learning of Pose and Depth

The self-supervised visual method uses a dual-network framework consisting of PoseNet for pose estimation and DepthNet for depth estimation. The depth estimates are obtained end-to-end by DepthNet. DepthNet is composed of an encoder and a decoder. Both are convolutional neural networks. The encoder transforms the image into high-level features, while the decoder converts these high-level features into depth estimates. The depth estimate is dense, with each pixel corresponding to the depth value of an object in the scene. The pose estimates are derived end-to-end from PoseNet. The input to PoseNet is a tensor that combines adjacent frame images along the RGB channel. A photometric model is employed to construct a self-supervised loss function, which integrates adjacent images, depth estimates, and pose estimates. Through multiple training epochs, PoseNet and DepthNet learn to accurately estimate both pose and depth. As shown in Equation (8), the relative pose estimate between adjacent frames preserves the luminance consistency of the target image ${\mathbf{I}}_t$ with respect to the source image ${\mathbf{I}}_s$. By combining the depth estimate of the target frame ${\mathbf{D}}_t$ with the camera’s intrinsic matrix $\mathbf{K}$, the source image ${\mathbf{I}}_s$ is projected onto the target frame’s perspective. This projection process generates a synthesized image ${\widehat{\mathbf{I}}}_s$ from the viewpoint of the target frame, maintaining photometric consistency with the source image. Here, ${\mathbf{p}}_s$ and ${\mathbf{p}}_t$ represent the pixel points of the source and target frames, respectively.

$${\mathbf{p}}_s\sim \mathbf{K}{\mathit{\xi }}_t^s{\mathbf{D}}_t\left({\mathbf{p}}_t\right){\mathbf{K}}^{-1}{\mathbf{p}}_t$$

(8)

The synthesized image ${\widehat{\mathbf{I}}}_s$ and the target image ${\mathbf{I}}_t$ not only maintain photometric consistency but also exhibit structural similarity. As shown in Equation (9), we incorporate this quantified similarity as a structural loss into the photometric loss ${\mathcal{L}}_{\mathrm{proj}}$ to preserve the structural details of the synthesized image. Here, $C_{\mathrm{SSIM}}$ represents the structural similarity loss [38], with ${\mu }_1$ and ${\mu }_2$ set to 0.15 and 0.85, respectively.

$${\mathcal{L}}_{\mathrm{proj}}={\mu }_1·|{\mathbf{I}}_t-{\widehat{\mathbf{I}}}_s|+{\mu }_2·{\displaystyle \frac{1-C_{\mathrm{SSIM}}({\mathbf{I}}_t,{\widehat{\mathbf{I}}}_s)}{2}}$$

(9)

In addition, to maintain depth scale consistency throughout the sequence, we introduce the 3D geometry consistency loss ${\mathcal{L}}_{\mathrm{DG}}$. The equation for this loss is shown in 10. $\widehat{\mathbf{D}}s$ refers to the reconstructed depth in the source frame, which should be consistent with the transformed depth obtained through $\mathit{\xi }{t}^s{\mathbf{D}}_t$, and the operation ${\mathit{\xi }}_t^s{\mathbf{D}}_t$ represents the transformation of the target frame’s depth estimate $\mathbf{D}t$ into the source frame’s perspective using the relative pose estimate ${\mathit{\xi }}_t^s$.

$${\mathcal{L}}_{\mathrm{DG}}={\displaystyle \frac{|{\widehat{\mathbf{D}}}_s-{\mathit{\xi }}_t^s{\mathbf{D}}_t|}{{\widehat{\mathbf{D}}}_s+{\mathit{\xi }}_t^s{\mathbf{D}}_t}}$$

(10)

As shown in Equation (11), we combine the photometric loss ${\mathcal{L}}_{\mathrm{proj}}$ and the 3D geometric consistency loss ${\mathcal{L}}_{\mathrm{DG}}$ to form the self supervised training loss ${\mathcal{L}}_{\mathrm{ss}}$, where ${\mu }_3$ is 0.5.

$${\mathcal{L}}_{\mathrm{ss}}={\mathcal{L}}_{\mathrm{proj}}+{\mu }_3{\mathcal{L}}_{\mathrm{DG}}$$

(11)

3.3.2. Virtual Features Derived from Learned Depth

In low-texture areas, ensuring an even distribution of detected keypoints across the entire image is challenging. Consequently, monocular SLAM struggles to achieve consistent scale in pose estimates through local feature matching alone. As shown in Figure 2a, this is a typical area with sparse texture. There is a lack of outdoor scenes such as vegetation and buildings. In close proximity to the camera, there are fewer feature points on the road. Therefore, it is difficult for the SLAM front-end to track sufficient feature points in close proximity areas. However, there are objects with obvious pixel gradients on the horizon. This enables the front end to stably track these distant features. It is worth noting that monocular SLAM exhibits scale ambiguity. It cannot distinguish between nearby and distant objects from a single image. Specifically, the positions of keypoints near the horizon remain largely unchanged as the camera moves. This leads SLAM to mistakenly assume that the camera has not moved significantly, resulting in degradation of scale estimation. As shown in Figure 2b, monocular SLAM exhibits scale degradation in the low-texture area. SLAM cannot correctly predict the depth of spatial points. Incorrect depth estimation of spatial points can also lead to incorrect keyframe pose estimation. At the same time, it is impossible to correct scale errors in the scene without loop nodes.

To enable monocular SLAM to achieve scale awareness of the scene, we integrate a self-supervised visual ego motion estimator into the proposed monocular SLAM framework. Depth estimates from DepthNet in the estimator serve as a pseudo-depth sensor. These depth estimates provide a relative depth value $D_i$ for each detected keypoint $u_L$. While this depth value lacks an absolute scale, it accurately represents the relative distance of each object in the image. The use of relative depth values enhances the performance of positioning and mapping.

As shown in Equation (12), we incorporate self-supervised depth labels into monocular SLAM to create a pseudo RGBD SLAM system. For each keypoint $u_L$, we use the virtual baseline $b_u$ and depth value $D_i$ to compute a virtual matching point $u_R$ in the right view, where $f_u$ represents the focal length.

$$\begin{array}{cc}{u}_R\hfill & =u_L-{\displaystyle \frac{f_u{b}_u}{{\mathbf{D}}_i}}\hfill \end{array}$$

(12)

The stereo keypoints and line features obtained through the pseudo depth sensor are referred to as virtual local features. The post-processing of these virtual local features follows the same procedures as in stereo SLAM. In the back-end optimization, we construct a re-projection error function based on both point and line features.

We combine the observed local features with the projected virtual features to formulate the optimization function.

3.3.3. The Motion Model from PoseNet

In traditional handcrafted feature-based SLAM (i.e., ORB-SLAM3), the tracking module first uses a constant velocity motion model ${\mathit{\xi }}_{lc}^{\mathrm{motion}}$ to estimate the relative pose between adjacent frames and then combines the pose ${\mathit{\xi }}_w^l$ of the last frame to calculate an initial pose ${\overline{\mathit{\xi }}}_w^c$ for the current frame. Subsequently, the initial pose of the current frame is used to project the 3D points in the mapping onto the image coordinates of the current frame. The 3D points in the mapping correspond to the keypoints of the last frame. Then we match the keypoints within a certain radius centered on the projected points in the current frame. Keypoints with similarity exceeding a certain threshold are considered to be matching keypoints with the keypoints of the last frame. Finally, as Equation (14) shows, we construct the pose-only optimization error $e\left({\overline{\mathit{\xi }}}_w^c\right)$ from the matching keypoints of adjacent frames. Then we use this optimization equation to find the optimal pose estimate ${\mathit{\xi }}_w^c$ of the current frame. However, the constant velocity motion model is too ideal and does not conform to real motion, which leads to a large difference between the initial pose and the ground truth, thereby affecting the pose optimization result. In view of this, we replace the constant velocity motion model with the pose estimate of the self-supervised PoseNet. The photometric loss and 3D geometry consistency loss theoretically originate from the multi-view geometry model. In summary, compared with the constant velocity motion model, the estimation of PoseNet is more consistent with the real motion model. In our implementation, we combine the rotation estimates from PoseNet with the translation from the constant velocity motion model to form the motion model.

$${\overline{\mathit{\xi }}}_w^c={\mathit{\xi }}_{lc}^{\mathrm{motion}}{\mathit{\xi }}_w^l$$

(13)

$${\mathit{\xi }}_w^c=arg\underset{{\overline{\mathit{\xi }}}_w^c}{min}{\displaystyle \frac{1}{2}}\sum {∥e\left({\overline{\mathit{\xi }}}_w^c\right)∥}^2$$

(14)

3.4. Line–Point Bundle Adjustment and Online BoW in the Back End

3.4.1. Line–Point Pose-Only Bundle Adjustment Optimization

Similar to point features, line features use the difference between projected and observed features to compute the optimization error. SLAM localization is optimized by adjusting the pose estimate to minimize this difference. As illustrated in Figure 3, the blue line $\left(\overrightarrow{\overline{{\mathbf{p}}_{\mathbf{i}}^{\mathbf{s}}}\overline{{\mathbf{p}}_{\mathbf{i}}^{\mathbf{e}}}}\right)$ represents the observed line segment, while the red line $\left(\overrightarrow{\widehat{{\mathbf{p}}^{\mathbf{s}}}\widehat{{\mathbf{p}}^{\mathbf{e}}}}\right)$ denotes the projected line segment.

As shown in Equation (15), we construct a line feature error function to refine the relative pose estimates between adjacent frames. Two types of line feature error functions are designed: the line feature distance error $E_{oul}$ and the line feature angle error $E_{ang}$.

$$E_{line}=E_{oul}+E_{ang}$$

(15)

As shown in Formula 16, we first compute the line coefficient ${\mathit{\iota }}_{\mathit{i}}$, which represents the infinite line passing through the starting point ${\overline{\mathbf{p}}}^{\mathbf{s}}$ and the ending point ${\overline{\mathbf{p}}}^{\mathbf{e}}$. In this coefficient, ${}^{\mathbf{h}}{\overline{\mathbf{p}}}^{\mathbf{s}}$ and ${}^{\mathbf{h}}{\overline{\mathbf{p}}}^{\mathbf{e}}$ denote the homogeneous coordinates of the endpoints of the observed line.

Next, we calculate the vertical distance $E_{ps}$ between the observed starting point ${\widehat{\mathbf{p}}}^{\mathbf{s}}$ and the projected line, as well as the vertical distance $E_{pe}$ between the observed ending point ${\widehat{\mathbf{p}}}^{\mathbf{e}}$ and the projected line. The sum of these two vertical distances constitutes the projection line distance error $E_{oul}$.

$${\mathit{\iota }}_{\mathit{i}}={\displaystyle \frac{{}^{\mathbf{h}}{\overline{\mathbf{p}}}^{\mathbf{s}}{\times }^{\mathbf{h}}\overline{{\mathbf{p}}^{\mathbf{e}}}}{∥{}^{\mathbf{h}}{\overline{\mathbf{p}}}^{\mathbf{s}}{\times }^{\mathbf{h}}\overline{{\mathbf{p}}^{\mathbf{e}}}∥}}$$

(16)

$$\begin{array}{c}{E}_{oul}=E_{ps}+E_{pe}={\mathit{\iota }}_{\mathit{i}}^{\mathit{T}}{\widehat{\mathbf{p}}}^{\mathbf{s}}+{\mathit{\iota }}_{\mathit{i}}^{\mathit{T}}{\widehat{\mathbf{p}}}^{\mathbf{e}}\hfill \end{array}$$

(17)

Additionally, as shown in Equations (18) and (19), the endpoints of the projected lines in the image adhere to the pinhole camera model. We project the 3D points in world coordinates into the image coordinates using the camera pose and intrinsic matrix. Here, ${\mathbf{P}}_{\mathbf{s}}^{\mathbf{w}}$ represents the starting point of the 3D line in world coordinates, while ${\mathbf{R}}_{\mathbf{w}}^{{\mathbf{c}}_{\mathbf{i}}}$ and ${\mathbf{t}}_{\mathbf{w}}^{{\mathbf{c}}_{\mathbf{i}}}$ denote the rotation and translation matrices that transform coordinates from the world frame w to the camera frame $c_i$. The index i indicates the i-th camera frame. $[{\mathbf{X}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}},{\mathbf{Y}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}},{\mathbf{Z}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}]$ are the components of the 3D line’s starting point ${\mathbf{P}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}$ in the i-th camera coordinate system. We then use the intrinsic matrix $\mathbf{K}$ to project ${\mathbf{P}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}$ onto the image coordinates ${\widehat{\mathbf{p}}}^{\mathbf{s}}$. This projection process is similarly applied to the ending point of the 3D line.

$${\mathbf{P}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}=\left[\begin{array}{c}{\mathbf{X}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}\\ {\mathbf{Y}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}\\ {\mathbf{Z}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}\end{array}\right]={\mathbf{R}}_{\mathbf{w}}^{{\mathbf{c}}_{\mathbf{i}}}{\mathbf{P}}_{\mathbf{s}}^{\mathbf{w}}+{\mathbf{t}}_{\mathbf{w}}^{{\mathbf{c}}_{\mathbf{i}}}$$

(18)

$${\widehat{\mathbf{p}}}^{\mathbf{s}}=\mathbf{K}{\mathbf{P}}_{\mathbf{s}}^{{\mathbf{c}}_{\mathbf{i}}}$$

(19)

$$\begin{array}{ccc}{E}_{ang}\hfill & =& E_{ss}+E_{es}\hfill \\ E_{ss}\hfill & =& {\displaystyle \frac{\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{s}}}\overline{{\mathbf{p}}^{\mathbf{e}}}}·\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{s}}}\widehat{{\mathbf{p}}^{\mathbf{e}}}}}{∥\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{s}}}\overline{{\mathbf{p}}^{\mathbf{e}}}}∥∥\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{s}}}\widehat{{\mathbf{p}}^{\mathbf{e}}}}∥}}-1\hfill \\ E_{es}\hfill & =& {\displaystyle \frac{\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{e}}}\overline{{\mathbf{p}}^{\mathbf{s}}}}·\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{e}}}\widehat{{\mathbf{p}}^{\mathbf{s}}}}}{∥\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{e}}}\overline{{\mathbf{p}}^{\mathbf{s}}}}∥∥\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{e}}}\widehat{{\mathbf{p}}^{\mathbf{s}}}}∥}}-1\hfill \end{array}$$

(20)

As illustrated in Figure 3, the line feature angle error measures the angular discrepancy between the endpoints of the projected line and the observed line. As shown in Equation (20), we calculate the inner product between the directed line segments $(\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{s}}}\overline{{\mathbf{p}}^{\mathbf{e}}}},\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{s}}}\widehat{{\mathbf{p}}^{\mathbf{e}}}})$ and $(\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{e}}}\overline{{\mathbf{p}}^{\mathbf{s}}}},\overrightarrow{\overline{{\mathbf{p}}^{\mathbf{e}}}\widehat{{\mathbf{p}}^{\mathbf{s}}}})$. This inner product represents the angle error. When the relative pose between the projected and observed lines is closer to the ground truth, the line feature angle errors $E_{ss}$ and $E_{es}$ approach zero.

The current frame acquires matched keypoints and line features through the feature tracking module and projects the 3D points and line segment endpoints into the current frame’s image coordinates using the pinhole camera model. The error equation is formed by the coordinate differences between the observed point–line features and the projected point–line features. This error is minimized by adjusting the pose of the current frame. When the error converges, the pose of the current frame reaches an optimal solution. The error equation based on point–line features includes the projection keypoint error $E_{po}$ and the projection line endpoint error $E_{line}$.

We use Equations (18) and (19) to transform the 3D coordinates of keypoint features ${\mathbf{P}}_{\mathbf{j}}^{\mathbf{w}}$ to the current image frame ${\mathbf{c}}_{\mathbf{i}}$ as $\widehat{{\mathbf{p}}^{\mathbf{j}}}$. The corresponding keypoint in the current frame is $\overline{{\mathbf{p}}^{\mathbf{j}}}$. The index j denotes the j-th keypoint in the current image frame ${\mathbf{c}}_{\mathbf{i}}$. We calculate the difference between $\widehat{{\mathbf{p}}^{\mathbf{j}}}$ and $\overline{{\mathbf{p}}^{\mathbf{j}}}$ as the projection point error, as shown in Equation (21).

$${\mathbf{E}}_{\mathbf{po}}^{\mathbf{i},\mathbf{j}}=\widehat{{\mathbf{p}}_{\mathbf{i}}^{\mathbf{j}}}-\overline{{\mathbf{p}}_{\mathbf{i}}^{\mathbf{j}}}$$

(21)

As shown in Equation (22), we combine the line feature error from Equation (15) and the point error from Equation (21) to form the total optimization error. The optimal pose of the current frame, ${\zeta }^{*}$, is determined using the Newton descent method. In this context, ${\mathsf{\Omega }}_{\mathbf{i},\mathbf{j}}$ and ${\mathsf{\Omega }}_{\mathbf{i},\mathbf{k}}$ represent the covariances of the observed keypoints and line features, respectively. $\mathcal{H}$ denotes the Huber kernel function, N is the total number of tracked keypoints, and M is the total number of tracked line features. The index k indicates the k-th line feature in the current image frame ${\mathbf{c}}_{\mathbf{i}}$.

$${\mathit{\zeta }}^{*}=\underset{\mathbf{1}}{argmin}\sum _{j=0}^N\mathcal{H}\left({{\mathbf{E}}_{\mathbf{po}}^{\mathbf{i},\mathbf{j}}}^T{\mathsf{\Omega }}_{i,j}^{-1}{\mathbf{E}}_{\mathbf{po}}^{\mathbf{i},\mathbf{j}}\right)+\sum _{k=0}^M\mathcal{H}\left({{\mathbf{E}}_{\mathbf{line}}^{\mathbf{i},\mathbf{k}}}^T{\mathsf{\Omega }}_{i,k}^{-1}{\mathbf{E}}_{\mathbf{line}}^{\mathbf{i},\mathbf{k}}\right)$$

(22)

3.4.2. Binary Descriptors and Online-Learning-Based BoW

Loop closure in SLAM systems is a critical function that identifies when a robot or autonomous vehicle revisits a previously explored area, allowing the system to correct accumulated localization errors. This module also enhances map consistency and improves overall system robustness. The loop closure module optimizes localization by leveraging the information of already mapped environments. This process is essential for maintaining the integrity and accuracy of the map, especially in dynamic or large-scale environments. The main component of loop closure is the bag-of-words (BoW) models. The BoW models can abstract scene information and obtain the global features of each scene. By comparing the euclidean distance of global features, we can find out the loop nodes in the sequence.

We have developed a learned local feature extractor at the front end that produces floating-point descriptors with values ranging between $(-1,1)$. The existing visual SLAM loop closure modules, however, rely on offline BoW models designed for binary descriptors. Additionally, deep learning methods often face challenges with generalization; a BoW model trained offline may not perform well in novel environments. To address these issues, we employ an online-trained BoW model (iBoW) [39] for the loop closure module.

As shown in [15], floating-point descriptors are difficult to organize into a tree-structured dictionary and thus are challenging to use in loop closure. Ref. [15] also indicates that the distribution of learned descriptor values is normally distributed. Therefore, a binary hashing transformation is applied to the learned descriptors. We binarize the descriptors by mapping vector values above zero to 1 and values equal to or below zero to 0. The transformed descriptors are 256-bit binary descriptors, with values being either 0 or 1. Subsequently, following the descriptor generation process of ORB, the binary descriptors are converted into integer descriptors. The conversion method involves transforming each 8-bit binary descriptor into a decimal integer, allowing 256 binary elements to be converted into 32 integer elements. Finally, the descriptors input into iBoW is 32-bit integer descriptors. The process is illustrated in Figure 4.

4. Experiments

4.1. Implementation Details

In our D3L-SLAM system, the learned local feature extraction module was implemented and trained using PyTorch, with a total of 200,000 training iterations. The RGB training dataset was sampled from the COCO dataset, and the batch size was set to 8. For improved inference efficiency, the trained local features were deployed using TensorRT. Additionally, the self-supervised visual odometry system was developed and trained using PyTorch (https://pytorch.org/), with a batch size of 4. Both learning methods were trained and evaluated on an NVIDIA RTX 3090 GPU (NVIDIA, Santa Clara, CA, USA).

4.2. Evaluation on the KITTI Odometry Dataset

We first conducted experiments using the publicly available KITTI dataset, which is a standard benchmark to evaluate visual SLAM for autonomous vehicles scenarios. For the proposed hybrid visual SLAM, we trained the self-supervised odometry system on KITTI odometry 11–20 sequences. We also evaluated the localization performance of the proposed SLAM and baselines on sequences 0–10. The traditional visual baselines (VISO-M, LDSO and ORB-SLAM3) do not include the training process. Moreover, LIFT-SLAM uses the COCO dataset [37] to supervise the training of LIFT extractor. We also use the COCO dataset to supervise the training of the proposed deep keypoint. We set the maximum number of keypoints to 3000 and the number of layers in the image pyramid to 3. Due to the scarcity of local features, many monocular SLAM systems struggle with sequence 01. Consequently, we did not assess the performance of our SLAM method on this sequence to ensure fairness. We calculated the mean square error of translation and rotation estimates on different length sequences $[100,200,\dots ,800]$ and took their mean as the mean square error of translation and rotation estimates. These two mean square errors are used as accuracy indicators for SLAM localization metric $t_{rel}$ and rotation metric $r_{rel}$. The specific calculation of these two metrics is shown in Equation (23), where i and j represent the index of two frames separated by a fixed distance. $len(i,j)$ represents the fixed distance, which is $[100,200,\cdots ,800]$. $Rot(.)$ and $Trans(.)$ represent the rotation and translation difference. ⊖ means calculating the relative pose estimate between two frames. Additionally, ${\mathit{\xi }}_j\ominus {\mathit{\xi }}_i$ represents the relative pose from the $i_{th}$ frame to the $j_{t}h$ frame. $\widehat{\mathit{\xi }}$ and $\mathit{\xi }$ represent the estimated and ground truth pose, respectively. N indicates the sum of fixed distances.

$$\begin{array}{cc}{r}_{rel}\hfill & ={\displaystyle \frac{1}{N}}\sum _{(i,j)\in \mathcal{S}}{\displaystyle \frac{Rot\left(\left({\widehat{\mathit{\xi }}}_j\ominus {\widehat{\mathit{\xi }}}_i\right)\ominus \left({\mathit{\xi }}_j\ominus {\mathit{\xi }}_i\right)\right)}{len(i,j)}}\hfill \\ t_{rel}\hfill & ={\displaystyle \frac{1}{N}}\sum _{(i,j)\in \mathcal{S}}{\displaystyle \frac{Trans\left(\left({\widehat{\mathit{\xi }}}_j\ominus {\widehat{\mathit{\xi }}}_i\right)\ominus \left({\mathit{\xi }}_j\ominus {\mathit{\xi }}_i\right)\right)}{len(i,j)}}\hfill \end{array}$$

(23)

To thoroughly evaluate the performance of the proposed SLAM, we compared it against several benchmark methods, including traditional feature-based visual SLAM approaches such as VISO-M [40,41], ORB-SLAM3 [42], and the direct method LDSO [43], as well as the learning-based method LIFT-SLAM [12]. VISO-M RMSE is not shown in

Table 2. Comparison of Pose Estimation Between the Proposed D3L-SLAM and Representative Traditional and Learning-Based Visual SLAM Methods. Sequence 01 is excluded from the evaluation due to inherent challenges that affect all monocular SLAM baselines, preventing the generation of reasonable comparative results. Root mean square error(RMSE) is the absolute localization error between predicted trajectory and ground truth.

Seq	VISO-M		LDSO			ORB-SLAM3			LIFT-SLAM			D3L-SLAM
	${\mathit{t}}_{\mathit{rel}}$	${\mathit{r}}_{\mathit{rel}}$	${\mathit{t}}_{\mathit{rel}}$	${\mathit{r}}_{\mathit{rel}}$	$\mathit{RMSE}$	${\mathit{t}}_{\mathit{rel}}$	${\mathit{r}}_{\mathit{rel}}$	$\mathit{RMSE}$	${\mathit{t}}_{\mathit{rel}}$	${\mathit{r}}_{\mathit{rel}}$	$\mathit{RMSE}$	${\mathit{t}}_{\mathit{rel}}$	${\mathit{r}}_{\mathit{rel}}$	$\mathit{RMSE}$
00	36.95	2.42	2.92	0.36	7.88	2.74	0.30	7.15	3.18	2.99	8.06	2.09	0.80	5.54
02	21.98	1.22	4.02	0.74	26.53	5.16	0.38	21.94	8.73	2.49	40.04	3.83	1.02	17.37
03	16.14	2.67	2.16	0.16	2.89	2.69	0.17	3.32	1.46	0.34	2.23	2.63	1.62	3.63
04	2.61	1.53	1.3	0.19	2.89	2.25	0.24	2.27	2.22	0.48	0.51	3.21	1.00	2.33
05	17.20	3.52	2.05	0.24	4.60	3.36	0.39	6.34	6.09	3.11	13.55	2.27	0.56	5.90
06	7.91	1.83	6.04	0.16	13.28	6.40	0.18	14.23	12.24	2.91	30.38	2.91	1.12	5.85
07	20.00	5.30	11.27	5.13	15.82	2.08	0.42	2.46	2.42	4.02	3.63	1.18	0.41	1.49
08	39.78	1.99	31.69	0.24	127.92	8.55	0.31	31.97	47.10	2.02	184.43	2.82	0.63	11.54
09	29.01	1.32	18.28	0.21	75.78	3.08	0.54	8.97	19.91	2.14	59.62	2.28	0.43	5.37
10	28.52	3.23	7.77	0.21	16.95	8.57	0.36	17.02	9.72	2.24	29.87	3.99	0.58	8.10
Avg	22.01	2.50	8.75	0.76	29.45	4.49	0.33	11.567	11.31	2.27	37.23	2.72	0.82	6.71

, because its absolute error in KITTI was not indicated in the original paper [40,41]. The results demonstrate that incorporating learned depth estimates significantly improves SLAM localization performance. As shown in Figure 5 and Table 2, sequence 08, which lacks loop nodes and is demanding in terms of feature tracking and scale degradation suppression, highlights the effectiveness of our approach. Our method achieved superior localization accuracy and closely matched the ground truth trajectory.

Monocular SLAM systems often struggle to accurately estimate the scale in long-distance open-loop sequences. In low-texture areas, the front-end of monocular SLAM may only detect keypoints at the distant skyline, leading to the incorrect assumption that the camera is stationary, which results in scale degradation. Our proposed SLAM method mitigates this issue by incorporating DepthNet’s depth estimates. Each valid keypoint is assigned a depth estimate, and we use the pseudo-baseline and keypoint depth to compute the corresponding virtual right-view keypoint, effectively converting the monocular SLAM into a stereo SLAM. This approach enhances scale awareness and prevents scale degradation caused by low-texture areas. Additionally, our line–point front end leverages line features for matching and pose optimization, further improving localization and mapping performance.

However, it is important to note that the introduction of depth estimation has led to a decrease in pose determination accuracy. This may be attributed to the inability of depth estimation to provide accurate global absolute metrics and its limited generalization performance in unfamiliar scenes, which affects rotation estimation accuracy. Despite this, the learned pose estimates enhance feature matching rates between adjacent frames, thereby improving front-end pose estimation performance.

We also present the mapping results with point and line details of our proposed D3L-SLAM in Figure 6a–d. The line–point front end provides comprehensive feature perception, while the learned depth estimates enhance scale perception. This combination allows the mapping structures to maintain relatively accurate spatial relationships.

4.3. Evaluation with Self-Collected Dataset

As illustrated in Figure 7, we constructed a car-driving experiment platform equipped with an RGB camera and a high-precision GNSS system to provide ground truth location data for evaluation.

Our car-driving dataset was collected using RGB images from Zed cameras and positioning data from Sinognss Navigation’s GNSS system. The data collection occurred in Changsha, China, covering three distinct scenarios: campus, urban street, and residential areas. The training dataset comprises 57,948 images, encompassing various scene types, while the testing dataset includes 14,972 images, focusing on campus and urban street scenes. Example images from these scenarios are shown in Figure 8. Four sequences were selected for testing: sequences 01, 03, and 04 represent urban data, and sequence 02 represents campus data.

Given the centimeter-level positioning accuracy of the GNSS system, we assessed the positioning accuracy of the proposed SLAM using root mean square error (RMSE). Since the proposed SLAM does not provide absolute scaled poses, trajectory scaling is necessary before evaluating pose estimates. The RMSE formula is given by

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}({\mathbf{p}}_i-{\mathbf{p}}_i^{\mathrm{g}})}$$

(24)

where $\mathbf{p}i$ denotes the three-dimensional location estimate at the $ith$ frame, N is the total number of frames, and ${\mathbf{p}}_i^{\mathrm{g}}$ represents the ground truth location.

As shown in Figure 9 and

Table 3. Comparison Of Pose Estimation Results Between The Proposed D3L-SLAM and Representative Visual SLAM Baselines (i.e., ORB-SLAM3 and LDSO).

Method	01 Seq	02 Seq	03 Seq	04 Seq	Avg
ORB-SLAM3	22.30	8.43	13.42	25.50	17.41
LDSO	70.36	117.10	53.96	24.23	66.41
D3L-SLAM	8.72	9.42	11.09	10.21	9.86

, the trajectory of the proposed D3L-SLAM closely aligns with the ground truth. In terms of average positioning error, D3L-SLAM demonstrates the best performance. Traditional handcrafted visual SLAM methods often suffer from scale degradation in expansive urban scenes due to the vast distances and sparse textures in such environments. These methods may struggle to accurately reflect camera motion in low-texture areas, leading to scale ambiguity.

In contrast, the self-supervised monocular depth estimation framework in D3L-SLAM achieves a globally consistent scale for the scene through training, offering relatively accurate relative depth information. This enables the SLAM system to maintain a consistent scale in mapping. The integration of learned pose priors ensures that feature matching between adjacent frames adheres to the true motion model, enhancing localization accuracy. Additionally, incorporating line features improves perception in low-texture areas, mitigating localization degradation.

However, it is important to note that the positioning accuracy of sequence 02 is not as high as that of traditional ORB-SLAM3. This discrepancy may be attributed to the presence of numerous small-sized dynamic objects, such as people and bicycles, in the campus environment. The instability of line features on these small objects can negatively impact feature tracking performance.

4.4. Ablation Study

To validate the contribution of each component in D3L-SLAM, we perform an ablation study. The results are summarized in

Table 4. The ablation study into the key modules in our D3L-SLAM system.

Method	DK	Depth	Pose	Line	Metric	00	02	03	04	05	06	07	08	09	10	Avg
Ours1	✓				$t_{rel}$	2.56	3.76	4.87	1.07	1.90	7.24	1.00	7.66	2.95	5.58	3.86
					$r_{rel}$	0.31	0.27	0.17	0.24	0.26	0.17	0.28	0.28	0.29	0.23	0.25
Ours2	✓	✓			$t_{rel}$	2.13	4.07	2.76	2.88	2.12	3.03	1.38	3.13	2.75	4.27	2.85
					$r_{rel}$	0.79	1.06	1.46	0.91	0.58	1.15	0.41	0.66	0.53	0.62	0.82
Ours3	✓	✓	✓		$t_{rel}$	1.98	3.70	2.58	3.53	2.15	2.57	1.59	3.03	2.67	4.27	2.81
					$r_{rel}$	0.63	1.01	1.43	1.16	0.54	1.07	0.69	0.65	0.51	0.62	0.83
Ours4	✓	✓	✓	✓	$t_{rel}$	2.09	3.83	2.63	3.21	2.27	2.91	1.18	2.82	2.28	3.99	2.72
					$r_{rel}$	0.80	1.02	1.62	1.00	0.56	1.12	0.41	0.63	0.43	0.58	0.82

. In the table, “DK” refers to deep keypoints, which utilize learned local features as the SLAM front-end. “Depth” denotes the use of learned depth estimates as a pseudo depth sensor. “Pose” signifies the incorporation of pose priors into the front end. “Line” indicates the addition of line features at the front end and line segment errors at the back end.

From the results labeled as Ours1, it is evident that deep keypoints substantially enhance both localization and rotation accuracy compared to the handcrafted feature-based ORB-SLAM. This improvement suggests that learned local features are more effective in handling scenes with limited texture. Comparing Ours1 and Ours2, it is clear that learned depth estimates significantly boost SLAM’s localization accuracy. These globally consistent depth estimates prevent scale degradation in mapping, even in low-texture areas, by providing relative depth information for each keypoint. However, the rotation accuracy shows a decrease, which may be attributed to inaccuracies in depth estimation for fine details such as edges, affecting rotation estimates. From Ours2 and Ours3, it can be observed that incorporating pose priors enhances the feature matching rate between adjacent frames. This improvement in matching accuracy results from the learned pose priors better aligning with the real motion model, as opposed to the uniform models used in traditional visual SLAM. Finally, comparing Ours3 and Ours4 demonstrates that integrating line features enhances SLAM performance, likely due to the robustness of line features in low-texture environments.

In addition to performance comparisons, we also test the efficiency of our proposed SLAM. The experiment baselines are the handcrafted traditional SLAM: ORB-SLAM3 and SuperPoint-SLAM [44], which is similar to our proposed SLAM. To ensure fairness, all these methods are set with a pyramid of 3 levels and a total number of keypoints set to 3000. SuperPoint-SLAM uses libtorch [45] to combine superpoint with ORB-SLAM2, while our framework is based on the C++ version TensorRT. The test data used are KITTI, and we measure the median and mean feature tracking time of the three methods. From the bar chart Figure 10, it can be seen that the efficiency of the traditional handcrafted method ORB-SLAM3 is the highest, which is because traditional SLAM only needs to run on CPU and RAM, and ORB is a low-level image feature. Therefore the ORB-SLAM3 is more efficient. Our proposed method uses various deep learning models running on the GPU: DepthNet, PoseNet, and Deep Keypoint, thus it is less efficient than traditional methods. However, we use the C++ version TensorRT to deploy the learning models. In contrast, SuperPoint-SLAM runs on libtorch. It converts each single image to a tensor on the GPU, and then it inferences an image through the learned model. Libtorch transfers the output keypoints and descriptors from the GPU to the CPU for subsequent processing. TensorRT, on the other hand, caches multiple images on the GPU before inference. This caching mechanism can greatly reduce the data transfer time from GPU to CPU, enhancing the efficiency of learning-based SLAM.

5. Conclusions

The core technology of navigation for unmanned platforms and autonomous driving systems is SLAM, and visual SLAM is widely used due to its low cost advantage. However, monocular visual SLAM often struggles with accurate positioning and mapping in large-scale scenes with low texture due to a lack of local features. Although loop closure methods can correct mapping scale, accurate localization remains challenging in large-scale environments without sufficient loop nodes. This paper proposes a hybrid monocular SLAM system that integrates a self-supervised ego motion estimator with a multi-view geometry model. Our approach combines self-supervised depth and pose estimates. The depth estimates assign depth values to each keypoint, enabling the front end to perceive the relative distances between objects. We achieve this by designing a virtual baseline that transforms the pseudo-RGBD front end into a virtual stereo front end. Additionally, we replace the uniform motion model in the tracking thread with pose priors to better reflect the real motion model, enhancing feature matching accuracy and, consequently, localization precision. To further improve SLAM performance in low-texture environments, we developed a hybrid line–point front end and added more constraints in the optimization back end. The experiments conducted on both publicly available and self-collected datasets, compared to typical benchmarks, demonstrate that our proposed D3L-SLAM significantly improves localization accuracy. However, we observed a decrease in rotation accuracy, likely due to the fact that the learned depth estimates are not actual depth measurements, which can introduce ambiguity and increase rotation estimation errors. Despite this, the rotation accuracy of D3L-SLAM remains comparable to that of traditional SLAM systems.

The future work will focus on enhancing rotation estimation accuracy by incorporating IMU data to create a virtual “stereo+IMU” SLAM. We also plan to introduce stereo data during the training of the self-supervised ego motion estimator to optimize scale estimation, enabling the SLAM framework to obtain absolute scale poses. In addition, we also plan to apply the proposed SLAM to the unmanned aerial vehicles and autonomous vehicles, to achieve accurate positioning in complex lighting environments.