Skip to Content
You are currently on the new version of our website. Access the old version .
SensorsSensors
  • Article
  • Open Access

17 December 2024

26 Pages

Advanced Monocular Outdoor Pose Estimation in Autonomous Systems: Leveraging Optical Flow, Depth Estimation, and Semantic Segmentation with Dynamic Object Removal

and
Electrical, Computer, and Biomedical Engineering, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada
*
Author to whom correspondence should be addressed.

Abstract

Autonomous technologies have revolutionized transportation, military operations, and space exploration, necessitating precise localization in environments where traditional GPS-based systems are unreliable or unavailable. While widespread for outdoor localization, GPS systems face limitations in obstructed environments such as dense urban areas, forests, and indoor spaces. Moreover, GPS reliance introduces vulnerabilities to signal disruptions, which can lead to significant operational failures. Hence, developing alternative localization techniques that do not depend on external signals is essential, showing a critical need for robust, GPS-independent localization solutions adaptable to different applications, ranging from Earth-based autonomous vehicles to robotic missions on Mars. This paper addresses these challenges using Visual odometry (VO) to estimate a camera’s pose by analyzing captured image sequences in GPS-denied areas tailored for autonomous vehicles (AVs), where safety and real-time decision-making are paramount. Extensive research has been dedicated to pose estimation using LiDAR or stereo cameras, which, despite their accuracy, are constrained by weight, cost, and complexity. In contrast, monocular vision is practical and cost-effective, making it a popular choice for drones, cars, and autonomous vehicles. However, robust and reliable monocular pose estimation models remain underexplored. This research aims to fill this gap by developing a novel adaptive framework for outdoor pose estimation and safe navigation using enhanced visual odometry systems with monocular cameras, especially for applications where deploying additional sensors is not feasible due to cost or physical constraints. This framework is designed to be adaptable across different vehicles and platforms, ensuring accurate and reliable pose estimation. We integrate advanced control theory to provide safety guarantees for motion control, ensuring that the AV can react safely to the imminent hazards and unknown trajectories of nearby traffic agents. The focus is on creating an AI-driven model(s) that meets the performance standards of multi-sensor systems while leveraging the inherent advantages of monocular vision. This research uses state-of-the-art machine learning techniques to advance visual odometry’s technical capabilities and ensure its adaptability across different platforms, cameras, and environments. By merging cutting-edge visual odometry techniques with robust control theory, our approach enhances both the safety and performance of AVs in complex traffic situations, directly addressing the challenge of safe and adaptive navigation. Experimental results on the KITTI odometry dataset demonstrate a significant improvement in pose estimation accuracy, offering a cost-effective and robust solution for real-world applications.

1. Introduction

Pose estimation is a crucial aspect in robotics, autonomous driving, and computer vision, enabling the accurate determination of an object’s orientation and position in space. While significant research has been dedicated to pose estimation using LiDAR or stereo cameras, there remains a notable gap in the literature concerning monocular vision systems—those utilizing only a single camera. Most existing solutions rely heavily on LiDAR sensors or stereo camera setups, which, although highly accurate, present significant limitations in weight, cost, and complexity. In contrast, with its practicality and cost-effectiveness, monocular vision is a prevalent choice for many drones, cars, and other autonomous vehicles. However, developing robust and reliable pose estimation models that leverage monocular images remains underexplored. The practicality and cost-effectiveness of monocular vision cannot be overstated. Many drones, cars, and other autonomous vehicles are equipped with only one camera due to the simplicity and affordability of such configuration. Despite this, developing robust and reliable pose estimation models that leverage monocular images remains underexplored. The lack of research in this area is evident when reviewing the KITTI benchmark leaderboard, where most high-performing models are based on LiDAR or stereovision datasets, leaving monocular approaches relatively underrepresented. Our work aims to fill this gap, providing a practical and cost-effective solution for the industry.
Our work aims to bridge this research gap by presenting a novel approach for pose estimation using monocular images. This is particularly crucial for applications where deploying additional sensors is not feasible due to cost or physical constraints. By focusing on a single-camera setup, we seek to develop a model that meets the performance standards set by multi-sensor systems and leverages the inherent advantages of monocular vision, such as reduced weight and cost. In this paper, we introduce an advanced framework for monocular visual odometry that integrates state-of-the-art preprocessing techniques, including optical flow, depth estimation, and semantic segmentation. Optical flow provides essential motion information between consecutive frames, depth estimation offers spatial context by predicting distances within the scene, and semantic segmentation enhances the understanding of object boundaries and scene context to remove the negative impact of dynamic objects on pose estimation. Combining these techniques, our approach aims to deliver high accuracy and robustness in pose estimation from a single camera input. This research contributes to the field by addressing the practical need for effective monocular pose estimation models. It highlights the potential of monocular systems to provide reliable performance in real-world applications, paving the way for broader adoption of cost-effective and lightweight vision-based solutions in various autonomous systems. Through extensive experiments on the KITTI odometry dataset, we demonstrate the effectiveness of our proposed framework, underscoring its potential to fill the existing gap in monocular pose estimation research. The main contributions of this paper can be summarized as follows:
  • Novel Monocular Pose Estimation Framework: This paper introduces an advanced framework for pose estimation using monocular images, addressing the gap in research where most high-performing models rely on multi-sensor setups like LiDAR or stereo cameras. The proposed approach focuses on monocular vision, which is cost-effective and practical for various autonomous systems.
  • Integration of Optical Flow, Depth Estimation, and Semantic Segmentation: The proposed framework combines optical flow, depth estimation, and semantic segmentation to improve the accuracy and robustness of visual odometry. These preprocessing steps allow the system to gather more environmental information, such as motion, spatial depth, and scene context.
  • Dynamic Object Removal: An essential contribution is the dynamic object removal technique that filters out moving objects (e.g., cars, pedestrians) from the scene, reducing noise in pose estimation and focusing on static background elements. This improves pose accuracy in dynamic environments.
  • Cost-effective and Lightweight Solution: By focusing on monocular vision and minimizing the need for additional sensors, this paper presents a lightweight and adaptable solution, making it highly suitable for applications in resource-constrained environments such as drones and autonomous vehicles.
  • Experimental Validation: The framework’s performance is validated through experiments on the KITTI odometry dataset, demonstrating significant improvements in pose estimation accuracy compared to existing methods. The experimental results highlight the superiority of the proposed approach, particularly when combining optical flow, depth estimation, and semantic segmentation.
These contributions collectively advance the field of monocular pose estimation by offering a practical, efficient, and robust solution for autonomous systems operating in GPS-denied and dynamic environments. The rest of this paper is organized as follows: Section 2 provides an overview of related work focusing on preprocessing techniques, including optical flow, depth estimation, and semantic segmentation. Section 3 introduces vision-based pose estimation. Section 4 discusses the methodology used in this research, explaining the proposed pose estimation framework, its architecture, and the integration of optical flow, depth, and semantic segmentation. Section 4 covers the experimental setup, including dataset details, training procedures, and evaluation metrics. The results of the experiments, as well as a comparison of different preprocessing methods, are presented in Section 5. Finally, Section 6 concludes this paper by summarizing our findings and discussing potential future research directions.

3. Vision-Base Pose Estimation

The architecture proposed in this paper leverages advanced computer vision techniques such as optical flow, depth estimation, and semantic segmentation, enhancing pose estimation accuracy. The proposed framework leverages these techniques to process image sequences from monocular cameras, addressing the challenges associated with dynamic environments and GPS-denied areas to estimate sensor pose changes from a sequence of frames captured by a single camera. In this section, we present the literature on integrating various computer vision techniques,
Zhi et al. [41] introduced an unsupervised monocular visual odometry method incorporating multiscale modeling to enhance pose and depth estimation accuracy, particularly in scenarios with varying motion patterns. The proposed framework utilizes densely linked atrous convolutions to expand the receptive field size while preserving image details. It integrates a nonlocal self-attention mechanism to capture long-range dependencies effectively. These architectural components enable the model to effectively model objects of different image scales, improving visual odometry accuracy, especially in rotating scenes. Despite the effectiveness of these components, their study highlights the inefficiency of rescaling for feature fusion in nonlocal modeling, suggesting a potential area for further optimization in future research. Liu et al. [42] proposed a Competition-Cooperation Transformer Network for the joint estimation of pose, depth, and optical flow, likely incorporating a transformer-based architecture. This architecture is designed to capture complex relationships between the visual perception tasks by integrating competitive and cooperative mechanisms. The network structure may consist of multiple transformer layers with attention mechanisms that enable the model to effectively process spatial and temporal information. This facilitates accurate pose, depth, and optical flow estimation in a unified framework. In the study by Zhao et al. [43] on Deep Direct Visual Odometry, the architecture likely involves a deep learning framework tailored for direct camera motion estimation from consecutive image frames. This architecture may consist of convolutional neural networks (CNNs) or recurrent neural networks (RNNs) designed to process visual data efficiently and accurately predict camera poses. The model may incorporate multiple layers with varying complexities to capture intricate spatial and temporal dependencies in the input data, enabling real-time and precise pose estimation for applications such as autonomous driving and robotics. In Song et al. [44], the ContextAVO method introduces a sophisticated architecture that leverages local context guidance to refine poses in deep visual odometry systems. The architecture likely incorporates convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to process sequential image data and extract relevant features for pose estimation. Additionally, the model may include attention mechanisms to focus on critical regions in the input data, enhancing the accuracy and robustness of pose refinement in challenging visual odometry scenarios. Tu et al. [45] proposed EMA-VIO, a deep visual-inertial odometry system that integrates external memory attention mechanisms. The architecture likely consists of a deep neural network that fuses visual and inertial sensor data with external memory modules to capture long-term dependencies. The attention mechanism in EMA-VIO enables the model to focus on relevant information for accurate motion estimation, making it robust in challenging environments. In Xu et al. [46], the AirVO method is introduced as an illumination robust point-line visual odometry system with a unique architecture. AirVO combines point and line features in a carefully designed neural network structure to estimate camera motion accurately, particularly in challenging lighting conditions. The architecture likely includes modules for feature extraction, point-line correspondence establishment, and motion estimation, showcasing a sophisticated design tailored to handle varying illumination levels effectively. Additionally, the network may incorporate attention mechanisms or memory units to capture long-term dependencies and enhance the robustness of odometry estimation in diverse lighting environments. Zhu et al. [47] proposed DeepAVO, which refines poses in deep Visual Odometry through feature distillation. The architecture likely involves a deep neural network that processes visual data to estimate poses, focusing on distilling informative features to enhance accuracy. The network may consist of convolutional layers for feature extraction, followed by modules for feature distillation, enabling the model to refine pose estimates efficiently. Additionally, the architecture may incorporate attention mechanisms or recurrent layers to capture temporal dependencies in the visual data, contributing to the robustness and precision of pose estimation in challenging scenarios. Yang et al. [48] introduced the D3VO framework, which leverages deep networks for monocular visual odometry by incorporating deep depth, pose, and uncertainty estimation. The proposed self-supervised monocular depth estimation network enhances accuracy by aligning training image pairs under similar lighting conditions and modeling photometric uncertainties. This framework performs better than traditional methods and achieves results comparable to stereo/LiDAR odometry and visual-inertial odometry while utilizing only a single camera.
The literature summarized in Table 2 highlights several limitations in existing vision-based pose estimation methods that underscore the need for further advancements. Key challenges identified include inefficiencies in feature fusion (as noted in Zhi et al.’s unsupervised monocular visual odometry), high computational complexity (Liu et al.’s transformer-based joint estimation), and limited robustness in dynamic environments (Zhao et al.’s Deep Direct Visual Odometry). Additionally, approaches like EMA-VIO (Tu et al.) depend on inertial sensors, adding complexity to the system, while others, such as DeepAVO (Zhu et al.), are sensitive to visual noise and lighting inconsistencies. The reviewed methods generally excel in specific contexts but struggle to provide a cost-effective, scalable, and adaptable outdoor monocular pose estimation solution. These limitations lead to the development of the proposed pipeline in this paper, which seeks to address the gaps by introducing a robust, AI-driven monocular visual odometry system. This system integrates optical flow, depth estimation, and semantic segmentation to enhance performance in GPS-denied environments without additional sensors, making it scalable and adaptable across different platforms.
Table 2. Comparing Vision-based VO methods.

4. The Proposed Monocular Pose Estimator

This section introduces a novel pose estimation architecture designed explicitly for monocular camera setups. This architecture leverages advanced computer vision techniques and transformer models to estimate sensor pose changes from a sequence of frames captured by a single camera. The system is structured into distinct yet interconnected components, each optimized for handling specific tasks related to optical flow, depth estimation, semantic segmentation, and dynamic object filtering. These components work synergistically to accurately estimate the camera’s 6 degrees of freedom (6DoF) pose. The proposed system is built with robustness and efficiency by employing deep learning models for complex spatial and temporal feature extraction. Throughout this section, we will explain the architecture in detail, outlining the role of each block in the pipeline, their contributions to the overall system, and the benefits of combining these models for pose estimation. In the following subsections, we summarize the architecture’s key components, their functionalities, and how they interact to produce high-precision pose estimates. We will also highlight the advantages of this architecture, such as its ability to integrate depth, motion, and semantic data, as well as its capacity to adapt to various dynamic environments.

4.1. Architecture

Figure 1 describes the proposed pose estimation system that leverages advanced computer vision techniques and transformer models to estimate sensor pose changes from a sequence of frames captured by a monocular camera. Each block in the pipeline serves a specialized function, utilizing deep learning models optimized for their tasks. Following, we describe each block and its contributions to the system, as well as the advantages of this architecture.
Figure 1. Proposed Pipeline Architecture.
  • Data Loader: This component retrieves data from the dataset in manageable sizes for processing (denoted by n batches of pair images). It ensures efficient data handling during the training or inference stages.
  • Optical Flow Estimator: CoTracker [49] is presumably a transformer-based model designed for tracking objects across frames by estimating their motion (optical flow). It provides a dense field of motion vectors that represent the displacement of pixels between consecutive frames. The estimator’s output goes to a ten-by-ten grid matrix for every five frames. The track of the motion is drawn on each frame. Finally, the original image on the frame is removed to reduce the processing for the pose estimator, as shown in Figure 2.
Figure 2. Optical flow processed output sample for one sequence of frames.
  • Depth Estimator [50]: This block estimates the depth from the camera to each point in the image. The cited approach suggests an integrated model that combines information from optical flow and direct depth estimation to produce a depth map; a sample output of the depth estimation process is shown in Figure 3.
Figure 3. Sample output of depth estimation.
  • Semantic Estimator: HRNet [32,51], which stands for High-Resolution Network, is a neural network architecture for semantic segmentation that maintains high-resolution representations throughout the network. This allows it to predict fine-grained semantic labels, which is crucial for understanding different objects and regions within the frame; a sample output of the semantic segmentation process is shown in Figure 4.
Figure 4. Sample output of the semantic segmentation.
  • Masking: Here, based on the information from optical flow, depth, and semantic estimations, a masking operation focuses on certain regions or objects of interest by filtering out irrelevant parts of the image.
  • Dynamic Object Filtration: This step filters out dynamic objects, which could be crucial for focusing the pose estimation on the static background or separating moving objects’ influence from the camera’s motion. The categories of objects to be removed include sky, car, pedestrian, bicycle, and motorcycle, as shown in Figure 5.
Figure 5. Sample output of dynamic object and sky removal.
  • Image Stacking: This block likely stacks or aligns multiple frames to comprehensively represent the scene, considering changes in view due to camera and object movements.
Figure 6 shows the full step-by-step preprocess and the sample outcome. It shows the original image, the image with the removed sky, cars and bicycle, semantic map, depth map, optical flow movement tracks, optical flow masked with a semantic map, weighted optical flow with a depth map, and finally, the result of the whole preprocessing the optical flow with masked semantic and depth.
Figure 6. Step-by-step preprocessing samples.
  • Pose Estimator: The pose estimator network shown in Figure 7 processes a rich set of visual data to estimate a camera’s 6 degrees of freedom (6DoF) pose. The input to the network consists of an RGB optical flow image, a semantic segmentation map, and a depth map. These inputs are combined to comprehensively represent the scene across multiple frames. Initially, the input data is segmented into a sequence of frames, each representing five channels: three for the RGB optical flow, one for the semantic segmentation, and one for the depth map. These frames are then stacked in pairs to create a sequence of combined frames for each time step, labeled as t − 1, t, and t + 1, each with dimensions of (5, 610, 1446). The pairs of frames (t − 1, t) and (t, t + 1) are then processed to create stacked images with dimensions of (10, 610, 1446), which are fed into Convolutional Neural Networks (CNNs) to extract spatial features. The CNNs process these stacked frames to produce feature maps that capture the spatial structure of the scene. These feature maps are then passed to Long Short-Term Memory (LSTM) networks, which are specialized types of Recurrent Neural Networks (RNNs). The LSTM networks are designed to capture temporal dependencies and dynamics within the sequence of frames. Two layers of LSTMs, denoted as LSTM 1 and LSTM 2, are used to process the feature maps sequentially, capturing the evolution of the scene over time. The output from the LSTM layers consists of 6DoF pose estimates for each time step, represented as vectors of rotational and translational components (θ, x). These estimates are then compared to ground truth poses, which include both rotational matrices [R|t] for 12-DoF and individual rotational and translational components (θx, θy, θz, x, y, z) for 6-DoF. The network computes the Mean Squared Error (MSE) loss for the angle and translation estimates. These losses are combined and scaled by a constant factor to compute the final loss. The network is trained to minimize this loss, improving its accuracy in estimating the 6DoF poses from the given visual input data. This architecture effectively combines spatial and temporal information to provide robust pose estimations in dynamic environments.
Figure 7. Pose estimator architecture. I changed it and replaced the image.

4.2. Output Format

The final output provides the estimated pose changes, including translation ( ∆ x ,   ∆ y , ∆ z ), rotation Roll ( ∆ φ ), Pitch ( ∆ θ ) , Yaw ( ∆ ψ ), and velocity ( ∆ v ), which collectively describe the sensor’s movement.

4.3. Advantages of the Architecture

  • High Precision: Transformers across the pipeline allow for precise motion, depth, and semantics estimations due to their ability to capture long-range dependencies and contextual information within data.
  • Integrated Depth and Motion Estimation: By unifying flow, stereo, and depth estimation, the system likely benefits from a more cohesive and robust depth map informed by motion across frames.
  • Effective Semantic Segmentation: HRNet’s ability to maintain high-resolution information through the network provides detailed and accurate semantic segmentation, which can significantly improve the quality of masking and dynamic object filtration.
  • Dynamic Object Handling: Including a dedicated block for dynamic object filtration helps the system isolate the camera’s motion from the motion of objects in the environment, which is critical for accurate pose estimation.
  • Temporal Understanding: Stacking images over time enables the model to understand the temporal dynamics of the scene better, leading to more accurate pose estimations.
  • Versatility: The architecture can be adapted to various environments by retraining with suitable datasets.
The proposed architecture reflects a comprehensive approach to pose estimation, integrating various sophisticated computer vision techniques and the powerful capabilities of transformer models. It is designed to provide accurate and detailed pose estimations that could be used in various applications, from autonomous vehicles to augmented reality systems.

5. Experimental Results and Analysis

In this section, we present the results of our experiments and analyze the performance of the proposed adaptive framework for outdoor pose estimation. The experiments were conducted using the KITTI odometry dataset, which is widely recognized for its application in tasks like autonomous driving and visual odometry. We evaluated our framework using various preprocessing methods and hyperparameters, aiming to identify the most effective configuration for accurate pose estimation. Our results are detailed through comparisons of training and validation losses, sensitivity analyses, and evaluations of different learning rates, providing a comprehensive understanding of the model’s behavior and its performance across diverse conditions.

5.1. Experimental Setup

We conducted extensive experiments using the KITTI odometry dataset to evaluate the effectiveness of our proposed adaptive framework for outdoor pose estimation. These experiments were performed on an NVIDIA RTX 3090 GPU (Toronto, ON, Canada), ensuring efficient training and testing processes. The training was carried out over 200 epochs with a learning rate of 0.008, typically requiring 9 to 14 h per training session. The Adagrad optimizer was employed alongside the ReLU activation function to optimize the model parameters.

5.2. Dataset Division

The KITTI odometry dataset is a widely used benchmark in computer vision and robotics, particularly for tasks involving autonomous driving and visual odometry. This dataset provides raw data from a moving vehicle, including stereo camera images, 3D laser scans, and GPS/IMU data. A crucial aspect of visual odometry involves estimating the vehicle’s motion, represented by the transformation matrix [R|t], where R is a 3 × 3 rotation matrix, and t is a 3 × 1 translation vector. The [R|t] matrix is used to transform a point p in the coordinate frame of one camera position to another camera position as follows:
p = ´   R p + t
Here, p is the point in the original coordinate frame, R rotates this point, and t translates it. For example, given a sequence of images from the KITTI dataset, the visual odometry algorithm aims to estimate the successive [R|t] matrices that describe the vehicle’s trajectory through the environment. Accurate estimation of these matrices is essential for applications such as 3D mapping, navigation, and autonomous driving, providing a basis for understanding and interpreting the vehicle’s movement in real-world scenarios. The KITTI odometry dataset was divided into training, validation, and testing subsets to facilitate a robust evaluation of our models. Specifically, the training subset consisted of sequences [00, 01, 02, 05, 08, 09], which are relatively long and provide substantial data for learning robust features. For validation, sequences [04, 06, 07, 10] were used to tune the model and prevent overfitting. Finally, all available sequences were used for testing to comprehensively evaluate our model’s performance.

5.3. Preprocessing Methods

We explored four different preprocessing methods to assess their impact on pose estimation accuracy.
  • Base Model (Brown Color): This baseline model does not utilize preprocessing steps and directly inputs raw images into the neural network. It is the original architecture of DeepVO [52].
  • Optical Flow Only (Yellow Color): This method involves computing optical flow between consecutive frames to capture motion information, which is then used as input to the model.
  • Optical Flow with Depth Estimation (Green Color): In this approach, optical flow and depth maps are computed for each frame pair. The depth estimation provides additional spatial information about the scene, improving the model’s understanding of the environment.
  • Optical Flow with Depth and Semantic Segmentation (Purple Color): This comprehensive preprocessing method combines optical flow, depth estimation, and semantic segmentation. The semantic segmentation provides context about different objects in the scene, further enhancing the model’s capability to interpret complex environments.

5.4. Evaluation Metrics

The performance of each preprocessing method was evaluated using the mean squared error (MSE) for both translation and rotation. The combined error metric used in our analysis is:
E r r o r = T r a n s l a t i o n   M S E + k × R o t a t i o n   M S E
where k = 100 to appropriately scale the rotation error for aggregation with the translation error.

5.5. Training Results

The results of our experiments are summarized in Figure 8, Figure 9, Figure 10 and Figure 11 and Table 3, which depict the training and validation loss for each preprocessing method. In the training loss chart Figure 8, the base model (brown) demonstrates the highest loss throughout the epochs, indicating its inferior performance without preprocessing. The optical flow-only model (yellow) significantly improves over the base model. Further enhancements are observed with the optical flow combined with depth estimation (green). The model achieves the best performance using optical flow, depth, and semantic segmentation (purple), which consistently exhibits the lowest loss values. The validation loss chart in Figure 9 corroborates these findings. The base model again performs the worst, while the optical flow-only model performs better but shows some fluctuation in the early epochs. The optical flow and depth estimation model achieves a more stable and lower validation loss. The model integrating optical flow, depth, and semantic segmentation continues to outperform the others, maintaining the lowest validation loss across the epochs. These results indicate that incorporating advanced preprocessing techniques, particularly optical flow, depth estimation, and semantic segmentation, substantially improves the pose estimation performance. This validates the effectiveness of our proposed framework and its potential for robust outdoor pose estimation.
Figure 8. Train/Loss chart for the KITTI odometry dataset.
Figure 9. The validation/Loss chart for the KITTI odometry dataset shows that the pose estimator can learn more rapidly by providing extra scene information, especially semantic segmentation, to add correction weight to each class of objects and remove dynamic ones from the estimations.
Figure 10. Train and validation loss for different preprocessing stages, including no preprocessing, OF, OF with depth estimation, and OF with depth and semantic segmentation.
Figure 11. Proposed model’s tracking experience output for the KITTI odometry dataset. The X and Y-axis units are in meters.
Table 3. Test results on the KITTI dataset.
Table 3 presents the loss values of our visual odometry model across different preprocessing techniques, demonstrating a clear performance hierarchy. Without preprocessing, the baseline model (DeepVO) exhibits the highest losses across almost all sequences, with a training loss of 0.0058 and a validation loss of 0.0485. Introducing optical flow as a preprocessing step significantly reduces the losses, achieving a training loss of 0.0035 and a validation loss of 0.0426. Further enhancement is observed when depth estimation is incorporated alongside optical flow, lowering the training and validation losses to 0.0028 and 0.0254, respectively. The most substantial improvement is seen with semantic segmentation in addition to optical flow and depth estimation, resulting in the lowest training loss of 0.0026 and a validation loss of 0.0101. This comprehensive preprocessing approach consistently yields the lowest losses across all sequences, underscoring its effectiveness in enhancing model performance for visual odometry tasks.
Figure 9 shows the validation/loss chart for the KITTI odometry dataset, which shows that the pose estimator can learn more rapidly by providing extra scene information, especially semantic segmentation, to add correction weight to each class of objects and remove dynamic ones from the estimations.
Figure 10 presents a bar graph showing the train and validation loss for different preprocessing stages, including no preprocessing, OF, OF with depth estimation, and OF with depth and semantic segmentation. It shows adding extra information to the pose estimator helps it to have a more accurate 6DoF estimation. It also significantly lowers the enormous gap between training and validation loss. This means that preprocessing helps generalization considerably.
Figure 11 provides the proposed model’s tracking experience output for the KITTI odometry dataset. The X and Y-axis units are in meters. The drifting error appears after a few hundred meters. It is the main challenge in visual odometry. As presented, the translation estimation is accurate, but the rotation estimation still needs to be more accurate to map the trajectories precisely.

5.6. Sensitivity Analysis

In this analysis, we explore the impact of different learning rates on the training and validation performance of a 6DoF pose estimation model. The learning rates examined are 0.008, 0.005, 0.011, and 0.014. The objective is to determine the optimal learning rate that balances rapid convergence and stable performance across training and validation datasets.

5.6.1. Training Loss Analysis

The training loss curves for the different learning rates are displayed in Figure 12. Observing the curves provides insight into how each learning rate affects the model’s ability to minimize loss during training.
Figure 12. Train/Loss with different learning rates.
(1)
LR: 0.008: The training loss starts high and decreases steadily, indicating a good convergence rate. The curve shows a smooth decline, suggesting a stable training process.
(2)
LR: 0.005: This learning rate also demonstrates a steady decrease in training loss. The curve is similar to LR: 0.008 but with a slightly slower convergence rate due to the lower learning rate.
(3)
LR: 0.011: The training loss for this learning rate decreases rapidly initially but shows more fluctuations compared to the previous two. This suggests that the higher learning rate introduces instability while the convergence is fast.
(4)
LR: 0.014: The highest learning rate among the four shows the most rapid initial decrease in training loss. However, this curve also displays more pronounced fluctuations, indicating potential instability during training.

5.6.2. Validation Loss Analysis

The validation loss curves indicate how well the model generalizes to unseen data. These curves are shown in Figure 13.
Figure 13. Validation/Loss with different learning rates.
(1)
LR: 0.008: The validation loss decreases steadily but exhibits some fluctuations. This indicates a good generalization with some overfitting tendencies, which might be addressed with further regularization techniques.
(2)
LR: 0.005: The validation loss shows a smooth decline with fewer fluctuations than LR: 0.008, suggesting a better generalization capability with reduced overfitting.
(3)
LR: 0.011: Similar to the training loss, the validation loss for this learning rate decreases rapidly but with noticeable fluctuations. This highlights the tradeoff between fast convergence and stability.
(4)
LR: 0.014: The validation loss curve shows the most significant fluctuations, indicating that while the model converges quickly, it may suffer from substantial overfitting and instability issues.
Our analysis reveals that lower learning rates (0.005 and 0.008) offer a more stable training and validation process with less fluctuation in the loss curves. Among these, LR: 0.005, with a learning rate of 0.005, appears to provide the best balance between convergence speed and stability, showing smoother curves with fewer fluctuations. Higher learning rates (0.011 and 0.014) enable faster convergence but at the cost of increased instability and potential overfitting, as indicated by the more pronounced fluctuations in training and validation loss curves. Therefore, a learning rate of 0.005 is recommended for the 6DoF pose estimation task. This rate provides a good balance, ensuring stable and generalizable performance.

6. Conclusions

In conclusion, our experiments on the KITTI odometry dataset demonstrate the benefits of incorporating sophisticated preprocessing methods into the pose estimation pipeline. Our results highlight the importance of leveraging multiple sources of information, such as optical flow, depth, and semantic segmentation, to achieve accurate and reliable pose estimation in challenging outdoor environments. Moreover, a substantial body of research and numerous papers focused on pose estimation using LiDAR or stereo cameras, which leverage multiple viewpoints or sensor types to enhance accuracy and robustness. However, few studies address pose estimation using mono images, where only a single camera is employed. This gap is significant because many drones, cars, and other vehicles are equipped with just one camera, making it essential to develop models that can deliver reliable and accurate pose estimation from this limited data source. Our work aims to fill this gap by proposing a novel approach for pose estimation using mono images. Focusing on a single-camera setup addresses a critical need in the field, ensuring that our model can be widely applicable to various platforms and systems that rely on a single visual input. This contribution is significant for cost-effective and resource-constrained applications where deploying additional sensors may not be feasible.

Author Contributions

Conceptualization, A.G. and R.K.; methodology, A.G. and R.K.; software, A.G.; validation, A.G. and R.K.; formal analysis, A.G. and R.K.; investigation, A.G. and R.K.; data curation, A.G.; writing—original draft preparation, A.G.; writing—review and editing, A.G. and R.K.; visualization, A.G.; supervision, R.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Luo, A.; Yang, F.; Luo, K.; Li, X.; Fan, H.; Liu, S. Learning Optical Flow with Adaptive Graph Reasoning. Proc. AAAI Conf. Artif. Intell. 2022, 36, 1890–1898. [Google Scholar] [CrossRef] [Scilit]
  2. Jeong, J.; Lin, J.M.; Porikli, F.; Kwak, N. Imposing Consistency for Optical Flow Estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 3171–3181. [Google Scholar] [CrossRef] [Scilit]
  3. Bai, S.; Geng, Z.; Savani, Y.; Kolter, J.Z. Deep Equilibrium Optical Flow Estimation. In Proceedings of the 2022 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 610–620. [Google Scholar] [CrossRef] [Scilit]
  4. Huang, Z.; Shi, X.; Zhang, C.; Wang, Q.; Cheung, K.C.; Qin, H.; Li, H. FlowFormer: A Transformer Architecture for Optical Flow; Springer Nature Switzerland: Cham, Switzerland, 2022. [Google Scholar]
  5. Wan, Z.; Dai, Y.; Mao, Y. Learning Dense and Continuous Optical Flow From an Event Camera. IEEE Trans. Image Process. 2022, 31, 7237–7251. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  6. Deng, C.; Luo, A.; Huang, H.; Ma, S.; Liu, J.; Liu, S. Explicit Motion Disentangling for Efficient Optical Flow Estimation. In Proceedings of the 2023 IEEE International Conference on Computer Vision, Paris, France, 1–6 October 2023; pp. 9487–9496. [Google Scholar] [CrossRef] [Scilit]
  7. Luo, A.; Yang, F.; Li, X.; Liu, S. Learning Optical Flow with Kernel Patch Attention. In Proceedings of the 2022 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 8896–8905. [Google Scholar] [CrossRef] [Scilit]
  8. Zhao, S.; Zhao, L.; Zhang, Z.; Zhou, E.; Metaxas, D. Global Matching with Overlapping Attention for Optical Flow Estimation. In Proceedings of the 2022 the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 17571–17580. [Google Scholar] [CrossRef] [Scilit]
  9. Hu, L.; Zhao, R.; Ding, Z.; Ma, L.; Shi, B.; Xiong, R.; Huang, T. Optical Flow Estimation for Spiking Camera. In Proceedings of the 2022 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 17823–17832. [Google Scholar] [CrossRef] [Scilit]
  10. Tu, Z.; Li, H.; Xie, W.; Liu, Y.; Zhang, S.; Li, B.; Yuan, J. Optical flow for video super-resolution: A survey. Artif. Intell. Rev. 2022, 55, 6505–6546. [Google Scholar] [CrossRef] [Scilit]
  11. Guizilini, V.; Lee, K.-H.; Ambrus, R.; Gaidon, A. Learning Optical Flow, Depth, and Scene Flow Without Real-World Labels. IEEE Robot. Autom. Lett. 2022, 7, 3491–3498. [Google Scholar] [CrossRef] [Scilit]
  12. Liang, Y.; Liu, J.; Zhang, D.; Fu, Y. MPI-Flow: Learning Realistic Optical Flow with Multiplane Images. In Proceedings of the 2023 IEEE International Conference on Computer Vision, Paris, France, 1–6 October 2023; pp. 13811–13822. [Google Scholar] [CrossRef] [Scilit]
  13. Han, Y.; Luo, K.; Luo, A.; Liu, J.; Fan, H.; Luo, G.; Liu, S. RealFlow: EM-Based Realistic Optical Flow Dataset Generation from Videos; Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer Nature: Cham, Switzerland, 2022; Volume 13679 LNCS, pp. 288–305. [Google Scholar] [CrossRef] [Scilit]
  14. Luo, A.; Yang, F.; Li, X.; Nie, L.; Lin, C.; Fan, H.; Liu, S. GAFlow: Incorporating Gaussian Attention into Optical Flow. In Proceedings of the 2023 IEEE International Conference on Computer Vision, Paris, France, 1–6 October 2023; pp. 9608–9617. [Google Scholar] [CrossRef] [Scilit]
  15. Xu, H.; Zhang, J.; Cai, J.; Rezatofighi, H.; Tao, D. GMFlow: Learning Optical Flow via Global Matching. In Proceedings of the 2022 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 8111–8120. [Google Scholar] [CrossRef] [Scilit]
  16. Shi, X.; Huang, Z.; Li, D.; Zhang, M.; Cheung, K.C.; See, S.; Qin, H.; Dai, J.; Li, H. FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation. arXiv 2023, arXiv:2303.01237. Available online: http://arxiv.org/abs/2303.01237 (accessed on 2 June 2024).
  17. Dong, Q.; Cao, C.; Fu, Y. Rethinking Optical Flow from Geometric Matching Consistent Perspective. In Proceedings of the 2023 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; Volume 2023, pp. 1337–1347. [Google Scholar] [CrossRef] [Scilit]
  18. Zheng, Z.; Nie, N.; Ling, Z.; Xiong, P.; Liu, J.; Wang, H.; Li, J. DIP: Deep Inverse Patchmatch for High-Resolution Optical Flow. In Proceedings of the 2022 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 8915–8924. [Google Scholar] [CrossRef] [Scilit]
  19. Sui, X.; Li, S.; Geng, X.; Wu, Y.; Xu, X.; Liu, Y.; Goh, R.; Zhu, H. CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow. In Proceedings of the 2022 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 17581–17590. [Google Scholar] [CrossRef] [Scilit]
  20. Lyu, X.; Liu, L.; Wang, M.; Kong, X.; Liu, L.; Liu, Y.; Chen, X.; Yuan, Y. HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, Online, 2–9 February 2021; Volume 3B, pp. 2294–2301. [Google Scholar] [CrossRef] [Scilit]
  21. Li, Y.; Ge, Z.; Yu, G.; Yang, J.; Wang, Z.; Shi, Y.; Sun, J.; Li, Z. BEVDepth: Acquisition of Reliable Depth for Multi-View 3D Object Detection. Proc. AAAI Conf. Artif. Intell. 2023, 37, 1477–1485. [Google Scholar] [CrossRef] [Scilit]
  22. Li, Z.; Chen, Z.; Liu, X.; Jiang, J. DepthFormer: Exploiting Long-range Correlation and Local Information for Accurate Monocular Depth Estimation. Mach. Intell. Res. 2023, 20, 837–854. [Google Scholar] [CrossRef] [Scilit]
  23. Bhat, S.F.; Alhashim, I.; Wonka, P. AdaBins: Depth Estimation Using Adaptive Bins. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 4008–4017. [Google Scholar] [CrossRef] [Scilit]
  24. Hoyer, L.; Dai, D.; Wang, Q.; Chen, Y.; Van Gool, L. Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation. Int. J. Comput. Vis. 2023, 131, 2070–2096. [Google Scholar] [CrossRef] [Scilit]
  25. Ramamonjisoa, M.; Firman, M.; Watson, J.; Lepetit, V.; Turmukhambetov, D. Single Image Depth Prediction with Wavelet Decomposition. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 11084–11093. [Google Scholar] [CrossRef] [Scilit]
  26. Yuan, W.; Gu, X.; Dai, Z.; Zhu, S.; Tan, P. Neural Window Fully-connected CRFs for Monocular Depth Estimation. In Proceedings of the 2022 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; Volume 2022, pp. 3906–3915. [Google Scholar] [CrossRef] [Scilit]
  27. Li, Z.; Liu, X.; Drenkow, N.; Ding, A.; Creighton, F.X.; Taylor, R.H.; Unberath, M. Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers. In Proceedings of the 2021 IEEE International Conference on Computer Vision, Montreal, BC, Canada, 17 October 2021; pp. 6177–6186. [Google Scholar] [CrossRef] [Scilit]
  28. Zhao, C.; Zhang, Y.; Poggi, M.; Tosi, F.; Guo, X.; Zhu, Z.; Huang, G.; Tang, Y.; Mattoccia, S. MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer. In Proceedings of the 2022 International Conference on 3D Vision (3DV), Prague, Czech Republic, 12–16 September 2022; pp. 668–678. [Google Scholar] [CrossRef] [Scilit]
  29. Chen, Y.; Zhao, H.; Hu, Z.; Peng, J. Attention-based context aggregation network for monocular depth estimation. Int. J. Mach. Learn. Cybern. 2021, 12, 1583–1596. [Google Scholar] [CrossRef] [Scilit]
  30. Miangoleh, S.M.H.; Dille, S.; Mai, L.; Paris, S.; Aksoy, Y. Boosting Monocular Depth Estimation Models to High-resolution via Content-adaptive Multi-Resolution Merging. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 9680–9689. [Google Scholar] [CrossRef] [Scilit]
  31. Hoyer, L.; Dai, D.; Chen, Y.; Koring, A.; Saha, S.; Van Gool, L. Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 11125–11135. [Google Scholar] [CrossRef] [Scilit]
  32. Wang, J.; Sun, K.; Cheng, T.; Jiang, B.; Deng, C.; Zhao, Y.; Liu, D.; Mu, Y.; Tan, M.; Wang, X.; et al. Deep High-Resolution Representation Learning for Visual Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 3349–3364. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  33. Fan, M.; Lai, S.; Huang, J.; Wei, X.; Chai, Z.; Luo, J.; Wei, X. Rethinking BiSeNet For Real-time Semantic Segmentation. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; Volume 2, pp. 9711–9720. [Google Scholar] [CrossRef] [Scilit]
  34. Huynh, C.; Tran, A.T.; Luu, K.; Hoai, M. Progressive Semantic Segmentation. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; Volume 1, pp. 16750–16759. [Google Scholar] [CrossRef] [Scilit]
  35. Xu, J.; Xiong, Z.; Bhattacharyya, S.P. PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers. In Proceedings of the 2023 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; Volume 2023, pp. 19529–19539. [Google Scholar] [CrossRef] [Scilit]
  36. Sun, Y.; Zheng, W. HRNet- and PSPNet-based multiband semantic segmentation of remote sensing images. Neural Comput. Appl. 2023, 35, 8667–8675. [Google Scholar] [CrossRef] [Scilit]
  37. Strudel, R.; Garcia, R.; Laptev, I.; Schmid, C. Segmenter: Transformer for Semantic Segmentation. In Proceedings of the 2021 IEEE International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 7242–7252. [Google Scholar]
  38. Yuan, Y.; Huang, L.; Guo, J.; Zhang, C.; Chen, X.; Wang, J. OCNet: Object Context for Semantic Segmentation. Int. J. Comput. Vis. 2021, 129, 2375–2398. [Google Scholar] [CrossRef] [Scilit]
  39. Wu, H.; Liang, C.; Liu, M.; Wen, Z. Optimized HRNet for image semantic segmentation. Expert Syst. Appl. 2020, 174, 114532. [Google Scholar] [CrossRef] [Scilit]
  40. Xu, Z.; Zhang, W.; Zhang, T.; Li, J. HRCNet: High-resolution context extraction network for semantic segmentation of remote sensing images. Remote Sens. 2021, 13, 71. [Google Scholar] [CrossRef] [Scilit]
  41. Zhi, H.; Yin, C.; Li, H.; Pang, S. An Unsupervised Monocular Visual Odometry Based on Multi-Scale Modeling. Sensors 2022, 22, 5193. [Google Scholar] [CrossRef] [Scilit]
  42. Liu, X.; Zhang, T.; Liu, M. Joint estimation of pose, depth, and optical flow with a competition–cooperation transformer network. Neural Netw. 2024, 171, 263–275. [Google Scholar] [CrossRef] [Scilit]
  43. Zhao, C.; Tang, Y.; Sun, Q.; Vasilakos, A.V. Deep Direct Visual Odometry. IEEE Trans. Intell. Transp. Syst. 2022, 23, 7733–7742. [Google Scholar] [CrossRef] [Scilit]
  44. Song, R.; Zhu, R.; Xiao, Z.; Yan, B. ContextAVO: Local context guided and refining poses for deep visual odometry. Neurocomputing 2023, 533, 86–103. [Google Scholar] [CrossRef] [Scilit]
  45. Tu, Z.; Chen, C.; Pan, X.; Liu, R.; Cui, J.; Mao, J. EMA-VIO: Deep Visual–Inertial Odometry With External Memory Attention. IEEE Sens. J. 2022, 22, 20877–20885. [Google Scholar] [CrossRef] [Scilit]
  46. Xu, K.; Hao, Y.; Yuan, S.; Wang, C.; Xie, L. AirVO: An Illumination-Robust Point-Line Visual Odometry. In Proceedings of the 2023 IEEE International Conference on Intelligent Robots and Systems, Detroit, MI, USA, 1–5 October 2023; pp. 3429–3436. [Google Scholar] [CrossRef] [Scilit]
  47. Zhu, R.; Yang, M.; Liu, W.; Song, R.; Yan, B.; Xiao, Z. DeepAVO: Efficient pose refining with feature distilling for deep Visual Odometry. Neurocomputing 2022, 467, 22–35. [Google Scholar] [CrossRef] [Scilit]
  48. Yang, N.; von Stumberg, L.; Wang, R.; Cremers, D. D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry. In Proceedings of the 2020 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 19 June 2020; pp. 1278–1289. [Google Scholar] [CrossRef] [Scilit]
  49. Karaev, N.; Rocco, I.; Graham, B.; Neverova, N.; Vedaldi, A.; Rupprecht, C. CoTracker: It is Better to Track Together. arXiv 2023, arXiv:2307.07635. Available online: http://arxiv.org/abs/2307.07635 (accessed on 2 June 2024).
  50. Xu, H.; Zhang, J.; Cai, J.; Rezatofighi, H.; Yu, F.; Tao, D.; Geiger, A. Unifying Flow, Stereo and Depth Estimation. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 13941–13958. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  51. Lambert, J.W.; Liu, Z.; Sener, O.; Hays, J.; Koltun, V. MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 796–810. [Google Scholar] [CrossRef] [Scilit]
  52. Wang, S.; Clark, R.; Wen, H.; Trigoni, N. Deepvo: Towards end-to-end visual odometry with deep Recurrent Convolutional Neural Networks. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017; IEEE: New York, NY, USA, 2017; pp. 2043–2050. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.