Error Correction Using Bayesian GRU Network in Hybrid Visual Inertial Navigation System ^†

Abstract

Vision-based navigation systems (VINS) are increasingly utilised as an alternative to GNSS for UAVs operating in urban environments, but they suffer from performance degradation under visual fault conditions like illumination variation, rapid motion, texture-less environments, and weather effects. While hybrid architecture incorporating Kalman filters and machine learning (ML) improves performance, they often lack evidence of providing contingency for non-Gaussian error distributions, limiting operational safety. To address these shortcomings, an enhanced hybrid VINS architecture is proposed, featuring a Bayesian GRU-based error correction network (B-GRU) to provide a contingency while compensating model errors. To the best of the authors’ knowledge, this is the first attempt to estimate uncertainty using a B-GRU compensator while addressing data uncertainty for VINS applications. The system architecture integrates an Error-State Kalman Filter (ESKF) and the B-GRU, compensating for position errors with uncertainty prediction. The proposed approach is validated using datasets from MATLAB incorporated in an Unreal Engine simulated environment, replicating the complex fault conditions. The ML model is trained on various visual failure modes to adapt the variability in the signal patterns during flights with simulated datasets and tested across varied flight paths and lighting scenarios. The results demonstrate that the fusion strategy effectively corrects erroneous measurements arising from corrupted sensor data and imperfect models and achieves an improvement of 78.06% compared to SOTA hybrid VIO on the horizontal axis while capturing complex flight dynamics in an unseen environment. A comparative analysis demonstrates the effectiveness of B-GRU in mitigating failure modes with a predictive error boundary, achieving a 72% improvement in performance compared to the architecture that integrates GRU-based error compensation. This approach shows a step forward in enhancing positioning accuracy and contingency in challenging urban environments.

1. Introduction

Urban Air Mobility (UAM) aims to establish a safe and efficient system for short-range air transportation, utilising highly automated aircraft to carry transport passengers or cargo at lower altitudes within urban environments. Operational safety remains the top priority, with navigation safety being a critical prerequisite [1]. A significant challenge in urban navigation is ensuring the reliability of core sensors like the Global Navigation Satellite System (GNSS) for PNT. In urban canyons, GNSS signals are frequently affected by factors such as no-line-of-sight (NLOS) and multipath effects, necessitating the integration of alternative sensors with GNSS to achieve accurate positioning [2]. Vision-based navigation systems (VINS) are increasingly utilised as an alternative to GNSS for UAVs operating in urban environments, offering a promising solution to address navigation challenges [1]. Performance degradation is attributed to the current state-of-the-art systems in complex scenarios due to visual fault conditions like illumination variation, rapid motion, texture-less environments, and weather effects [3,4]. A common approach to improving navigation integrity involves incorporating fault detection in feature-based visual odometry (VO) algorithms. However, these methods often struggle to bound errors arising from various sources of measurement uncertainties in urban environments [1,5]. To address these challenges, hybrid architectures have been integrated into VIO systems to compensate for errors in corrupted VO sensors and Kalman filter (KF) model measurements using machine learning (ML), thereby enhancing performance while accounting for various source of uncertainties under complex urban environments [6]. While hybrid architectures incorporating Kalman filters and machine learning (ML) improve performance, they lack evidence of providing a contingency for non-Gaussian error distributions, limiting the operational safety.

Deep learning methods like LSTM with Monte Carlo Dropout [7] variational autoencoders (VAEs) [8] and RCNN with mixture density network [9] estimate the pose with confidence but lack robust statistical guarantees in visually degraded environments. Furthermore, the current state of the art involving such architectures faces challenges, including reliance on large datasets [4], performance degradation in out-of-distribution test scenarios and insufficient uncertainty estimation [7,8,9]. To address these shortcomings, an enhanced hybrid VINS architecture is proposed, featuring a Bayesian Gated Recurrent Unit (GRU)-based error correction network (B-GRU) that provides a contingency while compensating for model errors. To the best of the authors’ knowledge, this is the first attempt to estimate uncertainty using the B-GRU compensator while addressing data uncertainty for VINS applications. The system architecture integrates an Error-State Kalman Filter (ESKF) and the B-GRU, compensating for position errors with uncertainty prediction. This research employs a B-GRU for efficient, real-time uncertainty estimation in navigation, offering a simpler, faster alternative to more complex Bayesian LSTM and Bayesian Bi-LSTM models by leveraging Monte Carlo Dropout (MCD) for uncertainty quantification. The formulation is extended by providing the out-of-the-distribution analysis by conducting the comparative analysis based on datasets collected from two different environments replicating fault conditions: a photorealistic environment built with the MATLAB incorporated Unreal Engine 2022a, and Euroc datasets [10] covering a complex indoor environment in real time.

The remainder of this study is structured as follows: Section 2 includes the proposed system architecture with a brief description. Section 3 discusses the experimental settings for data generation to train and test the solution. Section 4 discusses the test and performance analysis of the proposed approach under different scenarios with comparisons. Finally, the conclusion is presented in Section 5.

2. Proposed Bayesian GRU Error Correction-Aided Hybrid Visual Inertial Navigation System

This section presents a high-level overview of the proposed novel framework of hybrid VIO. Monocular camera images are first processed using a feature-based algorithm to estimate the VO position [11]. The estimated VO position is then fused with raw IMU measurements, including linear acceleration and angular velocity, through an ESKF to obtain the VIO position estimate. To further improve the accuracy of the ESKF output and mitigate errors arising from model uncertainty, a B-GRU is employed to predict position increments along with confidence measures. These confidence measures define the error bounds associated with faults or uncertainties that are present in the environment, helping to enhance the VIO performance by reducing the epistemic uncertainty. Figure 1 presents a high-level diagram of the proposed hybrid VIO by utilising the error correction mechanism with confidence measures.

2.1. Bayesian GRU-Aided Error Correction

Bayesian GRU integrates the Bayesian interface into the GRU neural network by replacing fixed weight estimates with probability distributions, enabling uncertainty quantification in predictions. This approach enhances the model’s robustness and improves its generalisation capabilities. This study introduces a single layer of GRU incorporating the MCD sampling method in the Bayesian interface, based on an extensive study that emphasised MCD’s balance of accuracy and computational efficiency [12]. The MCD technique mitigates overfitting by randomly dropping nodes during the training process, effectively setting their connected weight to zero. During inference, the MCD algorithm operates multiple stochastic forward passes through a neural network for a given input of $x$, applying dropout with a specified probability to randomly deactivate a fraction of units during each pass. This results from a set of outputs and $y$ from the multiple passes, and the final prediction is obtained by averaging these outputs, as shown in Equation (1). The associated uncertainty is estimated from the sample distribution, with $T$ forward passes chosen as being sufficiently large to ensure statistical significance.

$$y={\displaystyle \frac{1}{T}}\sum _{t=1}^T{y}_t$$

(1)

It is crucial to select the input and output for the ML module, since the parameters directly determine the training efficiency and navigation accuracy. B-GRU is applied to compensate for measurement model errors associated with the imperfection of the model, due to manual tuning and nonlinearity presents in the data. The ${∆P}_{VIO}$ model predicts the position error between the ESKF measured position and the ground truth with the input parameters of ESKF estimated position and innovation, i.e., the difference between the predicted observations and observed value in the NED frame. The Bayesian GRU model error compensator learns from the state-evaluation process and associated uncertainties, and the MCD facilities model the uncertainty estimation, utilising 500 samples to provide the reliability of the estimated mean and 99.7th percentile or 3$\sigma$ confidence interval (CI). Notably, the increment of ESKF-VIO is selected as the ground truth in these cases for the evaluation of the prediction of the Bayesian GRU model.

2.2. Hybrid Vio Navigation System

The proposed novel architecture is designed with a combination of B-GRU error correction and ESKF to estimate the improved positioning performance under dynamic conditions. Although the ESKF formulation follows the foundation established in [6,13,14], the difference in this work lies in the way it is integrated within a hybrid framework with confidence measure prediction. In this formulation, the UAV state $x$ is defined as

$$x={\left[p v q {\alpha }_b {\omega }_b\right]}^T$$

(2)

Let the nominal state vector of the ESKF be defined as the position, velocity, attitude and imu bias, where $p=\left[p_N,p_E,p_D\right]$ denotes the position in the platform; $v=\left[v_N,v_E,v_D\right]$ denotes the platform’s velocity; and $q=\left[q_N,q_E,q_D\right]$ represents the orientation in the form of quaternion. The acceleration bias is represented by ${\alpha }_b=\left[{{\alpha }_b}_N,{{\alpha }_b}_E,{{\alpha }_b}_D\right]$, while the gyroscope bias is represented by ${\omega }_b=\left[{{\omega }_b}_N,{{\omega }_b}_E,{{\omega }_b}_D\right].$ In the ESKF framework, the full state is decomposed into the nominal state $x$, and the error state $\delta x$, such that the true state can be expressed as

$$x_k=x+\delta x$$

(3)

The key idea of the ESKF is to propagate the nominal state using the nonlinear system model, while separately estimating the associated error through linearised dynamics. In this work, the ESKF operates in the north–east–down (NED) reference frame. The measurement update equations are derived as

$$\begin{gathered} K={\widehat{P}}_{k-1}{H}^T{\left(H{\widehat{P}}_{k-1}{H}^T+R\right)}^{-1} \\ \delta \widehat{x}={K(y}_k-h\left({\widehat{x}}_t\right)) \\ \widehat{P}=\left(I-KH\right){\widehat{P}}_{k-1} \end{gathered}$$

(4)

Finally, the ESKF measurement position is corrected following the equation and the estimate improved the proposed hybrid VIO position.

$$P_{VIO-corrected}=P_{VIO}-{∆P}_{VIO}$$

(5)

3. Experimental Setup and Dataset Generation

This section outlines the experimental setup to evaluate the performance of the proposed B-GRU-aided hybrid VIO. The simulation environment and data collection methodology were adopted from earlier work [6,13,14] based on MATLAB-incorporated Unreal Engine simulated urban environment. The simulated datasets involve diverse challenging conditions, including illumination variation, rapid motion, altitude variation, no-texture, and weather effects for shorter and extended flights. A monocular camera (720 × 1280 px, 10 Hz, 1109 px focal length) and simulated IMU (ICM 20649 at 100 Hz) were used. This study only investigates the sunny weather condition dataset and compares the performance against prior work [6,14]. Additionally, the Euroc dataset is utilised to assess generalisation, with an estimation of the error boundary in an unseen environment which is also compared with previous work [6,14]. Figure 2 illustrates examples of training and testing datasets that are utilised for the evaluation of the proposed novel approach.

3.1. Training Phase

The ML modules are trained with 12 trajectories in sunny weather conditions replicating fault scenarios like flight dynamics, environmental structures and illumination variations that can trigger multiple sources of data and model uncertainty within the navigation system involving imperfect modelling, sensor noise, data association errors, feature tracking errors and feature location errors mentioned in [6,14]. This diversity allows for the model to learn and adapt to different fault conditions, improving the overall performance and generalisation ability in the presence of multiple fault conditions. Notably, this research has utilised the GRU-aided KF model correction in Hybrid VIO architecture [14] from a previous study for comparison. The proposed B-GRU is optimised and trained using the ‘TensorFlow Keras’ Bayesian optimisation tuner, selected for its superior accuracy over other tuning methods, based on an extensive review conducted using simulated datasets. A fully connected layer outputs the predicted increments with uncertainty estimation using 500 Monte Carlo samples. The model uses a batch size of 128, a Monte Carlo dropout rate of 0.2, a Relu activation function and Relu recurrent activation function and the ADAMAX optimiser to minimise the discrepancies between the predicted and actual errors, using the mean squared error (MSE) loss.

3.2. Testing Phase

The testing dataset is designed with a combination of seen and unseen fault scenarios which are different from training datasets. During the testing phase, ML models operate in the prediction phase, remaining active throughout the experiment. The trained Bayesian GRU measurement error compensator predicts the KF estimated position error with uncertainty, based on KF’s predicted position and the observation differences from the measurement model in the current navigation frame.

3.3. Uncertainty Metrics

A good uncertainty estimation model should provide uncertainty intervals (UIs) that are as small as possible, at the same time including the ground truth value. In this study, Bayesian GRU is predicting the model error uncertainty, where the ground truth value is the position error (PE) associated with ESKF-VIO. The mean and $3\sigma$ confidence boundary are selected to evaluate the model, capturing the uncertainty in the error distribution of the target data.

4. Performance Evaluation

To evaluate the performance and adaptability of the proposed novel hybrid VIO framework, two experiments are carried out using benchmark environments adopted from prior work [6,13,14] to enable a direct comparison with the earlier GRU-aided hybrid VIO baseline. In contrast to the previous study, which mainly focused on trajectory improvement, this paper evaluates both the positioning accuracy and quality of uncertainty estimation. The experiments therefore examine: (1) out-of-the-distribution analysis to evaluate model adaptability in an unseen complex environment and (2) performance evaluation under a scarcity of features fault condition with varying lighting.

4.1. Out-of-the-Distribution Problem

The experiment evaluates whether the proposed B-GRU can generalise beyond the MATLAB-based training environment and provide uncertainty bounds in the unseen dataset. For this purpose, the Euroc MH05_difficult sequence is used as a benchmark because it contains high speed, illumination changes and motion blur. The B-GRU model is trained on the MATLAB environment and tested directly on Euroc without retraining. This experiment is intended to assess both the out-of-distribution adaptability and reliability of the predicted uncertainty intervals.

Table 1. Comparison with SOTA methods that used MH05_deficult sequence from Euroc dataset.

Techniques	RMSE (m)				Horizontal RMSE (m)	3D 95th Percentile (m)	3σ (OR%)			3σ (OR%)
	N	E	D	3D	3D		N	E	D	3D
End-to-End VIO [4]	-	-	-	1.96	-	-	-	-	-	-
DeepVIO [15]	-	-	-	0.52	-	-	-	-	-	-
Self-VIO [3]	-	-	-	0.29	-	-	-	-	-	-
UA-VO [7]	-	-	-	-	-	-	-	-	-	21.95
ESKF-VIO	1.75	0.96	0.87	1.94	1.45	1.63	35.5	39.4	29.2	-
GRU-aided Hybrid VIO [14]	0.40	0.65	0.16	0.65	0.56	0.71	-	-	-	-
B-GRU-aided Hybrid VIO	0.29	0.44	0.10	0.43	0.37	0.56	16.0	17.9	17.6	-

presents the comparative results. The proposed framework achieves the best overall performance among the considered hybrid VIO, reducing the 3D RMSE by 33.8% relative to GRU-aided hybrid VIO [14] and by 78.06% relative to the referenced SOTA hybrid VIO baseline [4]. In contrast, SOTA architectures suffered from performance degradation above 1 m in the 3D position, due to insufficiency in adapting the dynamics present in the data, e.g., Self-VIO [3], while using all the series of Euroc datasets for training their models. While this approach helps us to achieve centimetre-level accuracy by addressing challenges like the presence of rapid motion and motion blur in the flight that led to feature tracking errors and feature location errors, it struggles to maintain the confidence boundaries required [7] for uninterrupted flight integrity. In contrast, this study evaluates the proposed solution on the full dataset without a split and demonstrates significant improvement and more robustness than the induced outliers. Figure 3 illustrates that the proposed method mitigates the large error produced by feature-based VO and reduces the fluctuation remaining in GRU-aided hybrid VIO [6,14] around 60–90 s, due to rotation in the loop and a high speed of 0.88 $\mathrm{m}{\mathrm{s}}^{-1}$.

Figure 4 represents the proposed model’s ability to capture uncertainty, providing an uncertainty boundary in each coordinate. For comparison with the existing uncertainty-aware approach [7], this study has 3σ uncertainty intervals, corresponding to the 99.7th percentile. There is a gap in evaluating the performance of models using the entire Euroc dataset, due to the complexity of the data. Unlike the existing solutions concerning uncertainty in VO systems [7] where the dataset was split into 100 test samples, this study evaluates the performance of entire datasets involving 2273 frames: 22212 samples in IMU data. The proposed method has effectively predicted the error boundary compared with ESKF-VIO and mitigated erroneous measurements arising from various sources of uncertainty, with the improvement of UI of 55% in N, 55% in E and 40% in D coordinate. Additionally, it demonstrated approximately 16.0% in N, 17.9% in E and 17.6% in D coordinate of data that remain out of range (OR) compared to ESKF-VIO PE.

4.2. Performance Evaluation Under Fault Conditions

This section evaluates the proposed approach’s effectiveness in enhancing trajectory accuracy under complex flight dynamics including a texture-less environment, using unseen test data from the same simulation environment. The same dataset is retained for consistency with previous work [6,14], but the objective is to analyse the benefit of B-GRU correction. As shown in Figure 5, the proposed approach outperforms the reference systems across all coordinates. In contrast with earlier work [6], utilising B-GRU reduces the long train error induced after 200 s, due to corrupted sensor measurements and feature tracking errors caused by rotation with vibration during landing. Incorporating a confidence-aware correction approach significantly reduces errors, achieving horizontal 95th percentile errors of 3.1811 m, as shown in

Table 2. RMSE and 95th percentile summary of the performance under unseen scenario.

Techniques	RMSE (m)					95th Percentile
	N	E	V	3D PE	Horizontal PE	3D PE	Improvement	Horizontal PE (m)	Improvement
VO	4.04	5.13	18.11	27.56	6.53	33.20	-	11.36	-
ESKF-based VIO	3.7	4.83	16.00	24.19	6.15	29.63	40%	9.01	21%
GRU-aided Hybrid VIO [14]	1.76	3.46	1.96	4.31	4.16	11.34	66%	12.12	-
Multi-ML Hybrid VIO [6]	1.19	2.74	1.01	3.13	2.81	9.33	72%	8.88	22%
B-GRU-aided Hybrid VIO	1.350	0.649	0.990	3.13	1.4976	3.7519	88%	3.1811	72%

, and achieving a 74% improvement over GRU-aided hybrid VIO and a 65% improvement over traditional ESKF-VIO under complex conditions. The results indicates that the Bayesian formulation not only improves the positioning accuracy but provides robustness to the aleotoric uncertainty in complex cases.

5. Conclusions

This study presents a proposed hybrid VIO integrated with B-GRU measurement error compensation that significantly addresses model uncertainty and provides CI of bounded errors to enhance the navigation integrity in complex scenarios. Using MCD sampling, the model captures uncertainty from visual degradation, flight dynamics, and sensor noise, providing $3\sigma$ CI of errors present in the navigation environment. A series of experiments were conducted to validate the proposed approach under unseen diverse conditions, including challenging light and high flight dynamics. The solution is able to capture induced visual sources of uncertainties such as feature extraction error, feature tracking error, data association error, mismatch in dynamic, and imperfect modelling compared to ESKF-VIO, and provides $3\sigma$ UI improvement of approximately 50% in the horizontal coordinate, i.e., indoor (Euroc). Another experimental result highlights the proposed framework’s ability to mitigate position uncertainty while removing additional induced unmodelled noise, even when the sensor data is corrupted, or the model is imperfect, achieving 72% improvements (simulated urban environment). In summary, the combined performance and uncertainty gain estimates indicate that the proposed framework offers a promising vision-based A-PNT solution for safe and reliable UAV navigation under complex urban environments. This research is a step forward towards integrity monitoring using AI. Future work will focus on extending validation to real-time outdoor flights.