Event/Visual/IMU Integration for UAV-Based Indoor Navigation ^†

Abstract

Unmanned aerial vehicle (UAV) navigation in indoor environments is challenging due to varying light conditions, the dynamic clutter typical of indoor spaces, and the absence of GNSS signals. In response to these complexities, emerging sensors, such as event cameras, demonstrate significant potential in indoor navigation with their low latency and high dynamic range characteristics. Unlike traditional RGB cameras, event cameras mitigate motion blur and operate effectively in low-light conditions. Nevertheless, they exhibit limitations in terms of information output during scenarios of limited motion, in contrast to standard cameras that can capture detailed surroundings. This study proposes a novel event-based visual–inertial odometry approach for precise indoor navigation. In the proposed approach, the standard images are leveraged for feature detection and tracking, while events are aggregated into frames to track features between consecutive standard frames. The fusion of IMU measurements and feature tracks facilitates the continuous estimation of sensor states. The proposed approach is evaluated and validated using a controlled office environment simulation developed using Gazebo, employing a P230 simulated drone equipped with an event camera, an RGB camera, and IMU sensors. This simulated environment provides a testbed for evaluating and showcasing the proposed approach’s robust performance in realistic indoor navigation scenarios.

1. Introduction

Indoor navigation has attracted significant research attention due to its critical applications, including, but not limited to, inventory management, industrial inspections and monitoring, and disaster management. The absence of global navigation satellite systems (GNSSs) in indoor environments poses a significant challenge for UAV navigation. Various sensors have been explored for indoor navigation. For instance, ultra-wideband (UWB) technology has been used for UAV navigation in indoor environments [1,2,3]. UWB localizes systems by utilizing signal parameters such as time of arrival and received signal strength from fixed nodes [4]. Despite its precision, UWB needs additional infrastructure, which can be costly. Light detection and ranging (LiDAR), traditionally used in either 2D or 3D formats, offers real-time mapping capabilities. However, 2D LiDAR cannot capture height information, while 3D LiDAR provides comprehensive mapping but with a high cost, large size, and high power consumption [5,6,7,8,9].

Cameras, including monocular, stereo, and RGB-D types, are widely used for visual navigation. Monocular navigation methods include direct approaches, feature point matching, and semi-direct approaches [10]. LSD-SLAM and ORB-SLAM are notable examples of these methods [11,12]. Stereo cameras capture scenes from multiple directions to provide accurate 3D information, while RGB-D cameras offer depth information through an infrared transmitter and receiver [13,14]. Despite their advantages, these cameras struggle with illumination changes, motion blur, and limited dynamic range [15,16].

Event cameras, a novel type of sensor, have emerged due to their high temporal resolution, low latency, low power consumption, and high dynamic range [17]. Unlike traditional cameras, event cameras respond to changes in brightness asynchronously, providing a variable data rate for the sequence of events. This has enabled pose estimation using only event cameras [18,19], though real-time application remains challenging due to optimization costs [20,21,22]. Consequently, several approaches have combined event cameras with standard cameras and IMUs, effectively enhancing feature tracking and ensuring robust navigation [23,24,25,26,27].

This study proposes a novel event-based visual–inertial odometry algorithm for precise indoor navigation. In our approach, standard images are utilized for feature detection and tracking, while events are aggregated into frames to track features between consecutive standard frames. The fusion of IMU measurements with feature tracks enables the continuous estimation of sensor states. The proposed method is evaluated and validated using a controlled office environment simulation. The structure of this paper is as follows: Section 2 details the proposed approach, Section 3 discusses the experiments, and Section 4 presents the conclusions.

2. Event-Based VIO Approach

The overall methodology is summarized in the flowchart shown in Figure 1. The proposed VIO approach consists of two primary threads: the front end and the back end. The front-end thread involves detecting and tracking features in standard images, clustering events into spatiotemporal windows between standard frames, and accumulating these events into intermediate frames to improve feature tracking. These tracks are then filtered and triangulated to determine their 3D locations. IMU measurements are fused with the feature tracks in the back-end thread to continuously estimate sensor states. This methodology is implemented using the open-source library dv-mono-vio-sample by iniVation [28].

For each standard frame, events are clustered into overlapping spatiotemporal windows, which are then accumulated into event frames. Feature detection uses the FAST corner detector [29] with a bucketing grid to ensure even distribution, and feature tracking employs Lukas–Kanade tracking. Outliers in feature tracks are filtered using two-point RANSAC. In the back end, feature tracks and IMU measurements are fused to estimate sensor states using OKVIS [30], a nonlinear optimization technique.

The visual–inertial odometry problem is formulated as a joint optimization of a cost function, including weighted reprojection errors for both standard and accumulated event frames and inertial error terms. The Google Ceres optimizer [31] is used for optimization.

3. Experiments

In order to evaluate the proposed approach, a 3D indoor environment of an office was simulated based on a robot operating system (ROS) and Gazebo [32]. Gazebo is a robust, open-source simulation tool that is relied on for a wide range of applications due to its flexibility, the availability of common features, and its support for complex, distributed environments. It employs a modular approach with multiple physics engines, an extensive library of sensors, various user interfaces, and a graphical interface, making it capable of simulating both indoor and outdoor environments with high credibility [33]. The simulated environment covers an area of 600 m², as illustrated in Figure 2.

The P230 UAV model was used in our simulation and equipped with simulated sensors, including an RGB camera, an event camera, and an IMU. The event camera data were simulated using the DVS Gazebo plugin [34]. Both RGB and event cameras capture images at a resolution of 240 × 180 pixels and feature a horizontal field of view (FOV) of 103◦ and a vertical FOV of 97.7°. Figure 3 shows the simulated P230 UAV. The standard frame data were recorded at a rate of 25 Hz, whereas the IMU data were captured at 50 Hz.

The dataset was collected by sending commands to the UAV to follow a pre-determined trajectory. These commands were sent to the UAV using an offboard control node based on the PX4 SITL framework [35] and MAVROS [36]. The PX4 SITL framework is utilized to supply the software stack that facilitates interactions with a simulated UAV frame, while MAVROS is a mavlink extensible communication node for ROS, which retrieves the current state of the model and updates the setpoint position ROS topic accordingly.

The proposed approach fuses events, standard frames, and IMU measurements. Features are detected and tracked on the standard frame image; however, to enhance tracking accuracy, intermediate frames are generated from events, and tracking is executed on these frames. IMU measurements are then combined with the feature tracks to estimate the camera states. An example of the detected and tracked features on both events and standard frames is illustrated in Figure 4. Figure 5 and Figure 6 compare the estimated trajectory with respect to ground truth in 2D and 3D, respectively. The error statistics of the position for the proposed solution are detailed in

Table 1. Position (m) error statistics.

	Mean	RMSE	Max
X	0.09	0.18	0.28
Y	0.1	0.19	0.38
Z	0.1	0.26	0.27
Total		0.21

. The trajectory results compared to the ground truth show a mean RMSE of 0.21, with maximum deviations of 0.28 in the X direction, 0.38 in the Y direction, and 0.27 in the Z direction.

4. Conclusions

In this study, an event-based visual–inertial odometry algorithm was developed for precise indoor UAV navigation, addressing the complexities of GNSS-denied environments. Our approach combines events, standard frames, and IMU measurements, utilizing standard images for feature detection and tracking and intermediate frames from events to enhance tracking accuracy. IMU measurements are fused with feature tracks to estimate camera states continuously. The proposed approach was evaluated and validated using a controlled office environment simulation developed in Gazebo. A P230 simulated drone, equipped with an event camera, RGB camera, and IMU sensors, was used to test the algorithm. The UAV followed a pre-determined trajectory using commands sent via an offboard control node based on the PX4 SITL framework and MAVROS. The results demonstrate significant improvements in pose estimation accuracy, with a mean RMSE of 0.21 compared to ground truth, and maximum deviations of 0.28 in the X direction, 0.38 in the Y direction, and 0.27 in the Z direction. These findings confirm the effectiveness of our algorithm in providing accurate and reliable indoor navigation for UAVs.