LSTM Network-Assisted Binocular Visual-Inertial Person Localization Method under a Moving Base

Abstract

In order to accurately locate personnel in underground spaces, positioning equipment is required to be mounted on wearable equipment. But the wearable inertial personnel positioning equipment moves with personnel and the phenomenon of measurement reference wobble (referred to as moving base) is bound to occur, which leads to inertial measurement errors and makes the positioning accuracy degraded. A neural network-assisted binocular visual-inertial personnel positioning method is proposed to address this problem. Using visual-inertial Simultaneous Localization and Mapping to generate ground truth information (including position, velocity, acceleration data, and gyroscope data), a trained neural network is used to regress 6-dimensional inertial measurement data from the IMU data fragment under the moving base, and a position loss function is constructed based on the regressed inertial data to reduce the inertial measurement error. Finally, using vision as the observation quantity, the point feature and inertial measurement data are tightly coupled to optimize the mechanism to improve the personnel positioning accuracy. Through the actual scene experiment, it is verified that the proposed method can improve the positioning accuracy of personnel. The positioning error of the proposed algorithm is 0.50%D, and it is reduced by 92.20% under the moving base.

1. Introduction

With the continuous development of modern society, the development process of underground space is accelerating, and people spend more than 90% of their time living and working indoors or underground. Globally, there are more than 1 million emergencies such as earthquakes, fires, and mining accidents every year, which can cause huge casualties and material losses when the events occur and during the rescue process. Therefore, the demand for public safety, emergency rescue, and other fields is driving the explosive growth of high-precision and high-efficiency navigation and location services in underground space.

The commonly used underground space localization techniques include UWB (Ultra Wide Band) localization based on wireless signals, WLAN (Wireless Local Area Networks), and infrared, as well as sensor-based IMU localization and visual localization. Wireless signal-based localization methods are highly susceptible to occlusion and interference in underground space and have poor stability. Methods using inertial sensor localization, such as the PDR algorithm [1,2,3], ZUPT algorithm [4], etc., have the advantages of small size, full autonomy, all-weather use, and are not easily affected by external environmental interference and high update frequency. However, the single inertial sensor will drift due to quadratic integration and changes in human physique and walking posture. The visual positioning method has the advantages of relatively low cost, high positioning accuracy, and no cumulative error under long-time measurement [5]. However, the camera will have motion blur and feature-matching failure when it moves too fast or is obscured.

It is difficult to achieve high-accuracy positioning in underground space using a single positioning method [6]. A combined positioning approach that incorporates multiple sensors can make full use of the complementary advantages between sensors to improve the accuracy and robustness of positioning [7]. The observation information using vision can effectively estimate and correct the accumulated errors in IMU readings, while for fast movements in a short period, inertial sensors can provide some better estimates and maintain a better estimate of the position even during the period when the vision is obscured, resulting in invalid data; therefore, the positioning approach using a combination of vision and IMU has advantages.

Depending on the back-end optimization, research on visual-inertial localization [8,9] can be divided into filter-based and optimization-based approaches. The following categories of filter-based methods are available: extended Kalman filter (EKF)-based algorithms such as MonoSLAM [10]; particle filter-based methods such as FastSLAM [11] and its monocular SLAM alternative; multi-state constrained Kalman filter (MSCKF) [12,13]; and stereo multi-state constrained Kalman filter(S-MSCKF) [14].

The filter-based approach estimates both the pose of the camera and the position of the landmark in a state vector, which is a potential source of scalability inefficiency and the Markovian nature of the filter-based approach, i.e., the inability to establish the relationship between a moment and all previous states, so the mainstream visual-inertial tight coupling research is currently based on a nonlinear optimization approach.

SVO [15] is a semi-direct visual odometry, which combines the feature point method and the direct method, and finally calculates the bit pose based on optimization. The Flying Robotics Laboratory of the Hong Kong University of Science and Technology [16] proposed VINS-mono (visual-inertial system), a tightly coupled monocular visual-inertial SLAM system with a better system framework, including five parts: observation preprocessing, initialization, joint local visual-inertial optimization, global map optimization, and closed-loop detection. Campos et al. proposed the ORB-SLAM3 [17] system, which is one of the most accurate and robust algorithms available. ORB-SLAM3 is the first open-source algorithm that can support pure vision, visual-inertial, and multi-map reuse.

Visual odometry localization regarding pedestrians has received increasing attention in recent years. Alliez et al. [18] proposed a wearable real-time multi-sensor system for human localization and 3D mapping. The system incorporates IMU, laser SLAM, and visual SLAM sensors and was tested in a long-trajectory (185 m) underground space with a drift of 0.75 cm (0.41%). Kachurk et al. [19] proposed a wearable cooperative localization system for real-time indoor localization. The application scenario is mainly when GPS fails, such as in smoky rooms, stairs, and indoor environments with extreme lighting. The maximum error was 1.1% and the average error was 0.55% in a distance test of 950 m. Ruotsalainen et al. [20] investigated an infrastructure-free simultaneous localization with mapping and context recognition for tactical situational awareness. Based on the derived measurement errors, a particle filtering approach is proposed to combine with a monocular camera and foot-mounted inertial measurement unit. The error was 2.54 m in a 340 m closed-loop test of going up and down stairs and walking, which is a much higher accuracy compared to the 6.53 m error of pure PDR. Wang et al. [21] proposed a new method to exploit the short-term reliability of pedestrian trajectory derivation (PDR) to aid visual personnel localization and reduce localization errors. The method proposes that the PDR-assisted visual monitoring system can automatically select the best effect switching between VIO and PDR to provide a more accurate position indoors based on the short-term reliability of PDR. Chai et al. [22] presented a novel pedestrian dead reckoning (PDR)-aided visual-inertial SLAM, taking advantage of the enhanced vanishing point (VP) observation. The VP is integrated into the visual-inertial SLAM as an external observation without drift error to correct the system drift error. In addition, pedestrian dead reckoning velocity was employed to constrain the double integration result of acceleration measurement from the IMU. The accumulated drift error of the proposed system was less than 1 m. For inherent cumulative global drift and potential divergence facing texture-less indoor regions, Dong et al. [23] proposed a visual-inertial odometry assisted by pedestrian gait information for smartphone-based indoor positioning. The method mainly builds two additional state constraints, pedestrian velocity and step displacement, obtained by the pedestrian dead reckoning (PDR) algorithm for the visual-inertial tracking system. For each step, the corresponding residual term of step length and velocity constraints is constructed and added to the cost function for nonlinear sliding-window optimization. Niu et al. [24] proposed a pedestrian POS solution for infrastructure-free environments based on the fusion of foot-mounted IMU and a stereo camera. The stereo camera detects loop closure for mitigating the accumulated error; the foot-mounted pedestrian dead reckoning (Foot-PDR) provides reliable continuous trajectories using the IMU when the visual-based system degrades or crashes. In the office building and parking lot tests, the position error was 0.237 m and 0.227 m. The positioning accuracy of all articles is shown in Table 1.

Table 1. Pedestrian positioning method and accuracy. X means not mentioned.

References	Methods	Test Distance	Accuracy Error	ATE
[18]	incorporates IMU, laser, and visual	185 m	0.41%	X
[19]	incorporates IMU, laser, and visual	950 m	0.55%	X
[20]	Particle filtering method to combine inertial measurement units on monocular camera hinges	340 m	0.75%	X
[21]	selection of the best effect switch between VIO and PDR	1400 m	0.61%	X
[22]	PDR-aided visual-inertial	X	X	0.635 m
[23]	a visual-inertial odometry assisted by pedestrian gait information for smartphone-based	123 m	0.88%	X
[24]	fusion of foot-mounted IMU and stereo camera	X	X	0.227 m

The advent of deep learning offers new possibilities for extracting information from IMU data. Chen et al. [25] first proposed an LSTM structure that outputs relative displacements in two-dimensional polar coordinates and splices them to get the position. Hang Yan et al. proposed Robust IMU Double Integration (RIDI) [26] and Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, & New Methods(RoNIN) [27]. The two methods assumed that the orientation can be achieved from the Android device, thus rotating the IMU data to gravity-aligned frames. The former optimized the bias by regressing the velocity but still used the corrected IMU data to compute a quadratic integration to obtain the position, while the latter integrated the regressed velocity directly. Liu et al. [28] used the EKF to fuse the raw IMU data with the relative displacement measurements from the Residual Network.

For underground space personnel positioning, the wearable visual-inertial positioning undershirt equipment used in this paper is shown in Figure 1, whose inertial measurement reference will produce a moving base phenomenon with human motion, which will produce positioning error if solved according to the traditional Strapdown inertial navigation algorithm. Therefore, this paper proposes a neural network-assisted visual-inertial personnel positioning method to solve the inertial measurement errors caused by the moving base and improve positioning accuracy. The novelty of this work lies in proposing an LSTM network to regress six-dimensional IMU data and constructing two location loss functions. The regressed six-dimensional IMU data are then fused with visual to improve person localization. The contributions of this paper are as follows.

Figure 1. Wearable visual-inertial positioning undershirt equipment.

Regression of 6-dimensional inertial measurement data from the IMU data fragment under the moving base using LSTM neural network. The problem of inertial measurement error caused by the moving base is eliminated. Then, the point features are closely coupled with the inertial measurement data using vision as the observation quantity to optimize the mechanism and improve the personnel positioning accuracy.

A relative position loss function and an absolute position loss function are designed. The nature of inertial data itself is used to constrain: position information can be obtained after two integrations. Thus, the loss of neural network learning is reduced.

The remainder of this paper is organized as follows. Section 2 discusses in detail the general framework of the system, the basic mathematical description of visual-inertial, the inertial measurement error model under the dynamic base, and the neural network-assisted visual-inertial fusion localization method. Section 3 contains experiments on inertial measurement error under a dynamic base and experiments on personnel localization in an indoor–outdoor environment simulating an underground space, followed by the conclusions in Section 4.

2. Materials and Methods

In this paper, a neural network-assisted visual-inertial person localization method is proposed. The overall framework of the system is shown in Figure 2, which contains 3 parts: data preprocessing, initialization, and sliding window nonlinear optimization mechanism to estimate the positional posture. Based on RoNIN, a harness is used to attach the camera device to the body and an IMU is placed on the wearable undershirt to collect 6-dimensional inertial data. Running visual-inertial SLAM with the camera device on the body generates ground truth information (containing position information, velocity information, acceleration data, and gyroscope data). In this paper, the generated ground truth information is used as the label, the IMU data on the wearable undershirt as the training set, and the trained LSTM neural network is used to regress the 6-dimensional IMU data $(a,\omega )$. Each network window is one window with N frames of IMU data, and each frame of IMU data is 6-dimensional. The input dimension of the network is $N\times 6$, which consists of N samples of IMU data in gravity-aligned frames. Each window of the network output is N 6-dimensional IMU data, and two-position loss functions are proposed to improve the training accuracy. Finally, using vision as the observation quantity, point features and trained IMU data are combined in the sliding window to perform nonlinear optimization to solve the inertial measurement error caused by the moving base, thus improving the localization accuracy.

Figure 2. Neural network-assisted visual-inertial person localization framework diagram. The framework diagram is divided into three parts: data pre-processing, initialization, sliding window nonlinear optimization Mechanism to estimate the positional pose.

2.1. Basic Description of Visual-Inertial

Section 2.1.1 introduces and derives the point feature representation and the reprojection error of the normalized plane representation of the camera. Section 2.1.2 introduces the pre-integration and the derivation of the residual term of the pre-integration in this paper.

2.1.1. Point Feature Representation

According to the pinhole camera model [29], a point $p_w={[x_w,y_w,z_w]}^T$ in the camera world coordinate system can be mapped to a point $p_I={[u,v,1]}^T$ in the pixel plane. From Equation (1), the external reference matrix $T_{cw}$ (containing rotation matrix $R_{cw}$, translation vector $t_{cw}$) converts $p_w$ to $p_c$ under the camera coordinate system; From Equation (2), the internal reference matrix $K$ (containing parameter $f_x,f_y,c_y$) converts $p_c$ to $p_I$. Using the chi-square coordinates $s=1/z_c$ is introduced.

$$p_c=T_{cw}{p}_w=\left[\begin{array}{cc}{R}_{cw}& t_{cw}\\ 0& 1\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\end{array}\right]$$

(1)

$$p_I=\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=sK{p}_c=\frac{1}{z_c}\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right]$$

(2)

As shown in Figure 3, the $k$th spatial point $p_k$ is measured in the two camera pixel planes $z_{p_k}^{c_i}={\left[\begin{array}{ccc}{u}_{p_k}^{c_i}& v_{p_k}^{c_i}& 1\end{array}\right]}^T$, $z_{p_k}^{c_j}={\left[\begin{array}{ccc}{u}_{p_k}^{c_j}& v_{p_k}^{c_j}& 1\end{array}\right]}^T$, respectively, and the points in the pixel planes are transformed to the camera normalization planes ${\overline{p}}_{p_k}^{c_i}={\left[\begin{array}{ccc}{x}_{p_k}^{c_i}& y_{p_k}^{c_i}& 1\end{array}\right]}^T$, ${\overline{p}}_{p_k}^{c_j}={\left[\begin{array}{ccc}{x}_{p_k}^{c_j}& y_{p_k}^{c_j}& 1\end{array}\right]}^T$ by the ${\pi }^{-1}$() function, see Equations (3) and (4).

$${\overline{p}}_{p_k}^{c_i}={\pi }^{-1}\left(z_{p_k}^{c_i}\right)$$

(3)

$${\overline{p}}_{p_k}^{c_j}={\pi }^{-1}\left(z_{p_k}^{c_j}\right)$$

(4)

Figure 3. Point feature reprojection residuals diagram.

Converting ${\overline{p}}_{p_k}^{c_i}$ to $p_{p_k}^{c_j}$, see Equation (5). In this paper, the reprojection residuals are expressed in the camera-normalized plane as shown in Equation (6):

$$p_k^{c_j}=R_{bc}^T(R_{w{b}_j}^T(R_{w{b}_i}((R_{bc}\frac{1}{{\lambda }_k}[{\overline{p}}_{p_k}^{ci}]+p_{bc})+p_{w{b}_i})-p_{w{b}_j})-p_{bc})$$

(5)

where $p_k^{c_j}$ is the reprojection coordinate of the $k$ point feature under the frame $j$, ${\overline{p}}_{p_k}^{ci}$ is the projection of the $k$ feature point in the unit camera plane of the frame $i$. $(R_{w{b}_i},p_{w{b}_i})$, $(R_{w{b}_j}^T,p_{w{b}_j})$ is the position and pose of frames $i,j$, $\left(R_{bc},p_{bc}\right)$ is the external reference between IMU and camera. ${\lambda }_k$ is the inverse depth of the $k$ feature point.

$$r_p(z_{p_k}^{c_i})=\left[\frac{p_k^{c_j}}{\Vert p_k^{c_j}\Vert }-{\overline{p}}_k^{c_j}\right]=\left[\begin{array}{c}\frac{x_{p_k}^{c_j}}{z_{p_k}^{c_j}}-{\overline{x}}_{p_k}^{c_j}\\ \frac{y_{p_k}^{c_j}}{z_{p_k}^{c_j}}-{\overline{y}}_{p_k}^{c_j}\end{array}\right]$$

(6)

2.1.2. IMU Pre-Integration and Residual Measurement Model

In the continuous-time integration form, the position $p_{w{b}_k}$, velocity $v_k^w$, and attitude $q_{w{b}_k}$ of the IMU at moment $k$ are known, and the position $p_{w{b}_{k+1}}$, velocity $v_{k+1}^w$, and attitude $q_{w{b}_{k+1}}$ at moment $k+1$ can be recursively derived by measuring the acceleration $a$ and angular velocity $w$ of the IMU, as shown in Equation (7), where $g^w={\left[\begin{array}{ccc}0& 0& g\end{array}\right]}^T$.

$$\begin{array}{l}{p}_{\omega b_{k+1}}=p_{\omega b_k}+v_k^w\mathsf{\Delta }t+{\displaystyle {\mathsf{\int }\mathsf{\int }}_{t\in \left[k,k+1\right]}(R_{\omega b_t}{a}^{b_t}-g^{\omega })\delta t^2}\\ v_{k+1}^w=v_k^w+{{\displaystyle \mathsf{\int }}}_{t\in [k,k+1]}\left(R_{\omega b_t}{a}^{b_t}-g^w\right)\delta t\\ q_{\omega b_{k+1}}={{\displaystyle \mathsf{\int }}}_{t\in [k,k+1]}{q}_{\omega b_t}\otimes \left[\begin{array}{l}0\hfill \\ \frac{1}{2}{\omega }^{b_t}\hfill \end{array}\right]\delta t\end{array}$$

(7)

Typically, the frame rate of the IMU is higher than the frame rate of the cameras. The pre-integration expresses the relative observations of the IMU during the time between the two camera keyframes and can be used as a constraint on the keyframes for tightly coupled optimization. The pre-integration can decouple the IMU computation from the world coordinate system, avoiding the problem of changing Equation (7) for repeated integration of the IMU state quantities during the optimization process and reducing the computational effort. The integral model can be converted to a pre-integrated model by the transformation of Equation (8) as follows:

$$R_{\omega b_t}=R_{\omega b_k}\otimes R_{b_k{b}_t}$$

(8)

Substituting Equation (8) into Equation (7), the integral term in Equation (7) becomes the attitude concerning the moment of K instead of the attitude with respect to the world coordinate system:

$$\begin{array}{l}{p}_{\omega b_{k+1}}=p_{\omega b_k}+v_k^{\omega }\mathsf{\Delta }t-\frac{1}{2}{g}^{\omega }\mathsf{\Delta }{t}^2+R_{\omega b_k}{\displaystyle {\mathsf{\int }\mathsf{\int }}_{t\in \left[k,k+1\right]}(R_{b_k{b}_t}{a}^{b_t})\delta t^2}\\ v_{k+1}^{\omega }=v_k^{\omega }-g^{\omega }\mathsf{\Delta }t+R_{\omega b_k}{{\displaystyle \mathsf{\int }}}_{t\in [k,k+1]}\left(R_{b_k{b}_t}{a}^{b_t}\right)\delta t\\ R_{\omega b_{k+1}}=R_{\omega b_k}{{\displaystyle \mathsf{\int }}}_{t\in [k,k+1]}{R}_{b_k{b}_t}\otimes \left[\begin{array}{c}0\\ \frac{1}{2}{\omega }^{b_t}\end{array}\right]\delta t\end{array}$$

(9)

The pre-integrated quantity is related only to the IMU measurement, which is obtained by integrating the IMU data directly over a period of time:

$$\begin{array}{l}{\alpha }_{b_k{b}_{k+1}}={\displaystyle {\mathsf{\int }\mathsf{\int }}_{t\in \left[k,k+1\right]}(R_{b_k{b}_{k+1}}{a}^{b_t})\delta t^2}\\ {\beta }_{b_k{b}_{k+1}}={\displaystyle {\mathsf{\int }}_{t\in \left[k,k+1\right]}(R_{b_k{b}_{k+1}}{a}^{b_t})\delta t}\\ R_{b_k{b}_{k+1}}={\displaystyle {\mathsf{\int }}_{t\in \left[k,k+1\right]}{R}_{b_k{b}_t}\otimes \left[\begin{array}{c}0\\ \frac{1}{2}{\omega }^{b_t}\end{array}\right]\delta t}\end{array}$$

(10)

Further sorting of Equation (9) leads to the IMU pre-integrations ${\alpha }_{b_k{b}_{k+1}}$, ${\beta }_{b_k{b}_{k+1}}$, and $q_{b_k{b}_{k+1}}$ without considering the zero bias, and the first-order Taylor expansion approximations for the accelerometer zero bias and gyro zero bias $b_a$, $b_g$ and expressed as ${\widehat{\alpha }}_{b_k{b}_{k+1}}$, ${\widehat{q}}_{b_k{b}_{k+1}}$, and ${\widehat{\beta }}_{b_k{b}_{k+1}}$. Finally, the residual term of the pre-integration in this paper is shown in Equation (11):

$$r_b(z_{b_k{b}_{k+1,\chi }})=\left[\begin{array}{c}{r}_p\\ r_{\theta }\\ r_v\\ r_{ba}\\ r_{bg}\end{array}\right]={\left[\begin{array}{c}{R}_{b_k\omega }(p_{\omega b_{k+1}}-p_{\omega b_k}-v_k^{\omega }\Delta t+\frac{1}{2}{g}^{\omega }\Delta t^2)-{\widehat{\alpha }}_{b_k{b}_{k+1}}\\ 2{\left[{\widehat{q}}_{b_k{b}_{k+1}}\otimes (q_{b_k\omega }\otimes q_{\omega b_{k+1}})\right]}_{xyz}\\ R_{b_k\omega }(v_{k+1}^{\omega }-v_k^{\omega }+g^{\omega }\Delta t)-{\widehat{\beta }}_{b_k{b}_{k+1}}\\ b_a^{b_{k+1}}-b_a^{b_k}\\ b_g^{b_{k+1}}-b_g^{b_k}\end{array}\right]}_{15\ast 1}$$

(11)

2.2. Inertial Measurement Error Model under the Dynamic Base

To address the problem of inertial measurement errors due to wearable undershirt shaking, LSTM neural networks [30] are used to regress 6-dimensional IMU data and reduce inertial measurement errors from moving bases. In this paper, running visual-inertial SLAM relying on the camera device on the body can generate ground truth information. The generated ground truth information is used as the label and the IMU data on the wearable undershirt is used as the training set to regress the 6-dimensional IMU data ($a,\omega$) using the trained LSTM neural network. To reduce noise and improve the training accuracy, the ground truth information output from the visual-inertial SLAM is used as supervision, the IMU data in training is position-solved as a position loss function, and the longer time position is used as a loss function to ensure the accuracy for a long time. The wearable undershirt for data collection is shown in Figure 4.

Figure 4. Data Acquisition. The IMU data collected on the wearable are the training set, and the data collected by the camera device are the ground truth information.

2.2.1. Coordinate System Normalization

The different positions of the device placed on the wearable undershirt and the orientation of the device itself can have important effects on training. IMU sensor measurements are taken from the mobile device coordinate system (b-system), while ground truth motion trajectories are taken from the global coordinate system (n-system). In this paper, the coordinate system is defined in terms of the chest camera device orientation, using a coordinate system in which the Z-axis is aligned with gravity. In other words, any such coordinate system can be chosen as long as it remains consistent throughout the sequence. Coordinate conversion to a coordinate system with Z-axis aligned with gravity does not result in singularities or discontinuities, such as quaternions, as long as there is an appropriate rotational representation. The rotation of the IMU device direction $R_{gry}$ is in the same direction as the Z-axis ($R_z$) rotation:

$$\overrightarrow{a}=R_z{R}_{grv}{\overrightarrow{a}}_{dev}$$

(12)

During testing, the coordinate system was defined using the orientation of the camera equipment fixed on the chest with its Z-axis aligned with gravity.

$$\overrightarrow{a}=R_{grv}{\overrightarrow{a}}_{dev}$$

(13)

2.2.2. LSTM Network Architecture

In this paper, a special type of RNN, LSTM network (Long Short Term), is used. the memory of RNN is short term and there is a problem of gradient disappearance in RNN during backpropagation. The gradient is the value used to update the weights of the neural network, and if the gradient value becomes very small, it will not continue learning. Therefore, network layers with small gradient updates in the RNN stop learning, and these are usually the earlier layers. Since these layers do not learn, the RNN cannot remember what it has learned in longer sequences, so its memory is short-term. The LSTM is a proposed solution to overcome the short-term memory problem by introducing internal mechanisms called “gates” that regulate the flow of information. These gate structures can learn which data in a sequence are important to keep and which to delete. By doing so, they can pass relevant information along a long chain of sequences to perform predictions. The collected dataset is a long sequence, so the LSTM neural network was chosen to be used.

Figure 5 shows the architecture of the LSTM network:

Figure 5. LSTM architecture diagram.

From bottom to top, the first layer is the coordinate system of IMU on the device, i.e., (b-system); the second layer is the normalized coordinate system, i.e., transformed to (n-system); the third layer is the LSTM neural network; the fourth layer is the 6-dimensional IMU data output by the neural network, and the output position is solved by the fifth layer, which is supervised by the ground truth information in the uppermost layer and constitutes the loss function.

2.2.3. Loss Function

The general loss function takes the mean squared difference as the loss function; however, this effect is not good. In this paper, based on the inherent characteristics of IMU data, the location loss function is proposed as follows in Figure 6:

Figure 6. Schematic diagram of the position loss function.

Two different loss functions were used in the training process: the mean square error (MSE) and the location loss function. The MSE loss on the training dataset is defined as:

$$L{\mathrm{oss}}_{MSE}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{\Vert (a_i,{\omega }_i)-({\widehat{a}}_i,{\widehat{\omega }}_i)\Vert }^2}$$

(14)

Here ${\left\{a_i,{\omega }_i\right\}}_{i\le n}$ is the 6-dimensional IMU data provided by the visual-inertial SLAM, ${\left\{{\widehat{a}}_i,{\widehat{\omega }}_i\right\}}_{i\le n}$ is the output 6-dimensional IMU data after LSTM training, and $n$ is the number of data in the training set.

Position loss function:

$$Loss(d,\widehat{d})=\frac{1}{n}{\displaystyle \sum _{i=n}^n\left({\displaystyle \sum _{j=m}^{m+M}{\Vert d_{j\to j+1}-{\widehat{d}}_{j\to j+1}\Vert }^2}\right)}$$

(15)

where $d_i$ is the ground truth position provided by the visual-inertial SLAM and ${\widehat{d}}_i$ is the position solved by ${\left\{{\widehat{a}}_i,{\widehat{\omega }}_i\right\}}_{i\le n}$. $n$ is the batch size during training, $m$ is the start window of the sequence, and $M$ is the number of windows of the LSTM in the training set.

To learn long sequences of human motion relationships and reduce the cumulative error over time, an absolute position loss function is used:

$$L_{AL}^{MSE}\left(d,\widehat{d}\right)=\frac{1}{n}{\displaystyle \sum _{i=n}^n\left({\displaystyle \sum _{j=m}^{m+L}\left(d_m^{j+1}-{\widehat{d}}_m^{j+1}\right)}\right)}$$

(16)

where $d_m^{j+1}$ is the ground truth position provided by the visual-inertial SLAM from moment $m$ to moment $j+1$, and ${\widehat{d}}_m^{j+1}$ is the position solved from moment $m$ to moment $j+1$ based on inertial data.

2.2.4. Data Collection and Implementation

Data collection: IMU is fixed on the wearable undershirt. A total of 2 h of a pedestrian dataset consisting of walking and walking upstairs and downstairs with 3 test persons. In this paper, we use visual-inertial SLAM to provide position estimation and 6-dimensional IMU data for the entire dataset and use these results as supervised data in the training set and ground truth trajectories in the test set. We use 70% of the dataset as training, 15% as validation, and 15% test as a subset.

Data implementation: For network training, this paper uses overlapping sliding windows on each sequence to collect input samples. Each window contains N IMU data with a total size of N × 6. In the final system, we refer to the RoNIN settings. N = 50 is chosen as a window and the Pytorch model proposed by Adam Paszke [31] is used to implement the protocol. The fully connected layer is trained by Adam optimizer with an initial learning rate of 0.0001, zero weight decay, and Dropout probability set to 0.5. A total of about 150 epochs are required for full convergence on the RoNIN dataset. On the NVIDIA 1080Ti GPU, approximately 6 h of training time is required.

2.3. Neural Network-Assisted Visual-Inertial Person Localization Method under a Moving Base

In order to further improve the accuracy of personnel positioning under the dynamic base, we pre-integrate the processed IMU data in a visual-inertial fusion navigation framework to obtain the state quantities of the current moment: position, velocity, and rotation. Finally, the pre-integrated quantities between two camera keyframes are nonlinearly optimized within a sliding window together with the visual constraints.

2.3.1. Data Pre-Processing and Initialization

We pre-processed the collected IMU data to reject incorrect data values and fill in gaps. After LSTM network processing, the IMU data are pre-integrated to obtain the state quantities of the current moment: position, velocity, and rotation.

The initialization part includes purely visual SFM (structure from motion) and visual-inertial fusion alignment, which aims to provide good initial values for visual-inertial tight-coupling optimization within the sliding window below. The purely visual SFM is tracked by point feature extraction between consecutive frames, solving the relative transformation between frames by a pair of polar geometric models and selecting the two keyframes within the sliding window with the same observation but farthest away for triangulation to obtain the inverse depth of 3D points.

Since the IMU comes with zero bias and zero bias is included as a state variable to be estimated in the sliding window optimization mechanism below, the visual inertial joint initialization can be used to estimate the initial value of the gyroscope zero bias using the rotation constraint of the visual SFM. Accelerometer zero bias has little effect due to comparison with gravity action that is neglected here. The gravity direction, velocity, and scale information of the camera can be estimated using the translational constraint.

2.3.2. Sliding Window Nonlinear Optimization Mechanism for Estimating Poses

For the correlation of visual-inertial data, all visual-inertial state quantities in the sliding window can be optimized using a tightly coupled approach based on a graph optimization framework, and Figure 7 using a factor graph approach shows the constrained relationship between the camera, the waypoint, and the IMU.

Figure 7. Factor diagram for visual-inertial positioning system.

The state quantities to be estimated in the sliding window are shown in Equation (17), where $x_k$ in Equation (18) is the position, attitude, velocity, acceleration zero bias, and gyroscope zero bias state quantities of the $k$th data; and $\lambda$ is the inverse depth of all point features seen by the camera in the sliding window.

$$\chi ={\left[{\mathit{x}}_0,{\mathit{x}}_1,\cdots ,{\lambda }_0,{\lambda }_1,\cdots ,{\lambda }_m\right]}^{\mathrm{T}}$$

(17)

$${\mathit{x}}_k={\left[{\mathit{p}}_{v{b}_k},{\mathit{q}}_{u{b}_k},{\mathit{v}}_k^w,{\mathit{b}}_{a^k}^{b_k},{\mathit{b}}_g^{b_k}\right]}^{\mathrm{T}},k\in [0,n]$$

(18)

Based on the constructed visual constraint information between the two keyframes and the IMU pre-integration constraint information, it is possible to obtain the optimization objective function of Equation (19).

$$\begin{array}{r}\underset{\chi }{\min }\rho \left(\parallel r_p-J_p\mathcal{X}{\parallel }_{{\mathsf{\Sigma }}_p}^2\right)+{\displaystyle \sum _{i\in B}}\rho \left(\parallel r_b\left(z_{b_k{b}_{k+1}},\mathcal{X}\right){\parallel }_{{\mathsf{\Sigma }}_{b_k{b}_{k+1}}^2}\right)\\ +{\displaystyle \sum _{(k,j)\in F}}\rho \left(\parallel r_p\left(z_{p_j}^{c_k},\mathcal{X}\right){\parallel }_{{\mathsf{\Sigma }}_{p_j}^{c_k}}^2\right)\end{array}$$

(19)

where ${\mathit{r}}_p\left({\mathit{z}}_{p_j}^{c_k},\chi \right),{\mathit{r}}_b\left({\mathit{z}}_{b_k{b}_{k+1},\chi }\right)$ are the point feature reprojection residuals and IMU residuals represented by Equations (6) and (11), respectively; $\rho ()$ is the Huber robust kernel function; $B$ is the set of all pre-integrated constraint terms within the sliding window. $F$ is the feature of all observed points of the camera within the sliding window, ∑ and is the respective covariance term. The first term $r_{prior}-J_{prior\chi }$ is the a priori information term left after marginalizing keyframes in the sliding window, i.e., the constraint term left behind when the oldest frame is removed when a new keyframe is inserted. As mentioned above, the objective function and the state quantity to be optimized are known, and the Gauss–Newton method is used to optimize this nonlinear least squares problem, solving for the increment $\Delta \chi$ of the state quantity $\chi$. The optimal state quantity is solved by continuous iterative optimization, as shown in Equation (20):

$$\left({\mathit{H}}_{prior}+{\mathit{H}}_b+{\mathit{H}}_p\right)\mathsf{\Delta }\chi =\left({\mathit{b}}_{prior}+{\mathit{b}}_b+{\mathit{b}}_p\right)$$

(20)

where the relationship between Hessian matrix $H$ and Jacobi matrix $J$ is $H=J^T{\sum }^{-1}J$, and the right-hand term of the equation is $b=-J^T{\sum }^{-1}r$.

3. Results

In this section, in order to verify the effectiveness of the algorithm, experiments were conducted in real indoor and outdoor scenarios. There are two main parts: experiments and analysis of inertial measurement errors under a moving base; experiments and analysis of visual-inertial human localization under a moving base.

To verify the effectiveness of the algorithm, the trained 6-dimensional IMU data are solved to obtain the localization trajectory, and the trajectory of visual-inertial SLAM is used as the ground truth trajectory. The RONIN-LSTM method without position loss function and our method with position loss function are compared, and two evaluation criteria are used: absolute trajectory error (ATE) and relative trajectory error (RTE). The absolute trajectory error is positioned as the overall root mean square error (RMSE) between the estimated trajectory and the real trajectory, the relative trajectory error is defined as the average RMSE over a fixed time interval, the time interval is one minute in the evaluation of this paper, and for sequences less than one minute, the position error of the last frame is calculated and scaled.

To evaluate the superiority of the proposed algorithm more intuitively, two sets of representative training results are selected in this paper: the first set has better training results and the second set has worse training results. As shown in Figure 8, the blue line is the trajectory generated by the algorithm, the yellow line is the ground truth trajectory, and the green line is the trajectory generated by the RONIN-LSTM method. From the figure, we can see that the trajectory generated by our algorithm in the first picture (a) basically overlaps with the ground truth, and the trajectory generated by the RONIN-LSTM algorithm deviates more seriously. In the second picture (b), both methods deviate, but the trajectory of our algorithm obviously deviates less than that of RONIN-LSTM. As can be seen in Table 2, both images show a significant reduction in ATE and RTE after training using the position loss function.

Figure 8. Trajectories of the two methods on the dataset.

Table 2. ATE and RTE for both methods.

Model	01		02
	ATE	RTE	ATE	RTE
Our algorithm	0.68	0.76	2.28	1.95
RONIN-LSTM	2.65	1.05	3.23	2.22
Precision improvement	74.34%	27.62%	29.41%	12.16%

Experiment and Analysis of Visual-Inertial Personnel Positioning

In order to verify the reliability of the improved algorithm, indoor and outdoor experimental scenarios are selected, and the underground space is simulated with indoor–outdoor experiments and tested several times. The localization device is placed on the wearable undershirt, the camera device is used, as shown in Figure 9. The experiments are conducted by pedestrians wearing the undershirt on outdoor roads and indoor hallways. The experimental comparison results and analysis are as follows:

Figure 9. Testers’ experimental wear chart.

(1) Outdoor playground experiment. The experimental scene was chosen as a school playground, and the experimenter walked around the playground wearing localization equipment. To verify the effectiveness of the algorithm, the VINS-fusion algorithm and the improved localization method were used for comparison. The localization trajectories of the two methods are shown in Figure 10 and Figure 11. The total length of the trajectory is 400 m, and the accuracy is calculated by the offset distance and the total distance traveled. The endpoint of the improved positioning method is very close to the starting point, the offset is about 1.997 m, and the positioning accuracy error is 0.50%. As for the results of the VINS-fusion algorithm run, the positioning was relatively accurate for a short distance after the start, but the drift was more serious after that. After one lap, the endpoint of the positioning trajectory seriously deviates from the starting point. The offset is about 25.67 m and the positioning accuracy error is 6.41%. Positioning error is reduced by 92.20%. The comparison shows that the method proposed improves positioning accuracy.

Figure 10. VINS-fusion positioning trajectory map.

Figure 11. Our algorithm positioning the trajectory map.

(2) Outdoor road experiment. The total length of the trajectory is 725 m, the offset is about 3.674 m, and the positioning accuracy error is 0.51%. The experimental trajectory is shown in Figure 12.

Figure 12. Outdoor road positioning track map.

(3) Building experiment. Figure 13 shows the scene of the building. Experimenters wear positioning equipment along the building’s pre-determined trajectory from the starting point, back to the starting point after turning back into the building, that is, the starting point and the endpoint coincide, during which they have to pass up and down the stairs, the total length of the trajectory is 210 m, and the schematic diagram of the building experiment is shown in Figure 14. Figure 15 shows the positioning trajectory of the experimental run; from the running trajectory, the starting point and the starting point nearly coincide, the offset is about 0.983 m, and the positioning accuracy error is 0.47%.

Figure 13. Building environment point feature extraction.

Figure 14. Building an experimental walking diagram.

Figure 15. Building positioning track map.

The positioning accuracy of the algorithm under the moving base was verified by indoor and outdoor experiments.

4. Conclusions

In this paper, a neural network-assisted visual-inertial personnel localization method is proposed. Our goal is to enhance the accuracy of conventional visual-inertial under a moving base by means of LSTM networks. We developed an inertial measurement error model based on inertial measurement under a moving base, generated ground truth trajectories with visual-inertial SLAM, and regressed 6-dimensional IMU data from IMU data fragments of the shaky base using a trained neural network. Then, the localization accuracy was improved by combining with visual. The proposed algorithm was experimentally proven to be effective in improving the positioning accuracy of people under a moving base.

The algorithm proposed in this paper mainly addresses the problem of low accuracy of person localization under a moving base, and the effect is not obvious for general scenes. Due to the small dataset and testers, the localization accuracy will be degraded for those who are not trained by the LSTM network. In addition, the scenes in underground spaces may be complex and multiple actions of testers may occur. Therefore, the follow-up work should increase the diversity of the data (adding testers and their possible actions) so that the proposed algorithm can be adapted to more complex environments.