Sky-GVIO: Enhanced GNSS/INS/Vision Navigation with FCN-Based Sky Segmentation in Urban Canyon

Abstract

Accurate, continuous, and reliable positioning is critical to achieving autonomous driving. However, in complex urban canyon environments, the vulnerability of stand-alone sensors and non-line-of-sight (NLOS) caused by high buildings, trees, and elevated structures seriously affect positioning results. To address these challenges, a sky-view image segmentation algorithm based on a fully convolutional network (FCN) is proposed for NLOS detection in global navigation satellite systems (GNSSs). Building upon this, a novel NLOS detection and mitigation algorithm (named S−NDM) uses a tightly coupled GNSS, inertial measurement units (IMUs), and a visual feature system called Sky−GVIO with the aim of achieving continuous and accurate positioning in urban canyon environments. Furthermore, the system combines single-point positioning (SPP) with real-time kinematic (RTK) methodologies to bolster its operational versatility and resilience. In urban canyon environments, the positioning performance of the S−NDM algorithm proposed in this paper is evaluated under different tightly coupled SPP−related and RTK−related models. The results exhibit that the Sky−GVIO system achieves meter-level accuracy under the SPP mode and sub-decimeter precision with RTK positioning, surpassing the performance of GNSS/INS/Vision frameworks devoid of S−NDM. Additionally, the sky-view image dataset, inclusive of training and evaluation subsets, has been made publicly accessible for scholarly exploration.

1. Introduction

Currently, the synergistic use of global navigation satellite systems (GNSSs) and inertial navigation systems (INSs) has emerged as the predominant approach to navigate complex urban environments [1,2,3]. Within the domain of GNSSs, real-time kinematic (RTK) and precise point positioning (PPP) technologies have been extensively adopted to enhance GNSS/INS integrated solutions. Comparative studies indicate that RTK/INS fusion yields a superior accuracy over PPP/INS under identical observational conditions [4]. Nonetheless, urban environments, with pervasive obstructions like edifices and arboreal coverage, introduce non-line-of-sight (NLOS) errors into GNSS signals, compromising the accuracy of integrated GNSS/INS positioning. To address the impact of NLOS on GNSS/INS systems, researchers have experimented in recent years with additional sensors to detect and suppress NLOS, including fish-eye cameras, LiDAR, and 3-dimensional (3D) maps. Since sensor information can truly reflect the environment around a vehicle, it has become a hot topic to investigate additional sensor-based NLOS detection to improve the positioning accuracy and robustness of GNSS-related systems in urban canyons.

Cameras are increasingly utilized in vehicular motion estimation due to their energy efficiency and cost benefits. As an external sensor, cameras can provide rich environmental features for vehicle motion estimation [5,6,7]. Consequently, the integration of cameras with GNSS- and micro-electro-mechanical system (MEMS)-based inertial measurement units (IMUs) is a common strategy to attain precise localization in complex urban environments. Previous research [8] has introduced a monocular model of visual–inertial navigation systems (VINSs), integrating visual inertial odometry (VIO)-derived relative poses with global positioning system (GPS) data within a unified optimization structure. In contrast to VINS-Mono, the work in [9] combines differential GNSS results with the VIO model, in which the VIO model is transformed from the local frame to the global frame, achieving meter-level positioning accuracy in complex urban environments. Advancing from VINS-Mono, the work in [10] introduced the well-known GVINS model, which performs a joint optimization of GNSS pseudorange measurements, visual features, and inertial measurements through factor graph optimization techniques. While methods based on nonlinear optimization have advantages in handling system nonlinearity, multiple iterations of optimization increase computational complexity. Therefore, some researchers have started to focus on extended Kalman filter (EKF)-based methodologies. Building on [11], a paper [12] put forth a tightly-coupled Mono/MEMS-IMU/single-frequency GNSS-RTK model employing a multi-state constraint Kalman filter (MSCKF), which attained decimeter-level positioning accuracy in urban environments.

In the above multi-sensor fusion positioning systems, the GNSS is the only subsystem that provides absolute position information. Therefore, the quality control of the GNSS raw measurements determines the overall performance of the system. This underscores the significance of NLOS signal detection and mitigation, especially in the convoluted terrains of urban environments.

In the detection and mitigation of GNSS NLOS signals, strategies are divided into hardware-centric designs and algorithmic advancements. Compared to expensive hardware improvements such as antenna design in [13,14], many researchers have focused on algorithmic improvements. These include empirical weighting models based on the elevation angle [15], signal-to-noise ratio (SNR) [16], and methods that leverage multi-source information for satellite visibility. Notably, methods augmented by external sources like LiDAR [17,18], 3D maps [19,20], and cameras [21,22] have a refined GNSS NLOS signal detection accuracy. In particular, cameras present a cost-effective alternative to the high expenses and limited scope of LiDAR and the need for continuously updated 3D map databases. Infrared cameras [23] exhibit varying results for objects at different temperatures, making it easier to distinguish between sky and non-sky areas, which is advantageous for determining a satellite’s projection location on sky-view images. However, compared to regular fish-eye cameras, infrared cameras are more costly. Furthermore, these cameras have not yet seen widespread use in consumer market products, like smartphones or vehicle-mounted cameras. Subsequently, many research works began to use sky-pointing fish-eye cameras to capture sky-view images. These images were processed using segmentation algorithms [24,25,26] to distinguish between sky and non-sky areas. Finally, the satellites received by a GNSS receiver were projected onto the sky-view images, facilitating the visualization of GNSS NLOS satellites. As seen from the results in [23,27], this approach significantly enhances the performance of SPP/INS positioning in complex urban environments. However, these traditional segmentation algorithms may not adapt well to sky-view images with varying lighting conditions. Furthermore, we have observed that the use of sky-view images for GNSS NLOS detection has not been extended to tightly coupled GNSS/INS/Vision systems. Additionally, there is an absence of comparative performance analyses of sky-view images in different GNSS positioning modes, both domestically and internationally.

We aim to extend the sky-view image-aided GNSS NLOS detection and mitigation method (named S−NDM) to the tightly coupled GNSS/INS/Vision system, thereby enhancing vehicle positioning performance in urban canyons. Here, we particularly emphasize the progression from the work of [27] through the following points: (a) Different from the previous idea of improving the region growth algorithm, we use the algorithm of a neural network to achieve segmentation of sky-view images to adapt to different lighting conditions; and (b) While the original NLOS signal processing algorithm is only used in a tightly coupled SPP/INS framework, in this paper, we extend it to a tightly coupled SPP/INS/Vision and RTK/INS/Vision framework. In addition, we evaluate the performance of the algorithm in these two frameworks and verify the practicability of the algorithm. This paper emphasizes the following primary contributions:

• Adaptive Sky-view Image Segmentation: We introduce adaptive sky-view image segmentation based on fully convolutional networks (FCNs) that can adjust to varying lighting conditions, addressing a key limitation of traditional methods;
• Integration of Sky-GNSS/INS/Vision: We propose an integrated model that combines GNSS, INS, and Vision. Meantime, we extend the S−NDM method to this model (named Sky−GVIO), enabling a comprehensive approach to vehicle positioning in challenging urban canyon environments;
• Performance Evaluation: A comprehensive evaluation of S−NDM’s performance is conducted, with a focus on its effectiveness within GNSS pseudorange and carrier-phase positioning frameworks, thereby shedding light on its applicability across different GNSS-related integration positioning techniques;
• Open-Source Sky-view Image Dataset: An open-source repository of sky-view images, including the training and testing data, is provided at https://github.com/whuwangjr/sky-view-images, accessed on 27 July 2024, contributing a valuable dataset to the research community and mitigating the lack of available resources in this field.

The remainder of this paper is organized as follows: Section 2 gives an overview of the tightly coupled GNSS/INS/Vision system enhanced by S−NDM. The experimental description and result analysis are introduced in Section 3. Finally, Section 4 summarizes and concludes the study.

2. System Overview

The proposed model, Sky−GVIO, is described in this section, including sky-view image segmentation based on an FCN, the tightly coupled GNSS/INS/Vision integration system, and S−NDM, as shown in Figure 1. The tightly coupled model is a fusion of the observed values. Before the fusion, it is very important to process the original GNSS observations. We use the S−NDM algorithm to process GNSS NLOS signals. In addition, the INS mechanization is used for state prediction and the system covariance is also propagated. In the visual part, the feature extraction and tracking will be performed following [11]. Finally, we integrate the observation equations of GNSS, INS, and Vision into the MSCKF framework to obtain the navigation results.

2.1. Sky-View Image Segmentation

Sky-view images can be significantly affected by factors such as clouds and lighting conditions, making it challenging to achieve high-precision segmentation using traditional methods based on pixels [28], categories [29], regions [30], and so on. It is well known that FCNs represent mature pixel-level semantic segmentation networks [31]. The FCN network structure primarily consists of two parts: the fully convolutional part and the deconvolution part. The fully convolutional part comprises classical CNN networks, such as VGG and ResNet, which are used for feature extraction. The deconvolution part, on the other hand, upsamples the feature maps to obtain the original-sized semantic segmentation image.

In this paper, ResNet50 [32] is chosen for downsampling, based on the advantages of its residual structure in dense semantic segmentation tasks. The residual connections in ResNet50 effectively alleviate the vanishing gradient problem in deep networks, making it easier for the model to learn rich features during training. This structure not only accelerates model convergence but also enhances feature extraction capabilities, making it particularly suitable for handling complex sky-view images. For the upsampling part, the FCN-8s architecture was adopted. FCN-8s uses transposed convolution to upsample feature maps at 8×, 16×, and 32× scales back to their original size and combines these three scaled feature maps using skip connections. This method ensures the learning of features at different scales, thereby improving the model’s performance in the sky image segmentation task. By integrating information at different scales, FCN-8s can better capture image details, thereby improving the accuracy and robustness of semantic segmentation. In summary, the combination of ResNet50 and FCN-8s allows this study to effectively address the challenges of sky-view image segmentation.

For the sky-view image segmentation task, two types of labels are required (sky region and non-sky region), so the number of channels for the sky-view image segmentation algorithm based on an FCN is two. In addition, in this study, we made 440 training datasets by ourselves. As shown in Figure 2, we built a sky-view image segmentation model based on an FCN.

2.2. Tightly Coupled GNSS/INS/Vision Integration Model

First, this section provides a brief background on the GNSS observation model, the INS dynamic model, and the visual observation model to pave the way for homogeneous sensor fusion. In light of the aforementioned, the state model and the measurement model of the Sky−GVIO are described. Finally, NLOS detection is implemented using the segmentation results discussed in Section 2.1, and an LOS/NLOS model is established to mitigate NLOS effects.

(1)

GNSS Observation Model: The original pseudorange and carrier-phase observation equations in GNSS positioning are expressed as follows:

$$P=\rho +c(t_r-t^s)+I+T+{\epsilon }_P$$

(1)

$$L=\rho +c(t_r-t^s)-I+T+\lambda N+{\epsilon }_L$$

(2)

where $P$ and $L$ represent the pseudorange and carrier phase, respectively. The angular symbols $s$ and $r$ refer to satellites and receivers, respectively. $\rho$ denotes the geometric distance between the phase centers of the receiver and satellite antennas. $t_r$ and $t^s$ represent the receiver and satellite clock offsets, respectively. The speed of light is $c$. $I$ and $T$ refer to the ionospheric and troposphere delay, respectively. $\lambda$ represents the carrier wavelength. $N$ represents the carrier-phase ambiguity. ${\epsilon }_P$ and ${\epsilon }_L$ represent the pseudorange noise and carrier-phase noise, respectively.

For the SPP model, Equation (1) is sufficient. However, in the case of the RTK model, it can be represented as follows:

$$\left\{\begin{array}{c}\nabla \Delta P=\nabla \Delta \rho +\nabla \Delta I+\nabla \Delta T+\nabla \Delta {\epsilon }_P\\ \nabla \Delta L=\nabla \Delta \rho -\nabla \Delta I+\nabla \Delta T+\lambda \nabla \Delta N+\nabla \Delta {\epsilon }_L\end{array}\right.$$

(3)

where $\nabla \Delta$ denotes the double-differenced (DD) operator. The DD operation is used to not only eliminate satellite orbit errors and clock errors but also to mitigate receiver clock errors as well as tropospheric and ionospheric delays, making it a powerful technique in GNSS positioning.

(2)

INS Dynamic Model: Considering the noisy measurement of the low-cost IMU, the Coriolis and centrifugal forces due to Earth’s rotation are ignored in the IMU formulation. The inertial measurement can be modeled [21] in the $b$ (body) frame as follows:

$${\tilde{\mathit{a}}}_k={\mathit{a}}_k+{\mathit{b}}_{a_k}+{\left({\mathbf{R}}_{b_k}^n\right)}^T{\mathit{g}}^n+{\mathbf{n}}_a$$

(4)

$${\tilde{\mathit{\omega }}}_k={\mathit{\omega }}_k+{\mathit{b}}_{{\omega }_k}+{\mathbf{n}}_{\omega }$$

(5)

where $\left[{\tilde{\mathit{a}}}_k,{\tilde{\mathit{\omega }}}_k\right]$ is the output of the IMU at time $k$, and $\left[{\mathit{a}}_k,{\mathit{\omega }}_k\right]$ is the linear acceleration and angular velocity of the IMU sensor. ${\mathit{b}}_{a_k}$ and ${\mathit{b}}_{{\omega }_k}$ are the biases of the accelerometer and gyroscope at time $k$, respectively. In addition, ${\mathbf{n}}_a$ and ${\mathbf{n}}_{\omega }$ are assumed to be zero-mean Gaussian-distributed with ${\mathit{n}}_a\sim N\left(0,{\mathsf{\Sigma }}_{n_a}\right)$ and ${\mathit{n}}_{\omega }\sim N\left(0,{\mathsf{\Sigma }}_{n_{\omega }}\right)$. ${\mathbf{R}}_{b_k}^n$ denotes the rotation matrix from the IMU body (b) frame to the n frame. ${\mathit{g}}^n$ is the gravity in the n frame.

The linearized INS dynamic model [30] can be expressed as follows:

$$\left\{\begin{array}{c}\delta {\dot{\mathbf{p}}}^n=\delta {\mathbf{v}}^n\\ \delta {\dot{\mathbf{v}}}^n=-{\mathbf{R}}_b^n{(\tilde{\mathit{a}}-{\mathit{b}}_a)}^{\wedge }\delta \mathit{\theta }-{\mathbf{R}}_b^n\delta {\mathit{b}}_a-{\mathbf{R}}_b^n{\mathbf{n}}_a\\ \delta \dot{\mathit{\theta }}=-{(\tilde{\mathit{\omega }}-{\mathit{b}}_{\omega })}^{\wedge }\delta \mathit{\theta }-\delta {\mathit{b}}_{\omega }-{\mathbf{n}}_{\omega }\\ \delta {\dot{\mathbf{b}}}_a={\mathbf{n}}_{b_a}\\ \delta {\dot{\mathbf{b}}}_{\omega }={\mathbf{n}}_{b_{\omega }}\end{array}\right.$$

(6)

where $\delta {\dot{\mathbf{p}}}^n$, $\delta {\dot{\mathbf{v}}}^n$, and $\delta {\dot{\mathit{\theta }}}^n$ represent the derivative of the position, velocity, and attitude errors in the $n$ (navigation) frame, respectively. “$\wedge$” is the cross-product. $\delta {\dot{\mathbf{b}}}_a$ and $\delta {\dot{\mathbf{b}}}_{\omega }$ denote the accelerometer and gyroscope bias errors, respectively. ${\mathbf{n}}_{b_a}$ and ${\mathbf{n}}_{b_{\omega }}$ are assumed to be zero-mean Gaussian-distributed with ${\mathit{n}}_{b_a}\sim N\left(0,{\mathsf{\Sigma }}_{b_a}\right)$ and ${\mathit{n}}_{b_{\omega }}\sim N\left(0,{\mathsf{\Sigma }}_{b_{\omega }}\right)$ [7]. Therefore, the error state vector of INS can be expressed as follows:

$$\delta {\mathit{x}}_{ins}={\left[\begin{array}{ccccc}\delta {\mathbf{p}}^n& \delta {\mathbf{v}}^n& \delta \mathit{\theta }& \delta {\mathbf{b}}_a& \delta {\mathbf{b}}_{\omega }\end{array}\right]}^{\mathit{T}}$$

(7)

(3)

Visual Measurement Model: The core idea of the well-known MSCKF is to establish geometric constraints between multi-camera states by utilizing the same visual feature points observed by multi-cameras. Following this concept, we establish a visual model. For a visual feature point $f^j$ observed by a stereo camera at time $i$, its visual observation model [33] on the normalized projection planes of the left and right cameras can be represented as follows:

$$z_{cam,i}^j=\left[\begin{array}{c}{u}_{c_{0,i}}^j\\ v_{c_{0,i}}^j\\ u_{c_{1,i}}^j\\ v_{c_{1,i}}^j\end{array}\right]=\left[\begin{array}{cc}{\displaystyle \frac{1}{Z_j^{c_{0,i}}}}{\mathbf{I}}_{2\times 2}& {\mathbf{0}}_{2\times 2}\\ {\mathbf{0}}_{2\times 2}& {\displaystyle \frac{1}{Z_j^{c_{1,i}}}}{\mathbf{I}}_{2\times 2}\end{array}\right]\left[\begin{array}{c}{X}_j^{c_{0,i}}\\ Y_j^{c_{0,i}}\\ X_j^{c_{1,i}}\\ Y_j^{c_{1,i}}\end{array}\right]+{\mathit{\epsilon }}_{cam,i}^j$$

(8)

where the subscripts 0 and 1 represent the left and right cameras, respectively. $\left(u_{c_{k,i}}^j,v_{c_{k,i}}^j\right)\left(k\in 0,1\right)$ are the pixel coordinates of the feature points on the normalized plane. ${\mathit{\epsilon }}_{cam,i}^j$ is the visual measurement noise. $\left[X_j^{c_{k,i}},Y_j^{c_{k,i}},Z_j^{c_{k,i}}\right]$ represents the feature position in the camera (c) frame, which can be expressed as follows:

$$\left[\begin{array}{c}{X}_j^{c_{0,i}}\\ Y_j^{c_{0,i}}\\ Z_j^{c_{0,i}}\end{array}\right]={\left({\mathbf{R}}_{c_{0,i}}^n\right)}^T\left({\mathit{p}}_j^n-{\mathit{p}}_{c_{0,i}}^n\right)$$

(9)

$$\left[\begin{array}{c}{X}_j^{c_{1,i}}\\ Y_j^{c_{1,i}}\\ Z_j^{c_{1,i}}\end{array}\right]={\left({\mathbf{R}}_{c_{0,i}}^{c_{1,i}}\right)}^T\left({\mathit{p}}_j^{c_{0,i}}-{\mathit{p}}_{c_{1,i}}^{c_{0,i}}\right)$$

(10)

where ${\mathbf{R}}_{c_{0,i}}^n$ and ${\mathit{p}}_{c_{0,i}}^{\mathit{n}}$ are the rotation matrix and position of the left camera in the n frame, respectively. ${\mathbf{R}}_{c_{0,i}}^{c_{1,i}}$ is the rotation matrix of the left camera to the right camera, and ${\mathit{p}}_{c_{1,i}}^{c_{0,i}}$ is the translation matrix of the left camera to the right camera, which can be accurately corrected in advance [34]. ${\mathit{p}}_j^n$ and ${\mathit{p}}_j^{c_{0,i}}$, respectively, are the positions of the visual feature point in the n frame and left c frame. We adopted the method proposed by [35] to construct the visual reprojection error between relative camera poses, and the visual state vector was described as follows:

$$\delta x_{cam}={\left[\begin{array}{ccccccc}\delta {\mathit{\theta }}_{c_1}^n& \delta {\mathit{p}}_{c_1}^n& \delta {\mathit{\theta }}_{c_2}^n& \delta {\mathit{p}}_{c_2}^n& \cdots & \delta {\mathit{\theta }}_{c_s}^n& \delta {\mathit{p}}_{c_s}^n\end{array}\right]}^T$$

(11)

where $\delta {\mathit{\theta }}_{c_i}^n$ and $\delta {\mathit{p}}_{c_i}^n$, respectively, represent the rotation and position error at the time $i$. The subscript $s$ represents the total number of camera poses (defined as rotations and translations) in the sliding window. The reprojection residual of a visual measurement can be expressed as follows:

$$\delta {\mathit{z}}_{cam}={\mathit{z}}_{cam}-{\widehat{\mathit{z}}}_{cam}={\mathbf{H}}_{cam}\delta {\mathit{x}}_{cam}+{\mathit{\epsilon }}_{cam}$$

(12)

where ${\mathit{z}}_{cam}$ and ${\widehat{\mathit{z}}}_{cam}$, respectively, represent the observed and reprojected visual measurements. ${\mathbf{H}}_{cam}$ is the Jacobi matrix of the camera states involved. Specific details can be obtained from [35].

(4)

State and Measurement Model of the Tightly Coupled GNSS/INS/Vision: This paper employs MSCKF for tightly coupled GNSS/INS/Vision integration. Based on the above introductions of different sensor models, the complete state model for the integration of tightly coupled GNSS/INS/Vision is as follows:

$$\delta \mathit{x}={\left[\begin{array}{ccc}\delta {\mathit{x}}_{ins}& \delta {\mathit{x}}_{GNSS}& \delta {\mathit{x}}_{cam}\end{array}\right]}^T$$

(13)

For both SPP and RTK positioning modes, this paper has constructed their state models separately:

$$\delta x_{GNSS,SPP}={\left[\delta t_r\right]}^T$$

(14)

$$\delta x_{GNSS,RTK}={\left[\delta \nabla \Delta N\right]}^T$$

(15)

where $\delta \nabla \Delta N$ represents the double-difference carrier-phase ambiguity. In addition, the error state model of the INS and Vision have already been provided in Equations (7) and (11).

The state prediction model for the integration of tightly coupled GNSS/INS/Vision is as follows:

$$\left[\begin{array}{c}\delta {\dot{\mathit{x}}}_{ins}\\ \delta {\dot{\mathit{x}}}_{GNSS}\\ \delta {\dot{\mathit{x}}}_{cam}\end{array}\right]=\left[\begin{array}{ccc}{\mathbf{F}}_{ins}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}\end{array}\right]\left[\begin{array}{c}\delta {\mathit{x}}_{ins}\\ \delta {\mathit{x}}_{GNSS}\\ \delta {\mathit{x}}_{cam}\end{array}\right]+\left[\begin{array}{c}{\mathit{n}}_{ins}\\ {\mathit{n}}_{GNSS}\\ \mathbf{0}\end{array}\right]$$

(16)

In Equation (16), ${\mathit{F}}_{ins}$ is the continuous-time state transition matrix of INS. ${\mathit{n}}_{ins}$ and ${\mathit{n}}_{GNSS}$ are the process noise of the INS and GNSS, respectively. Based on Equation (6), the special form of ${\mathit{F}}_{ins}$ can be written as follows:

$${\mathit{F}}_{ins}=\left[\begin{array}{ccccc}\mathbf{0}& \mathbf{I}& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& -{\mathbf{R}}_b^n{(\tilde{\mathit{a}}-{\mathit{b}}_a)}^{\wedge }& -{\mathbf{R}}_b^n& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& -{(\tilde{\mathit{\omega }}-{\mathit{b}}_{\omega })}^{\wedge }& \mathbf{0}& -\mathbf{I}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}\end{array}\right]$$

(17)

where $\mathbf{I}$ is the identity matrix.

To deal with discrete time measurement from the INS, the fourth-order Runge–Kutta [10] numerical integration of Equation (17) is used to propagate the estimated state variables. It is worth noting that only the IMU state variables are propagated; the visual and GNSS state variables are only copied. Meanwhile, we also need to propagate the covariance of the state:

$$\begin{gathered} {\mathit{P}}_{k,k-1}={\mathit{\Phi }}_{k,k-1}{\mathit{P}}_{k-1}{\mathit{\Phi }}_{k,k-1}^T+{\mathit{Q}}_{k-1} \\ {\mathit{\Phi }}_{k,k-1}=\mathit{\Phi }(t_{k-1},t_k)=\exp ({\displaystyle {\int }_{t_k}^{t_{k+1}}\mathit{F}(\tau )d\tau }) \end{gathered}$$

(18)

where ${\mathit{\Phi }}_{k,k-1}$ represents the discrete state transition matrix, $\mathit{F}(\tau )$ is the continuous time state transition matrix at time $\tau$ ($\tau \in \left(t_k,t_{k+1}\right)$), and ${\mathit{Q}}_{k-1}$ is the discrete time noise covariance. ${\mathit{P}}_{k-1}$ represents the error state covariance matrix before augmentation. ${\mathit{P}}_{k,k-1}$ represents the one-step prediction error covariance matrix from time $t_{k-1}$ to time $t_k$.

It is worth noting that every time a new image is recorded, the state and covariance matrix will be augmented with a copy of the current camera pose estimate. The initial value of the camera pose is derived from the INS mechanization, and the covariance matrix ${\mathit{P}}_k$ after augmentation can be expressed as follows:

$${\mathit{P}}_k=\left[\begin{array}{c}{\mathbf{I}}_{15+\gamma +6m}\\ \mathbf{J}\end{array}\right]{\mathit{P}}_{k-1}{\left[\begin{array}{c}{\mathbf{I}}_{15+\gamma +6m}\\ \mathbf{J}\end{array}\right]}^T$$

(19)

where $\gamma$ and $m$ represent the number of variables related to the GNSS and Vision at a certain moment. When the GNSS is recorded, we only need to remove and add state variables and the corresponding covariance. $\mathbf{J}$ is the Jacobi matrix, which has the following form:

$$\mathbf{J}=\left[\begin{array}{cccccc}{\left({\mathbf{R}}_c^b\right)}^T& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& {\mathbf{0}}_{3\times (\gamma +6m)}\\ -{\mathbf{R}}_b^n{({\mathit{P}}_c^b)}^{\wedge }& \mathbf{0}& \mathbf{I}& \mathbf{0}& \mathbf{0}& {\mathbf{0}}_{3\times (\gamma +6m)}\end{array}\right]$$

(20)

where ${\mathbf{R}}_c^b$ and ${\mathit{P}}_c^b$ are the rotation matrix and translation matrix between the camera and IMU, which are calibrated offline [36].

Based on the previous equations, the measurement equations for the integration of tightly coupled SPP/INS/Vision are formulated as follows:

$$\begin{array}{l}\left[\begin{array}{c}\delta {\mathit{P}}_{SPP}\\ \delta {\mathit{z}}_{cam}\end{array}\right]=\left[\begin{array}{c}{\mathbf{H}}_{P,SPP}\\ {\mathbf{H}}_{cam}\end{array}\right]\left[\begin{array}{c}\delta {\mathit{x}}_{ins}\\ \delta {\mathit{x}}_{GNSS,SPP}\\ \delta {\mathit{x}}_{cam}\end{array}\right]+\left[\begin{array}{c}{\mathit{\epsilon }}_{P,SPP}\\ {\mathit{\epsilon }}_{cam}\end{array}\right]\\ \left[\begin{array}{c}\delta {\mathit{P}}_{SPP}\\ \delta {\mathit{z}}_{cam}\end{array}\right]=\left[\begin{array}{c}P-{\widehat{P}}_{ins}\\ {\mathit{z}}_{cam}-{\widehat{\mathit{z}}}_{cam}\end{array}\right]\end{array}$$

(21)

where $\delta {\mathit{P}}_{SPP}$ is the error of the pseudorange observation for SPP, and $\delta {\mathit{z}}_{cam}$ is the error of the visual observation in Equation (12). ${\mathbf{H}}_{P,SPP}$ is the Jacobi matrix of the pseudorange error, and ${\mathbf{H}}_{cam}$ is the Jacobi matrix of the involved camera states in Equation (12). Then, $\delta {\mathit{x}}_{ins}$, $\delta {\mathit{x}}_{GNSS,SPP}$, and $\delta {\mathit{x}}_{cam}$ are the error state vectors of INS, SPP, and Vision, which can be found in Equations (7), (11), and (14), respectively. In the same way, ${\epsilon }_{P,SPP}$ and ${\epsilon }_{cam}$ denote the pseudorange observation error noise for SPP and the visual observation error noise, respectively. In addition, $P$ and ${\widehat{P}}_{ins}$ are, respectively, the actual measured pseudorange in Equation (1) and the pseudorange predicted by INS mechanization. ${\mathit{z}}_{cam}$ and ${\widehat{\mathit{z}}}_{cam}$, respectively, represent the observed and reprojected visual measurements in Equation (12).

The measurement equations for the integration of tightly coupled RTK/INS/Vision are formulated as follows:

$$\begin{array}{l}\left[\begin{array}{c}\begin{array}{c}\delta {\mathit{P}}_{RTK}\\ \delta {\mathit{L}}_{RTK}\\ \delta {\mathit{z}}_{cam}\end{array}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}{\mathbf{H}}_{P,RTK}\\ {\mathbf{H}}_{L,RTK}\\ {\mathbf{H}}_{cam}\end{array}\end{array}\right]\left[\begin{array}{c}\delta x_{ins}\\ \delta x_{GNSS,RTK}\\ \delta x_{cam}\end{array}\right]+\left[\begin{array}{c}\begin{array}{c}{\mathit{\epsilon }}_{\nabla \Delta P,RTK}\\ {\mathit{\epsilon }}_{\nabla \Delta L,RTK}\\ {\mathit{\epsilon }}_{cam}\end{array}\end{array}\right]\\ \left[\begin{array}{c}\begin{array}{c}\delta {\mathit{P}}_{RTK}\\ \delta {\mathit{L}}_{RTK}\\ \delta {\mathit{z}}_{cam}\end{array}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}\nabla \Delta \mathit{P}-\nabla \Delta {\widehat{\mathit{P}}}_{ins}\\ \nabla \Delta \mathit{L}-\nabla \Delta {\widehat{\mathit{L}}}_{ins}\\ {\mathit{z}}_{cam}-{\widehat{\mathit{z}}}_{cam}\end{array}\end{array}\right]\end{array}$$

(22)

where $\delta {\mathit{P}}_{RTK}$ and $\delta {\mathit{L}}_{RTK}$ represent the observation errors of the DD pseudorange and the DD carrier phase in RTK positioning, respectively. $\nabla \Delta \mathit{P}$ and $\nabla \Delta \mathit{L}$ can be found in Equation (3). In addition, $\nabla \Delta {\widehat{\mathit{P}}}_{ins}$ and $\nabla \Delta {\widehat{\mathit{L}}}_{ins}$ are the DD pseudorange predicted and the DD carrier phase predicted by INS mechanization, respectively. Then ${\mathbf{H}}_{P,RTK}$ and ${\mathbf{H}}_{L,RTK}$ are the Jacobi matrices of the DD pseudorange error and DD carrier-phase error. $\delta {\mathit{x}}_{GNSS,RTK}$ is the error state vector of RTK positioning, which can be found in Equation (15). ${\mathit{\epsilon }}_{\nabla \Delta P,RTK}$ and ${\mathit{\epsilon }}_{\nabla \Delta L,RTK}$ denote the DD pseudorange observation error noise and DD carrier-phase observation error in RTK positioning.

$$\left\{\begin{array}{c}{R}_k^{P,LOS}=f\times ({10}^{-{\displaystyle \frac{SNR-S_1}{a}}}(({\displaystyle \frac{A}{{10}^{-\frac{S_0-S_1}{a}}}}-1){\displaystyle \frac{SNR-s_1}{s_0-s_1}}+1))\times {\sigma }_P^2\\ R_k^{L,LOS}=f\times ({10}^{-{\displaystyle \frac{SNR-S_1}{a}}}(({\displaystyle \frac{A}{{10}^{-\frac{S_0-S_1}{a}}}}-1){\displaystyle \frac{SNR-s_1}{s_0-s_1}}+1))\times {\sigma }_L^2\end{array}\right.$$

(23)

$$\left\{\begin{array}{c}{R}_k^{P,NLOS}=K\times R_k^{P,LOS}\\ R_k^{L,NLOS}=K\times R_k^{L,LOS}\end{array}\right.$$

(24)

In Equation (23), $f=1/{\sin }^2(ele)$. $ele$ and $SNR$ refer to elevation angles and the SNR of satellites, respectively. The work [37] gives the parameters of $s_1=50$, $A=30$, $s_0=10$, and $a=20$; these are empirical values. ${\sigma }_P$ and ${\sigma }_L$ represent the standard deviation of the pseudorange and carrier phase, respectively, which are 0.3 m and 0.03 m in this paper. The $K$ is the scale factor, which is 10 in this paper. If the satellite’s projection is located in the sky semantic region, then a stochastic model of the satellite observations will be modeled by using Equation (24), and if not, it will be modeled by using Equation (23).

2.3. The Sky-View Image-Aided GNSS NLOS Detection and Mitigation Method (S−NDM)

For the accurate modeling of the GNSS noise covariance in the integration of tightly coupled GNSS/INS/Vision, it is essential to differentiate between LOS and NLOS satellites. Therefore, we obtain a sky-mask after performing the semantic segmentation of the sky-view images using an FCN, and subsequently, based on the projection model mentioned in [27] and the satellite information, including the elevation and azimuth angles provided by the satellite ephemeris, we ultimately identify the LOS and NLOS conditions around the GNSS receiver. Figure 3 shows the overall flow of the S−NDM algorithm. In this process, if the satellite’s projection is located in the sky region, then the satellite will be classified as an LOS satellite (represented by a blue dot), and if not, it will be classified as an NLOS satellite (represented by a red dot). Through this satellite visualization strategy, we can obtain the judgment conditions of Equations (22) and (23). This satellite visualization strategy is more reliable than the strategy of judging conditions by an experience threshold, as in [38]. This is also the difference between LOS/NLOS signal modeling in this paper and traditional methods.

3. Experiments

This section delineates the experimental methodology undertaken to assess the effectiveness of sky-view image-assisted GNSS NLOS detection across different positioning models. The study categorizes the models into SPP-related and RTK-related tightly coupled models for a comparative analysis. To evaluate the positioning performance, we calculated the root mean square error (RMSE) in the three directions of east (E), north (N), and up (U).

3.1. Experiment Description

As is shown in Figure 4, the data acquisition platform consists of a GNSS receiver (Septentrio mosaic-X5 mini, Septentrio, Brussels, Belgium), GNSS antenna (NovAtel GNSS-850, NovAtel, Calgary, AB, Canada), two forward-looking cameras (FLIR BFS-U3-31S4C-C, FLIR Systems, Wilsonville, OR, USA), a sky-pointing fish-eye camera (FE185C057HA-1, FUJINON, Fuji, Japan), a tactical-grade IMU (NovAtel SPAN-ISA-100C, NovAtel , Calgary, AB, Canada), an MEMS-IMU (ADIS-16470, ADI, Boston, MA, USA), and a time synchronization board. The time synchronization board unifies the time of all sensors to GPS time through pulse per second (PPS) generated by the GNSS receiver. The sampling rates of GNSS, MEMS-IMU, forward-looking cameras, and fish-eye camera are 1 Hz, 100 Hz, 10 Hz, and 1 Hz, respectively. In addition, the NovAtel SPAN-ISA-100C interacts with NovAtel’s ProPak7 receiver via a highly reliable IMU interface. The tightly coupled multi-GNSS post-processing kinematic (PPK)/INS bidirectional smoothing position results can be obtained through commercial IE 8.9 software and used as a reference truth value.

Table 1. The parameters of the IMU sensors used in the vehicle experiments.

IMU Sensor	Grade	Sampling Rate (Hz)	Bias Stability		Random Walk
			Gyro. 1 (°/h)	Acc. 1 (mGal)	Angular (°/$\sqrt{\mathit{h}}$)	Velocity (m/s/$\sqrt{\mathit{h}}$)
SPAN-ISA-100C	Tactical	200	0.05	100	0.005	0.018
ADIS-16470	MEMS	100	8	1300	0.34	0.18

¹ “Gyro.” and “Acc.” stand for gyroscope and accelerometer, respectively.

lists the specific parameters of the two IMUs. We use the Linux operating system and an Intel Core i7-9750H processor running at 2.6 GHz and with 32 GB memory. In addition, the data are processed on a laptop with NVIDIA GeForce RTX 3060 Ti GPU (graphics processing unit) based on Ubuntu 18.04 (Canonical Ltd., London, UK). Meanwhile, we used opencv3.4.9 [39] to process the images in the tightly coupled system.

We collected vehicular data in a typical urban canyon area in Wuhan on 3 September 2023. The experimental trajectory and surrounding landscape, featuring high-rise structures, dense foliage, and overpasses, are illustrated in Figure 5. The number of satellites and the position dilution of precision (PDOP) values for this route are shown in Figure 6. Combined with the LOS/NLOS satellite conditions presented in Figure 5, it is conceivable that our testing environment is plagued by severe GNSS NLOS, multipath, and cycle slip issues. These complications not only impair GNSS−based positioning accuracy but also challenge the reliability of GNSS/INS/Vision integration model, which depends on GNSS for absolute position information. The choice of such a challenging environment underscores the purpose of the experiment, which is to validate the efficacy and dependability of S−NDM algorithm and Sky−GVIO model introduced in this study.

In addition, we briefly introduce our sky-view image dataset, including the training dataset and the testing dataset. The training dataset contains 440 images, and the testing dataset contains 2000 images, each with a resolution of 1432 × 1164 pixels. These data were collected in typical urban canyons in two different areas of Wuhan, which contain trees, tall buildings, and light poles. Therefore, this dataset is very suitable for sky-view image segmentation experiments. Figure 7 shows the sky-view images that we have labeled semantically, with blue representing the sky area and black representing the non-sky area.

3.2. The Results of Sky-View Image Segmentation and GNSS NLOS Detection

The segmentation of sky-view images in urban canyons is challenging due to dynamic environmental factors such as cloud cover and varying light conditions, which can degrade the accuracy of traditional image segmentation techniques. The result of poor image segmentation accuracy will lead to errors in NLOS detection. Figure 8 presents a comparison of the results of sky-view image segmentation between traditional segmentation algorithms and the method proposed in this paper.

We compare our FCN−based method with representative image segmentation methods, including Otsu, Kmeans, and region growth. For fair comparison with the other competitors, all tests were performed on our collected dataset. As can be seen in Figure 8, clouds and light cause obvious errors in Figure 8b–d, especially in areas close to buildings. The performance of Otsu and Kmeans is relatively similar, but region growth shows a higher incidence of misclassification. Furthermore, in images featuring elevated bridges, an incorrect selection of seed points leads to the misidentification of sky regions as non-sky regions, which can be critically detrimental to NLOS identification. In contrast, our proposed FCN−based approach attains a high-precision segmentation outcome. This is because FCN captures global contextual information for the input image by using convolution and pooling layers. This allows the network to better understand the relationships between different objects and the overall structure of the image, which allows more accurate segmentation.

The FCN−derived segmentation results are utilized for NLOS detection, with the visibility of satellites illustrated in Figure 9. In Figure 9, we observe that there are no identification errors in the visualization results of LOS and NLOS satellites, underscoring the reliability of our S−NDM algorithm.

In the case of mild urban canyon environments (as shown in Figure 9a,c), LOS satellites dominate. However, in the case of deep urban canyons (as shown in Figure 9b) and environments with elevated bridges (as shown in Figure 9d), fewer than four LOS satellites are detectable, and the satellite configurations are suboptimal, highlighting the complexity and challenges of our experimental test environments.

3.3. The Quantitative Analysis of Sky-View Image Segmentation

Considering the strict requirements for precision and real-time measurements in vehicle positioning, we conducted quantitative tests on the efficiency and accuracy of different segmentation algorithms. The results are presented in

Table 2. Performance comparison of sky-view images using different methods.

Methods	Kmeans	Otsu	Region Growth	Ours
FPS	0.34	5.47	3.69	10.85
Accuracy	49.50%	36.45%	44.96%	98.54%

. The efficiency of these algorithms is reflected by FPS (frames per second), which refers to the number of images processed per second. The accuracy of the algorithm is reflected by “Accuracy”, which refers to the percentage of the number of correctly segmented images in the total number of processing results. The experiment is carried out on the training dataset, which is convenient for us to calculate the performance index of these algorithms.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+FN+TN+FP}}$$

(25)

where $TP$ denotes the number of pixels with correct sky region, $TN$ denotes the number of pixels with correct non-sky region, $FP$ denotes the number of pixels with incorrect sky region, and $FN$ denotes the number of pixels with incorrect non-sky region.

It can be seen from Table 2 that the FCN−based image segmentation algorithm is more efficient, which is due to FCN supporting an acceleration in GPU processing. The accuracies of the other central processing unit (CPU)-based machine learning methods are less than 50%. In addition, the update cycle of the tightly coupled GNSS, INS, and Vision model based on MSCKF is 1 s (which is synchronized with the GNSS sampling frequency). Therefore, FCN’s FPS fully meets requirements. Compared with machine learning methods, FCN based on deep learning is also more advantageous in terms of accuracy. Therefore, in terms of efficiency or accuracy, the FCN−based sky-view image segmentation algorithm proposed in this paper is meaningful.

3.4. The Experimental Results of Positioning

To verify the effectiveness of the Sky−GVIO model for car positioning in the urban canyon region and to evaluate the performance improvement of the two modes based on SPP and RTK positioning enhanced by S−NDM, we conducted several experimental comparisons and compared the results with those of state-of-the-art methods. It should be noted that we use the RTK float solution.

The time series of position errors for different SPP−related tightly coupled models are shown in Figure 10, and the corresponding RMSEs are summarized in

Table 3. Position RMSEs of VINS−mono, GVINS, SPP/INS/Vision, Sky−SPP/INS/Vision, RTK/INS/Vision, and Sky−RTK/INS/Vision models.

Method		Position RMSE (m)
		East	North	Up
Ours	SPP/INS/Vision	3.24	2.14	3.39
	Sky−SPP/INS/Vision	2.07	1.51	2.47
	RTK/INS/Vision	0.21	0.13	0.36
	Sky−RTK/INS/Vision	0.16	0.11	0.27
Others	VINS−mono	-	-	-
	GVINS	2.50	1.75	2.82

. The positioning accuracy of SPP/INS/Vision in E, N, and U directions is 3.24, 2.14, and 3.39 m. Different from SPP/INS/Vision, Sky−SPP/INS/Vision identifies LOS/NLOS satellites under the GNSS challenge environment and models them to inhibit the impact of NLOS on GNSS observations. As expected, the positioning accuracy is improved to 2.07, 1.51, and 2.47 m in E, N, and U directions when SPP/INS/Vision enhanced by S−NDM. Compared with SPP/INS/Vision, the positioning accuracy of Sky−SPP/INS/Vision is improved by 36%, 29%, and 27% in E, N, and U directions, respectively. As seen from results, Sky−SPP/INS/Vision can maintain meter-level positioning accuracy. Therefore, SPP−related Sky−GVIO is more suitable for mobile phone navigation and pedestrian navigation in urban canyons.

The time series of position errors for different tightly coupled RTK−related models are shown in Figure 11, and the corresponding RMSEs are summarized in Table 3. The positioning accuracy of RTK/INS/Vision in E, N, and U directions are 0.21, 0.13, and 0.36 m. It can be seen that the positioning accuracy of Sky−RTK/INS/Vision is 0.16, 0.11, and 0.27 m in E, N, and U directions, which outperforms RTK/INS/Vision. Compared with RTK/INS/Vision, the positioning accuracy of Sky−RTK/INS/Vision is improved by 24%, 15%, and 25% in E, N, and U directions, respectively. These considerable improvements in the positioning accuracy mainly stem from GNSS NLOS detection and mitigation enhanced by sky-view images, which makes the weighting of GNSS observations more reasonable. Meanwhile, we find that the S−NDM algorithm acts on SPP/INS/Vision better than RTK/INS/Vision in terms of positioning accuracy. This is because the positioning accuracy of SPP is lower than that of RTK positioning, and there is a lot of room for its enhancement; in addition, the NLOS has a greater effect on the pseudorange caused by the NLOS.

In addition, we compared our results against those using state-of-the-art methods, including VINS−mono [8] and GVINS [10]. As we all know, VINS−mono is a very famous tightly coupled model. However, without external information for correction, VINS−mono will accumulate drift errors, resulting in gradually larger errors in E, N, and U directions as shown in Figure 12. Due to large errors obtained by VINS−mono, statistics are not shown in Table 3. GVINS, for which GNSS pseudorange measurement, GNSS doppler measurement, visual constraints, and inertial constraints were jointly optimized, is also mature tightly coupled GNSS/INS/Vision model, which is often used to compare models of the same type. The time series of position errors for GVINS are presented in Figure 13, and the corresponding RMSEs are summarized in Table 3. The positioning accuracy of GVINS in E, N, and U directions is 2.50, 1.75, and 2.82 m, which outperforms SPP/INS/Vision in this paper. This is because GVINS adds Doppler measurements. However, GVINS did not have strict quality control in GNSS preprocessing, especially in regards to GNSS NLOS. Therefore, Sky−SPP/INS/Vision model proposed in this paper has higher accuracy than GVINS.

4. Conclusions

To mitigate the impact of NLOS on GNSS observations, this paper proposes an S−NDM algorithm. The S−NDM employs an FCN to segment sky-view images, followed by satellite projection for NLOS detection. Building on this, S−NDM is integrated into a tightly coupled GNSS/INS/Vision positioning model, named Sky−GVIO. The experimental results show that the proposed S−NDM algorithm achieves a notable performance in terms of segmentation accuracy, efficiency, and NLOS detection. In addition, we also evaluate the positioning performance of various tightly coupled positioning methods, and the experimental results demonstrate that the proposed Sky−GVIO model effectively improves the positioning accuracy in urban canyons. Specifically, Sky−RTK/INS/Vision achieves sub-decimeter accuracy, which is highly valuable for users requiring high-precision positioning services, and Sky−SPP/INS/Vision achieves meter-level accuracy, which is beneficial for low-cost applications, such as smartphones and pedestrian navigation.

In the future, we still have the following work to do:

(1)

Enhance the utilization of fish-eye camera data beyond GNSS NLOS detection, potentially integrating fish-eye camera observations into the proposed model;

(2)

Reach centimeter-level accuracy. By adding prior information (such as high-precision maps), the whole system can be made more robust and the positioning accuracy can increase.