Point–Line-Aware Heterogeneous Graph Attention Network for Visual SLAM System

Abstract

Simultaneous localization and mapping (SLAM), as an important research topic in robotics, is useful but challenging to estimate robot pose and reconstruct a 3-D map of the surrounding environment. Despite recent success of several deep neural networks for visual SLAM, those methods cannot achieve robust results in complex industrial scenarios for constructing accurate and real-time maps due to the weak texture and complex geometric structure. This paper presents a novel and efficient visual SLAM system based on point–line-aware heterogeneous graph attention network, which combines points and line segments to solve the problem of the insufficient number of reliable features in traditional approaches. Firstly, a simultaneous feature extraction network is constructed based on the geometric relationships between points and points and points and lines. To further improve the efficiency and accuracy of the geometric association features of key regions, we design the point–line-aware attention module to guide the network to pay attention to the trivial features of both points and lines in images. Moreover, the network model is optimized by a transfer-aware knowledge distillation strategy to further improve the system’s real-time performance. Secondly, to improve the accuracy of the point–line matching, we design a point–line heterogeneous graph attention network, which combines an edge aggregation graph attention module and a cross-heterogeneous graph iteration module to conduct learning on the intragraph and intergraph. Finally, the point–line matching process is transformed into an optimal transport problem, and a near-iterative method based on a greedy strategy is presented to solve the optimization problem. The experiments on the KITTI dataset and a self-made dataset demonstrate the better effectiveness, accuracy, and adaptability of our method than those of the state of the art in visual SLAM.

1. Introduction

Simultaneous localization and map construction technology, as the key to autonomous movement of robots, is widely used in unmanned driving, virtual reality, mobile robots, and other fields [1,2,3,4,5]. Compared with laser SLAM, vision-based SLAM has a low power consumption, low cost, miniaturization, and other advantages, and its theoretical and application value is very prominent [6]. Visual SLAM constructs a map of the surrounding environment by obtaining the plane image information of the real world through a camera. The pose state of the camera is inferred by the extracted feature information or pixel grayscale. Visual SLAM methods can be classified into four categories: feature-point-based methods [6,7,8], feature-line-based methods [9,10,11,12], feature-plane-based methods [12,13], and the combination of the above methods [14,15,16,17,18,19,20], according to the kind of features used to estimate the trajectory. The existing empirical methods cannot deal effectively with complex industrial scenarios due to occlusion, illumination, and deformation issues. In recent years, research on visual SLAM based on deep learning has attracted widespread attention. Serra et al. [21] used deep convolution to extract a point description with the L2 norm as a similarity measure to enhance the robust matching between key points and their local image features. Since a single feature descriptor cannot generate an effective key-point detection, Shend et al. [22] proposed an end-to-end trainable matching network, RF-Net, based on a receptive field to achieve a more efficient key-point detection. In order to solve the problem of the low localization accuracy caused by the lack of key-point shape perception in the joint learning of feature detectors and descriptors, Luo et al. [23] applied a deformable convolutional network with a dense spatial transformation to enhance the dynamic receptive field and improve the ability to express local shapes. Sarlin et al. [24] proposed a graph neural network, SuperGlue, based on attention aggregation, which used the optimal transmission model for matching optimization and realized the pose estimation in both indoor and outdoor environments. Combining a convolutional network with a recurrent network, Tang et al. [25,26] proposed a geometric correspondence network (GCN), which used an end-to-end learning method to detect key points and generate descriptors for improving the accuracy of the pose estimation. Aimed at the problem of decreased localization accuracy caused by the partial occlusion of line segments, Pautrat et al. [27] used a self-supervised network for line detection and for extracting line-segment descriptors, which improved the robustness of line-segment matching. The above-mentioned feature detection algorithms based on deep learning fused multilevel features, which did not deeply explore the association and constraint relationship between point and line features.

In industrial production scenarios, there are many complex background objects such as various buildings, pipelines, production equipment, and safety signs that lack corner points or contain repeated textures. The point features in the image are not specific enough, which makes them unable to provide an accurate position estimation. In addition, the mismatch of line features greatly increases the time complexity of the computation. In general, although existing methods have achieved certain results in feature detection and matching tasks, due to the uneven light, single texture, and complex scene structure in industrial scenarios, the pose estimation is easily degraded. How to efficiently fuse point and line information to build a more stable visual SLAM system is still a difficult problem that needs further research. Our main contributions can be summarized as follows:

• To solve the problem of weak point–line extraction ability in complex scenes, a point–line synchronous geometric feature extraction network, PL-Net, is proposed. We use an optimized residual block-feature pyramid network (ORB-FPN) to extract the feature map of the input image. In the point extraction branch, based on the point-aware module, the multiscale context is aggregated to obtain features with rich receptive fields. Moreover, the edge information is used for the line extraction branch to improve the accuracy of the line-segment detection. In order to make the network lightweight, a transfer-aware knowledge distillation method is proposed to compress the model for generating the point–line feature in the extraction task.
• Targeting a high accuracy and efficiency, a heterogeneous attention graph neural network (HAGNN) is presented, which uses an edge-aggregated graph attention network (EAGAT) to iterate the vertices of the heterogeneous graph constructed from points and lines. To enhance the performance of the point–line matching, a cross-heterogeneous graph interaction (CHGI) is used for harmonizing heterogeneous information between graphs.
• By transforming the point–line matching process into an optimal transport problem, a greedy inexact proximal point method for optimal transport, GIPOT, is proposed, which calculates the optimal feature assignment matrix to find the global optimal solution for the point–line matching problem.

2. SLAM System Framework

The framework of the SLAM system proposed in this paper is shown in Figure 1. Firstly, a new image is input into the PL-Net network to detect key points and line segments, and the corresponding descriptors are obtained through the point–line-aware attention module to enhance the feature expressiveness for both points and lines in images. Then, the point and line features of the two images are transferred to the point–line heterogeneous graphs, which are constructed by using the point and line features as the vertices and connecting a vertex to its neighbors within a fixed radius. Secondly, the attention network HAGNN obtains the enhanced features and inputs them into the GIPOT to generate point–line matching results and calculate the pose of the current frame. Finally, by reprojecting the features in the local map to the current frame, the projection error is calculated for the backend processing of SLAM to complete the map.

3. Methodology

3.1. Point–Line Feature Extraction Network

The point–line feature extraction network PL-Net is shown in Figure 2. Firstly, the ORB-FPN module was used to extract the features of each layer for the image, and the PG module completed the key-point extraction. Then, the center point map and displacement map were generated through the branch of the line-segment perception module. Finally, the point–line descriptor was generated through convolution and upsampling operations.

3.1.1. ORB-FPN Module

As shown in Figure 2, an optimized residual block (ORB) is designed based on the Nesterov acceleration gradient (NAG) algorithm to enhance the expressive ability of target features [26]. Then, we have:

$$y_{k+1}=x_k+\beta \left(x_k-x_{k-1}\right)$$

(1)

$$x_{k+1}=y_{k+1}-\alpha \nabla f\left(y_{k+1}\right)$$

(2)

where $x_k$ and $y_{k+1}$ denote the output and input of the first layer of the network, and α and $\beta$ represent the learning rate and momentum parameters, respectively. $\nabla f\left(y_{k+1}\right)$ is the gradient of the objective function f at $y_{k+1}$, and f is a smooth function satisfying the Lipschitz property. When the momentum parameter $\beta =0$, the NAG algorithm is equivalent to the standard gradient descent algorithm. When $\beta >0$, it optimizes the combination of α and $\beta$ to accelerate convergence.

In the neural network propagation process, the transmission of the signal from the first layer to the last layer is expressed as:

$$L_{i+1}=\sigma \left(U_i{L}_i\right)$$

(3)

where $L_{i+1}$ is the features of the $i+1$th layer in the network, and $\sigma$ represents the activation function. Suppose U is a symmetrical positive definite matrix; let $V=\sqrt{U}$ and $\mu =VL$; then, for the nonlinear activation function $\sigma \left(\mu \right)$, there is a function $g\left(\mu \right)$, when $g^{\prime }\left(\mu \right)=\sigma \left(\mu \right)$. We have:

$$\nabla {\sum }_{i}g\left(V_j^T\mu \right)=U\sigma \left(U^{T}L\right)=U\sigma \left(UL\right)$$

(4)

The objective function $f\left(\mu \right)$ is defined as:

$$f\left(\mu \right)=\frac{∥{\mu }^2∥}{2}-{\sum }_{i}g\left(V_j^T\mu \right)$$

(5)

where $V_i$ is the ith column of V. Then,

$$\nabla f\left({\mu }_i\right)={\beta }_i-V\sigma \left(V{\mu }_i\right)$$

(6)

Equation (2) can be expressed by:

$${\mu }_{i+1}={\mu }_i+\beta \left({\mu }_i-{\mu }_{i-1}\right)-\alpha \left(\left(1+\beta \right)\nabla f\left({\mu }_i\right)-\beta \nabla f\left({\mu }_{i-1}\right)\right)$$

(7)

Recovering L by $L=V^{-1}\mu$ leads to:

$$L_{i+1}=\left(\left(1+\beta \right)\left(1-\alpha \right)-\alpha \beta \right)L_i+\beta \left(1-\alpha \right)L_{i-1}+\alpha \left(1+\beta \right)\sigma \left(U{L}_i\right)$$

(8)

where $\sigma \left(U{L}_i\right)$ is the ith layer feed-forward network, and the ORB module structure is shown in Figure 2.

In order to aggregate the FPN multiscale feature information, a dual attention module (DAM) was designed to perform the feature aggregation. As shown in Figure 3, firstly, in order to obtain the position and channel information of the feature, the shallow feature map $x\in {\mathbb{R}}^{W\times H\times D_1}$ was passed through a global pooling operation and compression to generate the position vector $x_p\in {\mathbb{R}}^{W\times H\times 1}$ and the channel vector $x_c\in {\mathbb{R}}^{1\times 1\times D_1}$, respectively. Then, the position vector $x_p$ computed the weight of each position with a sigmoid activation function and multiplied it with the feature map $x_p$ to generate the spatial position feature map $F^p\in {\mathbb{R}}^{W\times H\times D_1}$, which was defined as:

$$F^p=\sigma \left(x\right)\otimes x$$

(9)

where x is the shallow feature map, and $\sigma$ is the sigmoid activation function. Similarly, the convolution of the ReLU activation function, defined as $f\left(x\right)=max(0,x)$, and the sigmoid activation function was performed on the channel vector $x_c$. The weight of each channel was calculated and multiplied by the feature map x to generate the feature map $F^c\in {\mathbb{R}}^{W\times H\times D_1}$, which was defined as follows:

$$F^c=\sigma (\delta \left(W_2\left(W_1\left(gp\left(x\right)\right)\right)\right)\otimes x$$

(10)

where $\delta$ is the ReLU activation function, and $W_1$ and $W_2$ are convolution operations with sizes $1\times 1\times D_1/16$ and $1\times 1\times D_1$, respectively. Finally, the final output feature map F was obtained by fusing the feature maps $F^p$, $F^c$, and $x^{\prime }$. Then,

$$F=[(x\oplus F^p\oplus F^c),x^{\prime }]$$

(11)

where ⊕ represents the addition of the corresponding elements of two matrices.

The input image was passed through the ORB-FPN module, which denoted the output of the backbone as $\{C_2,C_3,C_4,C_5\}$ with strides of $\{4,8,16,32\}$. $\{F_2,F_3,F_4\}$ were obtained after a $1\times 1$ convolution with the same 128-dimensional channel features. Finally, the ORB module was added to enhance the acceptance domain of the output feature by using the backbone network on $C_5$ to separate the important context information. After interpolation and maximum pooling of the extracted context features and the generated three feature maps, an elementwise summation was performed to obtain features F.

3.1.2. Key-point Detection Module

As shown in Figure 2, the key-point detection module consisted of three point perception modules and two $3\times 3$ convolutions with a stride length of 1. A batch normalization layer and ReLU layers were added between each convolutional layer. The output vector was processed through a sigmoid activation function, so that the pixel values of the saliency map were between 0 and 1. Then, through the key-point perception module and convolution processing, the convolution operation was used to discriminate whether the $8\times 8$ area prediction contained key points. Finally, the key points were detected by using the nonmaximum suppression (NMS) method in the key-point generation module (PG).

The point-aware module was used to capture the relationship between key points. As shown in Figure 4, the key-point extraction branch embedded a context enhancement module, which improved the feature expression ability. The output feature y${}^p$$\in {\mathbb{R}}^{W\times H\times D}$ was obtained by fusing the convolutional features of different scales, which took y${}^p$$\in {}^{W\times H\times D}$ as input. The above process was defined as:

$$y^p=W_1[x^p,BN\left(W_1{x}^p\right),BN\left(W_2{x}^p\right),BN\left(W_3{W}_1{x}^p\right)]$$

(12)

where $x^P$ is the input feature, $W_1$, $W_2$, and $W_3$ are convolution operations with sizes of $1\times 1$, $2\times 2$, and $3\times 3$, respectively. $BN$ is a normalization, and $\left[\right]$ is a splicing operation.

3.1.3. Line-Segment Detection Module

The line-segment detection module extracted the features of the plane image through the ORB-FPN module and then input it to the line-segment extraction module to generate a midpoint with two symmetrical endpoints as the line-segment detection result. The extraction of the line segment’s center point [28] used the classification model to judge whether the pixel was the center point of the line segment. Since the shape of the line segment was narrow and long, a large receptive field was required to classify the center of the line segment. Therefore, a hybrid convolution module was introduced by stacking three convolutional layers, a 3 × 3 deformable convolutional layer and two 3 × 3 dilated convolutional layers. The receptive field of the network was increased while reducing the parameters of the network. Then, three line-segment perception modules were used to enhance the feature representation ability. Finally, a deconvolution layer was used to restore the size of the output map to 512 × 512, which represented the center point of the line segment on the output feature map. The displacement regression task of the line-segment extraction branch was designed to predict the angle and length of the end point relative to the midpoint. It was composed of a 3 × 3 deformable convolution layer and two 3 × 3 convolutional layers with a stride of 1. The relevant displacement was indexed by the position through the output map. Finally, the CAL [29] was used for the line-segment generation, and the two endpoints of the line segment were defined as follows:

$$({x_l}_{{}_s},{y_l}_{{}_s})=({x_l}_{{}_c},{y_l}_{{}_c})+\frac{\alpha }{2}(cos\theta ,sin\theta )$$

(13)

$$({x_l}_e,{y_l}_{{}_e})=({x_l}_{{}_c},{y_l}_{{}_c})-\frac{\alpha }{2}(cos\theta ,sin\theta )$$

(14)

where $({x_l}_{{}_c},{y_l}_{{}_c})$ is the coordinates of the root node, $\alpha$ is the length of the line segment, and $\theta$ is the rotation angle.

In the line-segment detection, a line-aware module was introduced to effectively extract line shape features. As shown in Figure 5, the line-segment-aware module adopted the improved self-attention mechanism, which took the feature $x^L\in {\mathbb{R}}^{W\times H\times D}$ as input and fused the self-attention features to generate the final output feature $y^L\in {\mathbb{R}}^{W\times H\times D}$; the above process was defined as follows:

$$y^L=x^L\oplus W_1(\alpha (W_{{}^q}{x}^L\times W_{{}^k}{x}^L)\times W_{{}^v}{x}^L)$$

(15)

where $x^L$ is the input feature, $W_1$, $W_q$, $W_k$, and $W_v$ are the learned weight matrices, which were implemented as 1 × 1 convolutions, and ⊕ means that the corresponding elements of the two matrices are added.

3.1.4. Parallel Attention Module

To efficiently merge the two branches of point descriptors and line descriptors, we designed a parallel attention module (PAB) based on self-attention and channel attention. As shown in Figure 6, the output features of the key-point description branch contained line-edge information with a strong correlation. In order to improve the accuracy of the line descriptor, a lightweight attention mechanism was used to assign more weights for the features of useful regions. The output feature map of the point description branch was expressed as ${\mathrm{X}}_E\in {\mathbb{R}}^{C\times H\times W}$. A one-dimensional convolution of ${\mathrm{X}}_E$ was performed to obtain the spatial attention map $A_E\in {\mathbb{R}}^{C\times H\times W}$. The edge feature map ${\mathrm{X}}_E^S\in {\mathbb{R}}^{C\times H\times W}$ was calculated as: ${\mathrm{X}}_E^S=a({\mathrm{X}}_E\odot A_E)+{\mathrm{X}}_E$, where a is a learnable parameter that was initialized to 0. The CAEU module was designed to calculate a channel attention map, which recalibrated the weight of the channel and obtained the fused feature map ${\mathrm{X}}_F^S\in {\mathbb{R}}^{C\times H\times W}$ from ${\mathrm{X}}_F^S={\mathrm{X}}_E^S\otimes \delta (Conv1\times 1(Conv1\times 1\left(GAP\left({\mathrm{X}}_E^S\right)\right))$. Finally, the final output ${\mathrm{X}}_F^{SC}\in {\mathbb{R}}^{2C\times H\times W}$ of PAB was obtained by concatenating ${\mathrm{X}}_T^S$ and ${\mathrm{X}}_F^S$ together.

3.1.5. Network Output Distillation

In order to reduce the increasing computation cost caused by the introduction of the attention mechanism and the point–line perception module, we further compressed the point–line detection model PL-Net. A transfer-aware method is presented to transfer the information from the teacher model to the student network. The knowledge distillation strategy (KD) [30,31,32] was used to fine-tune the accuracy of the recovery model. As shown in Figure 2, the multiples tasks were combined with the teacher network and the student network to guide the training of the student feature extraction. In the training process, the adaptive weighted multitask distillation was realized, and ${\mathrm{X}}_{tea}$, ${\mathrm{Y}}_{tea}$, and ${\mathrm{Z}}_{tea}$ represented the key point of the teacher model, the center point of the line segment, and the output of the line-segment regression feature layer, respectively. The student model corresponded to the feature layers ${\mathrm{X}}_{stu}$, ${\mathrm{Y}}_{stu}$, and ${\mathrm{Z}}_{stu}$. The mean squared error (MSE) function was applied to the multitask distillation. The loss function of the training distillation was:

$$\left\{\begin{array}{c}{L}_p^S={∥{\mathrm{X}}_{tea}-{\mathrm{X}}_{stu}∥}^2\\ L_{root}^S={∥{\mathrm{Y}}_{tea}-{\mathrm{Y}}_{stu}∥}^2\\ L_{dis}^S={∥{\mathrm{Y}}_{tea}-{\mathrm{Y}}_{stu}∥}^2\end{array}\right.$$

(16)

where $L_p^S$, $L_{root}^S$, and $L_{dis}^S$ denote the key point, the line segment’s center point, and the distillation loss of the line-segment regression task, respectively. Then, the weighted distillation loss was defined as:

$$L_{MSE}^s=\sum _l{\omega }_l{L}_l^s$$

(17)

where $L_{MSE}^s$ is the multitask distillation loss, and ${\omega }_l$ represents the weight value of the verification loss.

3.1.6. Interlayer Knowledge Distillation

Different from the network output distillation, the student network was further enhanced by performing an interlayer knowledge distillation between the output of the teacher model and the student model. The aware modules of each student layer were associated with the relevant target-layer-aware modules for knowledge transfer. The layer’s knowledge distillation loss was defined as:

$$L_{FMD}=\sum _{(s_l,t_l)\in C}Dist\left(Tran{s}^t\right(F_{tl}^t),Tran{s}^s(F_{sl}^s\left)\right)$$

(18)

Then, the overall loss was obtained as:

$$L_{\mathrm{total}}=L_{MSE}^s+\beta L_{FMD}$$

(19)

where $Trans(·)$ means to convert the feature map of the perception module into a specific manual representation through the attention map. C is the perception module, and $F_{tl}^t$ and $F_{sl}^s$ are the feature layers of the lth layer of the student model and the teacher model, respectively. The distance function $Dist(·)$ was used for computing the distillation loss of the feature maps, and $\beta$ was a hyperparameter.

3.2. Heterogeneous Graph Attention Network

As shown in Figure 7, consider two images I and $I^{\prime }$, and the number of two feature sets m and n belonging to them, respectively. Let $d\in \mathbb{R}{}^D$ be the feature descriptor, where D is the dimension of the descriptor. We utilized an attention graph neural network to integrate the contextual cues and enhance its feature expression ability.

Position encoders were used for the two input features, and the key points and lines positions were embedded into a high-dimensional vector by adding position encoding to ${\widehat{F}}_1$ and ${\widehat{F}}_2$; thus, we had:

$$f_i=d_i+\mathrm{MLP}\left(P_i\right)$$

(20)

$$\mathrm{MLP}\left(P_i\right)=W_a\sigma (W_b{P}_i)$$

(21)

where $P_i$ is the position of the feature, $d_i$ is the descriptor information, and $\sigma (·)$ is the ReLU activation function.

3.2.1. Edge-Clustering Graph Attention Module

We propose an edge-aggregated graph attention network (EAGAT) based on GAT [33], which uses edge information for feature enhancement during the aggregation process. In order to make full use of the information of edge features, these different types of links used different attention mechanisms. For features of the same nature (points and points, lines and lines), the self-attention mechanism was used for the aggregation. For features of different nature (points and lines), the cascade method was used for the aggregation. Let the feature of the vertex $v_i^{\prime }$ in the graph be $f_i$, defined as:

$$\begin{array}{c}{f^{\prime }}_i=\sigma ({\displaystyle \sum _{j\in {\mathcal{N}}_s}}softmax(\frac{W^a{f}_i{W}^{\beta }{f}_j}{\sqrt{d_k}}{W}^{\gamma }{a}_{ji})W^{\epsilon }{f}_j\hfill \\ +{\displaystyle \sum _{j\in \mathcal{N}{}_{\mathrm{d}}}}softmax([W^a{f}_i\left|\right|W^{\beta }{f}_j\left|\right|W^{\gamma }{a}_{ji}\left]\right)W^{\epsilon }\left)\right||f_i\hfill \end{array}$$

(22)

where $W^a$, $W^{\beta }$, $W^{\gamma }$, and $W^{\epsilon }\in {\mathbb{R}}^{{f^{\prime }}_{\eta }\times f_{\eta }}$ represent the weight parameters, ${\mathcal{N}}_s$ is the feature set of the same nature, ${\mathcal{N}}_d$ represents the feature set of different properties, and $\sigma$ is the ReLU activation function.

3.2.2. Cross-Heterogeneous Graph Iteration Module

Due to the affinity learning problem of message passing between graphs in graph matching, a point–line heterogeneous graph message-passing method is proposed to enhance the node features through an interactive correlation. The edge features and node features were aggregated in two ways. For nodes of the same nature (points and points, lines and lines), we used linear attention for the aggregation, and for nodes of different properties (points and lines), we used the aggregation method of the self-attention mechanism. Then, $f_{si}^{\prime }\in {\mathbb{R}}^{{f^{\prime }}_{\eta }}$ is expressed as:

$$\begin{array}{c}{f^{\prime }}_{si}=\mathrm{LN}\{{\displaystyle \sum _{{}_{j\in \mathcal{N}{}_{di}}}}(softmax[\left(W_{\mathrm{v}}{f}_j\right){\left(W_k{f}_j\right)}^T]+W_q{f}_{si}\left)\right\}\hfill \\ +\mathrm{LN}\{{\displaystyle \sum _{{}_{j\in \mathcal{N}{}_{\mathrm{s}i}}}}(W_l(\left(W_q{f}_{si}\right){\left(W_k{f}_j\right)}^T+W_k{f}_j)+f_{si}\}\hfill \end{array}$$

(23)

where $v_{l/r}\in {\mathbb{R}}^{{f^{\prime }}_{\eta }}$ is the feature node of the two graphs, $W_{(·)}$ is the weight parameter, and $\mathrm{LN}$ represents a layer normalization (LN).

3.3. Greedy near Iterative Matching Module

The output $H_i$ of the last layer in the graph neural network is the feature of graph $I^{\prime }$. $H_j$ is the feature of graph $I^{\prime }$, and the point–line distance matrix $G\in {\mathbb{R}}^{+N_1\times N_2}$ can be expressed as:

$$G=f_{aff}(H_i,H_j^{\prime }),i\in v_1,j\in v_2$$

(24)

where $f_{aff}$ is the weighted bilinear function, defined as:

$$f_{aff}(H_i,H_j^{\prime })=exp(\frac{H_i^{T}K{H}_j^T}{\tau })$$

(25)

where the feature is an n-dimensional vector, namely $\forall i\in v_1$, $j\in v_2$ and $H_{\mathrm{i}}^T$, $H_{\mathrm{j}}^T\in {\mathbb{R}}^{n\times n}$, $K\in {\mathbb{R}}^{n\times n}$ is the weight matrix of the affinity function, and $\tau$ is the regularization parameter. In the matching process, due to the inconsistency of point and line types, a direct fusion may cause a mismatch of point and line types. For this reason, we regarded the unit block diagonal matrix as the initial coupling matrix ${\mathsf{\Gamma }}^{\left(1\right)}$ so that the relationship between point features and line features in the iterative process minimized the matching cost. Then, we had:

$${\mathsf{\Gamma }}^{\left(1\right)}\leftarrow \left[\begin{array}{cc}{11}_p& 0\\ 0& {11}_L\end{array}\right]$$

(26)

where P is the number of points after completion, and L is the number of lines after completion. Then, the point–line discrete distribution Sinkhorn distance [34,35] was defined as:

$$W_{\in }(u,v)=\underset{\mathsf{\Gamma }\in \sum \left(u,v\right)}{\min }〈C,\mathsf{\Gamma }〉+\lambda h\left(\mathsf{\Gamma }\right)$$

(27)

where u and v are probability vectors, $W_{\in }\left(u,v\right)$ is the distance between u and v, the matrix $C=\left[c_{ij}\right]\in {\mathbb{R}}^{+n\times n}$ is the cost matrix, and $c_{ij}$ is the distance between $u_i$ and $v_j$. The regularization term $h\left(\mathsf{\Gamma }\right)={\sum }_{i,j}{\mathsf{\Gamma }}_{ij}ln{\mathsf{\Gamma }}_{ij}$. The proximal point iteration method [35] was used to solve Equation (25). According to the proximal point iteration method, it is defined as the Bregman divergence:

$$D_h\left(x,y\right)={\sum }_{i=1}^n{x}_{i}ln\frac{x_i}{y_i}-{\sum }_{i=1}^n{x}_i-{\sum }_{i=1}^n{y}_i$$

(28)

After introducing the near-end point iteration, Equation (27) can be rewritten as:

$${\mathsf{\Gamma }}^{(t+1)}=\underset{\mathsf{\Gamma }\in \sum (u,v)}{\mathrm{argmin}}〈C,\mathsf{\Gamma }〉+{\beta }^t{D}_h(\mathsf{\Gamma },{\mathsf{\Gamma }}^{\left(t\right)})$$

(29)

Substituting the Bregman divergence Equation (28) into Equation (29), it becomes

$${\mathsf{\Gamma }}^{(t+1)}=\underset{\mathsf{\Gamma }\in \sum \left(u,v\right)}{\mathrm{argmin}}〈C^{\prime },\mathsf{\Gamma }〉+{\beta }^{t}h\left(\mathsf{\Gamma }\right)$$

(30)

where $C^{\prime }=C-{\beta }^{t}ln{\mathsf{\Gamma }}^{\left(t\right)}$; we used the greedy strategy to update the best row or column and defined the distance matrix

$$\rho (x,y)=y-x+log\frac{x}{y}$$

(31)

According to Equations (29) and (30), the affinity matrix was updated to find the best matching relationship, and the specific algorithm flow is shown in Algorithm  1.

Algorithm 1: GIPOT($\mu$, v, G).

3.4. Loss Function

In order to realize the matching of points and lines, the point–line extraction loss, descriptor extraction loss, and point–line matching loss were used as the loss functions during model training.

3.4.1. Point–Line Extraction Loss

In the training stage of the point and line extraction branch, the output included the root node’s confidence map, key-point map, and displacement map. The losses of these three tasks were combined into Equation (32), defined as follows:

$$L_{PLE}=L_{root}+L_p+L_{dis}$$

(32)

The ground truth of the root-point confidence map was constructed by marking the root-point positions on a zero map. A weighted binary cross-entropy loss $L_{root}$ was used to supervise this task, which was defined as:

$$L_{root}=-{\sum }_i{\tilde{R}}_{i}log{R}_i+(1-{\tilde{R}}_i)log(1-R_i)$$

(33)

where ${\tilde{R}}_i$ and $R_i$ are the prediction and true label of the root node of the line segment, respectively. The true value of the key point was marked with the ORB feature. $L_p(\mathrm{X},\mathrm{Y})$ was defined as:

$$L_p(\mathrm{X},\mathrm{Y})=\frac{1}{H_c{w}_c}\sum l_p\left(T^{ij},\widetilde{T^{ij}}\right)$$

(34)

$$l_p\left(T^{ij},\widetilde{T^{ij}}\right)=-log(\frac{exp(T_k^{ij})}{{\sum }_1^{64}exp(T_k^{ij})})$$

(35)

The displacement part of the line segment relative to the root node was used to locate the length and angle of the line segment, which used the L1 loss and L1 smoothing loss, respectively defined as:

$$L_{dis}=\sum _{i=1}^m\left\{\begin{array}{ccc}\hfill \left|{\theta }^i-{\widehat{\theta }}^i\right|+0.5\ast {({\rho }^i-{\widehat{\rho }}^i)}^2& & if\left|{\rho }^i-{\widehat{\rho }}^i\right|<1\hfill \\ \hfill \left|{\theta }^i-{\widehat{\theta }}^i\right|+\left|{\rho }^i-{\widehat{\rho }}^i\right|-0.5& & otherwise\hfill \end{array}\right.$$

(36)

where ${\theta }^i$ and ${\rho }^i$ represent the actual line segment’s length and angle, and ${\widehat{\theta }}^i$ and ${\widehat{\rho }}^i$ are the predicted line segment’s length and angle, respectively.

3.4.2. Point–Line Descriptor Loss

Denote the original image as I, and apply the homography transformation for I to form a new image $I^{\prime }$. Since the homography transformation is known, the corresponding relationship between key points and line segments on I and $I^{\prime }$ can be obtained. Therefore, the loss function can be defined as:

$$L_d(\theta ,\left\{d_a^{\theta },d_{+}^{\theta },d_{-}^{\theta }\right\})={[m+{∥d_a^{\theta }-d_{+}^{\theta }∥}^2-\underset{d_{-}^{\theta }\in d_{-}^{\theta }}{min}{∥d_a^{\theta }-d_{-}^{\theta }∥}^2]}_{+}$$

(37)

where the parameter m was set to 0.5, $d_a^{\theta }$ is the descriptor on I of the anchor point, $d_{+}^{\theta }$ is the matching descriptor on $I^{\prime }$ of the positive sample, and $d_{-}^{\theta }$ is the set of nonmatching descriptors on $I^{\prime }$ of the negative sample.

3.4.3. Matching Loss

For a matching network using the L2 loss, the loss function can be expressed as:

$$L_m=\frac{1}{\left|M_{gt}\right|}\sum _{(i,j)\in M_{gt}}\frac{1}{{\sigma }^2\left(i\right)}{∥M(i,j)-M_{gt}(i,j)∥}_2$$

(38)

where ${\sigma }^2\left(i\right)$ is the confidence variance of feature i. $M(i,j)$ is the matching probability of feature i and feature j, and $M_{gt}$ is the real-valued matrix obtained by the homography transformation.

3.4.4. Normalization

The total loss function was the sum of the above loss functions:

$$L_{sum}={\lambda }_1{L}_{PLE}+{\lambda }_2{L}_d+{\lambda }_3{L}_m$$

(39)

where ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ represent the coefficients of each loss function, respectively. ${\lambda }_{1,2,3}=\{0.25,0.25,0.5\}$.

4. Experiments and Evaluation

4.1. Model Training Details

A wide range of experiments were performed on different datasets to demonstrate the efficacy of our method. Our approach was evaluated with several evaluation criteria by comparing to the typical SLAM methods on the KITTI dataset [36]. We used the training set of Wireframe [37] with the ground truth to train our models and the other compared methods. The training process used data enhancement techniques such as random Gaussian noise, motion blur, and brightness level changes to improve the network’s robustness ability for changes in lighting and viewing angles. For end-to-end training of the point–line matching network, our network was implemented in Pytorch [38] using the Adam [39] optimizer to train the network with an initial learning rate of $1\times {10}^{-5}$ and a decay of the learning rate by 20 at each epoch. We trained our model on a GeForce GTX2080Ti GPU.

4.2. KITTI Dataset Evaluation

We tested the proposed algorithm on the KITTI dataset [36]. The quantitative evaluations for the different SLAM systems were the absolute trajectory error (ATE) [40] and the relative pose error (RPE) [41] based on translations and rotations.

Table 1. Comparison between ATE and RPE on different SLAM algorithms.

Seq	ORB-SLAM2			Our
	ATE (m)	${\mathit{RPE}}_{\mathit{trans}}$ (%)	${\mathit{RPE}}_{\mathit{rot}}$ (deg/m)	ATE (m)	${\mathit{RPE}}_{\mathit{trans}}$ (%)	${\mathit{RPE}}_{\mathit{rot}}$ (deg/m)
00	1.266	52.5	0.363	1.233	2.9	0.122
01	4.296	3.4	0.420	2.616	4.8	0.044
02	12.790	4.3	0.107	12.721	3.6	0.077
03	0.403	0.8	0.072	0.385	2.0	0.055
04	0.466	2.2	0.055	0.192	2.1	0.040
05	0.348	2.3	0.144	0.402	1.7	0.056
06	1.184	3.9	0.089	0.572	1.8	0.042
07	0.439	1.3	0.076	0.436	1.6	0.046
08	3.122	12.1	0.076	2.874	3.9	0.054
09	3.319	15.0	0.104	1.537	2.2	0.054
10	0.927	2.6	0.090	0.989	2.1	0.060

shows the performance of this system was better than ORB-SLAM2, especially in sequences with strong lighting, motion blur, and low texture areas, such as 06 and 09. It can be seen that the multifeature fusion not only improved the accuracy of the algorithm but also avoided the degradation problem that may occur in the pose solution algorithm when using a single feature.

Figure 8 shows the comparison results of ORB-SLAM2 and the method in this paper on KITTI’s partial sequences. It can be seen that the algorithm of this paper was equivalent to ORB-SLAM2 as a whole. However, in sequences such as 00, 02, and 09, it showed that our method had the best results. On the other hand, the ORB-SLAM2 lost track easily and did not have a whole trajectory.

Figure 9 depicts the variation curve of the pose estimation error with the number of training iterations. When we used more than 80 iterations, the rotation and translation errors were relatively small while the performance improved, which showed that the algorithm in this paper had a good convergence.

Figure 10 visualizes the statistical error property between our system and ORB-SLAM2 on sequence 09 of KITTI. It shows that our model achieved obviously a better performance than ORB-SLAM2 in terms of the root-mean-square error (controlled within 2 m), extreme value error, and the sum of the squared errors. Figure 10b shows the columnar statistical comparison of postural errors; our model was better than ORB-SLAM2 in the APE distribution range of the algorithm.

4.3. Real Data Evaluation

As shown in Figure 11, the feasibility of the algorithm was verified on a physical and virtual oil and gas station, where the virtual simulation platform with a quadruped robot was built with Unity3D.

To show the effects of point and line features on the SLAM system, we intercepted two frames of images for extracting the features by PL-Net and matching by HAGNN. As shown in Figure 12, it can be seen that the combination of point and line could make the SLAM algorithm obtain richer and more diverse feature information.

The virtual simulation platform used a quadruped robot to inspect the oil field equipment. As shown in Figure 13, this trajectory was compared with the ground truth by using the evaluation package to obtain the RPE and APE. It can be seen that the area with a larger error was basically distributed at the corner of the trajectory. The RMSE was 17.5 m. Overall, the trajectory of our method was consistent with the ground truth with a high accuracy.

4.4. GIPOT Experiment

In order to illustrate the convergence of GIPOT with different $\beta$, the Wasserstein distance of two one-dimensional Gaussian distributions was measured as an evaluation index. As shown in Figure 14a, the blue equation was $0.5N(70,8)+0.5(35,10)$, the red equation was $0.4N(80,9)+0.6N(40,10)$, where $N(\mu ,{\sigma }^2)$ is the probability density function of the one-dimensional Gaussian distribution, $\mu$ and ${\sigma }^2$ are the mean and variance, respectively. Figure 14b shows the convergence of GIPOT under different conditions. Compared with the Sinkhorn method, the convergence of the GIPOT iteration was quicker when $\beta$ was large. GIPOT could converge to the exact Wasserstein distance with a complexity comparable to that of Sinkhorn.

4.5. Ablation Study

We used two datasets to demonstrate our proposed knowledge distillation. The result of the teacher network and the student network are shown in Figure 15. Compared with the true value, both models could identify key points and line segments with high precision. Although there were some small missing line segments and connection errors in the results of the student network, the expression of the line-segment structure in the environment was basically accurate. The quantitative comparison is shown in

Table 2. Quantitative evaluation of PL-Net point-line detection knowledge distillation method on Wireframe dataset and YorkUrban dataset.

Method	Wireframe Dataset			YorkUrban Dataset			FPS
	${\mathit{F}}^{\mathit{H}}$	$\mathit{sAP}$	$\mathit{LAP}$	${\mathit{F}}^{\mathit{H}}$	$\mathit{sAP}$	$\mathit{LAP}$
Student	77.5	58.9	59.8	64.6	25.9	32.0	12.5
Teacher	80.6	57.6	61.3	67.2	27.6	34.3	7.2

. Although the performance of the student network was slightly lower than that of the teacher network, the operation speed was increased by 73%.

To verify the role of the multifeature fusion in the SLAM system, the root-mean-square error (RMSE) of the ATE index under different feature combinations was calculated. The experimental results are given in

Table 3. Results of ablation experiment in term of the RMSE of ATE (Unit: m).

Seq	P-SLAM	L-SLAM	PL-SLAM	ORB-SLAM2	LSD-SLAM	PTAM
00	1.203	6.233	1.233	1.266	5.347	2.842
01	3.934	12.367	2.616	4.296	—	3.358
02	7.689	—	12.721	12.790	—	13.742
03	0.393	5.457	0.385	0.403	7.431	2.302
04	0.347	13.824	0.192	0.466	—	2.773
05	0.863	—	0.402	0.348	1.293	0.456
06	0.884	—	0.572	1.184	—	1.024
07	0.255	—	0.436	0.439	—	0.423
08	3.122	—	2.874	3.122	—	3.358
09	2.625	4.783	1.537	3.319	11.395	2.048
10	0.447	5.824	0.989	0.927	2.841	0.768

, where the point–line feature combination method used in this paper significantly improved the accuracy of the pose estimation. To evaluate our design decisions, we evaluated four different variants with results. This ablation study, presented in

Table 4. Ablation of HAGNN.

	Known		Unknown
	Match Precision	Matching Score	Match Precision	Matching Score
No EAGAT	79.6	29.5	55.3	15.6
No CHGI	66.7	25.3	48.2	18.5
No HAGNN	63.2	19.4	51.2	10.3
Full	89.3	34.2	78.3	23.8

, showed that all HAGNN blocks were useful and brought substantial performance gains. “No EAGAT” replaced all EAGAT layers with CHGI layers, and the matching accuracy decreased by 9.7%. “No CHGI” replaced all CHGI layers with EAGAT layer, and the precision of the resulting matching decreased by 22.6%, “No HAGNN” replaced the graph neural network with a single linear projection, and the precision of the resulting matching decreased by 26.1%.

The proposed HAGNN was compared with two feature matching methods: the nearest neighbor (NN) method and SuperGlue. As show in

Table 5. Experimental results of the pose estimation. Matching PL-Net features with HAGNN resulted in a significantly higher pose accuracy (AUC), precision (P), and matching score (MS) than with handcrafted or other learned methods.

Feature	Matcher	Pose Estimation AUC			P	MS
		$@{\mathbf{5}}^{\circ }$	$@{\mathbf{10}}^{\circ }$	$@{\mathbf{20}}^{\circ }$
SIFT	NN	7.89	10.22	35.30	43.4	1.7
SIFT	SuperGlue	23.68	36.44	49.44	74.1	7.2
SuperPoint	NN	9.80	18.99	30.88	22.5	4.9
SuperPoint	SuperGlue	34.18	44.32	64.16	84.9	11.1
LSD + LBD	NN	5.43	7.83	28.54	32.5	1.3
SOLD${}^2$	NN	18.34	13.22	23.51	63.6	6.2
SuperPoint + SOLD${}^2$	Ours	35.86	44.73	64.43	85.3	12.3
Ours	Ours	36.67	44.26	64.73	86.6	12.7

, it can be seen clearly that HAGNN had a significantly higher pose estimation accuracy than all competitors, which showed a higher feature expression ability.

5. Conclusions

In this paper, we proposed a point–line-aware heterogeneous graph attention network for a visual SLAM system. Combining the point- and line-aware attention modules based on an attention-driven mechanism, the geometric association features of key regions was further extracted, and the model was simplified by a transfer-aware knowledge distillation strategy. By improving the accuracy of image point–line matching, a point–line heterogeneous graph attention network was proposed, which realized the feature aggregation by conducting learning on the intragraph and intergraph. Based on the optimal transport theory, we proposed a greedy inexact proximal point method that could effectively solve the point–line matching problem. Experiments on a public dataset and a self-made dataset showed qualitatively and quantitatively that our model had stronger robustness and a better generalization ability. One limitation of our feature matching was that it was not easy to estimate the pose error due to the interference of dynamic objects. Thus, in a future study, we will introduce the cross-frame semantic information in the network for dynamic environments.