UAVs-Based Visual Localization via Attention-Driven Image Registration Across Varying Texture Levels

Abstract

This study investigates the difficulties associated with image registration due to variations in perspective, lighting, and ground object details between images captured by drones and satellite imagery. This study proposes an image registration and drone visual localization algorithm based on an attention mechanism. Initially, an improved Oriented FAST and Rotated BRIEF (ORB) algorithm incorporating a quadtree-based feature point homogenization method is designed to extract image feature points, providing support for the initial motion estimation of UAVs. Following this, we combined a convolutional neural network with an attention mechanism and the inverse-combined Lucas-Kanade method to further extract image features. This integration facilitates the efficient registration of drone images with satellite tiles. Finally, we utilized the registration results to correct the initial motion of the drone and accurately determine its location. Our experimental findings indicate that the proposed algorithm achieves an average absolute positioning error of less than 40 m for low-texture flight paths and under 10 m for high-texture paths. This significantly mitigates the positioning challenges that arise from inconsistencies between drone images and satellite maps. Moreover, our method demonstrates a notable improvement in computational speed compared to existing algorithms.

1. Introduction

In recent years, Unmanned Aerial Vehicle (UAV) technology has rapidly advanced, resulting in its widespread use across several key sectors, such as meteorological observation, agricultural monitoring, geological surveying, and military operations [1,2,3,4]. These applications demand high-precision location estimates to ensure the autonomous flight and effective task execution of UAVs, making efficient positioning systems essential. UAV navigation systems currently employ various methods, including Global Navigation Satellite Systems (GNSS), Inertial Navigation Systems (INS), and radio navigation systems. GNSS is particularly prevalent due to its extensive coverage and established reliability [5]. However, its positioning accuracy can be adversely affected by external factors and may struggle to provide stability in environments with weak or absent signals [6]. Therefore, it is crucial to explore UAV positioning technologies that can adapt to a range of environmental conditions.

The rapid advancement of computer vision technology and visual sensors has underscored the benefits of vision-based positioning and navigation techniques in addressing various challenges [7]. These methods provide effective solutions for the safe and efficient operation of Unmanned Aerial Vehicles (UAVs), especially in environments where Global Navigation Satellite Systems (GNSS) signals are unreliable, making their potential applications highly promising. As a result, vision-based positioning has become a focal point of research. Vision-based positioning can be divided into two main approaches: absolute visual positioning and relative visual positioning [8]. Absolute visual positioning involves comparing UAV aerial images captured with prior satellite images to identify the most similar regions through a search process, enabling the UAV to ascertain its location on the satellite map [9]. While this approach offers high accuracy, it depends on prior satellite maps and is sensitive to environmental variations. Conversely, relative visual positioning estimates the motion of the drone by analyzing consecutive frames captured by the UAV, without relying on prior maps. However, this method is susceptible to cumulative errors during prolonged flights. Researchers are actively investigating improved visual positioning techniques to enhance UAV autonomous navigation in complex environments.

Oriented FAST and Rotated BRIEF (ORB) is an efficient algorithm for detecting and describing image feature points, widely used in applications of computer vision such as object detection and motion tracking [10]. The algorithm is based on Features from Accelerated Segment Test (FAST) [11] for rapid feature point detection and integrates Binary Robust Independent Elementary Features (BRIEF) [12] to construct binary descriptors. ORB introduces two key improvements over the original methods: first, it calculates the grayscale centroid of the neighborhood around each feature point, enabling orientation invariance and robustness to image rotation. Second, it employs a pyramid structure to achieve scale invariance, enabling robust feature detection across varying image scales. Furthermore, ORB refines the binary pattern generation process of BRIEF descriptors, improving feature matching accuracy while maintaining high computational efficiency. Compared to traditional feature extraction algorithms such as SIFT and SURF, ORB provides superior robustness at significantly lower computational costs. This makes it particularly suitable for real-time applications with stringent performance requirements, such as UAV navigation, image stitching, and target tracking.

Subsequently, numerous excellent image feature-based methods have been used for visual localization. For example, Patel et al. [13] utilized the SURF algorithm to align UAV aerial images with satellite remote sensing maps from Google Maps, determining the position of UAV based on the registration results. Majidizadeh et al. [14] employed a U-Net network to segment roads and buildings in aerial imagery; however, this algorithm was limited to urban settings due to its reliance on road segmentation. Zhong et al. [15] proposed a monocular visual odometry based on information entropy and Lucas-Kanade optical flow method for SLAM domain. Literature [16,17] proposed a visual SLAM algorithm for point and line feature fusion, which achieves simultaneous localization and maps in indoor low-texture scenes, but is not applicable to aviation scenes. Goforth et al. [18] implemented a neural network-based image registration technique to correct UAV motion, but this approach faced challenges with slow computation speeds and suboptimal performance in low-texture areas. Such limitations are also prevalent in visual odometry methods [19], which tend to accumulate errors over long distances and require adequate overlap between consecutive frames. Therefore, enhancing existing methods to mitigate cumulative errors is of paramount importance.

Building upon previous research, this study proposes a novel positioning method that combines relative visual positioning with absolute visual positioning. Initially, visual odometry techniques are employed to match consecutive frames of UAV aerial images, allowing for a preliminary estimation of the motion trajectory of UAV. Subsequently, the UAV aerial images are matched with satellite images of the target area to effectively eliminate cumulative errors. Experimental validation demonstrates that this method performs exceptionally well under low-texture image conditions, providing significant academic and practical value for UAV positioning research in low-texture environments such as wilderness and mountainous regions. The main contributions of this study are as follows:

• This study introduces a quadtree feature point uniformization algorithm applied to Oriented FAST and Rotated BRIEF (ORB) feature points. The aim is to reduce the negative impact of densely concentrated feature points in areas with significant texture variation on motion estimation. This enhancement improves the accuracy of motion estimation for UAVs during high-speed flight in low-texture environments;
• This study present a twin neural network registration algorithm that leverages an attention mechanism by integrating the Convolutional Block Attention Module (CBAM) into the feature extraction layer of neural network. The Lucas-Kanade algorithm is then employed to align the feature maps processed by the convolutional network. This enhanced algorithm shows a marked improvement in matching performance for low-texture images;
• This study propose a novel vision positioning method that combines relative and absolute visual positioning through image registration. This method effectively mitigates the cumulative error associated with relative visual positioning during long-distance flights, enabling rapid and accurate positioning in low-texture environments, such as agricultural fields and river landscapes.

The subsequent sections of this paper are organized as follows: Section 2 reviews related research, focusing on image feature extraction and matching methods, as well as their applications in visual positioning. In Section 3, we provide a detailed explanation of the proposed vision positioning method based on image registration, covering the structural details of the image registration network, the estimation of the initial motion of UAV, motion correction, and specific details of the positioning process. Section 4 presents the experimental design and result analysis. Finally, Section 5 concludes the study, summarizing the research process and experimental findings, while also outlining future research directions.

2. Related Works

Image matching is a fundamental element of UAV visual positioning and serves as a critical step in the image registration process. This process involves extracting and matching features, as well as estimating the transformation matrix between two images to align them within the same coordinate system. Numerous researchers have proposed various classic image matching algorithms, including those based on templates, grayscale, and features. Among these methods, features such as points, lines, and textures are crucial for efficiently representing the entire image due to their simplicity and stability, forming the foundation of mainstream image matching algorithms [20]. Harris et al. introduced the Harris corner detection operator, which defines corners based on the rate of intensity change in two orthogonal directions [21]. This operator employs second-order moments or autocorrelation matrices to expedite the search for local extrema while maintaining directional invariance. To further reduce computational complexity, faster feature extraction methods based on pixel grayscale comparisons in local regions were developed, including the SUSAN operator [22] and FAST [11]. Rublee et al. later introduced the Oriented FAST and Rotated BRIEF (ORB) algorithm, which has been widely adopted for visual tasks [10]. However, these methods are often sensitive to scale and affine transformations. To address these challenges, David Lowe proposed the classic scale-invariant feature transform (SIFT) algorithm [23], followed by enhanced algorithms like SURF [24] and ASIFT [25]. These algorithms demonstrate strong performance in terms of translation and rotation invariance and are robust against variations in illumination, noise, and slight changes in viewing angles. Nonetheless, traditional image matching algorithms typically rely on manually designed feature extraction and matching strategies, which, although effective in simple scenes, may struggle in complex backgrounds and highly variable images.

With the rapid development of deep learning, it has been widely applied in various fields such as natural language processing and computer vision [26,27,28]. In the field of image processing, deep learning-based methods have gradually overcome the limitations of traditional feature matching techniques, owing to their exceptional ability in deep feature learning and representation. These methods excel in learning and representing complex features, resulting in significant performance improvements in image processing. Consequently, they have increasingly supplanted traditional handcrafted feature approaches, positioning themselves at the forefront of contemporary image matching technology. Modern descriptors generally train deep networks on pre-cropped patches derived from SIFT keypoints. Notable examples of these descriptors include Deepdesc [29], L2-Net [30], and LogPolarDesc [31]. In recent research, Verdie et al. [32] developed a reliable keypoint detection framework called TILDE. This framework successfully addresses the challenges posed by significant illumination variations due to changes in weather, seasons, and time, facilitating the reliable detection of repeatable keypoints. Barroso-Laguna et al. [33] developed KeyNet, a keypoint detector that utilizes a multi-scale shallow structure. This method efficiently extracts keypoints and exhibits strong adaptability to changes in image perspective. Similarly, Daniel et al. [34] introduced SuperPoint, a self-supervised learning framework based on fully convolutional neural networks. This framework features both a localization decoder and a description decoder, enabling the simultaneous extraction of feature point descriptors. Building on this foundation, SuperGlue [35] proposed a multi-layer graph neural network that integrates an attention mechanism. This innovative approach consolidates contextual information from images, allowing the matching process to consider both local features of keypoints and global information across the entire graph. By transforming the matching challenge into an optimal transport problem, SuperGlue significantly enhances matching performance. Another promising approach is the use of end-to-end learning networks, such as LIFT [36] and LoFTR [37], which also demonstrate exceptional performance in feature point matching tasks. LoFTR leverages self-attention and cross-attention layers from the Transformer architecture to extract feature descriptors from two images. Initially, it establishes pixel-level dense matching at a coarse level, followed by a refinement process at a finer level, effectively achieving reliable matching results even in low-texture regions.

D2-Net [38] tackles the challenge of pixel-level matching in complex imaging conditions through dense feature extraction and description, resulting in sparse feature points and their corresponding descriptors. Hou et al. [39] applied D2-Net to map matching for UAV visual positioning, demonstrating that the average positioning error under seasonal variations meets the localization requirements for UAVs. However, this method lacks robustness during long-distance flights and does not consider the effects of low-texture environments. In urban landing scenarios, Xu et al. [40] introduced the YOLOX network with an attention mechanism, achieving notable performance. Meanwhile, Marius-Mihail Gurgu [41] employed SuperGlue for localization in non-urban environments, such as fields and forests; however, this approach relies on retrieving the rotational angle of the onboard camera from image metadata. The 2chADCNN model [42] integrates a dual-channel fully convolutional neural network with an attention mechanism to focus on season-invariant features, yielding favorable results in scenarios with seasonal changes. Nevertheless, this method still depends on prior satellite maps for practical localization, which limits its applicability. An innovative approach proposed in [43] combines MEMS-based inertial sensors with visual odometry by enhancing the visual odometry pipeline. This method replaces the feature matching step with the Lucas-Kanade algorithm and employs an Extended Kalman Filter (EKF) to fuse data from both systems for accurate estimation of position and heading, while correcting for errors in the inertial sensors. Building on these prior works, we aim to leverage the benefits of visual odometry while addressing its cumulative error challenges.

3. Proposed Methodology

Image registration methods based on visual odometry focus primarily on aligning two homogeneous images. These methods generally perform better than absolute visual positioning across various environments, including urban, indoor, and outdoor settings. However, a significant drawback of visual odometry is its cumulative error over long-distance flights, which does not affect absolute visual positioning. Building on insights from previous research, this paper introduces a novel algorithm for image registration and UAV visual positioning that leverages an attention mechanism. This innovative approach aims to effectively integrate visual odometry with absolute visual positioning. By doing so, it can significantly reduce cumulative errors during long-distance flights, thereby improving localization accuracy.

The following sections will provide a detailed description of the method we propose. We begin by outlining the overall workflow of our approach, which consists of three key stages. Section 3.1 introduces the first stage, focusing on the improved Oriented FAST and Rotated BRIEF (ORB) algorithm and its application in estimating the initial motion of the drone. Section 3.2 thoroughly discusses the second stage, highlighting the principles and implementation details of the attention-based twin neural network we designed. Section 3.3 corresponds to the third stage, where we explain the process of motion correction for the drone and the detailed implementation of the final positioning.

Figure 1 illustrates the flowchart of the proposed localization algorithm, which consists of three key stages. First, in the initial stage, we employ an improved visual odometry algorithm to estimate the initial motion matrix of the drone by matching feature points between consecutive image frames captured by the UAV. This innovative approach significantly enhances both the accuracy and speed of motion estimation. Specifically, visual odometry algorithms are prone to cumulative errors. To address this, we improve the Oriented FAST and Rotated BRIEF (ORB) algorithm and extract feature points from consecutive frames of drone images. The FLANN algorithm is then used to match the feature points and estimate the initial motion. This stage is discussed in detail in Section 3.1, with the goal of quickly estimating the UAV’s initial motion, while the cumulative errors remain unaddressed at this point. Then, in the second stage, the initial motion of the UAV estimated in the first stage allows us to obtain the current UAV aerial image as well as its corresponding satellite image. We design an attention-based twin neural network to extract image features from both the UAV’s aerial image and the satellite tile image. This network will be described in detail in Section 3.2. The extracted image features from the drone’s aerial images and satellite tile images are then fed into the third stage. Finally, in the third stage, the image features of the UAV aerial image and the satellite tile image obtained in the second stage are used with the inverse composition Lucas-Kanade algorithm to achieve precise registration between the satellite image and the UAV aerial image, generating a correction matrix to eliminate the cumulative errors present in the visual odometry process. By combining the initial motion matrix of the drone with the correction matrix, the final motion of the UAV for each frame can be determined. Through the continuous estimation of motion, the UAV can accurately localize itself within the satellite map, achieving precise positioning. This stage is described in detail in Section 3.3.

To further clarify, the core idea of our method is to combine relative visual localization and absolute visual localization for UAVs, overcoming two major limitations of existing techniques: (1) eliminating the impact of cumulative errors in visual odometry, and (2) significantly reducing the time required for searching the satellite image database in absolute visual localization methods. The first stage is essentially part of the UAV relative visual localization. However, during long-distance flights, cumulative errors cannot be eliminated. To address this, we have designed an improved Oriented FAST and Rotated BRIEF (ORB) feature extraction algorithm to rapidly estimate the UAV’s initial motion by extracting image features. The second and third stages are essentially part of the UAV absolute visual localization. Compared to traditional absolute visual localization methods, since the initial motion of the drone has already been estimated in the first stage, there is no need to search through the entire satellite image database. We designed an attention-based twin neural network to extract and register image features from both the UAV aerial images and the corresponding satellite tile images, thereby correcting the UAV’s motion and eliminating cumulative errors. This approach effectively improves both the accuracy and speed of the localization process.

3.1. Estimation of Initial Motion for UAVs

Affine transformations maintain the linearity of images, which helps reduce distortions in the representation of roads and building boundaries [44]. As a result, these transformations are commonly employed in the kinematic and dynamic modeling of drones [45]. Affine transformations offer six degrees of freedom, enabling various geometric adjustments such as rotation, translation, and scaling. To estimate the continuous motion of the drone, it is crucial to establish the initial motion matrix $A_{rel}$ based on the relationships between affine transformations of consecutive images captured by the drone. Next, the aerial images from the current frame are aligned with satellite tile images using a neural network alongside the Lucas-Kanade algorithm, resulting in a correction matrix $A_{corr}$. This correction is then applied to the initial motion of the drone, leading to a more precise estimation of its movement. The affine matrix $A_{rel}$ between two frames of neighboring images can be defined as follows:

$$A_{rel}=\left(\begin{array}{ccc}\alpha & -\beta & t_x\\ \beta & \alpha & t_y\\ 0& 0& 1\end{array}\right)$$

(1)

where $\alpha$ and $\beta$ represent the parameters for the rotation and height changes of the drone, $\alpha =s·cos\theta$, $\beta =s·sin\theta$, s indicates the scaling factor between the UAV aerial images and the reference map, and $\theta$ denotes the yaw angle between two consecutive frames. The parameters $t_x$ and $t_y$ correspond to the translation values of the drone in the x and y directions, respectively. The scaling factor s is determined by the ratio of the ground sample distance (GSD) of the UAV aerial image to that of the satellite image. Taking the experimental dataset from the Shenyang region in China as an example, the $GS{D}_s$ of the Google satellite map at zoom level 17 is 1.76 m per pixel. The $GS{D}_u$ of the UAV aerial image, can be calculated from the metadata of the aerial images and is approximately 0.6 m per pixel. Therefore, the scaling factor is approximately 0.34, meaning the drone’s aerial image is scaled to 34% of its original size.

To accurately track the movement of the drone on the satellite map, it is essential to first establish a correlation between the initial frame captured at takeoff and the satellite map. At the moment the initial frame $f=1$ is captured, the position of the drone and heading are assumed to be known. With this information, the homography matrix $A_{rel}^f$ can be computed, accounting for the approximate location, orientation, and scale of the initial frame on the satellite map. As the drone navigates, the relationship between the current frame f and the previous frame $f-1$ is described by the homography matrix, which captures the changes in position. The position $L^f$ of the drone in the current image can be expressed as:

$$L^f=A_{rel}^f·A_{rel}^{f-1}·A_{rel}^{f-2}·...·L^1$$

(2)

where $A_{rel}^f$ denotes the affine transformation that occurs between frames f and $f-1$. $L^1$ represents the position of the first frame of the UAV aerial image in the satellite image. This demonstrates an iterative process, where the position of the first frame image, $L^1$, determines the position of the second frame image, $L^2$, and so on, until the position of the current frame, $L^f$, is reached. This transformation accounts for the rotation and translation of the drone as it moves between these two adjacent frames.

Based on the aforementioned principles, this study designs an improved Oriented FAST and Rotated BRIEF (ORB) algorithm to detect feature points, thereby estimating the motion of drone. Initially, feature points are detected in both the current frame f and the previous frame $f-1$. Next, the FLANN [46] algorithm is used to match the detected points, while the RANSAC [47] algorithm is applied to eliminate any mismatched pairs. For the matched feature point pairs, the pixel coordinates for frame $f-1$ are denoted as $(x,y)$, while those in frame f are represented as $(x^{\prime },y^{\prime })$. The relationship between the affine matrix $A_{rel}$ of the adjacent frames and the feature point coordinates can be expressed as:

$$\left(\begin{array}{c}\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\\ 1\end{array}\right)=A_{rel}\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)=\left(\begin{array}{ccc}\alpha & -\beta & t_x\\ \beta & \alpha & t_y\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)$$

(3)

Morphing Equation (3) yields:

$$\begin{array}{c}{x}^{\prime }=\alpha x-\beta y+t_x\hfill \\ y^{\prime }=\beta x+\alpha y+t_y\hfill \end{array}$$

(4)

To accurately determine the initial motion $A_{rel}$, at least three pairs of feature points are needed. Thus, acquiring a sufficient number of reliable matching point pairs between adjacent frames is essential. However, feature points tend to cluster around corners and areas with significant texture variations in the images. Consequently, many feature points may lie outside the overlapping regions of consecutive images, while only a limited number are detected within the overlapping areas. This uneven distribution can lead to incorrect calculations of the initial motion parameters. Figure 2 illustrates the distribution of feature points in the UAV aerial images.

To resolve the problem of concentrated feature point distribution illustrated in Figure 2, it is essential that the overlapping regions of consecutive images are sufficiently large to ensure accurate motion estimation of the drone. However, when the drone flies at high speeds, the overlap between consecutive images is often significantly reduced. This study proposes a quadtree-based feature point uniformization algorithm to ensure a more balanced distribution of feature points across the entire image. This approach enables accurate motion estimation even during high-speed drone flight.

The quadtree-based feature point uniformization algorithm consists of four stages, as illustrated in Figure 3. The process begins by setting an initial cell size according to the aspect ratio of the UAV aerial images, followed by the initial cell partitioning. Next, the Oriented FAST and Rotated BRIEF (ORB) feature point detection algorithm is applied to extract feature points with each cell. In the second stage, each cell is further divided into smaller nodes. Initially, it is assumed that each cell contains only one node. If a node has more than one feature point, it is split into four sub-nodes. Conversely, nodes without any feature points are removed. In the third stage, nodes are recursively split until either the total node count reaches a predetermined threshold or further division is no longer feasible. In the fourth stage, the feature point with the highest Harris corner response value is selected from each node. Ultimately, other feature points are removed to achieve a uniform distribution of feature points.

3.2. Siamese Neural Network Based on Attention Mechanism

This study introduces a novel attention-enhanced Siamese neural network designed specifically for registration UAV aerial images with satellite tiles. By incorporating attention mechanisms, the model significantly boosts the feature extraction capability of the Siamese network. As depicted in Figure 4, the network consists of two identical branches, each composed of three convolutional blocks followed by a Convolutional Block Attention Module (CBAM). Inspired by the initial three blocks of the VGG16 architecture, the first two convolutional blocks in each branch consist of two ReLU-activated convolutional layers and a max-pooling layer. The third block includes two convolutional layers with ReLU activation, while the final layer in this block is unactivated. The Convolutional Block Attention Module (CBAM) module consists of two submodules, namely the channel attention module and the spatial attention module, connected in series [48]. Within each branch, the Convolutional Block Attention Module (CBAM) module refines feature representation by employing two attention sub-modules: channel attention and spatial attention. The channel attention mechanism reduces the spatial dimension while keeping the channel dimension intact, which enhances the network’s capacity ability to capture and emphasize important channel features. spatial attention compresses the channel dimension while preserving the spatial dimension, which allows the model to focus on key positional information of objects within the image.

This attention mechanism enables the network to automatically identify and emphasize critical feature regions within UAV aerial images and satellite images, such as buildings in agricultural fields, exposed riverbeds, and large structures within urban areas. This ensures that these significant features retain their prominence even after the images undergo transformations such as translation and rotation. The integration of the Convolutional Block Attention Module (CBAM) not only enhances the ability of the model to capture subtle texture variations in the images but also significantly improves the overall performance of the model, providing robust support for the accurate registration of UAV aerial images and satellite tile images.

Next, a detailed description of the attention mechanism-based Siamese neural network is provided. First, UAV aerial images and satellite tile images are used as the two inputs to the Siamese neural network, which then generates feature maps $F\in R^{C\times H\times W}$ after passing through three convolutional blocks. Subsequently, the feature map F undergoes max-pooling and average-pooling operations in the channel attention module. The resulting one-dimensional vector is passed through a fully connected layer (MLP) for computation, and the sum of the results generates a one-dimensional channel attention. This one-dimensional channel attention is then multiplied by the input elements to obtain the adjusted feature map $F^{\prime }$ after the channel attention module. The calculation process of the channel attention module is as follows:

$$\begin{array}{cc}\hfill M_C\left(F\right)=& \sigma \left(MLP\left(AvgPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)\hfill \\ \hfill =& \sigma \left(W_1\left(W_0\left(F_{avg}^c\right)\right)+W_1\left(W_0\left(F_{max}^c\right)\right)\right)\hfill \end{array}$$

(5)

where $\sigma$ denotes the sigmoid function, $W_0$ and $W_1$ are the two weights of the MLP, $F_{avg}^c$ and $F_{max}^c$ are the features that have undergone average-pooling and max-pooling layers, respectively, within the channel attention module.

Then, the feature map $F^{\prime }$ undergoes max-pooling and average-pooling operations in the spatial attention module. The two resulting two-dimensional vectors from average pooling are concatenated and processed through a convolution operation to generate the two-dimensional spatial attention feature map $M_S\in R^{1\times H\times W}$. The obtained $M_S$ is then element-wise multiplied with the feature map $F^{\prime }$ to produce the final feature map $F^{″}$. The calculation process of the spatial attention module is as follows:

$$\begin{array}{c}\begin{array}{cc}\hfill M_S\left(F\right)=& \sigma \left(f^{7\times 7}\left(\left[\left(AvgPool\left(F\right)\right);MaxPool\left(F\right)\right]\right)\right)\hfill \\ \hfill =& \sigma \left(f^{7\times 7}\left(\left[F_{avg}^s;F_{max}^s\right]\right)\right)\hfill \end{array}\end{array}$$

(6)

where $f^{7\times 7}$ represents a convolution operation with the filter size of 7 × 7, $F_{avg}^s$ and $F_{max}^s$ are the features that have undergone average-pooling and max-pooling layers, respectively, within the spatial attention module.

The process of Convolutional Block Attention Module (CBAM) generating $F^{″}$ can be expressed as:

$$\begin{array}{c}{F}^{\prime }=M_C\left(F\right)\otimes F\hfill \\ F^{″}=M_S\left(F^{\prime }\right)\otimes F^{\prime }\hfill \end{array}$$

(7)

where ⊗ represents element-wise multiplication.

Finally, the UAV aerial images and satellite tile images are processed through the Siamese neural network to obtain the UAV aerial image feature map and the satellite tile image feature map, denoted as $F_U$ and $F_M$, respectively.

3.3. Image Registration Based on the Lucas-Kanade Algorithm

The role of the Lucas-Kanade inverse compositional algorithm in image registration is to estimate the displacement information between images using optical flow, while leveraging the inverse compositional method to improve computational efficiency. The core idea is based on the assumption that pixel intensity values between images remain consistent over short time intervals, and the motion is estimated using local gradient information, thereby achieving image alignment and registration.

In this study, we denote the feature maps of drone aerial images and satellite images as $F_U$ and $F_M$, respectively. To align these two original images, we employ the inverse compositional Lucas-Kanade algorithm. The motion model linking $F_U$ and $F_M$ is expressed as $W(\mathbf{x};\mathbf{p})$, where $\mathbf{x}=(x_i,y_i)$ represents the pixel coordinates in the original image. The total number of pixels in the image is N, and $\mathbf{p}={\left(p_1,p_2,\dots ,p_6\right)}^T$ signifies the parameter vector associated with the motion model. Equation (1) presents the six degrees of freedom affine matrix, which is crucial for this alignment process. In this section, we will detail the relationship between $\mathbf{p}$ and $A_{rel}$:

$$W(\mathbf{x};\mathbf{p})=\left(\begin{array}{ccc}1+p_1& p_2& p_3\\ p_4& 1+p_5& p_6\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)=\left(\begin{array}{ccc}\alpha & -\beta & t_x\\ \beta & \alpha & t_y\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)$$

(8)

Next, we apply a method based on photometric consistency loss to align the images. This process involves calculating the sum of squared differences in grayscale values between corresponding pixels in the two images. Our goal is to determine the optimal parameters $\mathbf{p}$ that minimize the grayscale differences between the drone aerial image and the satellite tile image, which has been altered by the motion model. This relationship can be expressed as follows:

$$\underset{\mathbf{p}}{min}{∥F_U\left(\mathbf{x}\right)-F_M\left(W\left(\mathbf{x};\mathbf{p}\right)\right)∥}_2^2$$

(9)

Finally, we correct $A_{rel}$ using the affine matrix $A_{corr}$ to obtain the final motion of the drone. The calculation formula is as follows:

$$A_{corr}^f·A_{rel}^f\to A_{rel}^f$$

(10)

where $A_{corr}^f$ represents the correction matrix between the drone image in the f-th frame and the corresponding satellite tile image.

3.4. Train

The training image dataset is sourced from a publicly available collection hosted on the United States Geological Survey’s Earth Explorer platform. It covers a region of 5.9 × 7.5 km in New Jersey, USA, and encompasses a variety of landscapes, including urban, forested, suburban, and rural areas. The dataset comprises images captured during spring, summer, and autumn, each with dimensions of 7582 × 5946 pixels and a spatial resolution of 1 m. Prior to neural network training, the raw images undergo a warping procedure, and the resulting processed images serve as the input data for the network. We begin by randomly selecting two images from the raw dataset. From these images, we crop a pair of patches at corresponding locations, with pixel sizes varying between 175 and 225. One patch remains unchanged, while the other undergoes warping using randomly generated affine parameters, which include scaling, translation, and rotation. These two patches—one processed and the other unchanged—constitute a training pair that is then fed into a twin neural network based on an attention mechanism. During the training phase, we generated a total of 3000 pairs of image patches as training data. The loss function employed for training is the angular loss, calculated as follows:

$$Loss\left(\mathbf{p},\widehat{\mathbf{p}}\right)={\displaystyle \frac{1}{4}}\sum _{i=1}^4{∥W\left({\mathbf{c}}_i;\mathbf{p}\right)-W\left({\mathbf{c}}_i;\widehat{\mathbf{p}}\right)∥}_2$$

(11)

where $\widehat{\mathbf{p}}$ refers to the true values of the warping parameters utilized to distort the images, while ${\mathbf{c}}_1,\dots ,{\mathbf{c}}_4$ indicate the positions of the four corners. The network weights are updated by backpropagating the angular loss, using the Adam optimizer with a learning rate set at 1 × 10⁻³.

4. Experiments

The experiments were carried out using Python 3.7, CUDA 10.2, and the deep learning framework PyTorch 1.7.1, with an NVIDIA GeForce 940MX graphics card as the hardware platform (NVIDIA Corporate, Santa Clara, CA, USA). To assess the effectiveness of the proposed method for accurately positioning drone-captured images relative to satellite maps across different texture conditions, this section presents experiments involving two flight paths in both high-texture and low-texture regions. The experimental dataset was gathered from the Changbai area in Heping District, Shenyang, Liaoning Province, China. The low-texture route covers a distance of 3.06 km and includes farmland, rivers, and a few buildings. In contrast, the high-texture route features a substantial number of buildings and roads, with the first high-texture route measuring 1.84 km and the second measuring 2.75 km. The satellite remote sensing maps utilized in this study are from Google Maps at level 17. The drone-captured images have a resolution of 750 × 750 pixels, with a 30% overlap between adjacent frames. It is crucial to note that the drone images and satellite images for all routes were captured at different times and using different sensors. As a result, visual discrepancies exist between the drone-captured images and the satellite images, including variations caused by vegetation, roads, buildings, and shadows. Figure 5 provides a comparative view of the drone-captured images and the satellite images.

This section outlines comparative experiments with various algorithms to assess the effectiveness of the proposed method. The comparison will involve calculating initial motion using different point feature detection algorithms, including SURF, SIFT, and an improved Oriented FAST and Rotated BRIEF (ORB) algorithm. Furthermore, we will employ distinct neural network architectures for image feature extraction. Specifically, we will extract features using both the original VGG16 model and a modified version that integrates the Convolutional Block Attention Module (CBAM) after the third convolutional block. The registration results from these methods will be compared to perform an ablation study of the Convolutional Block Attention Module (CBAM).

4.1. Feature Point Extraction and Matching

To determine the position of the drone, we first estimate the matrix $A_{rel}^1$, which links the images of the drone to the satellite maps based on known takeoff information. Subsequently, we conduct feature point detection and matching on consecutive frames captured by the drone, employing a 30% overlap between these frames. In cases where the overlap is minimal, the quality and uniform distribution of feature points become crucial for accurately estimating the initial motion of the drone. This study enhances the Oriented FAST and Rotated BRIEF (ORB) algorithm by implementing a quadtree feature point uniformization technique, which ensures an even distribution of extracted feature points throughout the image. Figure 6 shows the distribution of feature points in the images captured by the drone.

The experimental results reveal that using the Oriented FAST and Rotated BRIEF (ORB) algorithm alone results in most feature points being clustered on edges with prominent textures, such as those found in buildings, grass, and roads, while feature points are scarce in low-texture areas. In contrast, the enhanced ORB algorithm achieves a more uniform distribution of feature points, thereby mitigating the clustering effect. Following the feature point detection, we apply the FLANN algorithm for feature point matching. The matching results for adjacent frames with low-texture and high-texture images are presented in Figure 7 and Figure 8, respectively.

In this study, we employ the RANSAC algorithm for feature point pair selection. RANSAC requires a minimum of four correct matching pairs to obtain accurate initial motion parameters. The experimental results demonstrate that the number of successful matches between low-texture images using the Oriented FAST and Rotated BRIEF (ORB) algorithm alone is relatively low, with some scenes yielding fewer than four pairs. In contrast, the combination of the ORB algorithm and the quadtree uniformization technique significantly increases the number of matching pairs for low-texture images. High-texture images, characterized by their clear and distinct features, facilitate the identification of a larger number of correct matches. However, images processed with the quadtree uniformization algorithm may still present some erroneous matches. This issue arises because, while the algorithm enhances the even distribution of feature points, it can also reduce their quality, resulting in incorrect matches. Despite this, the high volume of successfully matched feature point pairs enables accurate calculations of the initial motion of the drone, even after filtering out the erroneous matches.

4.2. Image Registration and Localization for Low-Texture Areas

This section utilizes neural networks to extract features from both drone-captured images and satellite tiles. The inverse compositional Lucas-Kanade algorithm is then applied for image registration, allowing for the correction of the motion of the drone and precise determination of its location on the satellite map. Figure 9 illustrates the registration results for specific low-texture route images. The upper part of the figure displays the satellite tile image, while the lower part presents the drone image. The central section shows the registration results of the satellite tile using the drone image as a template, with a highlighted rectangular box to facilitate a clearer comparison of the outcomes. The proposed network structure builds upon the VGG16 model by incorporating the Convolutional Block Attention Module (CBAM) attention module, which enhances the preservation of essential features in critical areas, such as buildings and roads. Experimental results indicate that, despite variations between the drone-captured images and the satellite tiles, the registration results achieved with the proposed method significantly surpass those obtained using the standard VGG16 network structure.

Figure 10 illustrates the positioning results for low-texture routes, with the flight direction in the satellite image moving from the lower left corner to the upper right corner. The results indicate that from the ninth frame captured by the drone, the distance between the predicted coordinates obtained using the VGG16-based positioning method and the actual location progressively increases. In contrast, the predicted drone position coordinates derived from the proposed method remain significantly closer to the true location.

This study utilizes absolute positioning error as an objective metric to evaluate the positioning results. It specifically measures the two-dimensional Euclidean distance between the predicted coordinates and the actual GPS coordinates for each frame. Within the same network architecture based on the VGG16 structure, we compare the positioning errors associated with the SURF, SIFT, and improved Oriented FAST and Rotated BRIEF (ORB) algorithms. Furthermore, using the improved ORB algorithm as the feature point detection method, this study also examines the positioning errors obtained from both the VGG16 network structure and the proposed method. Figure 11 displays the positioning error line chart for the methods discussed. The data shows that, under the same network architecture, the positioning errors of the improved ORB (ORB_Qt) algorithm are notably lower than those of the SURF and SIFT algorithms. In low-texture areas, the errors for each frame using the VGG16 network structure tend to increase gradually. In contrast, the errors for each frame produced by the proposed method remain consistently small and stable. These results indicate that the positioning outcomes from the proposed network structure are clearly superior to those obtained with the VGG16 network structure.

4.3. Image Registration and Localization for High-Texture Areas

Figure 12 illustrates the registration results for images from high-texture routes. The top section features the satellite tile image, while the bottom section displays the UAV aerial images. The middle section presents the registration outcomes of the satellite tile image, using the UAV aerial image as a reference. The experimental results indicate that both the VGG16-based method and the proposed approach effectively register drone images with satellite tile images. This performance can be attributed to the presence of numerous richly textured buildings and roads within the images.

Figure 13 illustrates the positioning results for low-texture routes, where the flight direction is from left to right on the satellite map. The findings show that the predicted drone coordinates from both methods closely align with the actual positions, demonstrating a high level of positioning accuracy.

The comparison experiments for high-texture and low-texture routes utilized the same methodology. Figure 14 presents the positioning errors for high-texture routes. It is clear from the figure that the errors of the three feature point detection algorithms are largely consistent across the same network architecture. When applying the same feature point detection algorithm, the proposed network structure produces positioning errors similar to those of the VGG16-based method. However, the results from the proposed method exhibit greater stability.

Table 1. Comparative experimental results of the proposed method against several other benchmark methods. The proposed method demonstrates superior speed and accuracy in localization in both high-texture and low-texture areas compared to the other methods.

Experimental Route	SIFT [23]		SURF [24]		Method [18]		Ours
	Time (s)	Error (m)	Time (s)	Error (m)	Time (s)	Error (m)	Time (s)	Error (m)
Low-texture route I	198.429	281.332	346.932	543.122	53.288	33.06	33.176	38.670
Low-texture route II	140.862	6.919	260.785	6.925	47.517	17.82	26.538	6.502
High-texture route I	92.120	4.515	174.056	4.481	20.159	5.061	17.276	3.914
High-texture route II	118.591	7.539	168.616	7.599	28.367	6.324	18.186	7.512

gives the time required to compute the initial motion parameters for the four different methods on the four routes as well as the average localization error. The times indicated in Table 1 represent the duration required for the initial motion of the drone calculations, particularly focusing on feature point detection. The results clearly show that the feature point detection method used in this study significantly reduces computation time compared to other approaches, achieving a 90% reduction relative to SURF [24], an 83% reduction compared to SIFT [23], and a 41% reduction compared to Method [18]. Furthermore, the proposed method demonstrates superior performance in terms of average positioning error. On two high-texture routes, the localization error was consistently less than 10 m, with an average localization error of 5.713 m over a cumulative flight distance of 4.59 km. In contrast, on two low-texture routes, the lack of significant point, line, and surface features resulted in fewer extractable feature points, leading to a localization error of less than 40 m, with an average localization error of 22.6 m over a cumulative flight distance of 3.06 km.Overall, the positioning error approaches GNSS levels, indicating promising experimental results.

4.4. Ablation Study

To validate the logical soundness and practical effectiveness of the proposed visual localization algorithm, we conducted ablation experiments focused on the attention mechanism module. In these experiments, the first and third stages of the algorithm, as described in the main text, were left unchanged and served as the baseline framework (hereafter referred to as “BASE”), while modifications were applied only to the attention mechanism module in the second stage. The attention mechanism module comprises the Channel Attention Module (CAM) and the Spatial Attention Module (PAM). The method without the attention mechanism module is referred to as “BASE”, while the variants containing only the Channel Attention Module and only the Spatial Attention Module are referred to as “BASE+CAM” and “BASE+PAM”, respectively. The complete method, which incorporates both modules, is denoted as “BASE+CAM+PAM”. Quantitative results for the entire dataset are presented in

Table 2. Quantitative results of ablation experiments on four different flight paths. ✓ indicates the use of this module.

BASE	CAM	PAM	Low-Texture Route I	Low-Texture Route II	High-Texture Route I	High-Texture Route II
✓			58.11	16.56	9.523	10.66
✓	✓		46.31	10.23	5.061	90.34
✓		✓	44.57	8.223	5.166	7.135
✓	✓	✓	38.67	6.502	3.914	7.512

.

As shown in Table 2, the proposed complete localization method achieves higher localization accuracy across four different routes compared to methods without the attention mechanism module, with particularly significant improvements in low-texture regions. Specifically, in low-texture areas where image features are sparse or indistinct, conventional convolution operations struggle to effectively extract critical features. In contrast, the attention mechanism significantly enhances localization accuracy by focusing on the limited but crucial areas in the image, such as riverbanks and field edges. In high-texture regions, where image features are abundant, the localization accuracy of methods without the attention mechanism module approaches GNSS-level precision during long-distance flights. As a result, the addition of the attention mechanism provides only slight improvements in these regions. Overall, integrating the attention mechanism into the network architecture is a critical component of the proposed visual localization method. Aerial images are typically characterized by a large amount of information, complex content, the coexistence of global and local features, and diverse scales. Traditional convolution operations, constrained by their limited receptive fields, tend to be overly sensitive to local features while neglecting global context. The attention mechanism addresses this limitation by capturing long-range dependencies between spatial locations, enabling globally-aware feature extraction and enhancing the robustness of feature representations.

5. Conclusions

This paper introduces a drone visual positioning algorithm that combines image registration with an attention mechanism. By effectively integrating convolutional neural networks with the inverse compositional Lucas-Kanade registration algorithm, we enhance the accuracy of drone positioning. For feature point detection, we utilize Oriented FAST and Rotated BRIEF (ORB) and a quadtree-based method for uniformizing feature points, which significantly accelerates computation speed. This is crucial for meeting the demands of fast drone flights, especially in scenarios where consecutive frames exhibit low overlap. Furthermore, we have refined the feature extraction network by incorporating a Convolutional Block Attention Module (CBAM) in the feature extraction layer. This addition improves the accuracy of image feature extraction, particularly in cases where there are significant differences between low-texture UAV aerial images and satellite imagery. Experimental results indicate that our approach not only enhances the computational speed but also improves the accuracy of drone visual positioning algorithms, demonstrating substantial value for practical engineering applications.

Using UAV aerial imagery for visual positioning is a crucial method for navigation in environments where GNSS is unavailable. Visual odometry estimates the motion of the drone by analyzing sequences of aerial images. However, long-distance flights may result in cumulative errors. By combining visual odometry with absolute visual positioning methods, these errors can be effectively mitigated, leading to improved accuracy. We acknowledge that incorporating absolute visual positioning methods increases computational requirements, which can be challenging given the limited processing power of onboard drone systems. Consequently, future research will focus on optimizing computational models to reduce this burden and shorten positioning times. Additionally, from the experimental results, this is an acceptable outcome, with the localization accuracy being reasonably close to that of GNSS. However, this depends on the specific application scenario. For example, in low-texture areas, where precise tracking and target localization are required, further improvement in localization accuracy is necessary. In such cases, focusing on image dense features rather than point features would be beneficial, and employing an end-to-end approach could be a feasible solution. In high-texture areas, the image’s point features are abundant and diverse, so the focus should be on the stability of feature extraction. Moreover, it is important to design feature extraction methods that are robust to factors such as lighting conditions and seasonal variations.