Visual Geolocalization for Aerial Vehicles via Fusion of Satellite Remote Sensing Imagery and Its Relative Depth Information

Abstract

Visual geolocalization for aerial vehicles based on an analysis of Earth observation imagery is an effective method in GNSS-denied environments. However, existing methods for geographic location estimation have limitations: one relies on high-precision geodetic elevation data, which is costly, and the other assumes a flat ground surface, ignoring elevation differences. This paper presents a novel aerial vehicle geolocalization method. It integrates 2D information and relative depth information, which are both from Earth observation images. Firstly, the aerial and reference remote sensing satellite images are fed into a feature-matching network to extract pixel-level feature-matching pairs. Then, a depth estimation network is used to estimate the relative depth of the satellite remote sensing image, thereby obtaining the relative depth information of the ground area within the field of view of the aerial image. Finally, high-confidence matching pairs with similar depth and uniform distribution are selected to estimate the geographic location of the aerial vehicle. Experimental results demonstrate that the proposed method outperforms existing ones in terms of geolocalization accuracy and stability. It eliminates reliance on elevation data or planar assumptions, thus providing a more adaptable and robust solution for aerial vehicle geolocalization in GNSS-denied environments.

1. Introduction

In recent years, aerial vehicles have been widely used in agriculture, logistics, military operations, disaster rescue, and environmental monitoring [1,2,3,4,5]. The Global Navigation Satellite System (GNSS) serves as the primary approach for aerial vehicle geolocalization. However, aerial vehicles struggle to achieve autonomous geolocalization in GNSS-denied environments. Advances in computer vision have established vision-based perception as the preferred solution for this challenge. This approach follows a progressive three-tiered architecture: cross-domain image retrieval, heterogeneous image matching, and geographic location estimation. Related works in this domain are outlined below.

Cross-domain image retrieval can be defined as the process of utilizing aerial images to identify the most similar ones in a database of satellite remote sensing images, thereby establishing indexing relationships. Traditional methods have relied on hand-crafted features to capture the semantic content of images [6,7]. Compared to traditional systems, recent advancements in deep neural networks have significantly enhanced the performance of content-based retrieval. Unar et al. [8] utilized VGG16 and ResNet50 for feature extraction, improving the semantic information of the feature learner by employing multi-kernel feature learning to identify the most critical image-representation features. Meanwhile, Yamani et al. [9] combined Transformer encoders with CNN-based feature extractors to derive advanced semantic interpretations. Meng et al. [10] utilize DINOV2 [11] to extract features from aerial, satellite, and image sources, and VLAD [12] vectors are employed to combine sequential images with particle filters for the purpose of image retrieval.

Heterogeneous image matching establishes pixel-level feature correspondences between aerial and satellite remote sensing imagery. Traditional approaches relied on hand-crafted feature detection algorithms, such as Scale-Invariant Feature Transform (SIFT) [13], Speeded-Up Robust Features (SURF) [14], and Oriented FAST and Rotated BRIEF (ORB) [15]. Saleem et al. [16] proposed an improved SIFT algorithm based on normalized gradient enhancement. The RIFT algorithm, introduced by Li et al. [17], significantly enhanced feature point robustness and achieved rotation invariance. However, traditional algorithms exhibit limitations due to sensor discrepancies, inconsistent resolutions, and viewpoint variations inherent in cross-source imagery. In recent years, deep learning has revolutionized traditional feature extraction and matching techniques, gradually evolving from sparse feature extraction and matching to end-to-end dense matching. Han et al. [18] utilizes a convolutional neural network (CNN) to extract features from image patches. DeTone et al. [19] pioneered self-supervised joint feature detection and description with SuperPoint. Subsequently, Sarlin et al. [20] introduced SuperGlue, which incorporates a graph attention network (GAT) to model feature matching as a structural optimization problem between two views. SuperPoint combined with SuperGlue has become a paradigm for feature point extraction and matching. Lindenberger et al. [21] designed an adaptive pruning mechanism for LightGlue to address real-time bottlenecks, improving inference speed by 2-3 times while maintaining accuracy. Furthermore, Sun et al. [22] proposed LoFTR, which completely overturned the “detect-match” divide-and-conquer paradigm.

Geographic location estimation refers to calculating the position of the aerial vehicle using image-matching results to obtain longitude and latitude coordinates. Wang et al. [23] computed homography transformations based on the matching results of aerial and satellite imagery, subsequently employing the projection coordinates of the aerial image’s center pixel onto the satellite image as the aerial vehicle geolocalization. This methodology yields accurate positioning results exclusively when the onboard camera is oriented in a nadir-view configuration. Yao et al. [24] leveraged ground elevation data, in conjunction with feature matching results, to formulate a 3D-2D Perspective-n-Point (PnP) problem, employing algorithms such as EPnP [25] to determine the aerial vehicle’s geographic location. Following the establishment of a matching relationship between aerial image pixel coordinates and real-world geographic coordinates [26,27,28], a planar scene constraint is assumed, wherein all geographic points lie on a horizontal plane, thereby enabling the formulation and solution of a PnP problem. Li et al. [29] derived a homography transformation between aerial image pixel coordinates and road network geographic coordinates, predicated on the assumption of a horizontal plane. By computing the homography matrix H and subsequently decomposing it, the geographic location of the aerial vehicle can be determined.

Contemporary research predominantly focuses on cross-domain image retrieval and heterogeneous image matching, with relatively limited attention devoted to geographic location estimation. Current estimation methods primarily depend on either a priori geographic elevation data or planar terrain assumptions. The former approach is constrained by insufficient data accuracy and regional data gaps, while the latter introduces errors by disregarding actual topographic undulations and elevation variations. To address these limitations, this paper presents an innovative visual geolocalization method for aerial vehicles that integrates satellite remote sensing imagery with relative depth information. This approach significantly enhances geolocalization accuracy and stability in complex terrain environments. The principal contributions of this work are detailed below:

• We have conducted an error analysis of the planar assumption-based method for aerial vehicle geolocalization.
• We propose a novel method for aerial vehicle geolocalization that leverages relative depth consistency from satellite remote sensing imagery.

2. Materials and Methods

2.1. Problem Definition

Map projection transforms geodetic coordinates into planimetric map coordinates. The Pseudo-Mercator Projection (EPSG:3857), widely adopted by web mapping services like Google Maps, establishes its coordinate origin at the equator–prime meridian intersection. This system extends eastward along the positive X-axis and northward along the positive Y-axis. The X-axis and Y-axis take values in the range [−20,037,508.3427892, 20,037,508.3427892] in m units. Under the Pseudo-Mercator Projection, the geodetic coordinates of a terrestrial point are defined as $(\lambda ,\varphi )$, with longitude $\lambda$ constrained to $[-{180}^{\circ },{180}^{\circ }]$ and latitude $\varphi$ bound with $[-{85.06}^{\circ },{85.06}^{\circ }]$. The corresponding Mercator plane coordinates are derived through

$$\left\{\begin{array}{c}{x}_m=\mathrm{R}·{\lambda }_{rad}\hfill \\ y_m={\displaystyle \frac{\mathrm{R}}{2}}·ln\left({\displaystyle \frac{1+\sin {\varphi }_{rad}}{1-\sin {\varphi }_{rad}}}\right)\hfill \end{array}\right.$$

(1)

where $\mathrm{R}$ represents the radius of the Earth, which is 6378137 m, and ${\lambda }_{rad}$ and ${\varphi }_{rad}$ are the radians of the $\lambda$ and $\varphi$. This study establishes a global coordinate system with the following definitions:

• X/Y-axis Definition: The X-Y plane originates at the intersection of the Earth’s equator (0° latitude) and prime meridian (0° longitude), with axis orientations aligned with the Pseudo-Mercator coordinate system-X-axis positive eastward and Y-axis positive northward.
• Z-axis Definition: A local vertical axis originates from the aerial vehicle’s takeoff position, orthogonal to the X-Y plane, with a positive direction aligned with the upward vector in the East–North–Up (ENU) geodetic frame.

This study addresses the following problem: In the world coordinate system, given a satellite remote sensing reference image and the Mercator coordinates of each of its pixels, the aim is to geolocate the aerial vehicle based on aerial images.

2.2. Error Analysis in Aerial Vehicle Geolocalization Under Planar Assumption

Image-matching geolocalization establishes pixel-to-geographic coordinate mappings using satellite imagery. When elevation data are unavailable, geographic positions are calculated under a planar terrain assumption. However, actual terrain variations and building heights introduce localization errors. This section analyzes errors attributable to the planar assumption.

Let ${\mathbf{P}}_w={[X,Y,Z]}^{\top }$ denote the world coordinates of a geographical point, and $\mathbf{p}={[u,v]}^T$ represent its corresponding pixel coordinates in an aerial image. The elevation within the image coverage area follows a Gaussian distribution $Z\sim \mathcal{N}(a,b)$. Under the plane hypothesis, spatial points are approximated as ${\mathbf{P}}_w^{\prime }={[X,Y,0]}^{\top }$. Define the true pose of the aerial vehicle as $\mathbf{R}$ and $\mathbf{t}$, and the estimated pose as ${\mathbf{R}}^{\prime }$ and ${\mathbf{t}}^{\prime }$. The PnP problem is established based on the projection relationship between pixel points and geographical points for pose solution.

Actual Reprojection Error:

$$E=\sum _i{∥u_i-{\displaystyle \frac{1}{s_i}}\left[\mathbf{K}(\mathbf{R}{\mathbf{P}}_w^i+\mathbf{t})\right]∥}^2$$

(2)

Assuming a planar configuration, the reprojection error can be defined as follows:

$$E^{\prime }=\sum _i{∥u_i-{\displaystyle \frac{1}{s_i^{\prime }}}\left[\mathbf{K}\left({\mathbf{R}}^{\prime }{\mathbf{P}}_w^{\prime i}+{\mathbf{t}}^{\prime }\right)\right]∥}^2$$

(3)

Among them, $\mathbf{K}$ is the internal parameter matrix of the camera; $s_i$ is the depth value of ${\mathbf{P}}_w^i$ in the camera coordinate system under the true pose; and $s_i^{\prime }$ is the depth value of ${\mathbf{P}}_w^{\prime i}$ in the estimated pose under the planar assumption. When the re-projection errors E and $E^{\prime }$ converge to zero through optimization, the normalized camera coordinate direction vectors projected to the same point by the real model and the planar hypothetical model must be equal. The difference between the depth scale $s_i$ and $s_i^{\prime }$ is compensated by the pose error, and we can approximately obtain

$${\mathbf{R}}^{\prime }{\mathbf{P}}_w^{\prime }+{\mathbf{t}}^{\prime }=\mathbf{R}{\mathbf{P}}_w+\mathbf{t}$$

(4)

where ${\mathbf{P}}_w={\mathbf{P}}_w^{\prime }+Z{\mathbf{e}}_{\mathbf{z}}$, $Z\sim \mathcal{N}(a,b)$ and ${\mathbf{e}}_{\mathbf{z}}={[0,0,1]}^T$, respectively. Assume that ${\mathbf{R}}^{\prime }=\mathbf{R}\Delta \mathbf{R}$ and ${\mathbf{t}}^{\prime }=\mathbf{t}+\Delta \mathbf{t}$. $\Delta \mathbf{R}$ and $\Delta \mathbf{t}$ are the pose errors caused by the planar assumption, where $\Delta \mathbf{R}=exp\left({\varphi }^{\wedge }\right)$ and $\varphi \in \mathrm{so}\left(3\right)$. By substituting ${\mathbf{R}}^{\prime }$ and ${\mathbf{t}}^{\prime }$ into Equation (4), one obtains

$$\mathbf{R}exp\left({\varphi }^{\wedge }\right){\mathbf{P}}_w^{\prime }+\Delta \mathbf{t}=\mathbf{R}({\mathbf{P}}_w^{\prime }+Z{\mathbf{e}}_z)$$

(5)

By conducting a first-order Taylor expansion on the term associated with rotational disturbances, we can obtain

$$exp\left({\varphi }^{\wedge }\right)\approx \mathbf{I}+{\varphi }^{\wedge }$$

(6)

Assume that $\varphi ={[{\varphi }_x,{\varphi }_y,{\varphi }_z]}^{\top }$; then, we can obtain

$$\Delta \mathbf{t}=\mathbf{R}Z{\mathbf{e}}_z-\mathbf{R}\left[\begin{array}{c}-{\varphi }_{z}Y\\ {\varphi }_{z}X\\ -{\varphi }_{y}X+{\varphi }_{x}Y\end{array}\right]$$

(7)

The mathematical expectation of $\Delta \mathbf{t}$ is as follows:

$$\mathbb{E}(\Delta \mathbf{t})=\mathbb{E}\left(\mathbf{R}Z{\mathbf{e}}_z+\mathbf{R}\left[\begin{array}{c}{\varphi }_{z}Y\\ -{\varphi }_{z}X\\ {\varphi }_{y}X-{\varphi }_{x}Y\end{array}\right]\right)$$

(8)

Assume that $\mathbb{E}\left(X\right)={\mu }_x$ and $\mathbb{E}\left(Y\right)={\mu }_y$. We can then obtain

$$\mathbb{E}(\Delta \mathbf{t})=a\left[\begin{array}{c}{\mathbf{R}}_{13}\\ {\mathbf{R}}_{23}\\ {\mathbf{R}}_{33}\end{array}\right]+\mathbf{R}\left[\begin{array}{c}{\varphi }_z{\mu }_y\\ -{\varphi }_z{\mu }_x\\ {\varphi }_y{\mu }_x-{\varphi }_x{\mu }_y\end{array}\right]$$

(9)

It can be inferred that the expected value of the translation vector error $\Delta \mathbf{t}$ comprises two components: the first component is the compensation error attributable to elevation data, while the second component arises from coupling errors associated with rotational disturbances and the arrangement of two-dimensional geographic coordinates. The equation suggests that an increase in the mean elevation of geographic points correlates with a corresponding increase in the translation vector error $\mathrm{t}$ as determined by the PnP method under the assumption of a planar configuration.

2.3. Geographic Location Estimate Method for the Aerial Vehicle with Consistent Relative Depth

The coordinates of a geographical point in the world coordinate system are ${\mathbf{P}}_w={[X,Y,Z]}^T$. Here, X and Y are known Mercator coordinates, while the height information for Z is unknown. Satellite remote sensing images use orthographic projection, where each pixel in the image corresponds one-to-one with a geographic point. By performing depth estimation on the satellite images, we obtain relative depth information for each pixel, with output values ranging from 0 to 255. The relative depth value is inversely proportional to the distance from the geographic point to the shooting point; a smaller value indicates a greater distance, while a larger value indicates a closer distance. Additionally, the depth value is positively correlated with the relative height of the geographic point; a larger value corresponds to a higher elevation, while a smaller value corresponds to a lower elevation.

Within the ideal geometric model of satellite orthoimages, assuming the camera height is H, the absolute depth of a geographical point in the same image can be calculated using the formula:

$$D=cd$$

(10)

where D represents the absolute depth, d represents the relative depth, and c represents the scale factor which has a unit of m. Assuming the elevation value of the camera is H, and the elevation value of a certain geographical point is $Z(x,y)$, we can obtain

$$H=Z(x,y)+D(x,y)=Z(x,y)+cd(x,y)$$

(11)

Therefore,

$$Z(x,y)=H-cd(x,y)$$

(12)

Thus, when the relative depth is the same, the actual terrain elevation data should also be consistent. Select the geographical coordinate ${\mathbf{P}}_w={[X,Y,d]}^T$, ensuring that it maintains the same relative depth d. Find the coordinate system such that ${\mathbf{P}}_w$ in this coordinate system is ${[X,Y,0]}^{\top }$ and name this coordinate system the reference coordinate system. We can easily determine that ${\mathbf{R}}_{rw}=\mathbf{I}$, ${\mathbf{t}}_{rw}={[0,0,-d]}^T$. ${\mathbf{R}}_{rw}=\mathbf{I}$ and ${\mathbf{t}}_{rw}={[0,0,-d]}^T$ represent the rotation matrix and translation vector from the world coordinate system to the reference coordinate system, respectively, and the following can be obtained:

$${\mathbf{P}}_r={\mathbf{R}}_{rw}{\mathbf{P}}_w+{\mathbf{t}}_{rw}$$

(13)

In the case of ${\mathbf{P}}_r={[X,Y,0]}^{\top }$, assume that the homogeneous coordinate of ${\mathbf{P}}_w$ pixel corresponding to the aerial image is ${\mathbf{P}}_I={[u,v,1]}^{\top }$. Then, the coordinates ${\mathbf{P}}_r$ and ${\mathbf{P}}_I$ in the reference coordinate system satisfy the projection relationship as follows:

$${\mathbf{P}}_I=s\mathbf{K}({\mathbf{R}}_{cr}{\mathbf{P}}_r+{\mathbf{t}}_{cr})$$

(14)

where s is the scale factor, ${\mathbf{R}}_{cr}$ and ${\mathbf{t}}_{cr}$ are the rotation matrix and translation vector from the reference coordinate system to the aerial vehicle coordinate system, respectively. Let ${\mathbf{R}}_{cr}=[{\mathbf{r}}_1,{\mathbf{r}}_2,{\mathbf{r}}_3]$, where ${\mathbf{r}}_i$ represents the ${\mathbf{R}}_{cr}$ column i vector. We can then obtain

$${\mathbf{P}}_I=s\mathbf{K}(X{\mathbf{r}}_1+Y{\mathbf{r}}_2+{\mathbf{t}}_{cr})$$

(15)

Expanding Equation (15) yields

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=s\mathbf{K}[{\mathbf{r}}_1\hspace{1em}{\mathbf{r}}_2\hspace{1em}{\mathbf{t}}_{cr}]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]$$

(16)

Let

$$\mathbf{H}=s\mathbf{K}[{\mathbf{r}}_1\hspace{1em}{\mathbf{r}}_2\hspace{1em}{\mathbf{t}}_{cr}]$$

(17)

We can then determine that ${\mathbf{P}}_r^{\prime }={[X,Y,1]}^{\top }$ and ${\mathbf{P}}_I={[u,v,1]}^{\top }$ satisfy the homography transformation. $\mathbf{H}$ is the homologous transformation matrix from ${\mathbf{P}}_r^{\prime }$ to ${\mathbf{P}}_I$. Let $\mathbf{A}=\left[\begin{array}{cc}{\mathbf{r}}_1& {\mathbf{r}}_2\end{array}\right]$ and $\mathbf{B}={\left[{\mathbf{K}}^{-1}\mathbf{H}\right]}_{1,2}=[{\mathbf{b}}_1\hspace{1em}{\mathbf{b}}_2]$, where ${\left[\mathbf{M}\right]}_i$ represents the matrix $\mathbf{M}$ column i vector. Then, conduct Singular Value Decomposition (SVD) on matrices $\mathbf{A}$ and $\mathbf{B}$:

$$\left\{\begin{array}{c}\mathbf{A}=\mathbf{U}\mathbf{E}{\mathbf{V}}^{\mathbf{T}}\hfill \\ \mathbf{B}=\mathbf{U}\mathbf{W}{\mathbf{V}}^{\mathbf{T}}\hfill \end{array}\right.$$

(18)

where $\mathrm{W}=\left[\begin{array}{cc}{w}_1& 0\\ 0& w_2\\ 0& 0\end{array}\right]$. According to the conclusion of [29], let ${\mathbf{E}}^{\prime }=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 0& 0\end{array}\right]$ and ${\mathbf{A}}^{\prime }={\mathbf{UE}}^{\prime }{\mathbf{V}}^{\mathbf{T}}$. Then, we can obtain

$$\left\{\begin{array}{c}\widehat{{\mathbf{R}}_{cr}}=\left[\begin{array}{cc}{\widehat{\mathbf{a}}}_1\hfill & {\widehat{\mathbf{a}}}_2\hspace{1em}{\widehat{\mathbf{a}}}_1\times {\widehat{\mathbf{a}}}_2\hfill \end{array}\right]\\ \widehat{{\mathbf{t}}_{cr}}={\displaystyle \frac{2{\left[{\mathbf{K}}^{-1}\mathbf{H}\right]}_3}{w_1+w_2}}\\ \widehat{s}={\displaystyle \frac{w_1+w_2}{2}}\end{array}\right.$$

(19)

where ${\widehat{\mathbf{a}}}_1$ and ${\widehat{\mathbf{a}}}_2$ are the first two columns of ${\mathbf{A}}^{\prime }$.

The coordinates of the aircraft in the world coordinate system are

$$\widehat{{\mathbf{t}}_{wc}}={\mathbf{R}}_{wr}\widehat{{\mathbf{t}}_{rc}}+t_{wr}$$

(20)

where

$$\left\{\begin{array}{c}\widehat{{\mathbf{t}}_{rc}}=-{\widehat{{\mathbf{R}}_{cr}}}^{\mathrm{T}}\widehat{{\mathbf{t}}_{cr}}\\ {\mathbf{R}}_{wr}={{\mathbf{R}}_{\mathbf{r}\mathbf{w}}}^{\mathrm{T}}=\mathbf{I}\\ {\mathbf{t}}_{wr}=-{\mathbf{R}}_{rw}^{\top }{\mathbf{t}}_{rw}=-{\mathbf{t}}_{rw}\end{array}\right.$$

(21)

Substituting it into Equation (20) gives

$$\widehat{{\mathbf{t}}_{wc}}=-{\widehat{{\mathbf{R}}_{cr}}}^{\mathrm{T}}\widehat{{\mathbf{t}}_{cr}}+d{\mathbf{e}}_z$$

(22)

Writing ${\widehat{{\mathbf{R}}_{cr}}}^{\mathrm{T}}\widehat{{\mathbf{t}}_{cr}}=\left[\begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\right]$, we can obtain

$$\widehat{{\mathbf{t}}_{wc}}=\left[\begin{array}{c}{t}_x\\ t_y\\ t_z+d\end{array}\right]$$

(23)

$\widehat{{\mathbf{t}}_{wc}}$ represents the coordinates of the aerial vehicle in the world coordinate system. The theoretical analysis indicates that when selecting geographical points with identical relative depth, the position of the aerial vehicle in the pseudo-Mercator coordinate system, namely $(t_x,t_y)$, can be accurately determined. Subsequently, by applying Equation (24), the conversion from $(t_x,t_y)$ to longitude $\lambda$ and latitude $\varphi$ coordinates can be achieved, thereby identifying the geographic location of the aerial vehicle.

$$\left\{\begin{array}{c}\lambda ={\displaystyle \frac{t_x}{R}}\times {\displaystyle \frac{180}{\pi }}\hfill \\ \varphi =\left[2arctan\left(e^{{\displaystyle \frac{t_y}{R}}}\right)-{\displaystyle \frac{\pi }{2}}\right]\times {\displaystyle \frac{180}{\pi }}\hfill \end{array}\right.$$

(24)

2.4. Satellite Remote Sensing Image with Relative Depth-Assisted Aerial Vehicle Geolocalization Method

The methodology presented in this study consists of three core components: (a) a heterogeneous image-matching module, (b) a relative depth estimation module, and (c) a geographic location estimation module. As shown in Figure 1, the approach takes aerial images and their corresponding satellite remote sensing images as input. These aerial and satellite images are first processed by the heterogeneous image-matching module to extract matched feature pairs and their associated confidence levels. The satellite remote sensing image is then channeled into the relative depth estimation module to generate its corresponding relative depth map. Finally, the matched feature pairs, their confidence levels, and the satellite-derived relative depth map are input to the geographic location estimation module. This module integrates these inputs to accurately estimate the geographic location of the aerial vehicle.

2.4.1. Heterogeneous Image Matching

Heterogeneous image matching is facilitated by the use of LoFTR [22]. This method adeptly addresses cross-modal discrepancies through its end-to-end architecture, which operates without the need for detectors. It employs global modeling via Transformers and integrates fine-to-coarse granularity fusion. Notably, LoFTR is capable of producing dense and highly accurate matching point pairs even in scenes characterized by low texture and non-linear variations. The framework of this approach is illustrated in Figure 2.

The aerial image $I^A$ and the satellite remote sensing image $I^B$ are inputted into using the CNN feature extraction network to extract rough feature maps ${\tilde{F}}^A$ and ${\tilde{F}}^B$ of 1/8 size of the original image for quick screening of candidate matching regions. Fine feature maps ${\widehat{F}}^A$ and ${\widehat{F}}^B$ of 1/2 size retaining more details are used for precise localization.

Then, ${\tilde{F}}^A$ and ${\tilde{F}}^B$ are expanded, and positional encoding is added to the LoFTR module for processing to extract positional and context-dependent local features to obtain ${\tilde{F}}_{tr}^A$ and ${\tilde{F}}_{tr}^B$. ${\tilde{F}}_{tr}^A$ and ${\tilde{F}}_{tr}^B$ are input into the differentiable matching layer to compute the score matrix $\mathbf{S}$ between the features:

$$\mathbf{S}(i,j)={\displaystyle \frac{1}{\tau }}{\tilde{F}}_{tr}^A\left(i\right),{\tilde{F}}_{tr}^B\left(i\right)$$

(25)

When the dual-softmax micromatchable layer is then used, the matching confidence matrix ${\mathcal{P}}_{\mathbf{c}}$ is obtained:

$${\mathcal{P}}_{\mathbf{c}}(i,j)=\mathrm{softmax}{\left[\mathbf{S}(i,·)\right]}_j·\mathrm{softmax}{\left[\mathbf{S}(·,j)\right]}_i$$

(26)

The matches are then selected based on confidence thresholds and mutual proximity criteria:

$${\mathcal{M}}_{\mathrm{c}}=\left\{(\tilde{i},\tilde{j})\right|\forall (\tilde{i},\tilde{j})\in \mathrm{MNN}\left({\mathcal{P}}_{\mathbf{c}}\right),{\mathcal{P}}_{\mathbf{c}}(\tilde{i},\tilde{j})\ge {\theta }_c\}$$

(27)

The coarse matching prediction ${\mathcal{M}}_{\mathrm{c}}$ is obtained. For each coarse matching result $(\tilde{i},\tilde{j})\in {\mathcal{M}}_{\mathrm{c}}$, the position $(\tilde{i},\tilde{j})$ is centered in the fine feature maps ${\widehat{F}}^A$, ${\widehat{F}}^B$. Then, the center vectors of ${\widehat{F}}^A$ are correlated with all the vectors of ${\widehat{F}}^B$, respectively, to generate the heatmap, which represents the probability of matching each pixel in the neighborhood of $\tilde{j}$ with $\tilde{i}$. Finally, the expectation value on the probability distribution is calculated to bring the coarse matching result up to the sub-pixel matching level to obtain the final matching result ${\mathcal{M}}_{\mathrm{c}}=\left\{(\tilde{i},\widetilde{j^{\prime }})\right\}$.

2.4.2. Relative Depth Estimation

The relative depth estimation module uses the Depth Anything V2 model [30], which, in contrast to its predecessor V1 [31], uses fully synthetic images in place of all labeled real images and incorporates semantic information to train the teacher model. Reliable teacher models are trained based on high-quality synthetic images DINOv2-G. Pseudo-depth is generated using the teacher model on large-scale unlabeled real images. Student models, on the other hand, are trained based on pseudo-labeled real images for robust generalization. The training used five exact synthetic datasets (595,000 images) and eight large-scale pseudo-labeled real datasets (62 million images). Using the pseudo-labeling generated by the above exploits, ref. [30] trained four student models of different sizes (Small/Base/Large/Giant). Student models acquired the ability to make accurate depth predictions on real images by mimicking the teacher model’s behavior. The geometric constraints of nadir-view orthographic projection in satellite imagery position the imaging plane approximately perpendicular to the ground surface; consequently, relative depth differences between pixels directly correspond to changes in surface elevation. We therefore employ the Large variant to estimate satellite image depth, generating a 0–255 grayscale map representing relative depth. Lower values indicate smaller elevation values, while higher values signify greater elevation. Relative depth estimation is illustrated in Figure 3.

2.4.3. Geographic Location Estimation

Denote the feature point matching set of the aerial image ${\mathbf{I}}_{aer}\in {\mathbb{R}}^{H\times W\times 3}$ and the satellite remote sensing image ${\mathbf{I}}_{sat}\in {\mathbb{R}}^{H\times W\times 3}$ as $M=\left\{\left({\mathbf{p}}_i,{\mathbf{q}}_j\right)\mid {\mathbf{p}}_i\in {\mathcal{P}}_{\mathrm{aer}},{\mathbf{q}}_j\in {\mathcal{P}}_{\mathrm{sat}}\right\}$, where ${\mathcal{P}}_{aer}$ and ${\mathcal{P}}_{set}$ denote the feature point sets of the aerial image and the satellite remote sensing image, respectively. The feature point coordinates ${\mathbf{p}}_i$ and ${\mathbf{p}}_j$ are pixel coordinates. Meanwhile, let $C=\left\{c_{ij}\right\}$ be the matching confidence level of ${\mathbf{p}}_i$ and ${\mathbf{p}}_j$. Let the relative depth estimation result of the satellite remote sensing image be denoted as ${\mathbf{D}}_{sat}\in {\mathbb{R}}^{H\times W}$, a grayscale image composed of 0-255.

First, screen through the confidence threshold $\mathcal{I}$ and take 0.3. The filtered matching set is

$$S=\{({\mathbf{p}}_i,{\mathbf{q}}_j)\in M\mid c_{ij}>\tau \}$$

(28)

The relative depth values of aerial image feature points are determined by $D_{aer}={\left\{d_i\right\}}_{i=1}^M$, where $d_i={\mathbf{D}}_{sat}\left({\mathbf{q}}_j\right)(({\mathbf{p}}_i,{\mathbf{q}}_j)\in S)$. Feature points with the same depth value are selected using the depth consistency constraint. Since depth estimation has errors, the windowing mechanism is chosen. A dynamic depth window $W_d=[1,\sigma ]$ is constructed, where $\sigma$ is the standard deviation of the depth values of the feature points of the aerial image. The sliding window mechanism is then used to traverse $W_d$ and select the subset $S^{\prime }\subseteq S$.

Next, a triple constraint is imposed on the candidate subset. The first is the subset size constraint; to ensure that a single-response matrix computation can be performed, the number of matches in the subset needs to be larger than 4. The second is the confidence constraint; to ensure the matching quality of the subset, the average confidence of the matched pairs needs to be larger than the threshold value of ${\delta }_1$. ${\delta }_1$ does not use a fixed value and represents the average confidence level of the matching pairs. The third is the spatial distribution constraint; to ensure that the distribution of the feature points in the subset is uniform, we use the convex hull area of the feature points of aerial images in the selected subset to carve out the area of convex bags. The greater the convex hull area is over the area of the image, the more uniform its distribution is. The larger the area is, the more uniform its distribution is. The projected coordinates of the candidate subset $S^{\prime }$ on the aerial image are ${\{{\mathbf{p}}_i=(u_i,v_i)\}}_{i=1}^n$, and its convex hull area is computed by Graham’s scanning algorithm:

$$\mathrm{Area}\left(\mathrm{ConvHull}\left(S^{\prime }\right)\right)={\displaystyle \frac{1}{2}}\left|\sum _{i=1}^{n-1}(x_i{y}_{i+1}-x_{i+1}{y}_i)+(x_n{y}_1-x_1{y}_n)\right|$$

(29)

The ratio of convex hull area to image area needs to be satisfied:

$${\displaystyle \frac{\mathrm{Area}\left(\mathrm{ConvHull}\left(S^{\prime }\right)\right)}{H\times W}}>{\delta }_2$$

(30)

The parameter ${\delta }_2$ is adaptively determined through a dynamic selection strategy based on the depth window $W_d$. An increased $W_d$ value signifies substantial relative depth variations within the region, thereby necessitating a higher ${\delta }_2$ threshold to enforce enhanced spatial homogeneity of feature correspondences. Conversely, a diminished ${\delta }_2$ indicates limited depth disparity, permitting relaxation of the threshold. This adaptive mechanism is formally expressed as

$${\delta }_2={\delta }_{2_{min}}+\left({\displaystyle \frac{W_d-1}{\sigma -1}}\right)({\delta }_{2_{max}}-{\delta }_{2_{min}})$$

(31)

where $W_d=[1,\sigma ]$, ${\delta }_{2_{max}}=0.2$, and ${\delta }_{2_{max}}=0.4$.

At this point, the candidate subset $S^{\prime }$ are points approximately in the same plane. The candidate subset $S^{\prime }$ satisfying the above three constraints is subjected to the computation of the homography matrix. Let the mapping relation from the pixel coordinates to the Mercator coordinates of the satellite map be

$${\mathbf{q}}_j^{merc}=\mathcal{G}\left({\mathbf{q}}_j^{sat}\right)$$

(32)

where $\mathcal{G}:{\mathbb{R}}^2\to {\mathbb{R}}^2$ is the known geographical alignment function and ${\mathbf{q}}_j^{sat}=(u^{sat},v^{sat})$. The geographical alignment function is

$$\left\{\begin{array}{c}E=E_0+d·u^{sat}\hfill \\ N=N_0-d·v^{sat}\hfill \end{array}\right.$$

(33)

where $(E,N)$ are the Mercator coordinates and $(E_0,N_0)$ are the Mercator coordinates corresponding to the pixels in the upper left corner of the satellite remote sensing image. The spatial resolution of the d satellite remote sensing image represents the actual distance corresponding to a single pixel.

For $S^{\prime }=\left\{({\mathbf{p}}_i,{\mathbf{q}}_j)\right\}$, establish the projection relationship from the pixel coordinates of the aerial image to the Mercator coordinates:

$${\tilde{\mathbf{p}}}_i^{aer}=\mathbf{H}{\tilde{\mathbf{q}}}_j^{merc}$$

(34)

where ${\tilde{\mathbf{p}}}_i^{aer}={[u_i^{aer},v_i^{aer},1]}^T$ is the homogeneous coordinate of the feature points in the aerial image and ${\tilde{\mathbf{q}}}_j^{\mathrm{merc}}={[E_j,N_j,1]}^T$ is the Mercator homogeneous coordinate. $\mathrm{H}$ is the homography matrix. Expanding Equation (34), we can obtain

$$\left[\begin{array}{c}{u}_i^{aer}\\ v_i^{aer}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}{E}_j\\ N_j\\ 1\end{array}\right]$$

(35)

For $S^{\prime }$ that meets the conditions, solve the homography matrix $\mathbf{H}$ and calculate the projection error of the inner points. Select the one with the most minor error, $\mathbf{H}$. Then, based on the derivation in Section 2.3, decompose $\mathbf{H}$ to obtain $\mathbf{R}$ and $\mathbf{t}$:

$$\left\{\begin{array}{c}\mathbf{R}=[{\widehat{\mathbf{a}}}_1\hspace{1em}{\widehat{\mathbf{a}}}_2\hspace{1em}{\widehat{\mathbf{a}}}_1\times {\widehat{\mathbf{a}}}_2]\hfill \\ \hspace{1em}\mathbf{t}={\displaystyle \frac{2{\left[{\mathbf{K}}^{-1}\mathbf{H}\right]}_3}{w_1+w_2}}\hfill \end{array}\right.$$

(36)

The geographic location of the aerial vehicle in the Mercator coordinate system ${(t_x,t_y)}^{\mathrm{T}}$ is ${[-{\mathbf{R}}^{\mathrm{T}}\mathbf{t}]}_{1,2}$. The geographic location of the aerial vehicle can be obtained by bringing ${(t_x,t_y)}^{\mathrm{T}}$ into Equation (24).

3. Results

3.1. Dataset

3.1.1. Aerial Data of the Aerial Vehicle

To address the scarcity of publicly available aerial image datasets, we deployed a DJI M300 UAV equipped with an H20 gimbal camera (featuring an integrated RTK localization system) to capture aerial imagery over suburban Xi’an, China. This platform acquired images with precise geographic coordinates and attitude parameters. The resulting aerial video spans 7.9968 km, with altitudes ranging from 13 to 603 m, for a duration of 968 s, and a frame rate of 30 FPS. We compiled these data into a validation-ready benchmark dataset covering complex geomorphological areas featuring significant elevation variations, including representative terrain features such as building clusters, transportation networks, and vegetated zones. All images feature 1920 × 1080 resolution. The complete flight track recorded during the experiment is shown in Figure 4.

3.1.2. Satellite Remote Sensing Imagery Data

We used the Google Maps platform to acquire a high-resolution satellite remote sensing map covering the target area. As shown in Figure 5, the satellite image covers a rectangular area spanning about 31.2 km from east to west and extending about 5.8 km from north to south, with a geospatial extent of 181 km². The image adopts 24-bit true color coding (RGB three-band), and the pixel matrix dimension is 26,905 × 7032, which meets the requirement of wide coverage.

Table 1. Satellite remote sensing map boundary coordinates.

Position	Web Mercator (m)	WGS84 ${(}^{\circ })$
Top left corner	12,137,200, 4,051,300	109.0303, 34.1674
The upper right corner	12,165,200, 4,051,300	109.2819, 34.1674
Lower right corner	12,165,200, 4,043,300	109.2819, 34.1079
Lower left corner	12,137,200, 4,043,300	109.0303, 34.1079

shows the coordinate parameters of the four boundary points of the satellite remote sensing map containing two coordinate systems, the Web Mercator projected coordinate system (EPSG:3857) and the WGS84 geographic coordinate system (EPSG:4326). The spatial resolution of the map is 1 m/pixel.

3.1.3. Aerial Image—Satellite Remote Sensing Image-Matching Dataset

A matching pair of the aerial image and the satellite remote sensing map is ultimately created, as illustrated in Figure 6, after the corresponding area on the satellite remote sensing reference map is precisely matched based on the aerial image’s actual position. A satellite image of the same size as the aerial image is then cropped out.

In order to create a matched dataset of aerial and satellite images, we chose frame sequences at intervals of 100 frames due to the significant repetition between surrounding frames, excluding the takeoff and landing periods. The flight length was 7.9968 km, the altitude ranged from 400 to 603 m, and 180 frames were used for the aerial image–satellite image-matching dataset.

3.2. Experiments

3.2.1. Experimental Environment

The experimental environment in this study is equipped with a 14-core Xeon(R) Platinum 8362 processor with an NVIDIA GeForce RTX 3090 graphics card with 24 GB of video memory, 45 GB of RAM, Ubuntu 18.04 operating system, Python 3.8 programming language, and PyTorch 1.9.0 deep learning framework.

3.2.2. Calculation Method of Geolocalization Error

In the Mercator coordinate system, the aerial vehicle’s actual geographic location is ${[x,y]}^{\mathrm{T}}$, whereas its predicted location is ${[\tilde{x},\tilde{y}]}^{\mathrm{T}}$. We use $\Delta \mathrm{X}$, $\Delta \mathrm{Y}$ and $\Delta \mathrm{XY}$ to quantify the aerial vehicle geolocalization error:

$$\left\{\begin{array}{c}\Delta \mathrm{X}=|x-\tilde{x}|\hfill \\ \Delta \mathrm{Y}=|y-\tilde{y}|\hfill \\ \Delta \mathrm{X}\mathrm{Y}=\sqrt{\Delta {\mathrm{X}}^2+\Delta {\mathrm{Y}}^2}\hfill \end{array}\right.$$

(37)

3.2.3. Comparative Experiments

The comparison methods employed in this paper encompass two strategies: the first being the PnP-based position estimation, and the second being the position estimation based on single-response matrix decomposition. In the initial method, it is hypothesized that the target scene is planar, i.e., the z-coordinates of all points in the 3D point cloud are set to 0. On this basis, the UPnP [32] algorithm combined with the USAC [33] framework is applied to compute the initial value of the rotation matrix $\mathbf{R}$ and the translation vector $\mathbf{t}$. The initial value of the translation vector t is calculated by using the UPnP algorithm and the USAC framework. Subsequently, utilizing this initial value as a foundation, the Levenberg–Marquardt (LM) algorithm is employed to execute iterative optimization solely for the matched interior points, thereby enhancing the accuracy and robustness of the position estimation. The second method is also based on the planar assumption, but it directly solves for $\mathbf{R}$ and $\mathbf{t}$ by computing the singular-response matrix $\mathbf{H}$ for the 2D-2D matched point pairs and utilizing the decomposition of the singular-response matrix. The $\mathbf{R}$ and $\mathbf{t}$ values calculated by the two methods result in the geographic position of the vehicle in the Mercator coordinate system as ${[-{\mathbf{R}}^{\mathrm{T}}\mathbf{t}]}_{1,2}$. It is observed that out of 180 frames, 153 frames (or 85%) are successfully localized.

As shown in Figure 7, the localization error of the proposed method is compared with the other methods on the successfully localized sequence frames. From the overall graph trend, the algorithm proposed in this paper shows obvious advantages in terms of localization accuracy with lower error. The specific analysis is as follows: When the elevation information corresponding to the ground objects in the aerial image varies significantly, then the feature points exist at the same time points with significant elevation differences, such as tall buildings and the ground, e.g., frame 32,100 in Figure 8. Furthermore, methods based on PnP and the single-response matrix decomposition exhibit significant localization errors. This occurs because their planar assumption breaks down in scenes with substantial elevation variations, leading to inaccurate position estimation and thus larger localization errors. However, for those without significant elevation changes such as frame 79 and 95 in Figure 8, the localization errors of these methods are relatively small and the localization results are more accurate. In contrast, the methods proposed in this paper are more accurate for localization in different scenarios.

Table 2. Statistical characteristics of positioning errors of three geolocalization methods.

Error	Method	Min (m)	Max (m)	Mean (m)	Median (m)	Std (m)
$\Delta \mathrm{X}$	PnP	0.09	123.94	14.10	10.05	14.75
	H	0.14	160.69	21.76	16.12	21.94
	Ours	0.47	41.14	10.81	8.84	7.28
$\Delta \mathrm{Y}$	PnP	0.18	202.92	22.48	18.09	23.17
	H	0.21	206.45	24.59	14.67	28.73
	Ours	0.02	46.67	8.68	6.11	8.18
$\Delta \mathrm{XY}$	PnP	2.39	204.54	29.52	23.63	24.22
	H	1.65	207.93	36.51	27.00	32.41
	Ours	1.67	52.66	14.91	11.86	9.46

shows the statistical characteristics of the positioning errors of the three geolocalization methods. From the table, it can be seen that the maximum, mean, median, and standard deviation of the positioning errors of the proposed method are smaller than those of the other two methods, indicating that the proposed geolocalization method has high accuracy and stability.

3.2.4. Ablation Experiments

To evaluate the effectiveness of the feature-matching pair screening strategy in the proposed method, we designed ablation experiments. We quantitatively analyzed the following two filtering strategies: one is the confidence filtering strategy, which filters the matched subsets by setting a reasonable confidence threshold to eliminate the subsets with low confidence to reduce their negative impact on the accuracy of the position estimation; and the other is the convex hull area ratio filtering strategy, which filters the size of the convex hull area constituted by the feature points. It preferentially retains the subset of matches with larger convex packet areas, aiming to optimize the spatial distribution uniformity of the matched pairs in the image and thus improve the stability and accuracy of the position estimation.

The experimental results are shown in

Table 3. Effectiveness of average confidence and convex hull area ratio.

Confidence	Convex Hull	#Matched	$\overline{\mathbf{\Delta }\mathbf{XY}}/\mathbf{m}$
✕	✕	526	24.42
✓	✕	264	19.63
✕	✓	135	18.52
✓	✓	36	14.91

, where “#Matched” denotes the number of matched pairs in the final matched subset, “✓” indicates that the strategy is selected, and “✕” indicates that it is not selected. The confidence filtering strategy demonstrates significant mismatch filtering ability, and the geolocalization error of 24.42 m is reduced to 19.63 m. This fully confirms the key role of this strategy in suppressing mismatch interference and improving the accuracy of the position-solving technique. The convex hull area filtering strategy, on the other hand, reduces the geolocalization error from 24.42 m to 18.52 m by optimizing the distribution quality of the matching points, which verifies its effectiveness in enhancing the spatial coverage uniformity of the matching points.

When the two strategies work synergistically, the geolocalization error of 24.42 m is reduced to 14.91 m compared to the scenario involving the application of a single strategy. This synergistic phenomenon indicates a complementary mechanism between confidence screening and convex hull area screening in the pose optimization process, with the former aimed at eliminating matching noise interference and the latter focused on improving the geometric distribution of matches. The experimental results confirm that combining the proposed screening strategy enhances geolocalization accuracy.

3.2.5. Parameter Experiments

In the geolocalization algorithm, the appropriate choice of parameters is pivotal for the algorithm’s efficacy. This section presents a comparative analysis that was conducted through fixed thresholds and dynamically selected thresholds. The average confidence threshold ${\delta }_1$ measures the reliability of feature-matching point pairs; conversely, the convex hull area ratio threshold ${\delta }_2$ evaluates the extent of discretization within the matching point set concerning spatial distribution. Guided by empirical experience, we systematically explored various threshold values. Specifically, ${\delta }_1$ was set at 0.4, 0.5, and 0.6, while ${\delta }_2$ was set at 0.2, 0.3, and 0.4. Subsequently, a cross experiment was executed.

The results of the parameter experiments are presented in

Table 4. Comparison of experimental results under different parameters.

${\mathit{\delta }}_1$	${\mathit{\delta }}_2$	$\overline{\mathbf{\Delta }\mathit{X}\mathit{Y}}/\mathbf{m}$	$\mathbf{Recall}/\%$
0.4	0.2	22.86	92
0.4	0.3	19.38	90
0.4	0.4	18.92	80
0.5	0.2	18.44	87
0.5	0.3	16.95	85
0.5	0.4	16.02	66
0.6	0.2	16.23	70
0.6	0.3	15.16	61
0.6	0.4	15.34	52
Dynamic	Dynamic	14.91	88

, where recall indicates the ratio of successfully localized frames to the total number of frames. The table shows that a larger ${\delta }_1$ can enhance the reliability of the matching points; however, this may reduce the number of matching points that satisfy the specified conditions, consequently diminishing geolocalization accuracy and the percentage of successful localization frames due to a lack of reliable matching points. Conversely, a smaller ${\delta }_1$ will likely increase the proportion of false matching points within the subset, adversely affecting geolocalization accuracy. A larger ${\delta }_2$ will encourage a more uniform distribution of the matching points within the subgroup, yielding a more robust computation of the single-response matrix. Similarly, in specific scenarios, this may result in reduced subsets that meet the requirements and an increased likelihood of localization failure. A smaller ${\delta }_2$, on the other hand, will lead to an overly concentrated distribution of the point set, implying that the spatial information of the matching points is not sufficiently wealthy and dispersed, which can result in issues such as numerical instability during the computation of the single-response matrix, thus affecting the accuracy and reliability of geolocalization. The dynamic threshold selection method proposed in this paper has the lowest geolocalization error and a recall rate of 88%, achieving a balance between geolocalization accuracy and success rate.

4. Discussion

Utilizing a real flight dataset, we meticulously designed and executed comparison, ablation, and parametric experiments to thoroughly assess the performance of the proposed method. The comparative results demonstrate that the proposed method significantly outperforms other approaches in terms of localization accuracy and stability. Furthermore, ablation studies validate the effectiveness of the confidence screening and convex envelope area screening techniques. These findings collectively demonstrate that integrating these two screening strategies significantly enhances localization accuracy. Conversely, the parametric experiments reveal that selecting the average confidence threshold ${\delta }_1$ and the convex hull area ratio threshold ${\delta }_2$ significantly influences localization performance. Merely increasing or decreasing a single threshold cannot optimize the geolocalization. Instead, finding a balance between them is key. The dynamic threshold selection strategy adopted in this paper can achieve a balance between geolocalization accuracy and geolocalization success rate, and has strong robustness.

5. Conclusions

In this paper, we propose an innovative approach, fusing 2D Earth observation imagery with relative depth information for aerial vehicle geolocalization. The process begins by inputting both aerial and reference satellite remote sensing images into a feature-matching network to extract pixel-level matching pairs. Next, a depth estimation network estimates the relative depth of features within the satellite image. Finally, high-confidence matching pairs with similar depths and uniform distribution are selected to estimate the aerial vehicle’s location. Experimental results confirm that this method surpasses existing geolocalization accuracy and stability. In future work, we plan to explore more advanced feature-matching and depth estimation networks to enhance the reliability of feature matching and the accuracy of depth information. Additionally, we plan to investigate the integration of inertial navigation data into our system.