Fast Aerial Image Geolocalization Using the Projective-Invariant Contour Feature

Abstract

We address the problem of aerial image geolocalization over an area as large as a whole city through road network matching, which is modeled as a 2D point set registration problem under the 2D projective transformation and solved in a two-stage manner. In the first stage, all the potential transformations aligning the query road point set to the reference road point set are found by local point feature matching. A local geometric feature, called the Projective-Invariant Contour Feature (PICF), which consists of a road intersection and the closest points to it in each direction, is specifically designed. We prove that the proposed PICF is equivariant under the 2D projective transformation group. We then encode the PICF with a projective-invariant descriptor to enable the fast search of potential correspondences. The bad correspondences are then removed by a geometric consistency check with the graph-cut algorithm effectively. In the second stage, a flexible strategy is developed to recover the homography transformation with all the PICF correspondences with the Random Sample Consensus (RANSAC) method or to recover the transformation with only one correspondence and then refine it with the local-to-global Iterative Closest Point (ICP) algorithm when only a few correspondences exist. The strategy makes our method efficient to deal with both scenes where roads are sparse and scenes where roads are dense. The refined transformations are then verified with alignment accuracy to determine whether they are accepted as correct. Experimental results show that our method runs faster and greatly improves the recall compared with the state-of-the-art methods.

1. Introduction

Aligning an aerial image to a known Geographical Coordinate System (GCS), which is called aerial image geolocalization, is a fundamental task in the field of remote sensing. Roads are a kind of salient feature on the ground and are labeled on the existing Geographic Information System (GIS), so we consider it feasible to align aerial images to a certain GCS with roads. As is shown in Figure 1, we develop an efficient geolocalization method imposed directly on the aerial images by registering the road networks in the aerial images to the reference road networks in the GIS when the approximate positions of the aerial images, i.e., in which city the aerial images are captured, are known. The proposed method can geolocalize an aerial image over a search area larger than 1000 km² within seconds. Compared with traditional geolocalization methods, the proposed method is much less expensive and efficient, because the only sensor needed is the camera and the reference road vector map, which is highly compact and is readily available from many kinds of GISs (such as road vector maps from OpenStreetMap (https://www.openstreetmap.org)).

Though appealing, the task is quite demanding. The main challenges lie in the great appearance and geometric differences between the reference road vector map and roads extracted from the aerial images. On the one hand, despite the good performance of the state-of-the-art road extraction methods [1,2,3,4], much noise still exists, causing the extracted roads to be fragmented and incomplete; on the other hand, the reference road map may not be up-to-date, leading to some differences from the actual one. Such appearance differences make it difficult to measure the similarity robustly.

The geometric differences are mainly caused by the change of camera pose. Since the aerial images may be taken under any camera pose, it is necessary to use the 2D projective transformation to represent the mapping function between road points in the aerial images and the reference road vector map. Under the 2D projective transformation, most of the geometric features, including the lengths of line segments and the angles between two lines, are not invariant, leading to difficulties in establishing the correspondences between the roads in the aerial images and the reference road maps [5]. Because of this, many existing works on road based geolocalization [6,7,8,9] treat the transformation between the aerial image and the reference road network map as a similarity transformation, and extra attitude and altitude measurements from other sensors on the Unmanned Aerial Vehicles (UAVs) must be used to project the aerial images to the reference coordination.

Even though there are some shape matching methods [10,11,12] that can deal with the geometric deformation to some degree, they are not suitable for the task. The main drawback of these methods is that they can only deal with shape matching between two geometric primitive sets where the geometric primitives mostly correspond. In the geolocalization, the roads in the aerial images are only a small subset of the reference road maps, and it is hard to segment the corresponding part of the reference road vector map when the transformation is unknown. It is also inefficient to divide the reference road map into overlapping tiles and compare each tile with the roads in the aerial image with such shape matching methods because such tiles have difficulty capturing the correct corresponding area of the aerial image in the reference road map when the possible variation in translation, scale, and rotation is highly uncertain.

To address all the aforementioned challenges, we take a step further from our previous work [5] and develop a novel robust geolocalization algorithm. The core of the new method is a projective-invariant geometric feature, which consists of a road intersection together with all the road points surrounding it, called the Projective-Invariant Contour Feature (PICF) in this paper. The main motivation to design the PICF is that such a feature can split the same subsets of the point set under the 2D projective transformation, which can be proven mathematically. With such a property, we can encode the subsets of the point set in a projective-invariant manner. Compared with the “2-road-intersections-tuple” feature in [5], the PICF encodes much more geometric information and is more discriminative. This makes our algorithm more flexible. With much fewer initial feature correspondences and a much higher matching inlier ratio, each initial transformation recovered from the feature correspondences can be further refined with methods such as ICP [13,14,15]. Such a strategy eliminates the influence of the noisy estimation of the initial transformation, makes the geolocalization more robust, and improves the recall to a large extent.

Moreover, benefiting from the good invariance of the proposed PICF, a robust matching method, such as the methods proposed in [16,17,18], can be used to improve the matching quality further. Along with the PICF, we develop an effective matching algorithm considering the geometric coherence between matching pairs to improve the matching quality further. This is accomplished by minimizing the cost function, which includes the PICF descriptor loss and geometric inconsistency loss. The optimization problem can be solved sufficiently by employing the known graph cut method [18,19,20]. Finally, the correct transformation is determined by checking all the candidate matches. In this stage, a flexible way to recover the initial transformation from those correspondences is adopted. The homography transformation is estimated with all the correspondences when enough correspondences are found. Otherwise, we can recover the coarse transformation from just one matching pair and then refine it with a local-to-global ICP method. Such a strategy enables our method to deal with all cases efficiently no matter whether the roads in the scenes are dense or sparse.

2. Materials and Methods

2.1. Projective-Invariant Contour Feature

In this section, we first introduce the definition of the PICF and prove that the PICF is an equivariant representation under the 2D projective transformation group. Then, we introduce the details of the method to compute the PICF descriptor. Finally, we present a sufficient matching method, where the feature matching is modeled as an optimization problem to minimize both the matching distance between PICF descriptors and the geometric inconsistency between every two matching pairs.

2.1.1. Definition of the PICF

Given a set of 2D points $P=\left\{\left(x_1,y_1\right),\left(x_2,y_2\right),\dots ,\left(x_N,y_N\right)\right\}$ and a 2D point $\mathbf{p}\left(x_i,y_i\right)\in P$, we then give the following definitions. All notations and definitions can be easily understood by referring to Figure 2.

Definition 1.

The neighbor point of $\mathbf{p}$ in the direction of θ is the point $\mathbf{q}\left(x_j,y_j\right)$, which meets the following conditions:

(1)

$\mathbf{q}\in P$;

(2)

$\varphi (\mathbf{q}-\mathbf{p})=\theta$, where $\varphi (\mathbf{q}-\mathbf{p})$ is the angle of vector $\mathbf{q}-\mathbf{p}$ related to the X axis;

(3)

$\forall \mathbf{r}\in P$, if $\varphi (\mathbf{r}-\mathbf{p})=\theta$, then $∥\mathbf{q}-\mathbf{p}∥<∥\mathbf{r}-\mathbf{p}∥$.

Definition 2.

The contour C of $\mathbf{p}$ is the set of all the neighbor points of $\mathbf{p}$ in point sets P: $C=\left\{\mathbf{x}\right|\mathbf{x}=\mathbf{q}(\mathbf{p},\theta ,P),\theta \in [0,2\pi )\}$, where $\mathbf{q}(\mathbf{p},\theta ,P)$ is the neighbor point of $\mathbf{p}$ in the direction of θ.

2.1.2. Equivariance of the PICF

Equivariant representations are highly sought after as they encode both class and deformation information in a predictable way. Similar to the equivariance defined for the image in [21], the equivariance for the point set can be defined as follows:

Equivariance for the point set: Let G be a transformation group and $L_{g}P$ be the group action applied to a point set P. A mapping $\mathsf{\Phi }:E\to F$ is said to be equivariant to the group action $L_g,g\in G$, if:

$$\mathsf{\Phi }\left(L_{g}P\right)=L_g^{\prime }\left(\mathsf{\Phi }\left(P\right)\right),$$

(1)

where $L_g$ and $L_g^{\prime }$ correspond to the application of g to E and F, respectively, and satisfy $L_{gh}=L_g{L}_h$.

In the case of the PICF, the mapping $\mathsf{\Phi }$ aims to extract the contour of a given point $\mathbf{p}$, and we will show that this mapping is equivariant to the 2D projective transformation group.

The equivariance of contour $C(\mathbf{p},P)$ under the 2D projective transformation can be defined equally as follows:

Theorem 1.

Let $P^{\prime }$ be the point set after applying the 2D projective transformation H to P, ${\mathbf{p}}^{\prime }$ be the corresponding point of $\mathbf{p}$, and $C(\mathbf{p},P),C^{\prime }\left({\mathbf{p}}^{\prime },P^{\prime }\right)$ be the contour of $\mathbf{p}$ and ${\mathbf{p}}^{\prime }$, respectively, for any point $\mathbf{x}\in C(\mathbf{p},P)$; there exists a point ${\mathbf{x}}^{\prime }\in C^{\prime }\left({\mathbf{p}}^{\prime },P^{\prime }\right)$, which satisfies ${\mathbf{x}}^{\prime }=H\left(\mathbf{x}\right)$, and equally, for any point ${\mathbf{x}}^{\prime }\in C^{\prime }\left({\mathbf{p}}^{\prime },P^{\prime }\right)$, there exists a point $\mathbf{x}\in C(\mathbf{p},P)$, which satisfies $\mathbf{x}=H^{-1}\left({\mathbf{x}}^{\prime }\right)$, where $H^{-1}$ is the inverse transformation of H.

Before the proof of Theorem 1, we introduce another theorem.

Theorem 2.

Let l be the line passing through points ${\mathbf{x}}_1,{\mathbf{x}}_2$, $\mathbf{x}=\lambda {\mathbf{x}}_1+(1-\lambda ){\mathbf{x}}_2$ be another point on line l, and ${\mathbf{x}}_1^{\prime },{\mathbf{x}}_2^{\prime },{\mathbf{x}}^{\prime }$ be the corresponding points of ${\mathbf{x}}_1,{\mathbf{x}}_2,\mathbf{x}$ under the homography transformation H. ${\mathbf{x}}^{\prime }$ can be computed with ${\mathbf{x}}_1^{\prime },{\mathbf{x}}_2^{\prime }$ as ${\mathbf{x}}^{\prime }={\lambda }^{\prime }{\mathbf{x}}_1^{\prime }+(1-{\lambda }^{\prime }){\mathbf{x}}_2^{\prime }$, where:

$${\lambda }^{\prime }\in \left\{\begin{array}{c}(-\infty ,0],if\lambda \in (-\infty ,0]\\ (0,1],if\lambda \in (0,1]\\ [1,+\infty ),if\lambda \in [1,+\infty )\end{array}\right..$$

(2)

Theorem 2 shows us the fact that the relative order of points on a line stays the same under the homography transformation. The proof of Theorem 2 is provided in Appendix A.

Proof of Theorem 1.

Let ${\mathbf{x}}_1,{\mathbf{x}}_2,...,{\mathbf{x}}_{N_{\theta }}$ be the ordered points on the line $l_{\theta }=\mathbf{p}+\lambda {[cos\theta ,sin\theta ]}^T,\lambda \in {\mathbb{R}}^{+}$ in the point set P, where $∥{\mathbf{x}}_i-\mathbf{p}∥<∥{\mathbf{x}}_j-\mathbf{p}∥$, if $i<j$, and ${\mathbf{x}}_1^{\prime },{\mathbf{x}}_2^{\prime },...,{\mathbf{x}}_{N_{\theta }}^{\prime }$ be the corresponding points of ${\mathbf{x}}_1,{\mathbf{x}}_2,...,{\mathbf{x}}_{N_{\theta }}$ under the homography transformation H. As is illustrated by Proposition 2, ${\mathbf{x}}_1^{\prime },{\mathbf{x}}_2^{\prime },...,{\mathbf{x}}_{N_{\theta }}^{\prime }$ are collinear, and $∥{\mathbf{x}}_i^{\prime }-{\mathbf{p}}^{\prime }∥<∥{\mathbf{x}}_j^{\prime }-{\mathbf{p}}^{\prime }∥$ if $i<j$, which means ${\mathbf{x}}_1^{\prime }$ is the point closest to ${\mathbf{p}}^{\prime }$ in the direction of angle ${\theta }^{\prime }$. Therefore, ${\mathbf{x}}_1^{\prime }$ is a point on the contour of $C^{\prime }({\mathbf{p}}^{\prime },P^{\prime })$; that is to say, there exists a corresponding point ${\mathbf{x}}^{\prime }\in C^{\prime }\left({\mathbf{p}}^{\prime },P^{\prime }\right)$ for any point $\mathbf{x}\in C(\mathbf{p},P)$, which satisfies ${\mathbf{x}}^{\prime }=H\left(\mathbf{x}\right)$. Since the homography transformation is reversible, for any point ${\mathbf{x}}^{\prime }\in C^{\prime }\left({\mathbf{p}}^{\prime },P^{\prime }\right)$, there exists a point $\mathbf{x}\in C(\mathbf{p},P)$, which satisfies $\mathbf{x}=H^{-1}\left({\mathbf{x}}^{\prime }\right)$, where $H^{-1}$ is the inverse transformation of H.    □

2.1.3. The PICF Descriptor

At this point, we have introduced the definition of the PICF and proven its equivariance under the 2D projective transformation. Even though we know that the points on the contour $C,C^{\prime }$ correspond one-to-one, we still do not know the definite corresponding point of a certain point. Therefore, it is necessary to align the contour point sets to the same direction and scale in order to encode the corresponding PICFs with the same descriptors.

Let the local coordinate relative to point $\mathbf{p}$ of a point ${\mathbf{x}}_i$ on the contour of $\mathbf{p}$ be ${\mathbf{x}}_i^{\mathbf{p}}={\mathbf{x}}_i-\mathbf{p}$ and the local coordinate of its corresponding point ${\mathbf{x}}_i^{\prime }$ under the 2D projective transformation $\mathbf{H}=\left[\begin{array}{cc}\mathbf{A}\hfill & \mathbf{t}\hfill \\ {\mathbf{s}}^T\hfill & 1\hfill \end{array}\right]$ be ${\mathbf{x}}_i^{{\mathrm{p}}^{\prime }}={\mathbf{x}}_i^{\prime }-{\mathbf{p}}^{\prime }$. Since the homography transformation $\mathbf{H}$ projects point ${\mathbf{x}}_i$ to ${\mathbf{x}}_i^{\prime }$, we can get:

$${\mathbf{x}}_i^{{\mathrm{p}}^{\prime }}=\frac{\mathbf{A}{\mathbf{x}}_i+\mathbf{t}}{{\mathbf{s}}^T{\mathbf{x}}_i+1}-\frac{\mathbf{A}\mathbf{p}+\mathbf{t}}{{\mathbf{s}}^T\mathbf{p}+1}.$$

(3)

Since the projective transformation $\mathbf{H}$ can be approximated to the affinity transformation in most applications in remote sensing, which means ${\mathbf{s}}^T$ is close to the zero vector, the transformation between ${\mathbf{x}}_i^{\mathbf{p}}$ and ${\mathbf{x}}_i^{{\mathbf{p}}^{\prime }}$ can be approximated as:

$${\mathbf{x}}_i^{{\mathbf{p}}^{\prime }}=\mathbf{A}{\mathbf{x}}_i^{\mathbf{p}}.$$

(4)

Equation (4) emphasizes an important fact that the transformation between the local coordinate ${\mathbf{x}}_i^{\mathbf{p}}$ and ${\mathbf{x}}_i^{{\mathbf{p}}^{\prime }}$ can be approximated as a linear transformation, which can be presented with a $2\times 2$ matrix. Since an undetermined $2\times 2$ matrix has 4 parameters and each point-to-point correspondence accounts for two constraints, it is necessary to specify 2 point correspondences in order to constrain the transformation fully.

Even though we cannot obtain the exact correspondences between two contour point sets, we know that they correspond one-to-one; that is to say, for a certain point ${\mathbf{x}}_i^{\mathbf{p}}$, there exist a certain point ${\mathbf{x}}_j^{{\mathbf{p}}^{\prime }}$ that is corresponding to it, which satisfies ${\mathbf{x}}_j^{{\mathrm{p}}^{\prime }}=\mathbf{A}{\mathbf{x}}_i^{\mathbf{p}}$. Adding all the points up, we get:

$$\sum _j^n{\mathbf{x}}_j^{{\mathbf{p}}^{\prime }}=\sum _i^n{\mathbf{Ax}}_i^{\mathbf{p}}.$$

(5)

Let:

$${\overline{\mathbf{x}}}^{\mathbf{p}}=\frac{1}{n}\sum _i^n{\mathbf{x}}_i^{\mathbf{p}},{\overline{\mathbf{x}}}^{{\mathbf{p}}^{\prime }}=\frac{1}{n}\sum _i^n{\mathbf{x}}_i^{{\mathbf{p}}^{\prime }},$$

(6)

be the centroid of contour C and $C^{\prime }$, respectively; we get:

$${\overline{\mathbf{x}}}^{{\mathbf{p}}^{\prime }}=\mathbf{A}{\overline{\mathbf{x}}}^{\mathbf{p}}.$$

(7)

The line l passing through the center point $\mathbf{p}$ and the centroid ${\overline{\mathbf{x}}}^{\mathbf{p}}$ of contour C divides the contour into two parts $P_1=\left\{{\mathbf{x}}_1^{\mathbf{p}},{\mathbf{x}}_2^{\mathbf{p}},...,{\mathbf{x}}_{\overline{n}}^{\mathbf{p}}\right\}$ and $P_2=\left\{{\mathbf{x}}_{\overline{n}+1}^{\mathbf{p}},{\mathbf{x}}_{\overline{n}+2}^{\mathbf{p}},...,{\mathbf{x}}_n^{\mathbf{p}}\right\}$, and similarly, the line $l^{\prime }$ passing through the center point ${\mathbf{p}}^{\prime }$ and the centroid ${\overline{\mathbf{x}}}^{{\mathbf{p}}^{\prime }}$ of contour $C^{\prime }$ divides the contour $C^{\prime }$ into two parts $P_1^{\prime }=\left\{{\mathbf{x}}_1^{{\mathbf{p}}^{\prime }},{\mathbf{x}}_2^{{\mathbf{p}}^{\prime }},...,{\mathbf{x}}_{{\overline{n}}^{\prime }}^{{\mathbf{p}}^{\prime }}\right\}$ and $P_2^{\prime }=\left\{{\mathbf{x}}_{{\overline{n}}^{\prime }+1}^{{\mathbf{p}}^{\prime }},{\mathbf{x}}_{{\overline{n}}^{\prime }+2}^{{\mathbf{p}}^{\prime }},...,{\mathbf{x}}_n^{{\mathbf{p}}^{\prime }}\right\}$. Since the points correspond one-to-one and the line l and $l^{\prime }$ also correspond, the points in $P_1$ should also correspond one-to-one to those in $P_1^{\prime }$ or $P_2^{\prime }$. Similar to the definition of the centroid of the whole contour, we define the centroid of the half-contour as:

$$\left\{\begin{array}{c}{\overline{\mathbf{x}}}_1^{\mathbf{p}}=\frac{1}{\overline{n}}{\displaystyle \sum _{i=1}^{\overline{n}}}{\mathbf{x}}_i^{\mathbf{p}}\\ {\overline{\mathbf{x}}}_2^{\mathbf{p}}=\frac{1}{n-\overline{n}}{\displaystyle \sum _{i=\overline{n}}^n}{\mathbf{x}}_i^{\mathbf{p}}\end{array}\right.,\mathrm{and}\left\{\begin{array}{c}{\overline{\mathbf{x}}}_1^{{\mathbf{p}}^{\prime }}=\frac{1}{{\overline{n}}^{\prime }}{\displaystyle \sum _{i=1}^{{\overline{n}}^{\prime }}}{\mathbf{x}}_i^{{\mathbf{p}}^{\prime }}\\ {\overline{\mathbf{x}}}_2^{{\mathbf{p}}^{\prime }}=\frac{1}{n-{\overline{n}}^{\prime }}{\displaystyle \sum _{i={\overline{n}}^{\prime }}^n}{\mathbf{x}}_i^{{\mathbf{p}}^{\prime }}\end{array}\right..$$

(8)

Then, there exist two possible correspondences: ${\overline{\mathbf{x}}}_1^{\mathbf{p}}\leftrightarrow {\overline{\mathbf{x}}}_1^{{\mathbf{p}}^{\prime }},{\overline{\mathbf{x}}}_2^{\mathbf{p}}\leftrightarrow {\overline{\mathbf{x}}}_2^{{\mathbf{p}}^{\prime }}$ and ${\overline{\mathbf{x}}}_1^{\mathbf{p}}\leftrightarrow {\overline{\mathbf{x}}}_2^{{\mathbf{p}}^{\prime }},{\overline{\mathbf{x}}}_2^{\mathbf{p}}\leftrightarrow {\overline{\mathbf{x}}}_1^{{\mathbf{p}}^{\prime }}$. Therefore, we get two possible constraints ${\overline{\mathbf{x}}}_1^{{\mathbf{p}}^{\prime }}={\mathbf{A}\overline{\mathbf{x}}}_1^{\mathbf{p}}$ or ${\overline{\mathbf{x}}}_2^{{\mathbf{p}}^{\prime }}={\mathbf{A}\overline{\mathbf{x}}}_1^{\mathbf{p}}$.

With the correspondences of the centroids of the contours and centroids of the half-contours, we get two groups of possible constraints,

$$\left\{\begin{array}{c}{\overline{\mathbf{x}}}^{{\mathbf{p}}^{\prime }}=\mathbf{A}{\overline{\mathbf{x}}}^{\mathbf{p}}\\ {\overline{\mathbf{x}}}_1^{{\mathbf{p}}^{\prime }}={\mathbf{A}\overline{\mathbf{x}}}_1^{\mathbf{p}}\end{array}\right.,\left\{\begin{array}{c}{\overline{\mathbf{x}}}^{{\mathbf{p}}^{\prime }}=\mathbf{A}{\overline{\mathbf{x}}}^{\mathbf{p}}\\ {\overline{\mathbf{x}}}_2^{{\mathbf{p}}^{\prime }}={\mathbf{A}\overline{\mathbf{x}}}_1^{\mathbf{p}}\end{array}\right..$$

(9)

with which the $2\times 2$ transformation matrix can be determined. The contours can be aligned to the same coordinate with the transformations so that we can get a projective-invariant description of the contour.

As is shown in Figure 3, the detailed method to compute the PICF descriptor is:

(1)

Sample contour points: Divide the area centered at point $\mathbf{p}$ into N areas with angle step ${\theta }_s=\frac{2\pi }{N}$; for each area $[\frac{2\pi k}{N},\frac{2\pi (k+1)}{N})$; find the closest point ${\mathbf{x}}_{\mathbf{k}}$ to $\mathbf{p}$; and take ${\mathbf{x}}_{\mathbf{k}}$ as the sample point on the contour in direction $\frac{2\pi (2k+1)}{2N}$; compute the relative coordinate of ${\mathbf{x}}_{\mathbf{k}}$ to point $\mathbf{p}$ as ${\mathbf{x}}_k^{\mathbf{p}}$; if no point exists in the area, then set ${\mathbf{x}}_k^{\mathbf{p}}={\left[0,0\right]}^T$.

(2)

Calculate the transformation matrix: Compute the centroids of the contour ${\overline{\mathbf{x}}}^{\mathbf{p}}$ and the centroids of the half-contour ${\overline{\mathbf{x}}}_1^{\mathbf{p}},{\overline{\mathbf{x}}}_2^{\mathbf{p}}$ with Equations (6) and (8). Solve Equation (10) to get the two possible transformation matrices ${\mathbf{A}}_{\mathbf{1}}$ and ${\mathbf{A}}_{\mathbf{2}}$:

$$\left\{\begin{array}{c}{\mathbf{A}}_1{\overline{\mathbf{x}}}^{\mathbf{p}}={\mathbf{x}}_1\\ {\mathbf{A}}_1{\overline{\mathbf{x}}}_1^{\mathbf{p}}={\mathbf{x}}_2\end{array}\right.,\left\{\begin{array}{c}{\mathbf{A}}_2{\overline{\mathbf{x}}}^{\mathbf{p}}={\mathbf{x}}_1\\ {\mathbf{A}}_2{\overline{\mathbf{x}}}_2^{\mathbf{p}}={\mathbf{x}}_2\end{array}\right..$$

(10)

where ${\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}}$ are two constant vectors and are taken as ${\mathbf{x}}_1={[1,0]}^T,{\mathbf{x}}_2={[0,1]}^T$ in our implementation.

(3)

Align contour points: Project all contour points with ${\mathbf{A}}_{\mathbf{1}}$ and ${\mathbf{A}}_{\mathbf{2}}$, respectively, and get two transformed point sets $C_1=\{{\mathbf{A}}_1{\mathbf{x}}_k^{\mathbf{p}},k=1,2,...,N\}$ and $C_2=\{{\mathbf{A}}_2{\mathbf{x}}_k^{\mathbf{p}},k=1,2,...,N\}$.

(4)

Compute the PICF descriptor: Convert the coordinates of the points in point sets $C_i,i=1,2$ to the polar coordinates, and sort the points with their angles in ascending order, then we get $C_i=\{({\theta }_k^i,{\rho }_k^i),k=1,2,...,N\},i=1,2$. For each area $[\frac{2\pi k}{N^{\prime }},\frac{2\pi (k+1)}{N^{\prime }})$, compute the average distance of contour points in the area to the center point as: ${\overline{\rho }}_j^i=\frac{1}{N_j^i}{\displaystyle \sum _{{\theta }_k^i\in [j{\theta }_s^{\prime },(j+1){\theta }_s^{\prime })}}{\rho }_k^i$, where $N^{\prime }$ is the dimension of the descriptor, and ${\theta }_s^{\prime }=2\pi /N$’. Finally, the two possible PICF descriptors of point $\mathbf{p}$ in the point set P are obtained as: ${\mathbf{d}}_{C(\mathbf{p},P)}^i={\left[{\overline{\rho }}_1^i,{\overline{\rho }}_2^i,...,{\overline{\rho }}_{N^{\prime }}^i\right]}^T,i=1,2$.

2.1.4. Discussion

As proven in Section 2.1.2, the equivariance of the PICF under the 2D projective transformation holds strictly true when the direction angle $\theta$ is continuous. To represent the contour point set with the vector, as introduced in Section 2.1.3, the contour point set is sampled with discrete $\theta$. The strict equivariance does not hold in such a case, but the equivariance is approximately satisfied when the sample step ${\theta }_s$ is small enough. For each point on the contour, there exist points that are close enough to its corresponding point on the transformed contour as long as the sample step ${\theta }_s$ is small enough, which means the approximate corresponding point always exists. Therefore, in our implementation, the number of sample contour points is set to $4\times$ the dimension of the PICF descriptor to make the sample contour a better approximation of the full contour. Besides, we choose the closest points to the center in each small direction interval as the sample points of the contour, which makes the estimated PICF descriptors less sensitive to small breaks of the detected roads. These measures make approximate equivariance hold well in practical applications, which is also proven by our experiments.

2.2. Matching PICFs with Geometric Coherence

At this point, given a point set P and a center point $\mathbf{p}$, we can compute a projective-invariant descriptor, which encodes the local point cloud information. With such a descriptor, many fast retrieval algorithms, such as the FLANN [22], are sufficient to build the correspondences of two PICF sets. However, large amounts of outliers exist in the correspondences estimated from such methods, which leads to more unnecessary tries in the following transformation refining and validation. To solve the problem, we model the PICFs matching as an optimization problem to minimize the cost function, which penalizes the differences between the PICF descriptors and also the geometric inconsistency between two matching pairs.

2.2.1. Geometric Coherence between PICF Correspondences

Let $m_i=\{C_i^q({\mathbf{p}}_i^q,{\mathbf{A}}_i^q,P^q,{\mathbf{d}}_i^q)\leftrightarrow C_i^r({\mathbf{p}}_i^r,{\mathbf{A}}_i^r,P^r,{\mathbf{d}}_i^r)\}$ be a matching PICF pair, where ${\mathbf{A}}_i^r,{\mathbf{A}}_i^q$ are the transformation matrices in computing the PICF descriptors of ${\mathbf{p}}_i^q,{\mathbf{p}}_i^r$, respectively, and ${\mathbf{d}}_i^q,{\mathbf{d}}_i^r$ are the corresponding PICF descriptors. For a certain point ${\mathbf{x}}_j^{{\mathbf{p}}_i^q}\in C_i^q$ and its corresponding point ${\mathbf{x}}_{j^{\prime }}^{{\mathbf{p}}_i^r}\in C_i^r$, the transformation between them is:

$${\mathbf{A}}_i^q{\mathbf{x}}_j^{{\mathbf{p}}_i^q}={\mathbf{A}}_i^r{\mathbf{x}}_{j^{\prime }}^{{\mathbf{p}}_i^r}.$$

(11)

Multiply both sides of Equation (11) by ${\left({\mathbf{A}}_i^r\right)}^{-1}$; we get ${\mathbf{x}}_{j^{\prime }}^{{\mathbf{p}}_i^r}={\left({\mathbf{A}}_i^r\right)}^{-1}{\mathbf{A}}_i^q{\mathbf{x}}_j^{{\mathbf{p}}_i^q}$. Let ${\mathbf{A}}_i={\left({\mathbf{A}}_i^r\right)}^{-1}{\mathbf{A}}_i^q$, then for all PICF correspondences, their corresponding ${\mathbf{A}}_i$ are equal, as deduced in Section 2.1.3.

Considering the geometric consistency between matching pairs, the matching between two PICF sets $S^q=\{C_i^q({\mathbf{p}}_i^q,{\mathbf{A}}_i^q,P^q,{\mathbf{d}}_i^q),i=1,2,...,n\}$ and $S^r=\{C_i^r({\mathbf{p}}_i^r,{\mathbf{A}}_i^r,P^r,{\mathbf{d}}_i^r),$$i=1,2,...,m\}$ can be modeled as a problem to minimize the cost function:

$$E=\sum _{j=1}^n∥{\mathbf{d}}_j^q-{\mathbf{d}}_{i\left(j\right)}^r∥+\lambda \sum _{j=1}^n\sum _{k=1}^n∥(\mathbf{A}{}_{i\left(j\right)}^r{)}^{-1}{\mathbf{A}}_j^q-(\mathbf{A}{}_{i\left(k\right)}^r{)}^{-1}{\mathbf{A}}_k^q∥,$$

(12)

where $i\left(j\right),i\left(k\right)$ are the indexes of the corresponding PICF in $S^r$ of $j$th and $k$th PICF in $S^q$, respectively. The first term of Equation (12) is the data term, which models the matching distance between PICF descriptors, and the second term is the smooth term, which penalizes the geometric inconsistency between two matching pairs.

2.2.2. Refining Initial Matches with Geometric Coherence

It is difficult to solve Equation (12) directly; instead, we seek a two-stage solution. We first find the best matching for each PICF and then verify the correctness of the correspondences with both the matching distances and geometric consistency, as described in Section 2.2.1.

Let $M=\{m_i=\{C_i^r\leftrightarrow C_i^q\},C_i^r\in S^r,C_i^q\in S^q\}$ be the initial correspondences found with descriptors matching, $\varphi \left(m_i\right)$ be the matching distance of $m_i$, and ${\mathbf{A}}_i={\left({\mathbf{A}}_i^r\right)}^{-1}{\mathbf{A}}_i^q$ be the corresponding transformation matrix of $m_i$. We define a label function $L:m_i\to \{0,1\}$, which maps $m_i$ to 1 if the matching is correct, and 0 otherwise. Then, the PICF matching problem is equivalent to finding the labels of all initial correspondences that minimize both the matching distances and geometric inconsistency.

$$\begin{array}{c}E={\displaystyle \sum _{i=1}^n}e\left(L\left(m_i\right),K_d\left(\varphi \left(m_i\right)\right)\right)\hfill \\ +\lambda {\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^n}e\left(L\left(m_i\right),L(m_j),K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)\right)\hfill \end{array},$$

(13)

where $K_d\left(x,{\sigma }_d\right)=e^{-\frac{x^2}{2{\sigma }_d^2}}$, $K_s\left(x,{\sigma }_s\right)=e^{-\frac{x^2}{2{\sigma }_s^2}}$ are the kernel functions. We use the kernel function to normalize both the data term (matching distances) and the smooth term (geometric inconsistency) to improve the robustness. $∥{\mathbf{A}}_i-{\mathbf{A}}_j∥$ is the distance between ${\mathbf{A}}_i$ and ${\mathbf{A}}_j$ and is calculated with the method proposed in [23].

The data term $e\left(L\left(m_i\right),K_d\left(\varphi \left(m_i\right)\right)\right)$ is defined as:

$$e\left(L\left(m_i\right),K_d\left(\varphi \left(m_i\right)\right)\right)=\left\{\begin{array}{ccc}1-K_d\left(\varphi \left(m_i\right)\right)& if& L\left(m_i\right)=1\\ K_d\left(\varphi \left(m_i\right)\right)& if& L\left(m_i\right)=0\end{array}\right..$$

(14)

This energy term penalizes matching pairs whose labels are 1, but with large matching distances, and those whose labels are 0, but with small matching distances.

The smooth term $e\left(L\left(m_i\right),L\left(m_j\right),K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)\right)$ is defined as:

$$\begin{array}{c}e\left(L\left(m_i\right),L\left(m_j\right),K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)\right)=\hfill \\ \left\{\begin{array}{ccc}1& if& L\left(m_i\right)\ne L\left(m_j\right)\\ K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)& if& L\left(m_i\right)=L\left(m_j\right)=0\\ 1-K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)& if& L\left(m_i\right)=L\left(m_j\right)=1\end{array}\right.\hfill \end{array}.$$

(15)

In $e\left(L\left(m_i\right),K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)\right)$, if both correspondences are labeled as outliers, the penalty is $K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)$, thus “rewarding” label 0 if the geometric inconsistency is big. The penalty of considering a pair of correspondences as the inlier is $1-K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)$, which rewards the labels if the transformation matrices of the two correspondences are similar.

Equation (13) can be solved efficiently and globally via the graph-cut algorithm [18,19,20]. The complete matching algorithm considering geometric consistency is illustrated in Algorithm 1.

Algorithm 1 PICF matching algorithm.

Input:  $S^r$, the reference PICF set, and $S^q$, the query PICF set. Output:  $M\left\{m_i\left(c_i^r\leftrightarrow c_i^q\right)\right\}$, the PICF correspondence set.   1: Build the search index with all the reference PICF descriptors. 2: $M_{initial}\leftarrow$ EmptyMatchSet(). 3: for all$c_i^q\in S^q$do 4:  $c_i^r\leftarrow$ Find the matching PICF in $S^r$ with the minimum descriptor matching distance $\varphi \left(c_i^r\leftrightarrow c_i^q\right)$. 5:  $M_{initial}\leftarrow$Add a new matching pair $m_i\left(c_i^r\leftrightarrow c_i^q\right)$. 6:  ${\mathbf{A}}_i\leftarrow$ Compute the transformation matrix. 7: end for  8: $G\leftarrow$ EmptyGraph(). 9: for all$m_i\in M_{initial}$do 10:  $c_0,c_1\leftarrow K_d\left(\varphi \left(m_i\right)\right),1-K_d\left(\varphi \left(m_i\right)\right)$ 11:  $G\leftarrow AddTerm1(G,c_0,c_1)$ 12: end for 13: for all$m_i,m_j\in M_{initial}$do 14:  $c_{01},c_{10}\leftarrow 1,1.$ 15:  $c_{00}\leftarrow K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)$ 16:  $c_{11}\leftarrow 1-K_s\left(∥{\mathbf{A}}_i-{\mathbf{A}}_j∥\right)$ 17:  $G\leftarrow AddTerm2(G,c_{00},c_{01},c_{10},c_{11})$ 18: end for 19: $L\leftarrow$ Apply graph-cut to G. 20: $M\left\{m_i\left(c_i^r\leftrightarrow c_i^q\right)\right\}\leftarrow$ Get all the correspondences with label 1 from L.

2.3. PICF Based Fast Geolocalization

At this point, we propose the PICF, which is equivariant to the homography transformation group, and a way to describe the PICF by a vector invariant to homography transformation, with which the potential correspondences can be found effectively. Furthermore, to improve the matching, we model the PICFs matching as an optimization problem to minimize the cost function, which penalizes the differences between PICF descriptors and also the geometric inconsistency between two matching pairs, and solve it with the graph-cut algorithm sufficiently. In this section, we introduce our fast geolocalization method based on the PICF in detail. The overview of our method is shown in Figure 4.

Our fast geolocalization method contains the offline stage and the online stage. The offline stage is for processing the reference road vector map whose output is used in the online correspondence search scheme, allowing for a very fast operation. As is illustrated in Figure 4, in the offline stage, all the road intersections are detected and used as the centers of the PICFs; and then, the two possible PICF descriptors are computed for each PICF; finally, a KD-tree is built using all these PICF descriptors.

The proposed online search scheme starts by extracting the roads for a given aerial image using our method in [1]. The same as the offline stage, all the PICFs are detected, and their corresponding descriptors are calculated. The correspondences between the query PICF set and the reference PICF set are found with the geometric-consistent matching algorithm introduced in Section 2.2. These matching PICF pairs are then used to recover the homography transformations between the aerial images and reference road map with two different strategies:

(1)

If the number of matching pairs is larger than $N_c$ (20 in this paper), an initial homography transformation is estimated with all the matching pairs using the RANSAC method and then refined with the plain ICP. The refined transformation is then verified with the alignment accuracy between the query road map and the reference road map.

(2)

If the obtained feature correspondences are less than $N_c$ or the first strategy fails to recover the right transformation, the obtained pairs are afterward sorted with their matching scores. Then, a homography transformation is estimated with a single correspondence and refined using all the road points in the query road map with a local-to-global iterative closest point algorithm. Furthermore, the refined transformation is then verified with the alignment accuracy. Each correspondence in the sorted pair set will be tested until a homography transformation that aligns the query road map and the reference one accurately enough is found. If our method fails to recover the correct transformation after all the correspondences obtained with the geometric-consistent matching algorithm, the left correspondences in the initial matching will be checked in the same manner.

2.3.1. Initial Homography Transformation Estimation from One Matching Pair

Give a PICF correspondence $m_i=\{c_i^q({\mathbf{p}}_i^q,{\mathbf{A}}_i^q,P^q,{\mathbf{d}}_i^q)\leftrightarrow c_i^r({\mathbf{p}}_i^r,{\mathbf{A}}_i^r,P^r,{\mathbf{d}}_i^r)\}$, the coarse homography transformation $\mathbf{H}=\left[\begin{array}{cc}\mathbf{A}& \mathbf{t}\\ {\mathbf{s}}^T& 1\end{array}\right]$, which projects the query road point to the reference road point, can be computed as:

(1)

Compute matrix $\mathbf{A}$: $\mathbf{A}={\left({\mathbf{A}}_i^r\right)}^{-1}{\mathbf{A}}_i^q$;

(2)

Compute vector ${\mathbf{s}}^T$: the homography transformation is approximated to a affinity, so we get ${\mathbf{s}}^T=\left[0,0\right]$;

(3)

Compute the translation vector $\mathbf{t}$: since the contour center point ${\mathbf{p}}_i^q$ is projected to ${\mathbf{p}}_i^r$, we get $\mathbf{t}={\mathbf{p}}_i^r-{\mathbf{Ap}}_i^q$.

2.3.2. Transformation Refining

When the approximate homography transformation $\mathbf{H}$ is estimated from a given correct PICF correspondence, the estimated homography transformation has poor accuracy and should be refined further. This is because $\mathbf{H}$ is approximated to an affinity transformation, and the upper-left $2\times 2$ matrix $\mathbf{A}$ is estimated using only points on the contour.

The initial transformation estimated from one correspondence is refined so that it aligns the query road point set to the reference road point set accurately. The ICP algorithm is appropriate to solve such a problem. However, in our experiments, we find that the convergence rate is quite low when using the plain ICP algorithm, which may arise from the poor accuracy of the initial transformation. Since the initial transformation is estimated with the points near the center point of the PICF, it should be accurate enough for points near the center point. Inspired by [15], we use a local-to-global ICP algorithm, which first aligns the points near the center point and then aligns the points far from the center point gradually.

In the local-to-global ICP algorithm, the points whose distances to the center point are less than d in the query road point set are chosen to refine the initial transformation ${\mathbf{H}}_0$ with the standard ICP algorithm firstly. If the ICP algorithm is convergent, the refined transformation ${\mathbf{H}}_1$ is then used to align the points whose distances to the center point are less than $\sigma d(\sigma >1)$, resulting in a further refined transformation ${\mathbf{H}}_2$; otherwise, the correspondence used to estimate the initial transformation is regarded as an incorrect correspondence and is discarded. The optimization continues until all query road points are used.

2.3.3. Transformation Validation

After the transformation refining, the homography transformation estimated from a correct correspondence aligns the query road point set with the reference point set accurately, while the homography transformation estimated from an incorrect correspondence cannot. We use the inlier rate of road points in the query road map to measure the alignment accuracy. A road point is regarded as an inlier when the distance between its projection point under estimated transformation and the closest road point in the reference road map to its projection point is less than a certain threshold $\delta$. Therefore, the inlier rate can be computed as:

$$inlier\_rate=\frac{\underset{{\mathbf{x}}^q}{N}\left(∥{\mathbf{x}}^q-\underset{{\mathbf{x}}^r}{min}\left(∥H(\mathbf{H},{\mathbf{x}}^q)-{\mathbf{x}}^r∥\right)∥<\delta \right)}{N},$$

(16)

where $H(\mathbf{H},{\mathbf{x}}^q)$ is the projection point of ${\mathbf{x}}^q$ under $\mathbf{H}$.

One may argue that the transformation validation should be done with the initial transformations estimated from the correspondences before the transformation refining to remove incorrect correspondences further and to reduce unnecessary tries to find the correct transformation. However, in our experiments, we find that the inlier ratios under the coarse transformations are not sufficient to distinguish the right transformations from those that are wrong. As is shown in Figure 5, before the transformation refining, the inlier ratios are low under both the transformations estimated from the correct and incorrect correspondences, and there is not an obvious threshold of inlier ratio that can distinguish the correct transformations from these wrong ones. After the transformation refining, the inlier ratios of correct transformations are much higher than those of incorrect transformations in most of the cases.

3. Results

In this section, we firstly perform experiments on large-scale synthetic aerial image data sets to evaluate both the effectiveness of the PICF and the geolocalization method. Then, experiments on real aerial image data sets are conducted to test the performance of our algorithm on real applications.

Implementation details: Our implementation was written in C++, single-threaded. All the experiments ran on a computer with an Intel Xeon E5-2620v2 CPU and 24 GB RAM. In all the experiments, the road maps for aerial images were extracted with our road segmentation method proposed in [1] in advance. The parameters of our algorithm were set based on experiments on a validation synthetic aerial image data set and kept fixed in all experiments. The main parameters settings were: the dimension of the PICF descriptor $dim=48$; the weight for smooth energy term $\lambda =1/N$, where N is the number of initial correspondences; the parameters of the kernel function for the data term and smooth term ${\sigma }_{data}=48$ and ${\sigma }_{smooth}=2$, respectively; the maximum iterations for plain ICP $N_{icp}=120$; the distance to divide the query road points into different parts in local-to-global ICP $d=max\{w,h\}/4$, where $w,h$ are the width and height of the query road image, respectively; the ratio of the distance to expand the points to align in local-to-global ICP $\sigma =2$; the distance error threshold for a road point to be regarded as an inlier $\delta =20$; the minimum required inlier ratio to verify the geolocalization as correct $\alpha =0.7$.

Baseline and metrics: We compared our method with the state-of-the-art algorithm, the “2-road-intersections-tuple” (2RIT) based geolocalization method [5], in three aspects: precision, recall, and run time. Here, $precision=TP/(TP+FP)$, where $TP$ is the number of successfully and correctly geolocalized aerial images and $FP$ is the number of successfully, but wrongly geolocalized aerial images; $recall=TP/N$, where N is the number of all aerial images used. We used the error between the estimated homography and the ground truth one to verify whether the geolocalization was correct, which is computed with the same method as in [5]. In all experiments, the parameters for “2RIT” were set the same as reported in [5].

3.1. Experiments on Synthetic Aerial Image Data Sets

In this section, experiments on synthetic aerial image data sets are conducted to test the performance of our geolocalization method under different viewing angles, resolutions, and scenes. Since it is quite difficult to collect enough aerial images with the needed viewing angles, resolutions, and context, we followed the same manner as in [5] to validate our algorithm using synthetic aerial image data sets. The three reported synthetic aerial image data sets in [5], named “multi-poses data set”, “multi-GSDs data set”, and “multi-areas data set”, were used in our experiments to test the performance of our method under different viewing angles, resolutions, and scenes, respectively. Each data set contains synthetic aerial images from two different areas in Massachusetts. The roads in one area (the 13th area) are sparse, while roads in the other area (the 17th area) are much denser. Therefore, these data sets are representative.

As is shown in Figure 6, the precision of the two methods in both areas is higher than $90\%$ for all aerial images taken under different viewing angles. Meanwhile, our novel method improves the recall in both areas when the viewing angle is less than ${30}^{\circ }$, which is usually the case in real applications. The improvement of recall is especially significant in the 17th area where the roads are dense. Figure 6 also shows that the recall of both methods decreases with the increase of the viewing angle. However, the performance degradation of the method presented in this paper is much slower.

Figure 7 shows the matching inlier rates under different viewing angles in both areas. As can be seen, the inlier rate of the initial PICF correspondence set obtained with finding the nearest neighbor in the PICF descriptor space is acceptable. In both areas and under all view angles, the inlier rate of the initial correspondence set is no less than $10\%$, which indicates that the proposed PICF can encode the geometric structure effectively and is discriminative. It also proves that the PICF is projective-invariant. Furthermore, the geometric coherence based matching method is efficient to improve the initial matching quality, especially when the viewing angle is relatively small. As is shown in Figure 8, the improvement mainly comes from the ability of the geometric coherence based matching method to reject most of the wrong correspondences while preserving most of the right correspondences. The good performance is more obvious when the roads are dense in the search area (17th area). In such areas, the similar local structures of the road network are ubiquitous, leading to a low inlier ratio of initial matching. It is thus more meaningful to use the geometric coherence constraint. The improvement of the matching inlier rate is significant to reduce the unnecessary tries in the following geometric consistency validation and to improve the efficiency. Even though the geometric coherence based matching method is less effective when the viewing angle is large, since it may reject all the right matching pairs for some aerial images in such a case, leading to an inlier rate of zero, we try to recover the homography transformation from the rest candidates to ensure our algorithm can work in such a case.

We also conducted experiments on the “multi-GSDs data set” to evaluate the influence of Ground Sample Distance (GSD) on the geolocalization. The result is shown in Figure 9. It can be seen that the recall of the “2RIT” method decreases sharply when the aerial images are captured with small GSDs, which indicates that the “2RIT” is sensitive to the GSD. Our PICF based geolocalization method is less sensitive to the change of GSD since the recall varies much more slightly with the GSD compared with the “2RIT” method.

To study the performance of our method on different scenes, we conducted experiments on the “multi-areas data set”, which contains as many as 2000 aerial images from 20 areas. The precision and recall are shown in Figure 10. As can be seen, in all 20 areas, the precision of both methods is higher than $95\%$. The recall of our newly proposed method is improved significantly in 19 out of 20 areas. In 12 areas, the recall is higher than $90\%$. We highlight the improvement of recall in the first and seventh area. Several big cities lie in these two areas, leading to very dense roads. The similar structures of the road network in these two areas are ubiquitous. In such a case, the “2RIT” method may fail, since there exist too many candidates for each “2-road-intersections-tuple”, while our new algorithm performs well in such case. When a large amount of matching PICF pairs is found, it is robust and accurate enough to estimate the initial homography transformation using all these pairs. Such a strategy makes our algorithm effective at dealing with such cases. The only area where the recall of the new method is less than the “2RIT” method is the 13th area, and this is mainly caused by the too sparse roads in some aerial images, leading to few PICFs found in those aerial images.

The run time of geolocalization for both methods is shown in Figure 11. In 17 out 20 areas, the average geolocalization time method proposed in this paper is less than 1 s and is less than 2 s in all 20 areas, while the average run time for “2RIT” is usually several seconds or even tens of seconds in areas like the seventh area and the 12th area. In all areas, the average run time of our new algorithm is less than that of “2RIT”. Besides, the run time of the PICF based geolocalization method varies more slightly between different aerial images in the same area compared with “2RIT”.

3.2. Experiments on Real Aerial Image Data Sets

We also conducted experiments on real aerial images captured with cameras on UAVs to test the performance of our algorithm in real applications. We collected aerial images from three areas over China, including Suizhou, Xian, and Fengyang. The aerial images were captured in the form of videos. The details of the videos are shown in

Table 1. Details of the real flight aerial image data set.

Video Number	Localization	Maximum Altitude (m)	Trajectory Length (km)	Video Resolution
1	Suizhou, Hubei	1000	4.325	4096 × 2160
2	Suizhou, Hubei	1000	4.476	4096 × 2160
3	Suizhou, Hubei	1000	4.557	4096 × 2160
4	Suizhou, Hubei	1000	1.543	4096 × 2160
5	Suizhou, Hubei	1000	4.238	4096 × 2160
6	Suizhou, Hubei	1000	4.366	4096 × 2160
7	Suizhou, Hubei	1000	1.442	4096 × 2160
8	Suizhou, Hubei	1000	4.493	4096 × 2160
9	Suizhou, Hubei	1000	4.601	4096 × 2160
10	Suizhou, Hubei	1000	4.512	4096 × 2160
11	Suizhou, Hubei	1000	4.272	4096 × 2160
12	Suizhou, Hubei	1000	3.023	4096 × 2160
14	Xian, Shanxi	500	1.732	1920 × 1080
15	Xian, Shanxi	500	7.387	1920 × 1080
16	Xian, Shanxi	350	6.125	1920 × 1080
17	Xian, Shanxi	439	8.885	1920 × 1080
18	Xian, Shanxi	442	8.569	1920 × 1080
19	Xian, Shanxi	444	8.491	1920 × 1080
20	Fengyang, Anhui	400	4.831	3840 × 2160
21	Fengyang, Anhui	400	4.687	3840 × 2160
22	Fengyang, Anhui	400	2.032	3840 × 2160

. These videos were then downsampled with one frame every 48 frames. All the selected aerial images in Suizhou were then resized to the resolution of $1024\times 540$, and aerial images in the other two areas were resized to $960\times 540$, while the final GSDs of those aerial images ranged from 0.3 m/pixel to 1.0 m/pixel. The camera internal parameters were calibrated, and the distortion of the images was corrected offline.

Since the Field Of Vision (FOV) of the camera is less than ${70}^{\circ }$, the observed ground area in one single aerial image cannot cover enough roads. The roads in individual aerial images are not enough to determine their positions uniquely. Therefore, we stitched the aerial image sequences into bigger aerial mosaics to ensure that each mosaic contained enough roads. The final mosaics covered an area of 0.5 km² to 3.0 km². Finally, we obtained 21, 24 and 7 aerial image mosaics in Suizhou, Xian, and Fengyang, respectively. We then performed road segmentation to generate the corresponding road images for each mosaic, and those road images were then used to perform geolocalization.

All the reference road maps used in the geolocalization were downloaded from the Gaode map website (https://wprd01.is.autonavi.com/appmaptile?x=x&y=y&z=z&lang=zh_cn&size=1&scl=2&style=8&ltype=11) with QGIS, which were saved in the raster map format with the resolution of 1.0 m/pixel. Each reference road map was large enough to cover the area where the aerial images were captured. The detailed information for the reference maps is shown in

Table 2. Details of reference road map for each area. The longitude and latitude are measured under WGS 84.

Localization	Longitude Range	Latitude Range	Area
Suizhou, Hubei	[113.0000, 113.1792]	[32.0334, 31.6584]	54.99 km × 37.97 km
Xian, Shanxi	[108.9296, 109.1767]	[34.0707, 34.2053]	22.46 km × 14.90 km
Fengyang, Anhui	[117.4997, 117.8000]	[32.7357, 33.0812]	28.30 km × 38.120 km

.

We conducted geolocalization on the generated road images with both the “2RIT” method and our new method. Since the ground truth of homography transformation between the aerial image mosaics and the reference road map cannot be obtained, we verified whether the geolocalization is correct by projecting the aerial images onto the reference road map and checking visually whether they overlap with the reference road map well. The result is shown in

Table 3. Geolocalization result on the real flight aerial image data set.

Localization	Total Number of Mosaics	Number of Successful Geolocalized Mosaics
		2RIT	Ours
Suizhou, Hubei	21	14	16
Xian, Shanxi	24	14	20
Fengyang, Anhui	7	5	7

.

In all three areas, the proposed method outperformed the “2RIT”. The improvement is especially significant in Xian and Fengyang, where the recall increased by 42.86% and 28.57%, respectively. Compared with those in Suizhou, the GSDs of the mosaics in the two areas were higher. The width of roads may be more than 40 pixels in the road images for the main streets, making it difficult to estimate the tangents accurately. In such cases, the “2RIT” method failed. Our newly designed PICF is insensitive to road width, as is shown in the experiments on the synthetic data sets, which makes our new method more robust. Some cases where the “2RIT” failed while the proposed method succeeded are shown in Figure 12. The proposed method failed when no complete PICF was found in the scene. The PICF becomes less valuable when there exist too many directions in which no contour point is found, leading to the failure of geolocalization. A camera with a larger FOV may overcome this deficiency, which is left for further validation in our future work.

4. Discussion

The results of the experiments on both synthetic and real aerial image datasets show that our newly proposed PICF based geolocalization method improves the recall and run time to a large extent compared with the state-of-the-art “2RIT” method. Furthermore, the results indicate that the PICF based geolocalization method is more robust to differences of viewing angle and GSD between the aerial images and the reference road map. In this section, we discuss the main factors that contribute to these improvements in detail.

The improvement of recall mainly comes from two aspects. First, the proposed PICF is projective-invariant and encoded with a projective-invariant descriptor, which means the representation of the PICF is robust to viewing angle change. Moreover, the PICF can encode much more geometric information of the road network compared with the “2-road-intersections-tuple”, making it more efficient to establish initial correspondences, which can be clearly seen from Figure 7. Since few outliers exist in initial correspondences, it is possible to use time-consuming optimization methods, such as the ICP, to refine each estimated initial homography transformation in an acceptable run time. With more accurate homography transformation, the geometric consistency check is much more efficient, as is discussed in Figure 5. In the “2RIT”, there usually exist hundreds or even thousands of matching candidates, making it unreasonable to refine each hypothesis generated by the feature correspondences. Therefore, the “2RIT” depends heavily on the accuracy of the initial homography transformation estimation. A correct, but less precise hypothesis may be rejected wrongly when the estimation of the tangents of road branches is noisy. The other factor contributing to the improvement lies in the effectiveness of the proposed homography transformation refining strategy, especially the idea of recovering the initial homography transformation from just one matching pair and refining it with the local-to-global ICP method. Such a strategy makes our method feasible in more cases, even when the roads are sparse or the cover areas of the aerial images are relatively small.

The greater robustness of our new algorithm to changes of viewing angle and GSD also benefit from the good performance of the proposed PICF. On the one hand, the invariance of the PICF is proven mathematically. Therefore, the encoding of the PICF is robust to changes in viewing angle. On the other hand, no accurate geometric information, such as the tangent, is needed in the computing of the descriptor of the PICF, which makes the PICF descriptor less sensitive to road segmentation noise. In the “2RIT” method, the tangents of road branches are required to estimate precisely, which is hard when the road segmentation is not smooth or the GSD of the aerial image is small. Since when the GSD is small, the width of the road is usually dozens of pixels in aerial images, making it difficult to extract the centerlines of roads accurately. In such a case, the initial transformation is noise and may be rejected wrongly even when the correspondence between the “2-road-intersections-tuple” is correct.

Three factors contribute to the improvement of the efficiency. First, the PICF encodes more geometric information of roads and is more discriminative, leading to much fewer matching candidates. Second, the geometric coherence based matching method rejects most of the incorrect correspondences and improves the matching quality further. Last, the strategy of using all the matching pairs or only one matching pair to recover the initial transformation is elastic to deal with all cases despite the density of roads, making our method quite efficient to find the right location.

5. Conclusions

This paper presents an aerial image geolocalization method, which uses only the road vector map as the reference. The new method is capable of efficiently and effectively estimating the transformation aligning the query roads to the reference roads over a whole city within several seconds on a single CPU, making it possibly available in real scenarios. Experiments on both the synthetic and real aerial images show that our method can significantly improve the recall of geolocalization while maintaining high precision and efficiency compared with the state-of-the-art aerial image geolocalization methods based on road network matching. The experiments also highlight areas of potential improvement. In future work, we will further enhance the capability and useful ranges of our method by reducing its dependence on initial transformation estimations, especially on aerial images with few roads.