A Two-Stream Symmetric Network with Bidirectional Ensemble for Aerial Image Matching

Abstract

In this paper, we propose a novel method to precisely match two aerial images that were obtained in different environments via a two-stream deep network. By internally augmenting the target image, the network considers the two-stream with the three input images and reflects the additional augmented pair in the training. As a result, the training process of the deep network is regularized and the network becomes robust for the variance of aerial images. Furthermore, we introduce an ensemble method that is based on the bidirectional network, which is motivated by the isomorphic nature of the geometric transformation. We obtain two global transformation parameters without any additional network or parameters, which alleviate asymmetric matching results and enable significant improvement in performance by fusing two outcomes. For the experiment, we adopt aerial images from Google Earth and the International Society for Photogrammetry and Remote Sensing (ISPRS). To quantitatively assess our result, we apply the probability of correct keypoints (PCK) metric, which measures the degree of matching. The qualitative and quantitative results show the sizable gap of performance compared to the conventional methods for matching the aerial images. All code and our trained model, as well as the dataset are available online.

1. Introduction

1.1. Motivation

Aerial image matching is a geometric process of aligning a source image with a target image. Both images display the same scene but are obtained in different environments, such as time, viewpoints and sensors. It also a prerequisite of a variety of aerial image tasks such as change detection, image fusion, and image stitching. Since it can have a significant impact on the performance of the following tasks, it is an extremely important task. As shown in Figure 1, various environments have considerable visual differences of land-coverage, weather, and objects. The variance in the aerial images causes degradation of the matching precision. In conventional computer vision approaches, correspondences between two images are computed by the hand-crafted algorithm (such as SIFT [1], SURF [2], HOG [3], and ASIFT [4]), followed by estimating the global geometric transformation using RANSAC [5] or Hough transform [6,7]. However, these approaches are not very successful for aerial images due to their high-resolution, computational costs, large-scale transformation, and variation in the environments.

Another problem with aerial image matching is the asymmetric result. As aforementioned, there are tons of aerial image matching methods [1,2,3,4,5,6,7]. Notwithstanding, these methods [1,2,3,4,5,6,7] have overlooked the consistency of matching flow. i.e., most methods consider only one direction of the matching flows (from source to target). It causes asymmetric matching results and degradation of the overall performance. In Figure 2, it illustrates a failure case when the source image and the target image are swapped.

Many computer vision tasks have been applied and developed in real life [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24]. Because deep neural networks (DNNs) have shown impressive performance in real-world computer vision tasks [25,26,27,28], several approaches apply DNNs to overcome the limitation of traditional computer vision methods for matching the images. The Siamese network [29,30,31,32] has been extensively applied to extract important features and to match image-patch pairs [33,34,35]. Furthermore, several works [36,37,38] apply an end-to-end manner in the geometric matching area. While numerous matching tasks have been actively explored with deep learning, few approaches utilize DNNs in aerial image matching areas.

In this work, we utilize a deep end-to-end trainable matching network and design a two-stream architecture to address the variance in the aerial images obtained in diverse environments. By internally augmenting the target image and considering the three inputs, we regularize the training process, which produces a more generalized deep network. Furthermore, our method is designed as a bidirectional network with an efficient ensemble manner. Our ensemble method is inspired by the isomorphic nature of the geometric transformation. We apply this method in our inference procedure without any additional networks or parameters. The ensemble approach also assists in alleviating the variance between estimated transformation parameters from both directions. Figure 3 illustrates an overview of our proposed method.

1.2. Contibutions

To sum up, our contributions are three-fold:

• For aerial image matching, we propose a deep end-to-end trainable network with a two-stream architecture. The three inputs are constructed by internal augmentation of the target image, which regularizes the training process and overcomes the shortcomings of the aerial images due to various capturing environments.
• We introduce a bidirectional training architecture and an ensemble method, inspired by the isomorphism of the geometric transformation. It alleviates the asymmetric result of image matching. The proposed ensemble method assists the deep network to become robust for the variance between estimated transformation parameters from both directions and shows improved performance in evaluation without any additional network or parameters.
• Our method shows more stable and precise matching results from the qualitative and quantitative assessment. In the aerial image matching domain, we first apply probability of correct keypoints (PCK) metrics [44] to objectively assess quantitative performance with a large volume of aerial images. Our dataset, model and source code are available at https://github.com/jaehyunnn/DeepAerialMatching.

1.3. Related Works

In general, the image matching problem has been addressed in two types of methods: area-based methods and feature-based methods [39,40]. The former methods investigate the correspondence between two images using pixel intensities. However, these methods are vulnerable to noise and variation in illumination. The latter methods extract the salient features from the images to solve these drawbacks.

Most classical pipelines for matching two images consist of three stages, (1) feature extraction, (2) feature matching, and (3) regression of transformation parameters. As conventional matching methods, hand-crafted algorithms [1,2,3,4] are extensively used to extract local features. However, these methods often fail for large changes in situations, which is attributed to the lack of generality for various tasks and image domains.

Convolutional neural networks (CNNs) have shown tremendous strength for extracting high-level features to solve various computer vision tasks, such as semantic segmentation [27,41], object detection [26,42], classification [25,43], human action recognition [44,45], and matching. In the field of matching, E. Simo-Serra et al. [33] learned local features based on image-patch with a Siamese network and use the L2-distance for the loss function. X. Han et al. [35] proposed a feature network and metric network to match two image patches. S. Zagoruyko et al. [34] expanded the Siamese network in two-streams: surround stream and central stream. K.-M Yi et al. [46] proposed a framework that includes detection, orientation, estimation, and description by mimicking SIFT [1]. H. Altwaijry et al. [31] performed ultra-wide baseline aerial image matching with a deep network and spatial transformer module [47]. H. Altwaijry et al. [48] also proposed a deep triplet architecture that learns to detect and match keypoints with 3-D keypoints ground-truth extracted by VisualSFM [49,50]. I. Rocco et al. [36] first proposed a deep network architecture for geometric matching, and demonstrated the advantage of a deep end-to-end network by achieving 57% PCK score in the semantic alignment. This method constructs a dense-correspondence map using two image features and directly regress the transformation parameters. These researchers further proposed a weakly-supervision approach that does not require any additional ground-truth for training [37]. P. Seo et al. [38] applied an attention mechanism with an offset-aware correlation (OAC) kernel based on [36] and achieved a 68% PCK score.

Although these works show meaningful results, their accuracy or computational costs for aerial image matching require improvement. Therefore, we compose a matching network that is suitable for aerial images by pruning the factors that degrade performance.

2. Materials and Methods

We propose a deep end-to-end trainable network with a two-stream architecture and bidirectional ensemble method for aerial image matching. Our proposed network focuses on addressing the variance in the aerial images and asymmetric matching results. The steps for predicting transformation are listed as follows: (1) internal augmentation, (2) feature extraction with the backbone network, (3) correspondence matching, (4) regression of transformation parameters, and (5) application of ensemble to the multiple outcomes. In Figure 4, we present the overall architecture of the proposed network.

2.1. Internal Augmentation for Regularization

The network considers two aerial images (source image $I_S$ and target image $I_T$) with different temporal and geometric properties as the input. By using this original pair $(I_S,I_T)$ in the training process, the deep network is trained by considering the relation of only two images obtained in different environments. However, this approach is insufficient for addressing the variance in the aerial images. Collecting various pair sets to solve these problems is expensive. To address this issue, we augment the target image by internally jittering image color during the training procedure. The network can be trained with various image pairs since the color of the target image is randomly jittered in every training iteration as shown in Figure 5. This step has a regularization effect of the training process, which produces a more generally trained network. The constructed three inputs are passed through a deep network. Subsequently, the network directly and simultaneously estimates global geometrical transformation parameters for the original pair and augmented pair. Note that the internal augmentation is only performed in the training procedure. In inference procedure, we utilize a single-stream architecture without the internal augmentation process for computational efficiency.

2.2. Feature Extraction with Backbone Network

Given the input images $(I_S,I_T,I_{T^{\prime }})\in {\mathbb{R}}^{h\times w\times d}$, we extract their feature maps $(f_S,f_T,f_{T^{\prime }})\in {\mathbb{R}}^{h^{\prime }\times w^{\prime }\times d^{\prime }}$ by passing a fully-convolutional backbone network $\mathcal{F}$, which is expressed as follows:

$$\mathcal{F}:{\mathbb{R}}^{h\times w\times d}\to {\mathbb{R}}^{h^{\prime }\times w^{\prime }\times d^{\prime }},$$

(1)

where $(h,w,d)$ denote the heights, widths, and dimensions of the input images and $(h^{\prime },w^{\prime },d^{\prime })$ are those of the extracted features, respectively.

We investigate various models of the backbone networks, as shown in Section 3. SE-ResNeXt101 [43] add the Squeeze-and-Excitation (SE) block as the channel-attention module to ResNeXt101 [51], which has shown its superiority in [52]. Figure 6 shows the SE-block. Therefore, we leverage SE-ResNeXt101 as the backbone network and empirically show that it has an important role in improving performance compared with other backbone networks. We utilize the image features extracted from layer-3 in the backbone network and apply L2-normalization to extracted features.

2.3. Correspondence Matching

As a method for computing a dense-correspondence map between two feature maps [36], the matching function $\mathcal{C}$ is expressed as follows:

$$\begin{array}{cc}\hfill c_{S\to T}(i,j,k)& =\mathcal{C}(f_S(i_k,j_k),f_T(i,j))\hfill \\ \hfill & =f_T{(i,j)}_S^f(i_k,j_k),\hfill \end{array}$$

(2)

where $c_{S\to T}$ is the dense-correspondence map that matches the source feature map $f_S$ to the target feature map $f_T$. $(i,j)$ and $(i_k,j_k)$ indicate the coordinate of each feature point in the feature maps. Each element in $c_{S\to T}$ refers to the similarity score between two points.

We construct the dense-correspondence map of the original pair and augmented pair. To consider only positive values for ease of training, the negative scores in the dense-correspondence map are removed by ReLU non-linearity, followed by L2-normalization.

2.4. Regression of Transformation Parameters

The regression step is for predicting the transformation parameters. When the dense-correspondence maps are passed through the regression network $\mathcal{R}$, the network $\mathcal{R}$ directly estimates the geometric transformation parameters as follows:

$$\mathcal{R}:{\mathbb{R}}^{h^{\prime }\times w^{\prime }\times (h^{\prime }\times w^{\prime })}\to {\mathbb{R}}^{DoF},$$

(3)

where $(h^{\prime },w^{\prime })$ indicate the heights and widths of the feature maps, and $DoF$ means the degrees of freedom of the transformation model.

We adopt the affine transformation which has 6-$DoF$ and the ability to preserve straight lines. In the semantic alignment domain [36,37,38], thin-plate spline (TPS) transformation [53] which has 18-$DoF$ is used to improve the performance. However, it is not suitable in the aerial image matching domain, because it produces large distortions of the straight lines (such as roads and boundaries of the buildings). Therefore, we infer the six parameters that handle the affine transformation.

2.5. Ensemble Based on Bidirectional Network

The affine transformation is invertible due to its isomorphic nature. We take advantage of this characteristic to design a bidirectional network and apply an ensemble approach. Applying the ensemble method enables alleviating the variance in the aerial images and improvement in the matching performance without any additional networks or models.

2.5.1. Bidirectional Network

Inspired by its isomorphic nature, we expand the base architecture by adding a branch that symmetrically estimates the transformation in the opposite direction symmetrically. The network yields the transformation parameters in both directions of each pair, i.e., $({\widehat{\theta }}_{S\to T},{\widehat{\theta }}_{T\to S})$ and $({\widehat{\theta }}_{S\to T^{\prime }},{\widehat{\theta }}_{T^{\prime }\to S})$. To infer the parameters of another branch, we compute the dense-correspondence map in the opposite direction by using the same method as in Section 2.3. All dense-correspondence maps are passed through the identical regression network $\mathcal{R}$. Since we utilize a regression network for all cases, no additional parameters are needed in this procedure. The proposed bidirectional network only adds a small amount of computational overhead compared with the base architecture.

2.5.2. Ensemble

In general, the ensemble technique requires several additional different architectures and consumes additional time costs to train models differently. We introduce an efficient ensemble method without any additional architectures or models by utilizing the isomorphism of the affine transformation. Figure 7 illustrates the overview of the ensemble procedure. ${({\widehat{\theta }}_{T\to S})}^{-1}$, which is the inverse of ${\widehat{\theta }}_{T\to S}$, can be expressed as another transformation parameters in the direction from $I_S$ to $I_T$. To compute ${({\widehat{\theta }}_{T\to S})}^{-1}$, we convert ${\widehat{\theta }}_{T\to S}$ into the homogeneous form:

$$[a_1,a_2,t_x,a_3,a_4,t_y]⟹\left[\begin{array}{ccc}{a}_1& a_2& t_x\\ a_3& a_4& t_y\\ 0& 0& 1\end{array}\right].$$

(4)

In the affine transformation parameters $[a_1,a_2,t_x,a_3,a_4,t_y]$, $a_1\sim a_4$ represent the scale, rotated angle and tilted angle, and $(t_x,t_y)$ denotes the $(x$-axis, y-axis) translation. We compute ${({\widehat{\theta }}_{T\to S})}^{-1}$ by converting the homogeneous form, as shown in Equation (4). This inverse matrix denotes another affine transformation from $I_S$ to $I_T$. As a result, we fuse the two sets of affine transformation parameters as follows:

$${\widehat{\theta }}_{en}=\mu ({\widehat{\theta }}_{S\to T},{({\widehat{\theta }}_{T\to S})}^{-1}),$$

(5)

where $\mu (\ast )$ denotes the mean function for fusing two parameters. In the various experiments, we apply three types of mean: arithmetic mean, harmonic mean and geometric mean. Empirically, arithmetic mean shows the best performance. In the inference process, ${\widehat{\theta }}_{en}$ warps the source image into the target image. Note that we fuse only parameters that correspond to the original pair since we use the original two-stream network in the inference procedure and do not utilize the ensembled parameters in the training procedure to maximize the ensemble effects.

2.6. Loss Function

In the training procedure, we adopt the transformed grid loss [36] as the baseline loss function. Given the predicted transformation $\widehat{\theta }$ and the ground-truth ${\theta }^{gt}$, the baseline loss function $l(\widehat{\theta },{\theta }^{gt})$ is obtained by the following:

$$l(\widehat{\theta },{\theta }^{gt})=\frac{1}{N}\sum _{i,j=1}^{N}d{({\mathcal{T}}_{\widehat{\theta }}(x_i,y_j),{\mathcal{T}}_{{\theta }^{gt}}(x_i,y_j))}^2,$$

(6)

where N is the number of grid points, ${\mathcal{T}}_{\widehat{\theta }}(\ast )$ and ${\mathcal{T}}_{{\theta }^{gt}}(\ast )$ are the transforming operations parameterized by $\widehat{\theta }$ and ${\theta }^{gt}$, respectively. To achieve bidirectional learning, we add a term for training the additional branch to the baseline loss function. Formally, we define the proposed bidirectional loss of the original pair, ${\mathcal{L}}_{org}$, as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{org}=& l({\widehat{\theta }}_{S\to T},{\theta }_{S\to T}^{gt})+l({\widehat{\theta }}_{T\to S},{({\theta }_{S\to T}^{gt})}^{-1}).\hfill \end{array}$$

(7)

Note that additional ground-truth information for the opposite direction is not required due to the isomorphism of the affine transformation. For regularization of training, we add two terms utilizing the augmented pair:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{aug}=& l({\widehat{\theta }}_{S\to T^{\prime }},{\theta }_{S\to T}^{gt})+l({\widehat{\theta }}_{T^{\prime }\to S},{({\theta }_{S\to T}^{gt})}^{-1}),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{id}=& l({\widehat{\theta }}_{S\to T},{\theta }_{S\to T^{\prime }})+l({\widehat{\theta }}_{T\to S},{\theta }_{T^{\prime }\to S}).\hfill \end{array}$$

(9)

The augmented pair also share the ground-truth since the geometric relation between two images is equivalent to the original pair. The identity term in Equation (9) induces training to ensure that the prediction values from the original pair and the augmented pair are equal. Our proposed final loss function is defined by the following:

$$\begin{array}{cc}\hfill \mathcal{L}=& \alpha ·{\mathcal{L}}_{org}+\beta ·{\mathcal{L}}_{aug}+\gamma ·{\mathcal{L}}_{id},\hfill \end{array}$$

(10)

where $(\alpha ,\beta ,\gamma )$ are the balance parameters of each loss term. In our experiment, we set these parameters to (0.5, 0.3, 0.2), respectively.

3. Results

In this section, we present the implementation details, experiment settings, and results. For the quantitative evaluation, we compare the proposed method with other methods for aerial image matching. We further experiment with various backbone networks to obtain more suitable features for our work. We show the contributions of each proposed component in the ablation study section and the qualitative results of the proposed network compared with other networks.

3.1. Implementation Details

We implemented the proposed network using PyTorch [54] and trained our model with the ADAM optimizer [55], using a learning rate $5\times {10}^{-4}$ and a batch size of 10. We further performed data augmentation by generating the random affine transformation as the ground-truth. All input images were resized to $240\times 240$.

3.2. Experimental Settings

3.2.1. Training

We generated the training input pairs by applying random affine transformations to the multi-temporal aerial image pairs captured in Google Earth. Since no datasets were annotated with completely correct transformation parameters between two images, we built the training dataset, 9000 multi-temporal aerial image pairs, and corresponding ground-truths. Basically, multi-tempral image pairs consisted of the image pairs which were taken at different times (2019, 2017, and 2015) and by different sensors (Landsat-7, Landsat-8, WorldView, and QuickBird). The process of annotating ground-truth is as follows: (1) we employed the multi-temporal image pairs $(I,I^{\prime })$ with the same region and viewpoint. (2) The first images in the multi-temporal aerial image pairs were center-cropped. (3) The second images are transformed by the randomly generated affine transformation ${\theta }_{S\to T}^{gt}$ which was used as a ground-truth and subsequently center-cropped. (4) The center-crop process was performed to exclude the black area that serves as noise after transformation. Figure 8 illustrates the process of generating training pairs and ground-truths. In Algorithm 1, the training procedure is detailed. It has $O(N)$ complexity with respect to the number of training pairs N. We train our model for 2-days on a single NVIDIA Titan V GPU.

Algorithm 1: Training procedure.

Input: Training aerial image dataset D        Randomly initialized model ${\mathcal{M}}_w$    Output: Trained model ${\mathcal{M}}_w$    for epochs do        end

3.2.2. Evaluation

To demonstrate the superiority of our method quantitatively, we evaluated our model using the PCK [56], which was extensively applied in the other matching tasks [36,37,38,57,58,59,60]. PCK metric is defined as follows:

$$PCK=\frac{{\sum }_{i=1}^n{\sum }_{p_i}1[d({\mathcal{T}}_{\widehat{\theta }}(p_i),{\mathcal{T}}_{{\theta }^{gt}}(p_i))<\tau ·max(h,w)]}{{\sum }_{i=1}^n\left|p_i\right|},$$

(11)

where $p_i$ is the ith point, which consists of $(x_i,y_i)$, and $\tau ·\max (h,w)$ refers to the tolerance term in the image size of $h\times w$. Intuitively, the denominator and the numerator denote the number of correct keypoints and overall annotated keypoints, respectively. The PCK metric shows how well matching is successful globally according to given $\tau$ with a lot of test images. In this evaluation, we assess in the cases of $\tau =0.1,0.3,$ and $0.5$. The greater value of $\tau$ allows measuring degrees of matching more globally. To adopt the PCK metric, we annotated the keypoints and ground-truth transformation to 500 multi-temporal aerial image pairs. The multi-temporal pairs are captured in Google Earth and composed of major administrative districts in South Korea, like the training image pairs. The annotation process is as the following process: (1) we extracted the keypoints of multi-temporal aerial image pairs using SIFT [1], and (2) picked up the overlapping keypoints between each image pair. We annotate 20 keypoints per image pair, which generate a total of 10k keypoints for a quantitative assessment. This approach provides a fair demonstration of quantitative performance. In the evaluation and the inference procedure, we used a two-stream network, except for the augmented branch shown in Algorithms 2.

Algorithm 2: Inference procedure.

Input: Source and target images $(I_S,I_T)$       Trained model ${\mathcal{M}}_w$    Output: Transformed image $I_S^{\prime }$    # Feed-forward    ${\widehat{\theta }}_{S\to T},{\widehat{\theta }}_{T\to S}={\mathcal{M}}_w(I_S,I_T)$;    # Ensemble    ${\widehat{\theta }}_{en}=\mu ({\widehat{\theta }}_{S\to T},{({\widehat{\theta }}_{T\to S})}^{-1})$;    # Transform source image to target image    $I_S^{\prime }={\mathcal{T}}_{{\widehat{\theta }}_{en}}(I_S)$

3.3. Results

3.3.1. Quantitative results

Aerial Image Dataset

Table 1. Comparisons of probability of correct keypoints (PCK) in the aerial images. CNNGeo is evaluated in two versions: the pre-trained model provided in [36] and the fine-tuned model by the aerial images. Both models use ResNet101 as the backbone network.

Methods	PCK (%)
	$\mathbf{\tau }=0.05$	$\mathbf{\tau }=0.03$	$\mathbf{\tau }=0.01$
SURF [2]	26.7	23.1	15.3
SIFT [1]	51.2	45.9	33.7
ASIFT [4]	64.8	57.9	37.9
OA-Match [61]	64.9	57.8	38.2
CNNGeo [36] (pretrained)	17.8	10.7	2.5
CNNGeo (fine-tuned)	90.6	76.2	27.6
Ours; ResNet101 [62]	93.8	82.5	35.1
Ours; ResNeXt101 [51]	94.6	85.9	43.2
Ours; Densenet169 [63]	95.6	88.4	44.0
Ours; SE-ResNeXt101 [43]	97.1	91.1	48.0

shows quantitative comparisons to the conventional computer vision methods (SURF [2], SIFT [1], ASIFT [4] + RANSAC [5] and OA-Match [61]) and CNNGeo [36] on aerial image data with large transformation. Conventional computer vision methods [1,2,4,5,61] showed quite a number of critical failures globally. As shown in Table 1, the conventional methods show low PCK performance in the case of $\tau =0.05$. However, in the case of $\tau =0.01$, these methods showed lower degradation of performance compared with other deep learning based methods. This result implies that conventional methods enable finer matching if the matching procedure does not failed entirely. Although CNNGeo fine-tuned by aerial images shows somewhat tolerable performance, our method considerably outperforms this method in all cases of $\tau$. Furthermore, we performed an investigation of the various backbone networks to demonstrate the importance of feature extraction. Since the backbone network substantially affects the total performance, we experimentally adopted the best backbone network.

Ablation Study

The proposed method combines two distinct techniques: (1) internal augmentation and (2) bidirectional ensemble. We analyze the contributions and effects of each proposed component and compare our models with CNNGeo [36]. ’+ Int. Aug.’ and ’+ Bi-En.’, which signify the internal augmentation and bidirectional ensemble addition, respectively. As shown in

Table 2. Results of models with different additional components. We analyzed the contributions of each component with ResNet-101 backbone.

Methods	PCK (%)
	$\mathbf{\tau }=0.05$	$\mathbf{\tau }=0.03$	$\mathbf{\tau }=0.01$
CNNGeo [36]	90.6	76.2	27.6
CNNGeo + Int. Aug.	90.9	76.6	28.4
CNNGeo + Bi-En.	92.1	79.5	31.8
CNNGeo + Int. Aug. + Bi-En. (Ours)	93.8	82.5	35.1

, all models added by our proposed component improves the performances of CNNGeo for all $\tau$, while maintaining the number of parameters. We further compare the proposed two-stream architecture to single-stream architecture which is added to the proposed components (internal augmentation, bidirectional ensemble).

Table 3. Comparison of single-stream and two-stream architecture. We analyzed the effectiveness of the two-stream based regularization with ResNet-101 backbone.

Methods	PCK (%)
	$\mathbf{\tau }=0.05$	$\mathbf{\tau }=0.03$	$\mathbf{\tau }=0.01$
Single-stream (with Int. Aug. and Bi-En.)	92.4	79.7	33.5
Two-stream (Ours)	93.8	82.5	35.1

shows the excellence of the proposed two-stream architecture compared to the single-stream architecture. It implies that the proposed regularization terms by the two-stream architecture are reasonable.

3.3.2. Qualitative Results

Global Matching Performance

We performed a qualitative evaluation using the Google Earth dataset (Figure 9) and the ISPRS dataset (Figure 10). The ISPRS dataset is a real-world aerial image dataset that was obtained from different viewpoints. Although our model was trained from the synthetic transformed aerial image pairs, it is successful with real-world data. In Figure 9 and Figure 10, the samples consist of challenging pairs, including numerous difficulties such as differences in time, occlusion, changes in vegetation, and large-scale transformation between the source images and the target images. Our method correctly aligned the image pairs and yields accurate results of matching compared with other methods [4,5,36,61] as shown in Figure 9 and Figure 10.

Localization Performance

We visualized the matched keypoints for comparing localization performance with CNNGeo [36]. It is also important how fine source and target images are matched within the success cases. As shown in Figure 11, we intuitively compared localization performance. The X marks and the O marks on the images indicate the keypoints of the source images and the target images, respectively. Both models ([36] and ours) successfully estimated global transformation. However, looking at the distance of matched keypoints, ours was better localized.

4. Discussion

4.1. Robustness for the Variance of Aerial Image

Furthermore, we experimented on robustness for the variance of aerial images as shown in Figure 12. The source images were taken in 2004, 2006, 2015, 2016, and 2019, respectively. The target images were absolutely identical images. As a result, ours showed more stable results for overall sessions. Especially, source images which were taken in 2004 and 2006 have large differences of including object compared with the target image. It showed that ours had better robustness for the variance of the aerial images while the baseline [36] is significantly influenced by these differences.

4.2. Limitations and Analysis of Failure Cases

We describe the limitation of our method and analyze the case in which the proposed method fails. As shown in Section 3.3.1, our method quantitatively showed state-of-the-art performance. However, comparing $\tau =0.05$ with $\tau =0.01$ indicates a substantial difference in performance. Our method is weak in detailed matching even though it successfully estimates global transformation in most cases. This weakness can be addressed by additional fine-grained transformation as post-processing.

Our proposed method failed in several cases. As a result, we have determined that our method fails in mostly wooded areas or largely changed areas as shown in Figure 13 and Figure 14. In mostly wooded areas, repetitive patterns hinder the focus on a salient region. In the case of largely changed areas, massive differences, such as buildings, vegetation, and land-coverage between the source image and the target image are observed, which leads to degradation of performance. To address these limitations, a method that can aggregate local contexts for reducing repetitive patterns is required.

5. Conclusions

We propose a novel approach based on a deep end-to-end network for aerial image matching. To become robust to the variance of the aerial images, we introduce two-stream architecture using internal augmentation. We show its efficacy for consideration of various image pairs. An augmented image can be seen as an image which is taken in different environments (brightness, contrast, saturation), and by training these images with original target images simultaneously, it leads to the effect of regularizing the deep network. Furthermore, by training and inferring in two possible directions, we apply an efficient ensemble method without any additional networks or parameters, which considers the variances between transformation parameters from both directions and substantially improves performance. In the experimental section, we show stable matching results with a large volume of aerial images. However, our method also has some limitations as aforementioned (Section 4.2). To overcome these limitations, we plan to research the localization problem and the attention mechanism. Moreover, The studies applying Structure from Motion (SfM) and 3D reconstruction to image matching are very interesting and can improve performance of image matching, so we also plan to conduct this study in the future work.