A CNN-Based High-Accuracy Registration for Remote Sensing Images

Abstract

In this paper, a convolutional neural network-based registration framework is proposed for remote sensing to improve the registration accuracy between two remote-sensed images acquired from different times and viewpoints. The proposed framework consists of four stages. In the first stage, key-points are extracted from two input images—a reference and a sensed image. Then, a patch is constructed at each key-point. The second stage consists of three processes for patch matching—candidate patch pair list generation, one-to-one matched label selection, and geometric distortion compensation. One-to-one matched patch pairs between two images are found, and the exact matching is found by compensating for geometric distortions in the matched patch pairs. A global geometric affine parameter set is computed using the random sample consensus algorithm (RANSAC) algorithm in the third stage. Finally, a registered image is generated after warping the input sensed image using the affine parameter set. The proposed high-accuracy registration framework is evaluated using the KOMPSAT-3 dataset by comparing the conventional frameworks based on machine learning and deep-learning-based frameworks. The proposed framework obtains the least root mean square error value of 34.922 based on all control points and achieves a 68.4% increase in the matching accuracy compared with the conventional registration framework.

1. Introduction

Image registration is the process of geometric synchronization between a reference image and a current image from the same area. These images are acquired from different times and viewpoints by different sensors [1]. Thus, image registration is an essential preprocess step in many remote sensing applications because the main process, which includes change detection, image fusion, image mosaic, environment monitoring, and map updating can be drastically influenced by these differences [1,2]. Many types of image registration techniques have been developed in the areas of remote sensing over the past few decades. The registration frameworks can be classified into two categories—area-based frameworks and feature-based frameworks [1].

We introduce the two conventional image registration frameworks for the two categories—area-based frameworks and feature-based frameworks. In area-based frameworks, the registration problem is transformed into an optimization problem, where the similarity between reference and sensed images is maximized. Conventional area-based registration frameworks find correspondences at multiple key-points between input and reference images using similarity measures such as mutual information (MI) [3,4] or normalized cross-correlation (NCC) [5]. The detected correspondences are used in the estimation of the global geometric transform. However, they are sensitive to illumination changes and noise [1]. Liang et al. proposed spatial and mutual information (SMI) as the similarity metric for searching similar local regions using ant colony optimization [3]. Patel and Thakar employed mutual information (MI) based on maximum likelihood to expedite MI computation [4]. In contrast, feature-based frameworks are less susceptible to attacks and geometric distortions as they involve the matching of prominent features, such as points, lines, and regions. Scale invariant feature transform (SIFT) [6], speeded-up robust features [7], histogram of oriented gradients [8], and maximally stable extremal regions [9] are some of the widely applied feature detectors in practice. The SIFT-based framework is a well-known geometric transform approach [10]. Other approaches focus on the shape features or geometric structures. Ye et al. proposed the histogram of oriented phase congruency as a feature descriptor representing the structural properties of images, and then they used NCC as a reference matching similarity metric [11]. Yang et al. proposed a combination of shape context features and SIFT feature descriptors for remote sensing image registration [12]. There are approaches that integrate the advantages of the area-based and feature-based frameworks. The iterative multi-level strategy proposed by Xu at el. could re-extract and re-match features by adjusting the parameters [13]. The coarse-to-fine image registration framework by Gong et al. acquired coarse results from SIFT and then obtained precise registration based on MI [14].

Conventional feature-based frameworks require domain knowledge to design a feature extractor. This makes the handcrafted features less generic for diverse applications and data. Researchers often recommend feature-based frameworks if the images contain distinct artifacts. Feature-based frameworks are used in remote sensing image applications because the remote sensing images contain distinct artifacts [1]. To ensure the accuracy of feature-based frameworks, a well-designed feature extractor that can extract reliable features through trial and error is required. Aerial images used for remote sensing applications contain a large amount of appearance distortions caused by radiometric and geometric factors, attitude acquisition-related factors, seasonal factors, and so on. Consequently, many registration frameworks suffer poor correspondence between points detected by handcrafted feature extractors. In worst-case scenarios, these handcrafted feature extractors may be unable to detect a sufficient number of correspondence points to achieve satisfactory registration.

In recent years, deep learning has proven to be superior and robust in the field of remote sensing imaging—object detection [15,16], image classification [17,18], and image registration [19]. In particular, patch-based convolutional neural network (CNN) architectures have been extensively used in the area of image matching. Finding accurate correspondences between patches is instrumental to a broad range of applications, including wide-baseline stereo matching, multi-view reconstruction, image stitching, and structure from motion. Conventional patch matching methods use handcrafted features and distance measures. Zagoruyko and Komodakis proposed a CNN-based model that directly trains a general similarity function for comparing image patches from image data [20]. CNNs can generate powerful feature descriptors that are more robust to appearance changes than classical descriptors. These approaches divide the input image into a set of local patches and extract descriptors individually from each patch. The extracted descriptors are then compared with an appropriate distance measure to measure the similarity score even for a binary matching/unmatching decision. Han et al. proposed “MatchNet”, which extracts patch pair features from two identical CNNs via the Siamese network for image patch matching [21]. Alternatively, Zagoruyko and his colleagues proposed an image matching method by training the joint features of patches from two input images and evaluating the features extracted from two similar CNNs or two different CNNs [22].

Wang and his colleagues proposed a deep learning framework for remote sensing image registration [19]. They employed the deep belief network (DBN) to maintain the invariance feature against the distortion characteristics of remote-sensed images. Unlike conventional feature-based frameworks, their proposal directly trained an end-to-end mapping function by taking the image patch pairs as inputs using DBN and matching the labels as output. Furthermore, they attempted to reduce the computation cost in the training step. Their framework not only reduced the training time but also demonstrated better registration performance. As vectorized one-dimensional data from two-dimensional images are fed into the DBN, which may remove the spatial information for patch matching, they cannot handle geometric invariances in terms of rotation, translation, scale, shearing and so on. These variance factors in DBN may generate distortion in the registration result. To address this problem, Lee and Oh have proposed a MatchNet-based method which can improve the registration accuracy by maintaining the spatial information of features [23]. However, there still exists geometric distortion as shown in Figure 1. Rocco and his colleagues recently proposed the CNN architecture for geometric matching where they could handle global changes of appearance and incorrect matches between two matched images in a robust way [24]. However, it is not efficient to apply their model to applications which require a precise local patch matching process in each matched patch of two input images such as remote sensing image registration. Therefore, their robust model should be modified for remote sensing image registration.

In this paper, we propose a CNN-based registration framework for remote sensing that can improve the registration accuracy between two remote-sensed images acquired from different times and viewpoints. The framework can be summarized as follows: First, multiple key-points and their patches are extracted from two input images using scale–space extrema detection. Each patch contains one key-point at its center. Using the conventional network, finding the corresponding patch pair in the matching step would yield geometric distortions, such as translation, scale, and shearing because learning the invariance mapping function is difficult. For an accurate local patch matching process, we adopt the geometric CNN proposed in [24] to compensate the geometric distortion of each matched patch pair. From now on, the geometric CNN is called GMatchNet. A local geometric transformation is estimated from each matched patch pair. Using this local geometric transform, the corresponding center coordinate of each input patch is finely adjusted. Then, we compute the global geometric affine parameter set from all the adjusted coordinates by the random sample consensus algorithm (RANSAC). Finally, a registered image is generated after warping the input sensed image by the global affine parameter set. The proposed framework is evaluated on the KOMPSAT-3 dataset by comparing the conventional frameworks based on machine learning and deep-learning-based frameworks. We perform registration of images in which magnetic north is aligned with the universal transverse Mercator coordinate system. It is shown that the proposed high-accuracy registration framework can improve the accuracy of image registration by compensating the geometric distortion between matched patch pairs and can be applied to other registration frameworks based on patches.

The remainder of this paper is structured as follows: Section 2 introduces related work on image registration, deep learning, and patch matching. Section 3 details the proposed registration framework that uses the estimated geometric transformation in the corresponding patch pairs. Section 4 discusses the experimental results, and, finally, Section 4 summarizes the conclusions of the study.

2. High-Accuracy Registration Framework

The proposed framework consists of two different CNNs—MatchNet [21] and GMatchNet [24], as shown in Figure 2. First, multiple key-points and their patches were extracted from the reference image and the sensed image. Note that each patch (64 × 64 pixels) includes one key-point at its center. The next stage consists of three distinct processes for patch matching. For each reference patch of the reference image, multiple candidate lists were selected from the sensed image by MatchNet. Then, one-to-one matched labels for each reference patch were determined for its matched candidate lists based on cross correlation. The local affine parameter set was estimated between each input patch from the output of matched labels selection by GMatchNet, and the coordinate of the matched patch was finely adjusted using a local transformation. Then, the global geometric affine parameter set was computed from all the adjusted reference coordinates using the RANSAC algorithm. Finally, the warping process was performed to geometrically synchronize the reference image and sensed image.

2.1. Patch Extraction Based on Scale–Space Extrema

In the first stage of key-point detection, the locations and scales that can be repeatably assigned for different views of a same object were identified. Locations that are invariant to a change in the scale of the image can be detected by searching for stable features across all possible scales using a continuous function of scale known as the scale–space. Subsequently, the Laplacian of Gaussian (LoG) for the image with various standard deviation ($\sigma$) values was determined. The LoG operates as a blob detector that detects blobs in various sizes due to changes in $\sigma$. However, the LoG requires, to some extent, a heavier computational load. Therefore, the proposed framework adopts the difference of Gaussians (DoG), which approximates the LoG. The DoG is the difference between the Gaussian blurring of an image with two different standard deviations, denoted by $\sigma$ and $k\sigma$. When this DoG is generated, the local extrema are retrieved from the image, which results in the key-points. Lowe proposed several empirical parameter set, the number of octaves set to 4, number of scale levels set to 5, initial $\sigma$ set to 1.6, and $k$ set to $\sqrt{2}$ [6]. In the second step, the detected key-points as the central point were used to extract the image patches with a size of 64 × 64 pixels. Here, we assumed that the reference images and the sensed images are $I_1$ and $I_2$, respectively. If $I_1$ has $m$ key-points, then the patches are $P_1=\left\{p_1^1, p_1^2,\cdots ,p_1^m\right\}$. If $I_2$ has $n$ key-points, the patches are $P_2=\left\{p_2^1, p_2^2,\cdots ,p_2^n\right\}$. Thus, we can acquire the image patch pairs $\left\{\left(p_1^i,p_2^j\right)\right\}$ by combining the patches in images $I_1$ and $I_2$, where $i=1,2,\cdots ,m$, $j=1,2,\cdots ,n$.

2.2. Training Method for Matched Candidate List Generation

MatchNet is a deep network architecture that determines the correspondence of two images by analyzing the similarity of features in two input images. The structure of MatchNet is illustrated in Figure 3, and its layer parameters are listed in

Table 1. Layer parameters of MatchNet.

Name	Type	Output Dimension	Patch Size	Stride
Conv 1-1	Convolution	64 × 64 × 24	7 × 7	1
Pool 1	Max pooling	32 × 32 × 24	3 × 3	2
Conv 2-1	Convolution	32 × 32 × 64	5 × 5	1
Pool 2	Max pooling	16 × 16 × 64	3 × 3	2
Conv 3-1	Convolution	16 × 16 × 96	3 × 3	1
Conv 3-2	Convolution	16 × 16 × 64	3 × 3	1
Conv 3-3	Convolution	16 × 16 × 64	3 × 3	1
Pool 3	Max pooling	8 × 8 × 64	3 × 3	2
Bottleneck	Fully connected	256	-	-
FC1	Fully connected	512	-	-
FC2	Fully connected	512	-	-
FC3	Fully connected	2	-	-

. To compare the similarity of two patches, they are first passed through the same feature network. In the training stage, only one of the two feature networks is trained, while the other shares parameters.

The performance of MatchNet strongly depends on sufficient training dataset for optimizing parameters. However, it is difficult to obtain a labeled remote sensing image dataset. Thus, we adopt augmentation to construct a training dataset. The augmented dataset consists of remote sensing images transformed by a set of rotation matrices, where $\theta =\left\{0^{°},{90}^{°},{180}^{°},{270}^{°}\right\}$. Let $P_i$ and $M$ be the i-th image patch and the number of image patches, respectively. Then, $P_i$ can be transformed to an image set of $R_{\theta }\left(P_i\right)$. The patch size of MatchNet is 64 $\times$ 64. The matched patch pairs are $\left\{\left(P_i,R_{\theta }\left(P_j\right)\right),i=j \mathrm{and} \theta =0^{°}\right\}$ and unmatched patch pairs $\left\{\left(P_i,R_{\theta }\left(P_j\right)\right),i=j \mathrm{and} \theta \ne 0^{°}\right\}$ and $\left\{\left(P_i,R_{\theta }\left(P_j\right)\right),i\ne j\right\},$ where $i$ and $j$ $=1,2,\cdots M$. Therefore, the structure of a training sample is $\left\{\left(P_i,R_{\theta }\left(P_j\right)\right),y_{ij}^{\theta }\right\}$.

$$y_{ij}^{\theta }=\{\begin{array}{c}1, i=j \mathrm{and} \theta =0^{°}\\ 0, \mathrm{otherwise} \end{array}$$

(1)

Figure 4 illustrates examples of training patch pairs. The feature and metric networks were jointly trained in a supervised setting using the Siamese structure. The training dataset was constructed with a matched patch pairs to unmatched patch pairs ratio of 1:1 using the sampling method [21]. The cross-entropy error was minimized over a training set of $n$ patch pairs using the SGD with momentum. The cross-entropy was defined by

$$E=-\frac{1}{n}{\displaystyle \sum }_{i=1}^n\left[y_i\log \left(\widehat{y_i}\right)+\left(1-y_i\right)\log \left(1-\widehat{y_i}\right)\right]$$

(2)

The training parameters were set as follows: 200 iterations of MatchNet training and a learning rate of 0.01, a batch size of 32, and a momentum item parameter of 0.9 [21].

2.3. Matched Label Selection

In the matched label selection of the proposed framework, all image patch pairs $\left\{\left(p_1^i,p_2^j\right)\right\}$ from the sensed image $I_1$ and the reference image $I_2$ were fed to the trained CNN to predict multiple candidate lists. These lists were generated through patches with matched label sets. Owing to the remote sensing imaging mechanism and the small patch size, MatchNet is capable of finding more than one similar image patches between $I_1$ and $I_2$. This one-to-many matching leads to an ill-posed problem, which can be a major reason for the appearance of an inaccurate geometric affine parameter set. We adopted a local constraint using NCC to select one matching pair among the patches from the multiple candidate lists.

The NCC measures the similarity of two patches based on pixel intensity as the local constraint. In this study, we only selected the matched patch pair with the maximum NCC. The NCC of a patch pair $\left(p_1^i,p_2^j\right)$ was computed as follows:

$$c\left(p_1^j,p_2^j\right)=\frac{{{\displaystyle \sum }}_{x,y}\left[\left(p_1^i\left(x,y\right)-{\overline{p}}_1^i\right)\left(p_2^j\left(x,y\right)-{\overline{p}}_2^j\right)\right]}{\sqrt{{{\displaystyle \sum }}_{x,y}{\left(p_1^i\left(x,y\right)-{\overline{p}}_1^i\right)}^2{{\displaystyle \sum }}_{x,y}{\left(p_2^j\left(x,y\right)-{\overline{p}}_2^j\right)}^2}}$$

(3)

where $p_1^i\left(x,y\right)$ and $p_2^j\left(x,y\right)$ are the gray values of image patches $p_1^i$ and $p_2^j$ at location $\left(x,y\right)$, respectively. Further, ${\overline{p}}_1^i$ and ${\overline{p}}_2^j$ are the average gray values of image patches $p_1^i$ and $p_2^j$, respectively. One patch with the highest NCC value among the patches from multiple candidate lists was selected as the matched label.

2.4. Matched Patch Compensation with Local Geometric Transformation

As learning the invariance mapping function is difficult, geometric distortions, such as translation, scale, and shearing, appear between the matched patch pairs. It is necessary to correct the geometric distortion in the matched patch pairs. Figure 5a,b illustrate the matched patch pairs; however, two patches exhibit geometric distortions. To compensate for the geometric distortion, we adopted a pre-trained GMatchNet, which has been proposed for determining correspondences between two images in agreement with a geometric model, such as the geometric affine parameter set. Figure 6 shows a diagram of the GMatchNet architecture. The process of GMatchNet proceeds in four steps. First, input images $P_1$ and $P_2$ are passed through the Siamese architecture consisting of the convolutional layers, thus extracting feature maps $F_1$ and $F_2$. Second, feature maps across images are matched to a tentative correspondence map $F_{12}$. Third, a regression CNN that directly outputs the geometric affine parameter set $\widehat{\theta }$ is constructed. Finally, the network generates a new transformed image, $P_2^{\prime }$, by applying the transform $T_{\widehat{\theta }}$ to image $P_2$. We calculated the central coordinates of the newly generated image $P_t^2$ and used it to adjust the key-point position. In the case of GMatchNet, pre-trained weights were publicly available and could be used without any fine tuning since we could achieve the satisfied performance when those pretrained weights were applied to our framework. Figure 5 depicts an example of an adjusted key-point position. The red and yellow crosshairs indicate the central point of patch (a) and patch (b), respectively. In patch (c), the geometric distortion is compensated through GMatchNet, the previous yellow central position is shifted, and a new blue center position is assigned.

2.5. Global Constraints and Warping

The RANSAC algorithm estimates a model from a set of observed data through a random sampling and voting scheme often interpreted as an outlier detection method, which can further remove the falsely matched points globally associated with local constraints. By using the compensated matching labels from the previous step in the RANSAC algorithm, we calculated the global geometric affine parameter set $W$ with the RANSAC algorithm. Finally, we warped the sensed image using the global geometric affine parameter set $W$, generating the registered image $I_2^{\prime }$.

3. Results

In this study, we constructed datasets for both patch matching and registration using multispectral red, green, and blue images of cities around Seoul, South Korea, captured by the KOMPSAT-3 satellite with a resolution of 2.8 meter. Regions in Seoul are densely populated and their landscape is frequently changed by the emergence of new skyscrapers. On the other hand, areas around Seoul are agricultural areas with different colors depending on the seasonal conditions. The experiment was performed on a computer powered by an Intel (R) Core i7-8700K 3.40 GHz CPU with an NVIDIA GeForce GTX 1080 Ti GPU. In the following sections, we discuss the training and validation methods for patch matching via MatchNet and the evaluation metrics, and evaluate the performance of each remote sensing image registration framework.

We also explain the details of the dataset used for MatchNet. The training sets and validation sets for patch matching consisted of images from Suwon City. This dataset came with patches extracted using the scale–space extrema detection for extracting the key-points [6]. The size of the image patch used was 64 × 64 pixels. The resulting dataset was divided into 130k for training sets and 50k for validation sets. We used a sampler to generate an equal number of matched and unmatched patch pairs in each batch so that the network would not be overly biased toward the unmatched decision [25].

3.1. Evaluation Datasets and Metrics for Remote Sensing Image Registration Frameworks

The datasets for evaluation of remote sensing image registration consisted of images from Seoul and its surroundings from different times—three areas in the city and one area around it.

Table 2. Evaluation datasets description.

Class	Location	Size	Time
Area 1	Gwanak-gu	500 × 500	March 2014
		500 × 500	October 2015
Area 2	Guro-gu	500 × 500	April 2014
		500 × 500	October 2015
Area 3	Gwangmyeong City	500 × 500	April 2014
		500 × 500	October 2015
Area 4	Yeongdeungpo-gu	500 × 500	December 2014
		500 × 500	October 2015

lists the detailed information of those images. In the same area, the upper row represents the reference image and the lower row represents the sensed image. All satellite images were divided into 500 × 500 images. Each pair of images consisted of images from the same area captured at different times. The characteristics for each area are as follows: Area 1 dataset consists of images of residential areas, Area 2 dataset consists of images of residential and green lung areas, Area 3 dataset consists of images of industrial facilities, and Area 4 dataset consists of images of skyscrapers.

The metrics from [26] were employed in this study to objectively evaluate the proposed high-accuracy registration framework, which are as follows: the number of control points $\left(N_{red}\right)$; the root-mean-square error ($RMSE$) based on all control points and normalized to the pixel size $\left(RM{S}_{all}\right)$; the $RMSE$ computed by the control point residuals based on the leave-one-out method $\left(RM{S}_{loo}\right)$; the statistical evaluation of the residual distribution across quadrants $\left(P_{quard}\right)$; the bad point proportion with a norm greater than 1.0 $\left(BPP\left(1.0\right)\right)$; the statistical evaluation of the presence of a preference axis on the residual scatter plot $\left(S_{kew}\right)$; the statistical evaluation of the goodness of control points distribution across the image $\left(S_{cat}\right)$; and the weighted sum of the above seven measures, the cost function $\left(\varphi \right)$. Smaller values indicate better performance for six metrics except $N_{red}$. The cost function was used as an objective tool to evaluate the different control points for the pair of images. The equation of the cost function $\varphi$ is expressed as follows:

$$\varphi =\frac{2}{N_{red}}+RM{S}_{all}+2\times RM{S}_{loo}+1.5\times P_{quad}+2\times BPP+1.5\times S_{kew}+2\times S_{cat}$$

(4)

The registration accuracy was measured in terms of $RM{S}_{all}$ and $RM{S}_{loo}$. The quantity and quality of matching points were measured in terms of $N_{red}$ and $\varphi$, respectively. The lower the values of these metrics, the better $N_{red}$. We can observe that both $RM{S}_{all}$ and $RM{S}_{loo}$ equal or tend to the subpixel error, which are significant results of registration. $N_{red}$ measures the number of points that have been matched correctly. Further, a larger $N_{red}$ and a smaller $RMS$ imply a higher accuracy of point matching.

3.2. Evaluation of Remote Sensing Image Registration Framework

The proposed frameworks were compared with the conventional feature-based image registration framework, SIFT, and the state-of-the-art deep learning-based image registration framework. We used the DBN network structure proposed by Wang et al. [19]. The deep learning-based frameworks were experimented with two trained methods—the conventional method and proposed training method. We defined the improved accuracy, $I{A}_{\varphi }$, of $\varphi$ as follows:

$$I{A}_{\varphi }=\frac{SIF{T}_{\varphi }-DN{N}_{\varphi }}{SIF{T}_{\varphi }}\times 100 (\%)$$

(5)

where $SIF{T}_{\varphi }$ and $DN{N}_{\varphi }$ are the $\varphi$ values of the SIFT-based framework and each DNN-based framework, respectively.

$N_{red}$ measures the number of correct corresponding points. A larger $N_{red}$ and a smaller $RM{S}_{all}$ imply more accurate point matching.

Table 3. Quantitative comparison of model performance in evaluation datasets.

Class	Framework	${\mathit{N}}_{\mathit{r}\mathit{e}\mathit{d}}$	$\mathit{R}\mathit{M}{\mathit{S}}_{\mathit{a}\mathit{l}\mathit{l}}$	$\mathit{R}\mathit{M}{\mathit{S}}_{\mathit{l}\mathit{o}\mathit{o}}$	${\mathit{P}}_{\mathit{q}\mathit{u}\mathit{a}\mathit{r}\mathit{d}}$	$\mathit{B}\mathit{P}\mathit{P}\left(1.0\right)$	${\mathit{S}}_{\mathit{k}\mathit{e}\mathit{w}}$	${\mathit{S}}_{\mathit{c}\mathit{a}\mathit{t}}$	$\mathit{\varphi }$	$\mathit{I}{\mathit{A}}_{\mathit{\varphi }}$
Area 1	SIFT-based	154	4.550	3.942	0.949	0.525	0.249	0.920	1.431	-
	DBN-based [19]	869	2.671	2.671	1.000	0.527	0.061	1.000	1.105	29.5%
	Proposed high accuracy	435	2.161	1.792	1.000	0.474	0.044	1.000	0.855	75.2%
Area 2	SIFT-based	10	17.971	16.545	0.977	0.963	0.400	0.963	4.765	-
	DBN-based [19]	183	32.369	32.388	1.000	1.000	0.002	1.000	8.555	−79.54%
	Proposed high accuracy	329	4.652	4.058	0.987	0.722	0.026	1.000	1.478	68.98%
Area 3	SIFT-based	26	20.691	18.031	0.998	1.000	0.304	0.998	21.899	-
	DBN-based [19]	118	362.339	361.022	1.000	1.000	1.000	1.000	90.951	−315.32%
	Proposed	207	11.809	9.826	1.000	0.967	0.137	1.000	3.093	85.88%
Area 4	SIFT-based	12	619.558	506.167	1.000	1.000	0.966	0.907	136.576	-
	DBN-based [19]	85	265.764	265.770	1.000	1.000	0.807	1.000	67.004	50.94%
	Proposed high accuracy	36	121.066	111.815	0.979	0.861	0.212	0.990	29.187	78.63%
Average	SIFT-based	51	165.693	136.171	0.981	0.872	0.480	0.947	41.168	-
	DBN-based [19]	314	165.786	165.463	1.000	0.882	0.468	1.000	41.904	−80.3%
	Proposed high accuracy	252	34.922	31.873	0.992	0.756	0.105	0.998	8.653	68.4%

summarize the experimental results using eight metrics on four evaluation datasets. The last line in Table 3 illustrates the averaged results using eight metrics for the evaluation datasets in all areas. In the DBN-based framework of Wang et al. [19], although the number of control points $\left(N_{red}\right)$ was large, it had mismatched points and therefore an increased $RM{S}_{all}$ value. For qualitative assessment, we used the checkerboard mosaic image, which can demonstrate the subjective quality better than any other image in terms of edge continuity and region overlapping.

In Table 3, on the one hand, the DBN-based framework generated a large $N_{red}$, but the $RMS$ values increased due to the $RMSE$ for all control points. On the other hand, the proposed framework for the Area 1 dataset had a smaller $N_{red}$, but the lowest $RMS$ value representing the pixel error. In addition, the smallest $N_{red}$ of 0.855 was obtained for the quality of matching points $\varphi$. The performance of the proposed framework was 40.2% better than that of the SIFT-based framework in Area 1. Figure 7a,b illustrate the pair of images from Area 1, which were acquired by the KOMPSAT-3 satellite in March 2014 and October 2015. The green boxes indicate the same region in the three images and show smooth edges.

In Area 2, on the one hand, the DBN-based framework increased the $RMS$ values representing the quality of the matching point because the points did not match. Thus, the performance reduced by 79.54%. On the other hand, the proposed framework on the Area 2 dataset had a relatively large $N_{red}$ and the lowest $RMS$ value representing the pixel error. In addition, the smallest result of 1.478 was obtained from the quality of matching points $\varphi$. The performance of the proposed framework was 68.98% better than that of the SIFT-based framework in Area 2. Figure 8a,b illustrate the pair of images from Area 2 acquired by the satellite in April 2014 and October 2015. The green boxes in the three images represent the same region and demonstrate smooth edges. The red boxes in the three images indicate the same region and highlight the deviation of results of the conventional frameworks from those of the proposed framework.

In Area 3, on the one hand, using the DBN-based framework increased the $RMS$ values because the points did not match. Thus, the performance reduced by 315.32%. On the other hand, using the proposed framework on the Area 3 dataset produced the largest $N_{red}$ and the least $RMS$ value representing the pixel error. In addition, the smallest result of 3.093 was obtained from the quality of matching points $\varphi$. The proposed framework performed 85.88% better the SIFT-based framework in Area 3. The greatest performance improvement was observed in the industrial facility areas. Figure 9a,b illustrates the pair of images from Area 3 acquired by KOMPSAT-3 satellite in April 2014 and October 2015. The green boxes in the three images represent the same region and demonstrate smooth edges.

In Area 4, on the one hand, the DBN-based framework generated a large $N_{red}$ along with an increased $RMS$. On the other hand, the proposed framework produced the second-largest $N_{red}$ but the lowest $RMS$ representing the pixel error. In addition, the smallest result of 29.187 was obtained from the quality of matching points $\varphi$. The proposed framework performed 78.63% better than the SIFT-based framework in Area 4. The largest performance improvement occurred in the industrial facility areas. Figure 10a,b illustrate the pair of images from Area 4 acquired in December 2014 and October 2015. The changes observed in Figure 10 are large owing to the difference between skyscrapers and viewports. The SIFT-based framework failed to register the image. By contrast, the two proposed models successfully registered the images. Figure 10c,d are the image registration results of the SIFT-based framework and the DBN-based framework, respectively. Both frameworks failed to register the images.

In the KOMPSAT-3 image datasets, the DBN-based framework generated the largest $N_{red}$ along with a larger $RMS$ value representing the matching point quality because the points did not match. The DBN-based framework reduced the $RM{S}_{all}$ value representing the registration accuracy by 165.786, but the $RM{S}_{all}$ value of the proposed framework significantly reduced to 34.922. The DBN-based framework reduced the $\varphi$ value representing the matching points quality by 41.904, but that of the proposed framework significantly reduced to 8.653. The proposed framework achieved a performance improvement of 68.4%. The remarkable improvement in the performance of the proposed framework can be observed in the difference between the high-rise building and the image as the viewpoint shifts.

4. Conclusions

In this study, we proposed a CNN-based registration framework for remote sensing that can improve the image registration accuracy between two remote-sensed images acquired from different times and viewpoints. The matching step often produces geometric distortions, such as translation, scale, and shearing between the matched patch pairs given that the invariance mapping function is difficult to learn. To correct these distortions, we adopted a geometric CNN with a stronger invariance feature to find a local affine parameter set for each matched patch pairs. Therefore, we constructed multiple candidate lists, from which we estimated the local geometric transform. The proposed framework was evaluated on the KOMPSAT-3 dataset by comparing the conventional machine-learning-based frameworks and the proposed deep-learning-based framework. The proposed framework obtained the smallest RMSE of 34.922 based on all control points and achieved a 68.4% increase in the matching accuracy compared with the conventional registration framework. As the proposed framework is composed of two different networks, there is a computational complexity owing to the redundancy of the two feature networks. A unified network to alleviate the computational complexity can be the future direction of this research.