Self-Supervised Keypoint Detection and Cross-Fusion Matching Networks for Multimodal Remote Sensing Image Registration

Abstract

Remote sensing image matching is the basis upon which to obtain integrated observations and complementary information representation of the same scene from multiple source sensors, which is a prerequisite for remote sensing tasks such as remote sensing image fusion and change detection. However, the intricate geometric and radiometric differences between the multimodal images render the registration quite challenging. Although multimodal remote sensing image matching methods have been developed in recent decades, most classical and deep learning based techniques cannot effectively extract high repeatable keypoints and discriminative descriptors for multimodal images. Therefore, we propose a two-step “detection + matching” framework in this paper, where each step consists of a deep neural network. A self-supervised detection network is first designed to generate similar keypoint feature maps between multimodal images, which is used to detect highly repeatable keypoints. We then propose a cross-fusion matching network, which aims to exploit global optimization and fusion information for cross-modal feature descriptors and matching. The experiments show that the proposed method has superior feature detection and matching performance compared with current state-of-the-art methods. Specifically, the keypoint repetition rate of the detection network and the NN mAP of the matching network are 0.435 and 0.712 on test datasets, respectively. The proposed whole pipeline framework is evaluated, which achieves an average M.S. and RMSE of 0.298 and 3.41, respectively. This provides a novel solution for the joint use of multimodal remote sensing images for observation and localization.

1. Introduction

The joint observation of multimodal remote sensing images can bring unpredictable discoveries to the region and significantly improve the interpretation of the same scene. There is thus a remote sensing image registration, which is to align two or more multi-temporal/multi-sensor images in the same coordinate system for the observed scenarios. Registration is crucial since it determines the quality and accuracy of the remote sensing image fusion and change detection [1]. Therefore, a matching process needs to be implemented for remote sensing missions for joint observations [2,3,4,5].

The registration problem is usually solved by searching for matching information from the entire image or partial image patches using similarity measurement (e.g., area-based methods) [6], or by discriminative features (e.g., feature-based methods) [7], depending on the different approaches of building matching points.

Area-based methods search for correspondences between images by using a template. These methods can be roughly divided into three categories: correlation methods (NCC) [8], Fourier-based methods [9] and mutual information (MI) methods. Recently, Ye et al. [10] proposed a fast and robust template matching framework for multimodal remote sensing image matching framework based on pixel-wise structure feature representation. It used structure information from the entire template window, combined with a fast similarity measure, to detect correspondences between images. However, since these area-based methods are weak in dealing with large geometric deformation, they fail to process complex remote sensing image registration.

Feature-based methods first perform feature (e.g., points, lines, or regions) detection and description, and then determine correspondences by using the similarity of these features, which is more robust to image distortions compared with area-based methods [2,11]. In the past few decades, various feature-based methods have been developed; the most representative one is the scale invariant feature transformation (SIFT) [12] because it is invariant to translation, rotation and scale. Moreover, some SIFT-like methods, such as OS-SIFT [13] and improved SIFT [14], were also proposed for SAR-optical image matching. However, customizing universal feature matching algorithms for various remote sensing images remains a challenging task. Since these images are obtained by various sensors, multi-temporal observation or different imaging views, they have complicated geometric and radiometric differences. Additionally, those non-learned features (e.g., statistical information of edges, textures, corners and gradients) lack high-level semantic information. Therefore, those methods cannot guarantee that the extracted features are highly repeatable and distinct between multimodal remote sensing images.

Currently, deep learning (DL) methods have achieved great success [15,16,17,18]. They have been applied in remote sensing image processing tasks including image registration [19,20,21], change detection [22] and object detection [23], etc. The main reason is its completely data-driven scheme, trying to abstract the distribution structure from the input data.

Many studies have employed deep neural networks to produce some feature descriptors with rotation and scale invariance [24]. Burgmann et al. [25] investigated a DL-based method for automatically matching SAR and optical images by learning a common feature representation. Wang et al. [26] proposed a DL network to learn the mapping of registration pairs to matching labels. Yang et al. [19] proposed a multi-scale feature descriptor based on a pre-trained convolutional neural network (CNN) [27]. Zhou et al. [28] used deep learning techniques to extract multi-directional gradient features to depict the structure properties of images. The above methods introduce high-level features as matching primitives, which achieve a considerable matching performance in many cases. While these methods use deep neural networks to provide descriptors of salient features, they still use non-learned methods for feature detection rather than learning-based schemes.

To address the above problem, many researchers proposed several learning-based frameworks, discarding the keypoint detection step, by directly using deep neural networks to map matching responses based on patches [29]. Hughes et al. [30] proposed a three-step framework for sparse image matching of SAR and optical images, generating correspondences via a cross-correlation operator. Hughes et al. [31] proposed a pseudo-Siamese network architecture that predicts the correspondence of SAR and optical patch pairs based on learning in the training data. Li et al. [32] proposed a semantic template matching method that used a deep neural network to map the sensed image to the reference image, avoiding the keypoint detection step. Li et al. [32] proposed a semantic mapping image matching method based on deep neural networks, which transforms the matching problem into a semantic centroid correspondence. Although the above methods avoid the keypoint detection step, they are very time consuming in searching for the best correspondence for each patch.

There are some methods that combine feature detection and description in one network for natural images, such as D2-Net [33], Superpoint [34] and R2D2 [35]. D2-Net uses the original image as input to generate feature maps of keypoints. However, the accuracy of the keypoint locations is relatively low due to the detection on the feature maps. Superpoint applies a simulated training approach to obtain keypoint localization, which finds it difficult to obtain keypoint responses on multimodal remote sensing with complex features. R2D2 uses upsampling to maintain the size of the original image and regarded the final output as key information to generate keypoints, which will lose feature responses at corners and edges.

For these natural image-based matching methods, their keypoint detection and feature description mechanisms must be improved, e.g., keypoints detected based on maximum pooling have low repeatability on remotely sensed images with nonlinear radiometric differences. Integrating multiple tasks in a single network may not perform well for keypoint detection and feature description. Furthermore, these unified networks are optimized based on a fixed size image patches, which are difficult to perform global matching optimization for remote sensing images with large sizes. Therefore, each matching step requires an adjusted network structure to optimize remote sensing images globally. Based on the above description, the establishment of multimodal remote sensing image matching needs to overcome the following problems.

(1) The problem of repeatability for the keypoints. Establishing keypoints conditional on local features for one image using maximum pooling, which may be difficult in terms of obtaining a repeatable correspondence on another remote sensing image caused by the radiometric and geometric differences;

(2) The problem of cross-modal feature similarity. Since the heterogeneous information between multimodal remote sensing images, e.g., nonlinear radiometric differences, the feature descriptions may not have similarities and cannot be complete matching;

(3) The problem of global matching optimization. The network structure that integrates detection, description and matching is optimized for only one local fixed size patch. For remote sensing images with larger sizes, this may significantly increase the anomaly correspondence;

In this work, a two-step “detection + matching” network framework is proposed, where each network is applied to a different task to accommodate multimodal remote sensing image matching. To generate keypoints with repeatability, we construct a self-supervised keypoint detection network from both spatial and channel domains rather than local keypoint responses by maximum pooling. To obtain cross-modal similarity feature descriptions, an interactive fusion network is proposed for global optimization. The main contributions are summarized as follows:

(1) For the detection network, considering the differences in the input multimodal images, we build confidence at the same locations from the spatial and channel domains, which discards the maximum pooling keypoint detection mechanism. Overall, we design a self-supervised training manner that communicates the same locations with the same keypoint confidence, which shifts the keypoint detection from a single image response to the fitting of keypoint positions conditional on two images;

(2) In terms of matching networks, it is considered that constructing feature descriptors with local image patches ignores the global information and the interaction between image patch pairs. We develop a cross-fusion mechanism for exchanging high-level semantic information between image patches for feature description. Simultaneously, the network aims to generate a matrix of matching relationships from the overall image patches, unifying the “description + matching” steps into a single network to obtain optimization of overall remote sensing image matching.

The proposed method is evaluated on multimodal SAR-optical images. Experiments show that the keypoint detection network obtains keypoints with a high repetition rate on remote sensing images with geometric distortions and nonlinear radiometric differences. For the cross-fused matching network, state-of-the-art performance is also achieved compared to competitive methods.

2. Related Works

This section briefly introduces the structure of convolutional networks used for feature extraction in the detection network including depthwise separable convolution (DSC) and deformable convolutional network (DCN). For fusion mechanisms, we review multi-scale, high-level and low-level, local and global feature information fusion based on deep neural networks. Moreover, we introduce self-attention for cross-fusion matching.

2.1. Convolution Operation

2.1.1. Depthwise Separable Convolution

DSC [36] is composed of two processes, depth convolution and point-by-point convolution. First, each channel is separated to perform spatial convolution in the depth direction. Then all the results on channels are blended to perform point-by-point convolutions. Compared with the traditional CNN, the separable convolution only transforms the image once in the deep convolution, reducing the computational burden.

2.1.2. Deformable Convolutional Network

The feature extraction capability of DCN is enhanced by inserting offsets (e.g., deformable convolution) in the convolution layer [37], which serves to enable the network to learn the dynamic sensory domain. In the conventional CNN operation, the input feature map is sampled using a regular grid R. For each position, $p_0$ on the output feature map can be calculated by Equation (1).

$$y\left(p_0\right)=\sum _{p_n\in R}w\left(p_n\right)·x\left(p_0+p_n\right),$$

(1)

where $p_n$ is an enumeration of the positions listed in R. In the deformable convolution operation, the regular grid R is expanded by adding an offset, and the same position $p_0$ becomes:

$$y\left(p_0\right)=\sum _{p_n\in R}w\left(p_n\right)·x\left(p_0+p_n+\Delta p_n\right).$$

(2)

2.2. Fusion Module

The cross-fusion matching network aims to fuse multimodal image patches containing keypoints, which translates them into easy-to-match feature descriptors and generates a matching relationship matrix. Different feature fusion modules are introduced in deep neural networks to combine global and local information through global average pooling and feature connectivity, such as Spatial Pyramid Pooling [38], Receptive Field Block (RFB) [39], Atrous Spatial Pyramid Pooling (ASPP) [40], Pyramid Pooling Module (PPM) [41] and Feature Pyramid Network (FPN) [42]. All of these methods are aimed at the fusion of multi-scale, high-level and low-level, local and global features in the extraction process. However, the multimodal remote sensing image matching needs to improve the similarity of keypoint descriptors; therefore, they must capture each other’s information while achieving better cross-fusion and matching.

Self-attention [43] provides an effective way to model global contextual information through a triplet of (key, query, value), where the key and query are multiplied by points to obtain the corresponding attention weights, and the weights and value are multiplied by points to obtain the final output. For the multimodal remote sensing image matching network, self-attention layers in the Transformer are used for cross-modal feature description matching.

3. Methodology

3.1. Overall Framework Description

The proposed matching method consists of two parts, a detection network and a cross-fusion matching network, which are used for specific tasks. A detailed description of these components is shown in Figure 1.

In the detection network, let R, S be the reference and sensed images. DSCs are used to encode the input image to improve the computational efficiency of the network, while introducing DCN operations to enhance the geometric robustness of keypoints in the decoding stage. The weighted results from three scales are chosen to input into the peakiness measurement to obtain the candidate keypoint feature maps. The keypoints measurement is calculated conditional on the two images from the local spatial and channel domains respectively, additionally using $Softplus$ and confidence threshold $C(i,j)$ to activate the peaks to positive values. The detailed calculation procedure is described in this Section 3.2.

In the matching network, image patches containing the candidate keypoints are first fed into the position embedding operation, extracting position-dependent features. The image patches are cropped to be matched from their neighborhoods of keypoints for matching, and enhancing the local domain information of keypoints. Both the reference and sensed image patches are used as the input of the designed interactive fusion layers, which aim to exchange the information. Therefore, they capture each other’s information and transform it into an easy-to-match feature descriptor (feature description step). In the matching phase, multiple fully connected layers are used to optimize the feature descriptor by an overall constraint strategy, and normalize it to $\left[\mathrm{0,\; 1}\right]$ for obtaining the final matching matrix between $S\left(i\right)$ and $S\left(j\right)$ (matching step). The proposed cross-fusion network unifies “description + matching” steps into the cross-fusion network for global optimization.

3.2. Detection Network

Network architecture. Figure 1a depicts the structure of the detection network, where Conv, SConv and DCN denote CNN, DSC and DCN respectively. Images with a size of 224 × 224 pixels are as input. The convolutional kernels of the CNN are all of the sizes $3\times 3$, and each CNN layer is followed by ReLU [44] and BN [45]. The input images are first passed through Conv-0/1, then through SConv-0/…/20 to reduce the size of the feature map, where the previous outputs are reused and CNN operations are performed before each input. The network structure repeats the SConv-9/10/11 operation three times in the middle and then inputs to DCN-0/1/2.

SConv-8/20 and DCN-2 are rehabilitated to their original sizes by upsampling for obtaining undistorted features on-scale, and then are input to the network to generate the three keypoint feature maps, called ${\gamma }_1,{\gamma }_2 and {\gamma }_3$, respectively. For those multiple scale features, they are not assigned the same weight. This is because we have considered the abstraction of these features from low-level to high-level features. The final keypoint feature map is computed as follows:

$$O={\Delta }_1{\gamma }_1+{\Delta }_2{\gamma }_2+{\Delta }_3{\gamma }_3,$$

(3)

where ${\Delta }_1$, ${\Delta }_2$ and ${\Delta }_3$ are weights, and ${\Delta }_1+{\Delta }_2+{\Delta }_3=1$.

In addition, since the offsets $\Delta p_n$ are usually fractional, they are implemented by bilinear interpolation, and the number of channels is 128. The rest of the parameter settings are implemented in [37]. The detailed parameters of the network structure are shown in

Table 1. Detection network structure parameters. The abbreviations are as follows: C: CNN layer; S: DSC layer; D: DCN layer; mp: maximum pooling layer.

Layer	Size	Operation
c-0/1	$w\times h\times 32/\times 64$	3 × 3 c
c-2	$w\times h\times 32$	3 × 3 c
S-0/1/2	$w\times h\times 64/\times 64/\times 64$	2 × 2 c; mp
c-3	$w\times h\times 32$	3 × 3 c
S-3/4/5	$(w/2)\times (h/2)\times 64/\times 64/\times 64$	1 × 1 c; 3 × 3 c; mp
c-4	$w\times h\times 32/\times 64$	3 × 3 c
S-6/7/8	$(w/4)\times (h/4)\times 128/\times 128/\times 128$	1 × 1 c; 3 × 3 c; mp
S-9/10/11	$(w/8)\times (h/8)\times 256/\times 256/\times 256$	1 × 1 c; 3 × 3 c
S-12…20	$(w/8)\times (h/8)\times 512/$…$\times 512$	1 × 1 c; 3 × 3 c
D-0/1/2	$(w/8)\times (h/8)\times 128/\times 128/\times 128$	3 × 3 c

.

Keypoint detection. To obtain keypoints that are robust to scaling changes, three stages of feature maps in the network Sconv-8/20 and DCN-2 are used for keypoint detection. Subsequently, the outputs from the three stages are input to the upsampling network, which is used to recover the original size and assign the corresponding weights. Keypoints are determined from the peaks in the local spatial- and channel domains, as shown in Figure 2. Specifically, for each position $(i,j)$ and channel $(c=1,2,$…$,c)$ in the feature map output by the detection network, the local, spatial (${\alpha }_{ij}^c$) and channel (${\beta }_{ij}^c$) scores are calculated by:

$${\beta }_{ij}^c=\mathrm{softplus}\left(y_{ij}^c-\frac{1}{c}\sum _t{y}_{ij}^t\right),$$

(4)

where c is the feature map in the channel domain, $y_{ij}^c$ is a value on the feature map $(i,j)$. The activation function ($Softplus$) [46] serves to activate the keypoint feature map to a positive value.

$${\alpha }_{ij}^c=\mathrm{softplus}\left({\mathbf{y}}_{ij}^c-\frac{1}{\left|\mathcal{N}\right(i,j\left)\right|}\sum _{\left(i^{\prime },j^{\prime }\right)\in \mathcal{N}(i,j)}{y}_{i^{\prime }{j}^{\prime }}^c\right),$$

(5)

where $N(i,j)$ is the set of nine neighbors of the pixel $(i,j)$ and also its own pixel value.

To evaluate these two criteria, we create a single score map by maximizing the product of two scores in all feature maps c. The formula is as follows:

$${\gamma }_{ij}=\underset{c}{max}\left({\alpha }_{ij}^c{\beta }_{ij}^c\right).$$

(6)

Self-supervised detection.Figure 3 shows the self-supervised training process of the detection network. The SAR and optical images, which are pixel-aligned, are used as examples. A randomly selected image from these two images is a projection transformed by the matrix M, where the optical image is selected in the figure to be transformed for obtaining $o^{\prime }$. Then, s and $o^{\prime }$ are input to the detection network with shared weights to obtain the feature maps($Poin{t}_A$ and $Poin{t}_B$). The feature map $Poin{t}_A$ is transformed into the same coordinate as $Poin{t}_B$ using the projection transformation matrix M. Finally, the Huber loss function is used to determine the loss between the peak points, to attain the same confidence value for the keypoints at the same location. The Huber loss function is detailed as follows:

$$L_{\delta }(y,f\left(x\right))=\left\{\begin{array}{cc}\frac{1}{2}{(y-f\left(x\right))}^2\hfill & \mathrm{for}|y-f(x\left)\right|\le \delta \hfill \\ \delta |y-f\left(x\right)|-\frac{1}{2}{\delta }^2\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(7)

where $\delta$ is the optional hyperparameter, $f\left(x\right)$ is the predicted value, and y is the ground truth value.

3.3. Cross-Fusion Matching Network

Figure 4 depicts the proposed cross-fusion structure used for the keypoint feature description and matching. The reference and sensed patches containing keypoint are referred to as $F_o$ and $F_s$. For $F_o$ and $F_s$, the same position encoding as in [43] is employed for recording their position information on the original images. First, the position-encoded $F_o$ and $F_s$ are multiplied by their $query,key,value$ weights to obtain Q, K, and V, respectively, which are then fed to the respective encoders to convert the image blocks into a $1\times 256$ feature vector. To enhance the exchange of information between them, the Q, K, V of $F_o$ and $F_s$ are swapped to achieve cross-modal matching.

The goal of the network is to acquire a larger range of semantic information through a self-attention mechanism. $F_o$ and $F_s$ exchange their respective K, V, and the attention weights are calculated from the corresponding V and key vector K using Q to obtain the cross-fused similarity feature description. The specific computation process is as follows:

$$\mathrm{Attention}(Q,K,V)=softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V,$$

(8)

where $\sqrt{d_k}$ is the dimension of a Q and K, which is used to prevent the result from being too large.

Intuitively, the self-attention interaction fusion mechanism identifies relevant information that matches by comparing the similarity of the reference and sensed patches. The cross-fusion structure is interleaved eight times in the module. Finally, using numerous dense layers to represent matching possibilities, the two feature descriptors are normalized to $[0,1]$. The cross-entropy is used to supervise the learning process:

$$\mathrm{loss}=-\frac{1}{n}\sum _x[ylna+(1-y)ln(1-a)],$$

(9)

where $x,y$ and a denote the sample, label, and output value, respectively, and n denotes the total number of samples.

The matching network is trained by positive and negative samples generated from strictly pixel-aligned remote sensing image patches. The next section describes in detail the method of positive and negative sample generation.

3.4. Positive and Negative Sample Generation

Figure 5 outlines the random sample generation method. Optical(O) and SAR(S) images are applied as an example. Let O and S be the optical and SAR images, which are pixel-aligned. First, O and S are warped by using two different projection transformation matrices, where the warped images are $O^{\prime }$ and $S^{\prime }$. Then, a pair of corresponding points are selected in $O^{\prime }$ and $S^{\prime }$, and the image patches of the same size are cropped centered on these two points as positive samples, which ensures that the center pixels of these two patches are matched. The O and S centroids before transformation are used as the reference points for cropping, where the blue boxes indicate the positive samples after cropping. In contrast, negative samples are obtained by cropping the image at a fixed position without selecting a match point in O and S, where the yellow box indicates a negative sample cropped at a fixed position.

3.5. Matching and Parameter Setting

Two factors determine the image matching performance in image registration: (1) the repeatability of keypoints; (2) the robustness of the keypoint descriptor. Therefore, a dataset with multimodal remote sensing images covering different scenes is employed for training the detection network. The weight combinations $0,0.1,0.3,0.6,1.0$ are allocated to $w_1,w_2,w_3$ during the training and used to obtain stable keypoints. The parameter for the confidence threshold is set to 0.6.

The size of image patches input to the matching network invariably influences the robustness of keypoint feature descriptors, considering different imaging techniques are used in multimodal images. Therefore, patches with the size of 32 × 32, 48 × 48, 64 × 64, 96 × 96, 128 × 128 are selected for training to compare the performance of their keypoint descriptors and matching. Section 4 describes the detailed experimental results.

For the obtained correspondings, they are globally constrained by using the Randomized Sampling Consensus Algorithm (RANSAC) [47]. The point set is refined using the least squares algorithm to calculate the transformation matrix based on the obtained N matched point pairs. The sensed image is converted and aligned with the reference image according to the transformation matrix.

4. Experiments

In this section, we first introduce the dataset used to train the networks in Section 4.1. Then, we describe the details of evaluation metrics in Section 4.2. In Section 4.3, an ablation study is carried out to compare the performance between the network combination and scale weight parameters set in the detection network. Section 4.4 gives an ablation study for the cross-fusion matching network. Section 4.5 and Section 4.6 describe the performance of the proposed detection and matching network. In Section 4.7, the overall performance of our proposed method in multimodal image matching is evaluated.

4.1. Dataset

To train the network model, many optical and SAR images from the different areas are acquired to generate the dataset. The optical satellite sensor is SkySat, with a spatial resolution of 0.8 m. SAR images are acquired by Sentinel-1, which contains all the ground range detected scenes. Each scene is available in three resolutions and four band combinations (corresponding to the scene polarization). In the experiments, we used a combination of $VV+VH$ polarizations. According to the above training data generation method, 70,000 pairs of images are used for detection network training and 120,000 positive and negative samples are generated for matching network training. The original images used to generate the dataset (available at https://github.com/liliangzhi110/SARopticaldataset (accessed on 25 June 2022)), are released for benchmark evaluation.

For the test dataset, the optical satellite sensor is GF-2, with a spatial resolution of 0.8 m. SAR images are acquired by GF-3, which is C-band multi-polarization SAR satellite with a resolution of 1 m. In this paper, all experiments are conducted on an AMAX workstation with Ubuntu 18.04 LTS, RTX3090Ti GPU, and 128 GB RAM, where the initial learning rate is 0.0001 and the number of epochs is 300 with an Adam optimizer.

4.2. Evaluation Metrics

The performance of the detection network and the cross-fusion matching network is evaluated using the following evaluation protocol.

Repeatability. The pixels with the same position between two images are recognized as a keypoint pair, indicating that the keypoint pairs at this position is reliable. We use the repeatability ($n/N$) to evaluate the performance of the detection network, where $n,N$ are the number of repeatable and all keypoints obtained.

Mean Matching Accuracy (MMA). The performance of feature matching is evaluated by using the same definition of MMA as in D2-Net [33], which is the average percentage of correct matches in an image at a given threshold pixel.

Nearest Neighbor mean Average Precision (NN mAP). This metric evaluates the discriminative ability of the descriptor under multiple pixel thresholds. It is calculated by measuring area under curve of the Precision-Recall curve, using the nearest neighbor matching strategy.

Root-mean-square error(RMSR). We use RMSE to evaluate the overall performance of image matching. The calculation formula is as follows:

$$RMSE=\sqrt{\frac{1}{M}\sum _{i=1}^M{\left(x_i-x_j\right)}^2+{\left(y_i-y_j\right)}^2},$$

(10)

where m is the number of matching points, $x_i,x_j$ and $y_i,y_j$ are the coordinates of the correspondence on the x-axis and y-axis, respectively.

Match Score (M.S.) This metric is used to measure the overall performance of the whole pipeline (detection and cross fusion matching). It evaluates the overall performance by calculating the ratio of the ground truth correspondences that can be recovered for the entire process to the number of keypoints extracted within the same scene for the entire pipeline.

4.3. Ablation Study for the Detection Network

To aid in the design of the detection network described in Section 3, we made different combinations between the variants of the three-stage network. The three components were replaced with the original convolution, where the combinations are $CNN$, $CNN\cup DCN$, $CNN\cup DSCN$, $CNN\cup DSCN\cup DCN$.

In experiments, we analyzed the performance of keypoint detection by changing the combination of scale weighting parameters, where the scale parameter combinations could be enumerated as $W_1=(0.0,0,1.0)$, $W_2=(0.0,1.0,0.0)$, $W_3=(0.1,0.3,0.6)$, $W_4=(0.1,0.6,0.3)$, $W_5=(0.3,0.1,0.6)$, $W_6=(0.3,0.6,0.1)$, $W_7=(0.6,0.1,0.3)$, $W_8=(0.6,0.3,0.1)$ for the Sconv-8/20, and DCN-2 weights varying between 0 and 1. Therefore, an ablation study was performed between network combinations and scale weight parameters. The repeatability with different pixel thresholds was used to measure the detection performance. The specific parameter combination is shown in Figure 6.

Overall, the repeatability of $CNN\cup DCN\cup DSCN$ was higher than all the network combinations. $CNN\cup DCN$ could significantly improve the repeatability of keypoints compared to the $CNN\cup DSCN$ combination. It indicated that $DCN$ could cope with the image distortion better. The best experimental results were given by the combination of $W_3=(0.1,0.3,0.6)$, which was explained by the addition of feature multiscale information at different levels, while using the DSC layer to obtain geometric invariant features. In contrast, the single scale combination ($W_1=(0,0,1)$, $W_2=(0,1,0)$) were with a lower repeatability for keypoints. For the combination of multiple scales ($W_4=(0.1,0.6,0.3)$, $W_5=(0.3,0.1,0.6)$, $W_6=(0.3,0.6,0.1)$, $W_7=(0.6,0.1,0.3)$, $W_8=(0.6,0.3,0.1)$), the higher weight was given to DCN-2, the higher the number of keypoints obtained, which proved the importance of DSC layers for the detection network feature extraction.

Figure 7 provided additional insights about the keypoints generated with different weight combinations. The keypoint feature maps generated by the combination ($W_3=(0.1,0.3,0.6)$, $W_4=(0.1,0.6,0.3)$, $W_7=(0.6,0.1,0.3)$, $W_8=(0.6,0.3,0.1)$) were selected to compare the repeatability of the keypoints. The distribution showed that the keypoint feature maps with local maxima (local peaks) could be generated with all scale weight combinations. From the comparison of $W_3=(0.1,0.3,0.6)$ and $W_8=(0.6,0.3,0.1)$, they showed that the heatmap peaks of $W_3=(0.1,0.3,0.6)$ were more distinguishable, with better local smoothness. The heatmaps in $W_4=(0.1,0.6,0.3)$ and $W_7=(0.6,0.1,0.3)$ generated an irregular shape with peaks surrounding each other, lacking more distinguishable and concentrated keypoint peaks compared to $W_3=(0.1,0.3,0.6)$.

4.4. Ablation Study for the Matching Network

To fully understand the matching network, four different variants were evaluated. ViT was employed as the basis network ($ViTBase$) in ablation studies. The network variants between the ViT and interaction fusion modules were performed for the experiments ($InteF$). Furthermore, ablation studies were conducted on the layers of the interaction fusion module, with n = 6 ($InteF(layer=6)$), 8 ($InteF(layer=8)$) as the number of layers. For a fair comparison, the random data during training was set to be deterministic. The performance of the cross-fusion matching network was evaluated using MMA and NN mAP with different pixel thresholds. Detailed results of the ablation tests are described in

Table 2. Ablation test results in cross-modal networks.

	$\mathit{MMA}$						$\mathit{NNmAP}$
	≤2 px(%)	≤3 px(%)	≤4 px(%)	≤5 px(%)	≤6 px(%)	$(\%)$
$ViTBase$	0.106	0.129	0.564	0.293	0.426		0.675
$InteF$	0.116	0.171	0.635	0.398	0.559		0.743
$InteF(layer=6)$	0.134	0.136	0.572	0.387	0.573		0.739
$InteF(layer=8)$	0.107	0.130	0.580	0.373	0.569		0.724

.

Table 2 depicted the matching results on the test dataset. The best matching performance was achieved by $InteF$. The overall performance on the test dataset was worse while using $ViTBase$. The average MMA and NN mAP were greatly enhanced when the ViT was replaced with an interactive fusion module. For $InteF(layer=6)$, the MMA decreased at the 2–3 pixel thresholds, and the MMA decreased to 0.573 at 6 pixel thresholds. Overall, the NN mAP was reduced to 0.739. Similarly, for $InteF(layer=8)$, the NN mAP decreased to 0.724. The MMA at 2–3 pixel threshold decreased to 0.130. This suggested that increasing the number of interaction fusion layers caused the network to be less accurate, which could be owing to the considerable rise in similarity and loss in distinguishability produced by the increased number of layers.

4.5. Repeatability of the Detection Network

The performance of the detection network was evaluated with the the state-of-art methods (SIFT, Affine-SIFT [48], RIFT [49], Superpoint [34] and R2D2 [35]). Random SAR-optical image pairs($P_1\dots P_{12}$) were selected for testing, as shown in Figure 8. The network was assessed using the keypoint repeatability method. Confidence thresholds greater than 0.6 were considered keypoints in the proposed method.

Quantitative comparison.

Table 3. The repeatability of keypoints on $P_1\dots P_{12}$.

	Repeatability
	SIFT	ASIFT	RIFT	Superpoint	R2D2	Proposed
$P_1$	0.061	0.227	0.201	0.295	0.361	0.391
$P_2$	0.087	0.133	0.327	0.290	0.459	0.503
$P_3$	0.103	0.116	0.197	0.255	0.391	0.426
$P_4$	0.093	0.235	0.249	0.407	0.489	0.473
$P_5$	0.061	0.135	0.327	0.337	0.364	0.437
$P_6$	0.085	0.096	0.207	0.340	0.420	0.466
$P_7$	0.064	0.145	0.118	0.383	0.459	0.416
$P_8$	0.053	0.230	0.301	0.374	0.252	0.398
$P_9$	0.077	0.108	0.143	0.267	0.382	0.471
$P_{10}$	0.078	0.098	0.315	0.354	0.463	0.452
$P_{11}$	0.068	0.230	0.243	0.382	0.309	0.316
$P_{12}$	0.115	0.138	0.225	0.451	0.471	0.473

gave an overview of keypoint repeatability results on $P_1\dots P_{12}$. Overall, the proposed keypoint detection network achieved better performance. While Affine-SIFT generated a significant number of keypoints, the repeatability was poor. RIFT was higher than that of SIFT, which indicated that RIFT performed better for handling nonlinear radiation differences. However, its overall repeatability was lower than that of the proposed method, This might be that RIFT is built in a non-learning way for the specified sensor images, failing to obtain high reproducibility on the scene-rich test dataset. For Superpoint, its repeatability was lower than R2D2, which was since Superpoint used simulation data to train the keypoint, hardly reflecting the real remote sensing application scenes. For the R2D2 method, it found the correspondence by processing the images independently and did not consider the information differences between multimodal images, which made the repeatability of keypoints in multimodal images lower than our proposed method. The keypoint confidence threshold was used to filter keypoints on the proposed detection network, which determines the number of reproducible keypoints. Choosing a larger threshold could increase the repeatability of these keypoints, while reducing the overall number of keypoints. The confidence thresholds needed to be determined manually in practical applications. Therefore, we would study the algorithm for adaptive confidence threshold selection.

Qualitative comparison.Figure 8 showed the results of the detection on $P_1\dots P_{12}$. The results showed that corresponding keypoints were produced on all image pairs, especially for the regions with more pronounced texture changes. For $P_1$, $P_4$ and $P_{11}$, the ships differ in number on the two images, producing responses that were not reproducible due to the multi-temporal observations. For image pairs with simple geometry, such as $P_5\dots P_{10}$, their results were able to obtain responses at the same locations. For $P_2$, $P_3$, and $P_{10}$, the number of keypoints on SAR images was significantly higher than that of optical images, however, the responses of optical images could find their counterparts in SAR images. For $P_{12}$, the peak points produced at the same position were significantly lower than other effect pairs, which might be because of the significant radiometric difference on these two images, resulting in fewer responses of the keypoints at the same position.

4.6. Experimental Analysis of the Matching Network

In this section, numerous experiments were designed to test the effectiveness of the cross-fusion matching network. Individual pixel values were not relevant for feature description, hence image patches for feature matching required a certain scale size. Therefore, we first investigated the effect of input patch size on the performance of feature matching. Theoretically, larger patch sizes generated more identified and distinguished keypoint feature descriptors. To determine the optimal patch size, MMA with a 2.0 pixel threshold on the same dataset was evaluated in different patches sizes (32 × 32, 48 × 48, 64 × 64, 96 × 96 and 128 × 128), whose MMAs were 0.321, 0.539, 0.553, 0.596, 0.504, respectively. It illustrated that the matching performance gradually increased as the patch size increased and reached its highest value near the image patch with a size of 96 × 96, where the matching accuracy gradually leveled off. A patch with a size of 96 × 96 was selected as the following experimental size to improve the efficiency of the calculation while ensuring the matching performance.

To evaluate the effectiveness of the cross-fusion network for multimodal image matching, the proposed method was compared with SIFT, Affine-SIFT, RIFT, Superpoint, and R2D2. For SIFT and Affine-SIFT, they were matched by the Euclidean distance ratio between the corresponding feature nearest neighbors and second nearest neighbors with ratios of $0.60$, $0.70$, and $0.80$, respectively, where the best matching results were used for comparison. For RIFT, the matching parameters used in its online public code were used to conduct a comparison. For Superpoint and R2D2, the Euclidean distance was used to measure the similarity of feature descriptors. The MMA and NN mAP were used to measure the matching performance of the feature descriptors. The experimental results were shown in Figure 9.

The results showed that the proposed method significantly outperformed other methods in multimodal image matching, especially in the lower range of thresholds. It was worth mentioning that the learning-based methods overall outperformed SIFT, Affine-SIFT and RIFT, implying that the learning-based strategy might extract more robust features. Furthermore, the proposed method outperformed multi-task network-based detection and description methods (e.g., Superpoint and R2D2), demonstrating that the information fusion-based matching structure proved capable of communicating descriptor information, hence improving matching performance.

4.7. Overall Matching Performance

The effectiveness of the detection and matching networks had previously been assessed separately in prior studies; however, these experiments only examined the detection and matching networks separately and did not assess the total matching performance of the two networks. Accordingly, we jointly used them to match the test remote sensing images ($M_1\dots M_9$) with geometric and radiometric differences, where each pair of images had a size of 512 × 512, as shown in Figure 10.

The keypoints obtained between images contained some matching points that did not have repeatability. Therefore, RANSAC was used to eliminate these mismatched points. The proposed method would be compared with the state-of-art methods based on keypoints (such as SIFT, POS-SIFT, SAR-SIFT, Superpoint and D2-Net). For the SIFT-based methods, they were matched by the Euclidean distance ratio between the corresponding features, and the ratios were $0.6$, $0.7$, $0.8$, $0.9$ respectively. For Superpoint, the Euclidean distance was used to measure the similarity of feature matching. For D2-Net, the optimal combination of parameters was applied for comparison [35].

Table 4. The repeatability of matching points. $D_1,D_2,D_3,D_4,D_5,D_6$ are urban, suburban, industrial, pond, port and mountain scenes respectively.

	${\mathit{P}}_1$		${\mathit{P}}_2$		${\mathit{P}}_3$		${\mathit{P}}_4$		${\mathit{P}}_5$		${\mathit{P}}_6$		${\mathit{P}}_7$		${\mathit{P}}_8$		${\mathit{P}}_9$
	M.S.	RMSE	M.S.	RMSE	M.S.	RMSE	M.S.	RMSE	M.S.	RMSE	M.S.	RMSE	M.S.	RMSE	M.S.	RMSE	M.S.	RMSE
SIFT	0.158	7.55	0.185	9.5	0.136	9.93	0.169	8.01	0.198	8.1	0.105	8.19	0.165	9.65	0.182	8.95	0.157	9.95
POS-SIFT	0.165	9.39	0.116	7.81	0.145	9.85	0.097	8.47	0.084	8.05	0.149	7.65	0.148	9.38	0.164	9.35	0.142	7.55
SAR-SIFT	0.163	5.74	0.247	5.86	0.188	5.74	0.226	5.06	0.245	6.03	0.155	5.79	0.227	6.53	0.158	5.43	0.205	6.14
Superpoint	0.245	3.95	0.241	4.72	0.255	3.65	0.278	3.35	0.295	4.22	0.291	4.95	0.211	4.36	0.306	4.59	0.306	3.81
D2-Net	0.313	3.87	0.198	4.67	0.191	4.62	0.228	3.87	0.191	4.46	0.178	4.95	0.213	4.29	0.325	4.15	0.240	4.93
Proposed	0.362	3.10	0.288	3.06	0.246	3.52	0.276	3.38	0.348	3.94	0.232	3.22	0.246	3.26	0.348	3.49	0.306	3.76

listed the quantitative comparison among five methods. The M.S. and RMSE metrics were employed in the quantitative experiments. The proposed method provided higher M.S. and smaller RMSEs on these image pairs than existing comparative methods. SIFT presented the worst overall matching accuracy, which might be because the large radiometric differences between the SAR and optical images result in low repeatability of keypoints. Whereas the PSO-SIFT method used more constraints, which had a lower RMSE than SIFT, the overall matching results were unsatisfactory. For Superpoint, it obtained a lower M.S. on multimodal remote sensing images, which might be attributed to non-linear radiometric differences between images resulting in distinct feature descriptions. The Harris detector employed in SAR-SIFT was too sensitive to nonlinear radiometric differences, resulting in a low matching performance accuracy. The proposed method used a cross-fusion mechanism to make the network robust in both geometrical and radiometric variations, which obtained feature descriptions that fused the similarity between two images. Furthermore, the proposed method performed detection, description and matching on the original image, which enhanced image matching localization accuracy even more when compared to the R2D2 method for positioning on the output feature map.

Figure 10 presented the qualitative results of the proposed method’s overall performance, with blue lines indicating good matches and red indicating mismatches with a greater 4-pixel error. According to the results, the proposed method was capable of obtaining a uniformly distributed correspondence on all tested datasets. For SIFT and POS-SIFT, they could hardly obtain the correct correspondence, overall covered by the red line. SAR-SIFT for SAR and optical images obtained a higher number of correct correspondences. For Superpoint and D2-Net, the number of their correct correspondences was overall lower than that of the proposal. For the proposed method, the correspondence found on $M_2$, $M_4$, and $M_9$ was largely concentrated on construction areas, which could be since $M_2$, $M_4$, and $M_9$ were mostly covered by plants and water bodies with little textural differences. Overall, the proposed method was effective in providing correspondence on multimodal remote sensing images with geometrically and radiometrically invariant.

The above results demonstrated the effectiveness of the proposed detection and matching network for each step in the whole remote sensing image matching. Designing an applicable network for each step allowed the network to focus on the matching implementation of a single task. However, these separation steps in the implementation increased the complexity of data processing and required consideration of the appropriate parameters that should be selected in each step.

5. Conclusions

To obtain repeatable keypoints and cross-modal feature matching on multimodal remote sensing images, the self-supervised detection and cross-fusion matching networks are proposed in this paper. A detection network with weight-sharing allows keypoints to be automatically learned through self-supervised training. The cross-fusion matching network exchanges information between two images, transforming the multimodal image patches into robust feature descriptors. Specifically, this network unifies “description + matching” steps into one network for global optimization.

The experiment results demonstrate that the self-supervised detection network is capable of obtaining higher repeatability of keypoints on test datasets compared to SIFT, Affine-SIFT, RIFT, Superpoint and R2D2. We further evaluate the cross-fusion matching network, which can produce robust descriptors by the cross-fusion of image patches. For the overall performance, the average M.S and RMSE of the proposed method on the nine SAR-optical image pairs are about 0.298 and 3.41. Although two networks are proposed for the problems of detection and description in multimodal remote sensing image registration, this separate processing of tasks is still cumbersome. Therefore, we will study a unified network to achieve end-to-end matching in future studies.