SD-CapsNet: A Siamese Dense Capsule Network for SAR Image Registration with Complex Scenes

Abstract

SAR image registration is the basis for applications such as change detection, image fusion, and three-dimensional reconstruction. Although CNN-based SAR image registration methods have achieved competitive results, they are insensitive to small displacement errors in matched point pairs and do not provide a comprehensive description of keypoint information in complex scenes. In addition, existing keypoint detectors are unable to obtain a uniform distribution of keypoints in SAR images with complex scenes. In this paper, we propose a texture constraint-based phase congruency (TCPC) keypoint detector that uses a rotation-invariant local binary pattern operator (RI-LBP) to remove keypoints that may be located at overlay or shadow locations. Then, we propose a Siamese dense capsule network (SD-CapsNet) to extract more accurate feature descriptors. Then, we define and verify that the feature descriptors in capsule form contain intensity, texture, orientation, and structure information that is useful for SAR image registration. In addition, we define a novel distance metric for the feature descriptors in capsule form and feed it into the Hard L2 loss function for model training. Experimental results for six pairs of SAR images demonstrate that, compared to other state-of-the-art methods, our proposed method achieves more robust results in complex scenes, with the number of correctly matched keypoint pairs (NCM) at least 2 to 3 times higher than the comparison methods, a root mean square error (RMSE) at most 0.27 lower than the compared methods.

1. Introduction

Synthetic aperture radar (SAR) is used in a variety of applications such as disaster monitoring [1], agricultural monitoring [2], and geological surveys [3] due to its all-weather, all-day Earth observation capability. These applications require a variety of technical tools to support them, such as change detection [4], image fusion [5] and 3D reconstruction [6]. Image registration plays a vital role in these technical tools. Image registration is the process of transforming multiple images acquired at different times or from different viewpoints into the same coordinate system. However, implementing robust and highly accurate geometric registration of SAR images is a challenging task because SAR sensors produce varying degrees of geometric distortion when imaging at different viewpoints. In addition, SAR images suffer from severe speckle noise.

Existing SAR image registration methods can be roughly divided into two categories, i.e., area-based methods and feature-based methods. The area-based methods, also known as intensity-based methods, realize registration of image pairs by a template-matching scheme. Classical template-matching schemes include mutual information (MI) [7] and normalized cross-correlation coefficient (NCC) [8]. However, these area-based methods generally require high computation cost. Although the computational cost can be reduced to a large extent by precoarse registration [9,10] or local search strategies [11,12], the area-based methods still perform poorly in scenes with large geometric distortions.

In contrast to the area-based methods, the feature-based methods achieve registration by extracting feature descriptors of keypoints, and their registration speed is usually better than that of the area-based methods. The feature-based methods can be further divided into handcrafted approaches and learning-based approaches. The traditional handcrafted approaches for SAR image registration include SAR-SIFT [13,14,15], speeded-up robust features (SURF) [16,17], KAZE-SAR [18], etc. Although these methods reduce the interference of speckle noise in SAR image registration, there is still a large number of incorrectly matched point pairs in SAR image registration with complex scenes. One reason is that due to the complexity of texture and structural information in SAR images with complex scenes, the accuracy and repeatability of the detected keypoints is not well supported. Eltanany et al. [19] used phase congruency (PC) and a Harris corner keypoint detector to detect meaningful keypoints in SAR images with complex intensity variations. Xiang et al. [20] proposed a novel keypoint detector based on feature interaction in which a local coefficient of variation keypoint detector was used to mask keypoints in mountainous and built-up areas. However, it is difficult to detect uniformly distributed keypoints in SAR images with complex scenes using these methods. Another reason is that the handcrafted feature descriptors are unable to extract significant features such as line or contour structures in complex scenes. With the development of deep learning, methods such as convolutional neural networks (CNNs) have achieved exciting success in the field of SAR image interpretation. Quan et al. [21] designed a deep neural network (DNN) for SAR image registration and confirmed that deep learning can extract more robust features and achieve more accurate matching. Zhang et al. [22] proposed a Siamese fully convolutional network to achieve SAR image registration and used the strategy of maximizing the feature distance between positive and hard negative samples to train the network model. Xiang et al. [20] proposed a Siamese cross-stage partial network (Sim-CSPNet) to extract feature descriptors containing both deep and shallow features, which increases the expressiveness of the feature descriptors.Fan et al. [23] proposed a transformer-based SAR image registration method, which exploits the powerful learning capability of the transformer to extract accurate feature descriptors of keypoints under weak texture conditions.

Although CNN-based methods extract feature descriptors that contain salient information useful for registration, such as structure, intensity, orientation, and texture information, CNNs improve feature robustness by virtue of “translation invariance” in convolution and pooling operations. “Translation invariance” means that the CNN is able to steadily extract salient features from an image patch even when the patch undergoes small slides (including small translations and rotations) [24]. However, this “translation invariance” is fatal in image registration, as it leads to small displacement errors in the matched point pairs, as shown in Figure 1. In addition, the feature descriptors obtained by the CNN-based methods are mostly described in a vector form. However, in traditional deep learning networks, the activation of one neuron can only represent one kind of information, and the singularity of its dimensionality dictates that the neuron itself cannot represent multiple kinds of information at the same time [25]. Therefore, the multiple pieces of information of keypoints and the relationship between the information hidden in a large number of network parameters pose many problems for SAR image registration. For instance, (1) the network requires a large number of training samples to violently decipher the relationships between multiple kinds of information. However, the scarcity of SAR image registration datasets often leads to incomplete training of models or causes data to fall into local optima. (2) The presence of complex non-linear relationships in feature descriptors detaches them from their actual physical interpretation, resulting in poor generalization performance. (3) The feature descriptors in vector form are not capable of representing fine information about keypoints in complex scenes. When two keypoints at different locations have similar structure and intensity information, their feature descriptors may be very similar in vector space, so that the registration algorithm is unable to recognize the differences between the feature descriptors, which may result in mismatched point pairs. As shown in Figure 2, although the advanced registration algorithm Sim-CSPNet is used, keypoint A is still incorrectly matched to keypoint C.

Figure 1. Illustration of small displacement errors in the matched point pairs obtained by Sim-CSPNet [20].

Figure 2. Illustration of a mismatched point pair. (a) Registration results obtained by Sim-CSPNet [20]. (b) Enlarged regions of the mismatched point pair.

The advent of capsule network (CapsNet) [26] has broken the limitation that an active neuron in traditional neural networks can only represent certain kinds of information, achieving more competitive results than CNNs in various image application fields [25,27,28,29]. CapsNets encapsulate multiple neurons to form neuron capsules (NCs), enabling the evolution from “scalar neurons” to “vector neurons”. In this case, a single NC can represent multiple messages at the same time, and the network does not need to additionally encode the complex relationships between multiple pieces of information. Therefore, CapsNets require far less training data than CNNs, whereas the effectiveness of the deep features is no less than that of CNNs. Despite the many advantages of CapsNets over CNNs, there are still some limitations to the application of CapsNets for image alignment tasks; therefore, no relevant research has been published.

One limitation is that the traditional CapsNet adopts a single convolutional layer to extract deep features, which is not sufficient to obtain significant features. The most direct solution is to add more convolutional layers to extract deep features [29,30]. However, the small amount of training data cannot support the training of a deep model, and the increased backpropagation distance leads to reduced convergence performance of the network. In addition, as mentioned earlier, most feature descriptors are in the form of vectors, whereas the feature descriptors extracted by CapsNets are in a special matrix form (the rows of the matrix represent the NCs, and the columns of the matrix represent a variety of useful information). Therefore, how to measure the distance between the feature descriptors is one of the key difficulties that constrain the application of CapsNet to image registration.

In this paper, we propose a novel registration method for SAR images with complex scenes based on a Siamese dense capsule network (SD-CapsNet), as shown in Figure 3. The proposed registration method consists of two main components, namely a texture constraint-based phase congruency (TCPC) keypoint detector and a Siamese dense capsule network-based (SD-CapsNet) feature descriptor extractor. In SAR data processing, phase information has been shown to be superior to amplitude information, reflecting the contour and structural information of the image [31,32]. However, the phase information contains contour and structural information generated by geometric distortions such as overlays or shadows within complex scenes, which may affect the correct matching of keypoints. Therefore, we add a texture feature to the PC keypoint detector to constrain the keypoint positions. Specifically, when a keypoint has a large local texture feature, we consider that the keypoint may be located inside a complex scene, making it difficult to extract discriminative feature descriptors. Furthermore, considering the possible rotation between the reference and sensed images, the use of rotation-invariant texture features facilitates the acquisition of keypoints with high repeatability. Therefore, we adopt the rotation-invariant local binary pattern (RI-LBP) to describe the local texture features of the keypoints detected by the PC keypoint detector, and we discard a detected keypoint when its RI-LBP value is higher than the average global RI-LBP value. To break the limitation of traditional capsule networks in terms of deep feature extraction capability, we cascade three convolutional layers in front of the primary capsule layer and design a dense form of connection from the convolutional layer to the primary capsule layer, which makes the primary capsules contain both deep semantic information and shallow detail information. The dense connection form shortens the distance of loss backpropagation, which improves the convergence performance of the network and reduces the dependence of the model on training samples. In addition, in the parameter settings of most CapsNets, the dimension of the high-level capsules is greater than that of the primary capsules, allowing the high-level capsules to represent more complex entity information [29,30]. However, in our proposed method, high-level capsules are used as feature descriptors to describe important information of keypoints, such as structure information, intensity information, texture information, and orientation information; therefore, the feature descriptors extracted by SD-CapsNet require only four dimensions, greatly reducing the computational burden of the dynamic routing process. In addition, we define the L2-distance between capsules for feature descriptors in the capsule form. The effectiveness of the proposed method is verified using data for Jiangsu Province and Greater Bay Area in China obtained by Sentinel-1.

Figure 3. Schematic diagram of the proposed method.

The main contributions of this paper are briefly summarized as follows.

(1)

We propose a texture constraint-based phase congruency (TCPC) keypoint detector that can detect uniformly distributed keypoints in SAR images with complex scenes and remove keypoints that may be located in overlay or shadow regions, thereby improving the high-repeatability of keypoints;

(2)

We propose a Siamese dense capsule network (SD-CapsNet) to implement feature descriptor extraction and matching. SD-CapsNet designs a dense connection to construct the primary capsules, which shortens the backpropagation distance and makes the primary capsules contain both deep semantic information and shallow detail information;

(3)

We innovatively construct feature descriptors in the capsule form and verify that each dimension of the capsule corresponds to intensity, texture, orientation, and structure information. Furthermore, we define the L2 distance between capsules for feature descriptors in the capsule form and combine this distance with the hard L2 loss function to implement the training of SD-CapsNet.

The rest of this paper is organized as follows. In Section 2, the proposed method is introduced. Section 3 presents the experimental results of our proposed method, as well as a comparisons with other state-of-the-art methods. Section 4 includes a discussion on the effectiveness of the keypoint detector and feature descriptor extractor. Finally, conclusions are presented in Section 5.

2. Methodology

Figure 3 shows the overall architecture of the proposed method. The proposed method consists of four steps: keypoint detection, feature descriptor extraction, feature descriptor matching, and geometric transformation. First, the PC keypoint detector is used to capture keypoints with significant structural features, followed by constrained keypoint detection results using the texture feature obtained by RI-LBP. Second, SD-CapsNet is proposed to extract feature descriptors of keypoints. Thirdly, a novel distance metric is defined to measure the distance between feature descriptors in the form of capsules. Finally, random sample consensus (RANSAC) is used to remove mismatched point pairs, and least squares is used to calculate the projection transformation matrix. Researchers generally agree that after removing mismatched point pairs by RANSAC or fast sample consensus (FSC), all that remains are correctly matched point pairs. In this section, we will introduce the details and contributions of the first three modules.

2.1. Texture Constraint-Based Phase Congruency Keypoint Detector

Many current keypoint detectors are based on gradient information, e.g., DoG [33], Sobel [34], Harris [35], etc. However, these operators are unable to extract accurate and well-distributed keypoints in weakly textured regions of SAR images, as shown in Figure 4b. As shown in Figure 4c, PC is a detector based on local phase information and is therefore able to extract structural information in weakly textured regions of SAR images.

Figure 4. Comparison of phase congruency and gradient. (a) SAR image. (b) Gradient feature. (c) Phase congruency feature.

PC can be calculated on multiple scales and in multiple directions using log Gabor wavelets according to the following formula [36].

$$\left\{\begin{array}{c}\mathrm{PC}\left(x\right)=\frac{{\mathsf{\Sigma }}_{n}W\left(x\right)⌊A_n\left(x\right)\Delta {\varphi }_n\left(x\right)-T⌋}{{\Sigma }_n{A}_n\left(x\right)+\epsilon }\\ \Delta {\varphi }_n\left(x\right)=cos\left({\varphi }_n\left(x\right)-\overline{\varphi }\left(x\right)\right)-\left|sin\left({\varphi }_n\left(x\right)-\overline{\varphi }\left(x\right)\right)\right|,\end{array}\right.$$

(1)

where $A_n$ and ${\varphi }_n\left(x\right)$ are the amplitude and phase angle of the n-th Fourier component at location x, T is the estimated noise threshold, $W\left(x\right)$ is the weighting factor for the frequency spread, $\epsilon$ is a small constant to avoid division by zero, the symbol $\lfloor \rfloor$ denotes that the enclosed quantity is equal to itself if its value is positive and zero otherwise, and $\overline{\varphi }\left(x\right)$ is the mean phase angle.

Following the classical moment analysis formulas, the maximum (M) and minimum (m) moment of PC are computed as

$$\begin{array}{c}M=\frac{1}{2}\left[a+c+\sqrt{b^2+{(a-c)}^2}\right],\hfill \\ m=\frac{1}{2}\left[a+c-\sqrt{b^2+{(a-c)}^2}\right],\hfill \\ \left\{\begin{array}{c}a={\sum \left[\mathrm{PC}\left(\theta \right)cos\left(\theta \right)\right]}^2\hfill \\ b=2\sum \left[\mathrm{PC}\left(\theta \right)cos\left(\theta \right)\right]·\sum \left[\mathrm{PC}\left(\theta \right)sin\left(\theta \right)\right]\hfill \\ c={\sum \left[\mathrm{PC}\left(\theta \right)sin\left(\theta \right)\right]}^2\hfill \end{array}\right.\hfill \end{array}$$

(2)

where $\mathrm{PC}\left(\theta \right)$ denotes the phase congruency value determined at orientation $\theta$. The maximum and minimum moments of PC represent the edge and corner strength information, respectively. In this paper, we select the first 10,000 points of the sum of the PC maximum and minimum moment as the proposal keypoints, as shown in Figure 5a. Although the proposal keypoints are well distributed at the edges and corners, where the structural information is strong, many keypoints are also detected in the interior of complex scenes. However, complex scenes of SAR images may have varying degrees of geometric distortion (such as overlays or shadows) at different viewpoints, resulting in small possible displacement deviations between keypoint pairs. Therefore, we propose constraining the keypoint positions using texture information. Considering the possible rotation between the reference image and the sensed image, the RI-LBP is used to describe the local texture features of the keypoints.

Figure 5. Visualization of keypoint detection results. (a) Proposal keypoints obtained by the PC detector. (b) Texture features obtained by RI-LBP. (c) Keypoints obtained by the TCPC detector.

LBP is an operator used to describe the local texture features of images, which can be written as [37]

$$LB{P}_{P,R}=\sum _{p=0}^{P-1}s\left(g_p-g_c\right)2^p,s\left(x\right)=\left\{\begin{array}{cc}1,\hfill & x\ge 0\\ 0,\hfill & x<0\end{array}\right.$$

(3)

where $g_c$ and $g_p$ denote the gray values of the central pixel and its neighbors, respectively; R and P denote the radius of the circularly neighborand the number of neighbors, respectively; and p is the index of the neighbor.

Ojala et al. [38] proposed a rotation-invariant local binary pattern (RI-LBP) to extract a rotation-invariant texture feature.

$$\left.LB{P}_{P,R}^{ri}=min\{ROR\left(LB{P}_{P,R},k\right)\mid k=0,1,\dots ,P-1\right\}$$

(4)

where $ROR\left(x|k\right)$ indicates that the binary number (x) is cyclically shifted to the right k times.

Finally, the proposal keypoints obtained by the PC detector are identified as the final keypoints by RI-LBP according to the following rules.

$$K{P}_{final}=K{P}_i\times ⌊\overline{LB{P}_{P,R}^{ri}}-LB{P}_{P,R}^{ri}\left(K{P}_i\right)⌋,K{P}_i\in K{P}_{proposal}$$

(5)

where $K{P}_{final}$ and $K{P}_{proposal}$ denote the final keypoints and proposal keypoints, respectively; $\overline{LB{P}_{P,R}^{ri}}$ denotes the global average value of the RI-LBP feature; and the symbol $\lfloor \rfloor$ denotes that the enclosed quantity is equal to itself if its value is positive and zero otherwise. Then, the keypoints inside complex scenes are removed, as shown in Figure 5c.

2.2. Siamese Dense Capsule Network-Based Feature Descriptor Extractor

In CNN-based feature descriptor extractors, various pieces of useful information are mixed together, resulting in feature descriptors losing their actual physical significance. CapsNet-based feature descriptor extractors encapsulate multiple neurons into neuron capsules that can represent multiple kinds of information at the same time. Figure 6 shows the difference between two types of feature descriptors. In this paper, we divide the information useful for SAR image registration into four categories, i.e., structure, intensity, texture, and orientation information. In the feature descriptors extracted by CapsNet-based methods, each row represents a capsule that contains various types of information, and each column represents a type of information that contains multiple features of this type of information.

Figure 6. Illustration of the difference between two types feature descriptors. (The different shapes represent different types of information, and the different colors represent different features).

Inspired by a densely connected convolutional network (DenseNet) [39] and a cross-stage partial network (CSPNet) [40], we design a new way to connect capsule networks, whereby the convolutional layers are able to connect directly to the primary capsule layer. As shown in Figure 7a, in the traditional connection of CapsNets, losses need to be backpropagated layer by layer, and the backpropagation distance increases with the number of layers in the network. However, in our proposed densely connected CapsNet, the loss of the primary capsule layer can go straight to each convolutional layer, greatly reducing the backpropagation distance, and the backpropagation distance does not increase with the number of convolutional layers, as shown in Figure 7b. In addition, the primary capsules of SD-CapsNet can contain both detailed information from low-level convolutional layers and salient information from high-level convolutional layers.

Figure 7. Illustration of the backpropagation pathway. (a) Traditional connected pathway. (b) The proposed densely connected pathway.

Next, we use a dynamic routing mechanism to deliver the output of the primary capsules to the feature descriptors. A diagram of the dynamic routing mechanism is shown in Figure 8. In the dynamic routing mechanism, the process of transferring the primary capsule ($u_i$) to the feature descriptor ($f_j$) is defined as

$$f_j={\sum }_i{c}_{ij}{\widehat{u}}_{j|i}$$

(6)

where ${\widehat{u}}_{j|i}=W_{ij}{u}_i$. $W_{ij}$ denotes the matrix of connection weights from the i-th capsule to the j-th capsule, $c_{ij}$ represents coupling coefficient, which can be written as

$$c_{ij}=\frac{exp\left(b_{ij}\right)}{{\displaystyle \sum _k}exp\left(b_{ik}\right)}$$

(7)

where $b_{ij}$ represents the prior probability of the i-th capsule relative to the j-th capsule.

Figure 8. Illustration of the dynamic routing mechanism.

In most CapsNets, the dimensions of the high-level capsules are much larger than those of the primary capsules so that the high-level capsules represent more complex information. However, information with too many dimensions may be divorced from the actual physical interpretation and place an additional computational burden on the network. In SAR image registration tasks, the information that positively influences the registration results can be divided into four categories, i.e., intensity, orientation, texture, and structure information. Therefore, we set the information dimension of the feature descriptors to 4, and for consistency with the dimensions of other feature descriptors, the feature dimension is set to 32. Now, the size of the feature descriptors extracted by the proposed method is $32\times 4$, as shown in Figure 9a. In other words, the common vector form ($128\times 1$) of the feature descriptors is replaced with a specific matrix form ($32\times 4$).

Figure 9. Illustration of the feature descriptor form. (a) The size of the feature descriptor. (b) The mathematical representation of feature descriptors.

The goal of training is to maximize the distance between the positive matches and hard negative matches. Before calculating the loss function, the distance metric between the feature descriptors in capsule form needs to be determined. Due to the special form of feature descriptors, the useful information of keypoints is confused after capsules are reshaped into one-dimensional vectors. Therefore, classical Euclidean distance, L2 distance, and cosine distance are not applicable in capsule networks. For the two given feature descriptors ($F_1=\left[\overrightarrow{a_1},\overrightarrow{a_2},\overrightarrow{a_3},\overrightarrow{a_4}\right]$ and $F_2=\left[\overrightarrow{b_1},\overrightarrow{b_2},\overrightarrow{b_3},\overrightarrow{b_4}\right]$), the distance between $F_1$ and $F_2$ can be defined as

$$\begin{array}{cc}\hfill d(F_1,F_2)& ={∥F_1-F_2∥}_2\hfill \\ \hfill & ={∥\overrightarrow{a_1}-\overrightarrow{b_1},\overrightarrow{a_2}-\overrightarrow{b_2},\overrightarrow{a_3}-\overrightarrow{b_3},\overrightarrow{a_4}-\overrightarrow{b_4}∥}_2\hfill \\ \hfill & =\left|\sqrt{\odot }\left(\sum _{i=1}^4{\left(\overrightarrow{a_i}-\overrightarrow{b_i}\right)}^2\right)\right|\hfill \\ \hfill & =\left|\sqrt{\odot }\left(\sum _{i=1}^4\left(\left(\overrightarrow{a_i}-\overrightarrow{b_i}\right)\odot \left(\overrightarrow{a_i}-\overrightarrow{b_i}\right)\right)\right)\right|\hfill \end{array}$$

(8)

where ⊙ denotes the Hadamard product, which can be expressed as $\overrightarrow{z}=\overrightarrow{x}\odot \overrightarrow{y}=[x_1{y}_1,x_2{y}_2,\dots ,x_n{y}_n]$. Similarly, $\sqrt{\odot }$ can be expressed as $\overrightarrow{p}=\sqrt{\odot }\left(\overrightarrow{z}\right)=[\sqrt{z_1},\sqrt{z_2},\dots ,\sqrt{z_n}]$. $\left|·\right|$ denotes the module of the vector contained therein.

Then, we feed the distance function between feature descriptors into the hard L2 loss function [41]. For a given batch of SAR image pairs (${\left(R_i,S_i\right)}_{i=1\cdots n}$), feature descriptors (${\left(F_i^R,F_i^S\right)}_{i=1\cdots n}$) are extracted by SD-CapsNet. The non-matched feature descriptor ($F_{j_{min}}^S$) closest to $F_i^R$ can be calculated as

$$j_{min}={argmin}_{j=1\cdots n,j\ne i}d\left(F_i^R,F_j^S\right)$$

(9)

The non-matched feature descriptor ($F_{k_{min}}^R$) closest to $F_i^S$ can be calculated as

$$k_{min}={argmin}_{k=1\cdots n,k\ne i}d\left(F_k^R,F_i^S\right)$$

(10)

Finally, the triplet margin loss can be written as

$$L=\frac{1}{n}\sum _{i=1}^{n}max\left\{0,1+d\left(F_i^R,F_i^S\right)-min\left(d\left(F_i^R,F_{j_{min}}^S\right),d\left(F_{k_{min}}^R,F_i^S\right)\right)\right\}$$

(11)

As mentioned earlier, feature descriptors in capsule form contain intensity, texture, structure and orientation information. To verify this conclusion, we reconstruct the feature descriptors as images and apply separate perturbations to each type of information during the reconstruction process. The reconstruction process is implemented by a decoder, which contains three fully connected layers, the structure and parameters of which are shown in Table 1.

Table 1. Model specifications and parameters of the decoder.

Block	Layer	Input Size	Output Size
Input layer	One column of the feature descriptor		32 × 1
Fully connected layer	Linear (32, 256); “RelU”	32 × 1	256 × 1
Fully connected layer	Linear (256, 512); “RelU”	256 × 1	512 × 1
Fully connected layer	Linear (512, 1024); “Sigmoid”	512 × 1	1024 × 1
Output layer	Reshape	1024 × 1	32 × 32

As shown in Figure 10, we add perturbations in the magnitude range of [−0.25, 0.25] to each column of the feature descriptor with a perturbation interval of 0.05. When perturbation is added to the column where the intensity information is located, the intensity information of the SAR image changes globally. When a perturbation is added to the column where the texture information is located, the texture information of the SAR image is changed locally. We add perturbations to the column where the orientation information is located, and as the perturbations are added, the SAR image orientation is gradually rotated. We add perturbations to the column where the structure information is located, and as the perturbations are added, the structure information of the SAR image changes to different degrees.

Figure 10. Visualization of reconstruction results for feature descriptors under different perturbations.

3. Experimental Results and Analysis

3.1. Data Description and Parameter Settings

In this paper, we collect large-scale SAR image pairs of four scenes for the production of a training dataset, as shown in Figure 11. The training dataset comprises C-band data from the Gaofen-3 system. To completely separate the test data from the training dataset, six SAR image pairs in the C-band from Sentinel-1 system are selected for testing. The incidence angle of the reference image is approximately ${30.73}^{\circ }$ in the near range and ${46.07}^{\circ }$ in the far range. The incidence angle of the sensed image is approximately ${30.71}^{\circ }$ in the near range and ${46.05}^{\circ }$ in the far range. Both the reference and sensed images have the same resolution of 10 m × 10 m. Detailed information about the six SAR image pairs is listed in Table 2.

Figure 11. The training dataset of SAR images obtained by the Gaofen-3 system. (a) Desert. (b) Mountain. (c) Lake. (d) Urban.

Table 2. Detailed Information about SAR image pairs used for testing.

Site	Size	Resolution (Range × Azimuth)
Pair 1	836 × 583	10 m × 10 m
	663 × 488
Pair 2	1027 × 752	10 m × 10 m
	1339 × 911
Pair 3	1095 × 1201	10 m × 10 m
	1171 × 1143
Pair 4	918 × 683	10 m × 10 m
	976 × 666
Pair 5	1135 × 961	10 m × 10 m
	945 × 916
Pair 6	500 × 499	11 m × 14 m
	600 × 600

In our experiments, the PC is calculated at three scales and in six orientations, and the scaling factor between successive filters is set to 1.6. The RI-LBP is calculated in a circular neighborhood with a radius equal to three ($R=3$), and the number of sampled neighbors is set to eight ($P=8$). The threshold of RANSAC is set to 0.8. The number of kernels in the three convolutional layers is set to 64, 128, and 256, respectively. The dimension of the primary capsules is set to eight. “Adam” is employed as an optimizer, and the learning rate is set to 0.001. The detailed model specifications and parameters are listed in Table 3. The basic experimental environment settings are listed in Table 4. The comparison methods are introduced briefly as follows.

Table 3. Model specifications and parameters of the SD-CapsNet.

Block	Layer	Input Size	Output Size (Width × Height × Channel)
Input layer			32 × 32 × 1
Conv layer 1	Conv (3 × 3), stride (1), ReLU	32 × 32 × 1	32 × 32 × 64
Low-level capsule Layer	Conv (9 × 9), stride (2), ReLU	32 × 32 × 64	12 × 12 × 64
	Reshape	12 × 12 × 64	1152 × 8
Conv layer 2	Conv (3 × 3), stride (1), ReLU	32 × 32 × 64	32 × 32 × 128
Medium-level capsule Layer	Conv (9 × 9), stride (2), ReLU	32 × 32 × 128	12 × 12 × 128
	Reshape	12 × 12 × 128	2304 × 8
Conv layer 3	Conv (3 × 3), stride (1), ReLU	32 × 32 × 128	32 × 32 × 256
High-level capsule layer	Conv (9 × 9), stride (2), ReLU	32 × 32 × 128	12 × 12 × 256
	Reshape	12 × 12 × 256	4608 × 8
Primary capsule layer	Concat	1152 × 8, 2304 × 8, 4608 × 8	8064 × 8
Feature descriptor layer	Dynamic routing	8064 × 8	32 × 4

Table 4. The basic experimental environment settings.

Platform	Linux
Torch	V 1.10.1
CPU	Intel(R) Xeon(R) Silver 4210R
Memory	64G
GPU	Nvidia GeForce RTX 3090
Video memory	24G

• SAR-SIFT [13] proposed SAR-Harris instead of DoG to detect keypoints and used the circular descriptor to describe spatial context information;
• SOSNet [42] used second-order similarity regularization for local descriptor learning, and proposed SOSLoss for registration model training with a small training sample;
• Sim-CSPNet [20] proposed a feature interaction-based keypoint detector and used a Siamese cross-stage partial network to generate feature descriptors.

3.2. Registration Results

To evaluate the registration performance of the proposed method, we provide the keypoint matching results with different methods, as shown in Figure 12 and Figure 13. Table 5 presents the quantitative comparison results. Note that for a fair comparison, SOSNet, Sim-CSPNet, and the proposed SD-CapsNet use the same training dataset.

Figure 12. The feature descriptor matching results with different methods for pair 1. (a) SAR-SIFT. (b) SOSNet. (c) Sim-CSPNet. (d) Our proposed SD-CapsNet.

Figure 13. The feature descriptor matching results with different methods for pair 2. (a) SAR-SIFT. (b) SOSNet. (c) Sim-CSPNet. (d) Our proposed SD-CapsNet.

Table 5. Comparison of the NCM, RMSE, and Time (s) of Six SAR Image Pairs with Different Methods.

Method	Pair 1			Pair 2			Pair 3			Pair 4			Pair 5			Pair 6
	NCM	RMSE	Time	NCM	RMSE	Time	NCM	RMSE	Time	NCM	RMSE	Time	NCM	RMSE	Time	NCM	RMSE	Time
SAR-SIFT	33	0.77	718.51	70	0.68	2496.3	46	0.77	2807.37	23	0.89	975.43	40	0.76	1625.38	14	0.87	369.9
SOSNet	94	0.77	28.78	60	0.72	40.42	61	0.73	39.89	84	0.79	18.06	118	0.84	19.19	17	0.82	15.39
Sim-CSPNet	24	0.61	3.15	121	0.63	5.63	105	0.65	7.01	48	0.76	5.08	43	0.74	5.68	51	0.71	3.60
SD-CapsNet	156	0.59	2.05	391	0.45	5.43	369	0.46	6.75	310	0.74	3.60	325	0.73	5.39	93	0.66	3.39

As shown in Figure 12, the distribution of matched keypoint pairs is non-uniform in the results of SOSNet and Sim-CSPNet, and the number of correctly matched keypoint pairs (NCM) in SAR-SIFT results is low. However, our proposed method obtains the most uniform and the largest number of correctly matched keypoint pairs. Figure 13 shows that the NCM of the registration results obtained by SAR-SIFT, SOSNet, and Sim-CSPNet is much lower than that of the registration results obtained by our proposed method because the comparison methods are not adjusted for complex scenes and therefore cannot be adapted to the registration task of SAR images with complex scenes. From Figure 14, Figure 15, Figure 16 and Figure 17, we can obtain the same conclusion that the proposed method is able to obtain the most uniform and the largest number of correctly matched keypoint pairs.

Figure 14. The feature descriptor matching results with different methods for pair 3. (a) SAR-SIFT. (b) SOSNet. (c) Sim-CSPNet. (d) Our proposed SD-CapsNet.

Figure 15. The feature descriptor matching results with different methods for pair 4. (a) SAR-SIFT. (b) SOSNet. (c) Sim-CSPNet. (d) Our proposed SD-CapsNet.

Figure 16. The feature descriptor matching results with different methods for pair 5. (a) SAR-SIFT. (b) SOSNet. (c) Sim-CSPNet. (d) Our proposed SD-CapsNet.

Figure 17. The feature descriptor matching results with different methods for pair 6. (a) SAR-SIFT. (b) SOSNet. (c) Sim-CSPNet. (d) Our proposed SD-CapsNet.

The quantitative comparison results of different methods are listed in Table 5. The NCM of SAR-SIFT is lower than that of the other comparison methods, whereas the running time is much longer than that of the other comparison methods. Although SOSNet is able to achieve larger NCMs, its RMSE is higher than that of the other comparison methods, and its running time is much longer than that of Sim-CSPNet and SD-CapsNet. Since Sim-CapsNet is designed with a keypoint detector based on feature intersection, it is able to detect keypoints with high repeatability and therefore has a lower RMSE than SAR-SIFT and SOSNet. However, Sim-CSPNet is unable to detect uniformly distributed matched keypoint pairs in SAR images with complex scenes. Our proposed SD-CapsNet achieves the highest NCM and lowest RMSE and uses the least amount of runtime compared to other comparison methods. Experimental results show that our proposed method is able to detect uniformly distributed keypoints in complex scenes and to extract accurate feature descriptors for keypoints.

To show the registration results more intuitively, we present final registration results in terms of checkboard overlay, as shown in Figure 18. In the registration results of SAR image pairs 1–3, SAR-SIFT and SOSNet produce incoherent regions, as shown in the red box regions in Figure 18. In the registration results for SAR image pairs 4 and 5, SAR-SIFT and Sim-CapsNet are unable to estimate accurate transformation models because they have fewer NCMs, resulting in incoherent regions, as shown in Figure 18(a4,c4,a5,c5). As shown in Figure 18(b5), although SOSNet obtains more NCMs, its RMSE is higher than that of other comparison methods; therefore, its results show incoherent regions. In the registration results of SAR image pair 6, the NCMs of SAR-SIFT and SOSNet are too small to accurately estimate the transformation model, resulting in significant incoherent regions, as shown in Figure 18(a6,b6). Our method outperforms other comparison methods in terms of NCM and RSME, so the checkerboard results of the proposed method have no obvious errors, as shown in Figure 18(d1–d6).

Figure 18. Checkboard overlay maps of the different methods. (a1–a6) Registration results of SAR-SIFT. (b1–b6) Registration results of SOSNet. (c1–c6) Registration results of Sim-CSPNet. (d1–d6) Registration results of SD-CapsNet.

Finally, we show a enlarged map of a selected region of the registration results for pair 5 to explain the link between the keypoint matching result and the registration result, as shown in Figure 19. In the keypoint matching results of SOSnet and Sim-CSPNet, no correctly matched keypoint pairs appear in the selected region; therefore, incoherent roads are produced in the region of the registration results. In the keypoint matching results of SAR-SIFT, only a few correctly matched keypoint pairs appear in the selected region, which still produce incoherent roads, despite improved registration results compared to SOSnet and Sim-CSPNet. However, our proposed method achieves more correctly matched keypoint pairs in selected regions, so the transformation model is calculated more accurately, and no incoherent roads emerge.

Figure 19. Enlarged maps of the registration results for pair 5 with different methods. (a) SAR-SIFT. (b) SOSNet. (c) Sim-CSPNet. (d) SD-CapsNet.

4. Discussion

4.1. Effectiveness of the TCPC Keypoint Detector

To verify the effectiveness of the proposed TCPC keypoint detector, ablation experiments are conducted on SAR image pair 6. Specifically, five different keypoint detectors are selected, and the same SD-CapsNet is used to implement feature descriptor extraction and matching. The feature descriptor matching results are shown in Figure 20. We can observe that the matched keypoint pairs obtained by the comparison method are not uniformly distributed in the SAR image. The NCMs of the comparison method are also much smaller than those of the proposed method according to visual inspection. Our proposed TCPC keypoint detector obtains more uniform and a larger number of correctly matched keypoint pairs.

Figure 20. The feature descriptor matching results with different keypoint detectors. (a) DoG. (b) FAST. (c) Harris. (d) SAR-Harris. (e) Feature interaction. (f) Our proposed TCPC.

We present quantitative comparison results of different keypoint detectors in Table 6. The network takes more registration time when using the DoG, FAST, Harris, and SAR-Harris detectors, indicating that they can detect more keypoints than feature interaction (FI) and TCPC keypoint detectors. However their NCM is much smaller than that of FI and TCPC keypoint detectors, indicating that the repeatability of keypoints is not high. The FI detector detects a smaller number of keypoints and therefore yields a smaller NCM than that of TCPC. Our proposed TCPC detector detects keypoints with the highest repeatability in an acceptable detection time, thereby obtaining the largest NCM.

Table 6. Quantitative comparison of feature descriptor matching with different keypoint detectors.

Method	DoG [33]	FAST [43]	Harris [35]	SAR-Harris [13]	FI [20]	Proposed TCPC
Detection time	0.10	0.01	0.57	44.27	2.40	1.17
Matching time	2.65	3.28	2.44	2.81	1.14	2.22
Total time	2.75	3.29	3.01	47.08	3.54	3.39
NCM	14	14	22	33	74	93
RMSE	0.78	0.74	0.75	0.69	0.66	0.66

4.2. Effectiveness of the SD-CapsNet Feature Descriptor Extractor

To verify the effectiveness of the proposed SD-CapsNet, ablation experiments are conducted on SAR image pair 6. Specifically, three different registration networks are selected, and the same TCPC is used to implement keypoint detection. In particular, CapsNet represents a capsule network without dense connections, with the same number of network layers and parameter settings as SD-CapsNet. The feature descriptor matching results with different feature descriptor extractors are shown in Figure 21. CapsNet and SOSNet are unable to obtain a uniformly distributed and sufficient number of correctly matched keypoint pairs.

Figure 21. The feature descriptor matching results with different feature descriptor extractors. (a) TCPC+SOSNet. (b) TCPC+Sim-CSPNet. (c) TCPC+CapsNet. (d) TCPC+SD-CapsNet.

We present quantitative comparison results of different feature descriptor extractors in Table 7. The traditional capsule network has a much smaller NCM than that of SD-CapsNet due to the outdated connection method. Experimental results show that the proposed method achieves the largest NCM and the lowest RMSE with the shortest running time compared to the other comparison methods.

Table 7. Quantitative comparison of feature descriptor matching with different feature descriptor extractors.

Method	TCPC+SOSNet	TCPC+Sim-CSPNet	TCPC+CapsNet	SD-CapsNet
RMSE	0.74	0.71	0.71	0.66
NCM	22	55	57	96
Time(s)	8.85	3.47	3.23	3.39

5. Conclusions

In this paper, we propose a TCPC keypoint detector and a SD-CapsNet feature descriptor extractor to address the problems of inaccurate keypoint detection and inadequate feature descriptor extraction in SAR image registration of complex scenes. First, we extract the SAR image texture features using RI-LBP to remove the keypoints detected by the PC detector in the strong texture regions. Secondly, we propose SD-CapsNet to extract feature descriptors of keypoints, which can shorten the backpropagation distance and make the primary capsules contain both deep semantic information and shallow detail information. In addition, we construct feature descriptors in the capsule form and verify that each dimension of the capsule corresponds to intensity, texture, orientation, and structure information. Finally, we define the L2-distance between capsules for the feature descriptors and adopt a hard L2 loss function to train the model. The extensive experimental results demonstrate that the proposed method can maintain stable registration performance on SAR images with complex scenes, and its NCM, RMSE, and running time are better than those of other state-of-the-art methods. However, because the proposed method masks keypoints in strongly textured regions, the distribution of keypoints may not be uniform in SAR images with large mountainous regions. In future work, we hope to use digital elevation models to improve registration accuracy in areas with large geometric distortions.