A Fast Matching Method for the SAR Images with Large Viewing Angles Based on Inertial Navigation Information and Neighborhood Structure Consensus

Abstract

In the field of multi-view SAR target location, the greater the difference in viewing angles, the higher the target location accuracy. However, this makes it difficult to match the same target between the SAR images with different viewing angles. The performance of traditional SAR image-matching algorithms will deteriorate or even fail to match the images correctly when the viewing angle is gradually increased. To solve this problem, a fast SAR matching method for the SAR images with large viewing angles based on inertial navigation information and neighborhood structure consensus (ININSC) is proposed in this paper. In this algorithm, the key targets are detected in the SAR images by using the maximum connected domain algorithm and the K-means clustering algorithm, and the connected domain centroid of the target is taken as a feature point. The approximate position of the key targets after the viewing angle change is found through inertial navigation information, and then accurate and fast matching is achieved by using the consensus of the neighborhood topological structure of the key targets. The measured data sets demonstrate that compared with traditional SAR image-matching algorithms, the proposed ININSC algorithm solves such a problem that SAR images cannot be accurately matched under the differences at large viewing angles and has better robustness and timeliness.

1. Introduction

Synthetic aperture radar (SAR) is a sensor system that actively emits microwave signals for observation [1]. Compared with traditional optical and infrared sensors, SAR can be operated in all-time and all-weather conditions [2]. Due to this feature, SAR has been widely applied in the military reconnaissance activities to accurately and quickly locate the key military targets. Traditional SAR target location relies on a monostatic radar detection platform, which has inherent limitations in practical applications [3]. It is also difficult to effectively ensure location accuracy [4]. Compared with the single-view SAR location technology, the multi-view SAR target location technology has more obvious advantages in location accuracy and can theoretically reduce the errors caused by external factors in the process of single-view location [5]. The first problem to be solved in multi-view SAR target location is how to determine the same target in the SAR images with different viewing angles. Theoretically, when multi-view SAR is used for target location, the greater the difference between multiple viewing angles, the higher the target location accuracy [6]. However, under the differences at large viewing angles, the radar cross section (RCS) of the target fluctuates greatly with the changes in viewing angles, which leads to the obvious differences in the shape, appearance and gray value of the same target in the SAR images with different viewing angles [7]. This is difficult to be processed by traditional SAR image-matching algorithms. Therefore, how to meet the requirement of the differences at large viewing angles in target location accuracy and correctly match the same target between the SAR images with large viewing angles has become an urgent problem to be solved in the field of multi-view SAR target location [8].

Image registration is the process of matching and superimposing two or more images obtained at different times, different sensors (imaging equipment) or under different conditions (weather, illumination, camera position and angle, etc.). There is the region-based registration method and feature-based registration method in the field of image registration [9]. The region-based registration method is mainly used to search and optimize the corresponding relationships by defining the template and using various similarity measures such as mutual information [10], normalized cross correlation [11] and cross cumulative residual entropy [12] based on the intensity information of the images. This method is relatively sensitive to geometric changes and radiation differences, so it is seldom adopted in the SAR image registration. The feature-based registration algorithm can realize the registration between the images with different image scales and rotation differences through accurate feature point matching [13]. This algorithm contains three steps: feature detection and extraction, the matching of features, and the calculation of transformation parameters. The scale-invariant feature transform (SIFT) algorithm is the most widely applied in the field of optical image registration [14]. However, the SIFT algorithm misjudges the speckle noise points widely existing in the SAR images as key points, which will greatly increase the time of false matching and calculation. Therefore, some improvements have been made on the basis of SIFT. Fan et al. [15] optimized the SIFT algorithm through the spatial relationship to improve the description of feature points. Dellinger et al. [16] proposed a gradient operation method which is similar to SIFT, designated for SAR image noise, and named it SAR-SIFT. Ma et al. [17] redefined the gradient algorithm based on the intensity difference, and proposed the position scale and orientation scale-invariant feature transform (PSO-SIFT) algorithm. In order to reduce the influence of multiplicative noise, Eltanany et al. [18] successfully detected key points in the complex SAR images by combining phase congruency (PC) with a Harris angular point detector. However, it is difficult to remove the influence of speckle noise from the complex scattering background in the SAR images, and the observation of ground targets at different time and locations also leads to great geometric distortion and different radiation information differences in the same scene. These factors lead to false matching or mismatching in the matching process of the SAR images with large viewing angles, and the matching of misjudgment points also consumes a lot of matching time [19]. In addition, the method based on SIFT is carried out with manually designed descriptors, which is difficult to adapt to the complex geometric and radiometric differences of SAR images and will lead to registration failure. In order to improve the accuracy of registration, Yao et al. [20] presented a novel registration method for oblique synthetic aperture SAR images based on complementary integrated filtering (CIF) and multilevel matching. Xiang et al. [21] achieved SAR image registration by extracting the inherent direction features of the image and focusing on the geometric invariant region. For slant-range (SR) synthetic aperture radar images, Xiang et al. [22] fixed the scale and rotation parameters and parallelized the phase correlation of cropped patches to generate accurate matches.

With the development of deep learning, some methods based on convolution neural networks (CNN) were proposed to extract deep features, such as the twin network, pseudo twin network, and two channel networks. The Siamese network, known as twin network, consists of two identical CNN-based subnetworks where all the weights and biases of the two subnetworks are tied together. The downside the of the Siamese network is that it can easily suffer from the interference of semantic distractors, in particular, the background, since it ignores the background information. The Siamese network can also be used to identify corresponding patches in SAR and optical images [23]. Zhang et al. [24] provided a method to match unsupervised SAR and optical images by using Siamese domain adaptation. Du et al. [25] proposed FM-CycleGAN to achieve feature matching consistency. These methods based on CNN can mine the nonlinear relationship between SAR images, and can adapt well to the geometric and radiometric differences between SAR images. However, with the deepening of the convolution layer, the details of the SAR image are gradually lost. Although significant features are retained, they usually appear in buildings and mountains with high backscattering intensity, and there may be large geometric distortion in these areas.

Therefore, a fast-multi-view SAR image registration algorithm based on inertial navigation information and neighborhood structure consensus (ININSC) is proposed. The acquisition of inertial navigation information is not affected by the outside world, and it can be continuously located in any environment [26]. Combined with the target location model of single-view SAR images, this algorithm can quickly solve the viewing angle differences between various SAR images in the same scene, and reduce the time complexity. Meanwhile, matching is conducted based on the consensus of neighborhood topology structure of important targets. A pair of correct matching points in the same scene is physically constrained, and other corresponding matching points in its neighborhood should maintain the consistent topology structure [27]. Therefore, the matching problem of feature points between various SAR images is changed into the problem of topology structure consensus within the neighborhood of feature points [28]. By designing a similarity score function, we compared the similarities between the neighborhoods focusing on feature points. The matching with the highest similarity is the correct feature matching. The experimental data were based on the echo data sets obtained from the field flight test in a national defense park of Nanjing from 2021 to 2022 by using the MiniSAR system independently developed by the research team of radar detection and imaging technology of Nanjing University of Aeronautics and Astronautics. By using the data sets, the proposed ININSC algorithm was compared with two traditional SAR image registration algorithms (SAR-SIFT and PSO-SIFT), which proves the robustness and timeliness of the proposed algorithm.

The algorithm proposed in this paper was mainly aimed at the problem of difficult matching between SAR images in the case of the large angle of view difference, and the main innovations are as follows.

(1)

In terms of extracting image feature points, different from the traditional SAR image registration algorithm, the traditional SAR image registration method is based on the SIFT algorithm, which inevitably regards some noise spots as feature points to be matched, and the matching process depends on the similarity between the registered images.

(2)

The way of extracting feature points in this paper will lead to the problem of less information on matching points. To solve this problem, a matching method based on the consistency of inertial navigation information and target neighborhood structure is proposed. Through this method, accurate matching is completed and the matching time is reduced.

The rest of this article is structured as follows. The Section 2 provides the implementation method of the ININC algorithm. The Section 3 is the experimental verification. The Section 4 analyzes the experimental results. The Section 5 summarizes the full paper.

2. Methods

The realization flow chart of the ININSC algorithm is shown in Figure 1, which is mainly divided into the following steps:

(1)

Image preprocessing and target detection from different perspectives

SAR images are preprocessed, and then the targets are preliminarily detected by the maximum connected domain algorithm. The feature points of the detected targets are expressed by the connected domain centroid, and then strong scattering region and target feature points are separated by the K-means clustering algorithm.

(2)

Transformation relationship solution based on inertial information

SAR imaging is performed at different viewing angles in the same scene, and inertial navigation information of the aerial carriers at different viewing angles is obtained. Based on the inertial navigation information and combined with the single-view SAR location model, the preliminary transformation relationship between the SAR images at different viewing angles is obtained through the coordinate transformation among a geodetic coordinate system, ENU (East-North-Up) coordinate system and geodetic rectangular coordinate system.

(3)

Structural similarity matching

According to the transformation relationship, the corresponding coordinates of the feature points of the A-view SAR image on the B-view SAR image is calculated, and the matching points corresponding to the feature points are screened by the nearest neighbor algorithm. The nearest neighbor algorithm and brute force matching algorithm are used to match with the feature points within the neighborhood of matching points, then the matching similarity score is calculated through the neighborhood structure consensus, finally the correct matching is output.

2.1. Target Detection and Extraction

When the aerial carrier images the same target at different positions, the SAR images of the same target will have great shape changes and viewing angle differences due to the position changes of the aerial carrier. The SAR images collected at different time and angles are quite different due to the imaging characteristics of SAR images and the influence of speckle noise and RCS changes at different viewing angles. This makes it difficult to extract the same features from reference images, and the extracted features are not stable enough. Moreover, the extracted features are simple and cannot meet the requirements of high-precision SAR image registration in complex scenes. In the ININSC algorithm, image registration is conducted based on point features. In the process of viewing angle transformation, the structural intersection of targets, the intersection of lines and the local modulus maximum of wavelet transform are affected by SAR image noise and RCS changes. These feature points are not stable. In order to obtain better stability of the extracted feature points in large viewing angle changes, the centroid of the target region of SAR images is taken as the point feature of the matching between the SAR images at different viewing angles.

2.1.1. Preprocessing of SAR Images

In the field of SAR image registration, the difficulty of registration is mainly caused by two factors; one is the influence of noise, and the other is the complex geometric and radiometric differences in the SAR image caused by RCS fluctuation under different viewing angles. The algorithm proposed in this paper mainly depends on the fact that the neighborhood structure of the target does not change in the process of changing the viewing angle of the SAR image, so it only needs to extract the centroid of the main target as the feature point. This is reasonable in military reconnaissance. The targets that need to be located are the ground combat center mainly composed of tanks, anti-aircraft guns, and transport loaders, which are easy to extract. The main reason for converting to the binary image is that it is convenient to extract the centroid of the reconnaissance target. This method can more quickly remove the influence of noise spots and background and is conducive to the subsequent use of the target neighborhood structure for matching.

The background and target are segmented in the SAR images according to the threshold value method, and the SAR images are preliminarily processed. The basic idea of threshold value method is to segment the images into two different regions according to the set threshold values. If the pixel value is smaller than the threshold value, it will be background. If the pixel value is greater than the threshold value, it will be target. In this way, the SAR gray images are changed into binary images to distinguish the target from the background. The threshold value segmentation method features simple calculation and fast segmentation if a threshold value is given. In this paper, the classic threshold value method (OTSU) is adopted to preprocess the SAR images.

OTSU automatically determines the threshold values by using the maximum between-class variance. Variance is used to measure the uniformity of gray distribution. The greater the between-class variance between the background and the target, the greater the difference between the background and target of the images. If part of the targets is misclassified as background or part of the backgrounds are misclassified as target, the difference between both of them will become smaller [29].

$M$ is the number of gray levels and the number of pixels with a gray value $i$ is $n_i$; the total number of pixels will be $N={\displaystyle {\sum }_{i=1}^M{n}_i}$. If the proportion of each gray value after normalization is $p_i=n_i/N$, the average gray level of SAR images will be $\mu ={\displaystyle {\sum }_{i=1}^{M}i{p}_i}$, and the variance will be ${\sigma }^2={{\displaystyle {\sum }_{i=1}^M\left(i-\mu \right)}}^2{p}_i$. If the segmentation threshold value is $T$, the image is divided into target $F_1$ and background $F_2$, $w_1$ and $w_2$ are the probabilities of the emergence of target $F_1$ and background $F_2$, ${\mu }_1$ and ${\mu }_2$ are the average gray values, ${\sigma }_1^2$ and ${\sigma }_2^2$ are the variances, then:

$$w_1={\displaystyle \sum _{i=1}^T{p}_i},w_2={\displaystyle \sum _{i=T+1}^M{p}_i},$$

(1)

$${\mu }_1=\frac{{\displaystyle \sum _{i=1}^{T}i{p}_i}}{w_1},{\mu }_2=\frac{{\displaystyle \sum _{i=T+1}^{M}i{p}_i}}{w_2},$$

(2)

$${\sigma }_1^2=\frac{1}{w_1}{\displaystyle \sum _{i=1}^T{\left(i-{\mu }_1\right)}^2}{p}_i, {\sigma }_2^2=\frac{1}{w_2}{\displaystyle \sum _{i=1}^T{\left(i-{\mu }_2\right)}^2}{p}_i$$

(3)

The between-class variance is:

$${\sigma }_{}^2=w_1{\left({\mu }_1-{\mu }_2\right)}^2+w_2{\left({\mu }_1-{\mu }_2\right)}^2$$

(4)

If the between-class variance in Equation (4) is a maximum, the gray value $T$ will be the threshold value of the segmented SAR images.

Based on the image morphology method, the SAR images segmented by the threshold value method are conducted with subsequent segmentation processing to reduce the influence of noise on SAR image target detection. The basic idea of morphology is to use structural elements to measure or extract the shapes or features of images. The basic operations include corrosion, expansion, opening operation and closing operation. The opening operation and closing operation are adopted in this paper. Isolated dots and burrs can be effectively removed by the opening operation, and the filtering effect is achieved. Closing operation can fill the gaps between the target images. The principle of opening operation and closing operation is shown in Figure 2. In Figure 2, black squares represent the target, and white squares represent the background.

The results of SAR image preprocessing using the threshold value method and morphology are shown in Figure 3.

In Figure 3, the target has been extracted from the background in the SAR images after preprocessing, but there are still many small noise spots in the processed images and the influence caused by scattering in the wood area.

2.1.2. Target Detection and Feature Extraction

After preprocessing the single-frame SAR images and observing the processing results, images can be divided into three-pixel categories: target pixel region, isolated misjudgment point and strong scattering region. In order to obtain the correct target detection, false alarm suppression is required for other pixel categories beyond the target pixel regions.

(1)

Elimination of isolated misjudgment points

SAR is affected by noise interference in the imaging process, which brings some isolated misjudgment points into the image after preprocessing. In fact, the target occupies a certain area in the image, and it does not exist as an isolated point in the image. Therefore, the maximum connected domain algorithm can be adopted to eliminate the isolated misjudgment points. The connected domain refers to an area composed of adjacent pixels. In the field of computer vision, it is a commonly used image processing technology, which is used to separate different regions in the image, to realize image segmentation, target detection, and other applications. The maximal general domain algorithm is an image processing algorithm based on the connected domain. By setting the maximal general domain pixel threshold, the connected domain greater than the set threshold is extracted. As shown in Figure 4a, isolated misjudgment points are successfully eliminated to obtain some regions with a certain area.

(2)

False alarm suppression in strong scattering region

Target region and strong scattering regions have certain areas in the SAR images, so they can hardly be distinguished using the maximum connected domain method. In practice, there is an obvious regional division between the target region and the strong scattering region dominated by woods. By calculating the connected domain centroid (as shown in Figure 4b, the centroid points of strong scattering region are concentrated and dense), the K-means clustering algorithm can be adopted to separate the target from the strong scattering region to solve this problem [30]. Clustering is a process of classifying and organizing data members that are similar in some aspects. K-means clustering is the most famous partition clustering algorithm. Because of its simplicity and efficiency, it has become the most widely used of all clustering algorithms. Given a set of data points and the required number of clusters $K$, $K$ is specified by the user, and the K-means algorithm repeatedly divides the data into $K$ clusters according to the distance function. As shown in Figure 4c, the target centroid is obtained after suppressing the strong scattering region, and target detection and feature extraction are completed.

The clustering processing result is shown in Figure 4. The key targets in the single-view SAR images are detected, and the target centroid is used as the feature point of subsequent matching.

2.2. Coordinate Transformation Based on Inertial Navigation Information

Inertial navigation system is a self-aid navigation system which is independent of external information and based on Newton’s laws of motion [31]. Speed, yaw angle and position information in the navigation coordinate system can be obtained by measuring the accelerated speed of the carrier in the inertial reference frame, integrating the time by the accelerated speed of the carrier, and transforming it into the navigation coordinates. Therefore, a model based on inertial navigation information and single-view SAR location is proposed by using the motion information of the aerial carrier to complete preliminary registration.

2.2.1. Single-View Location Model

Assuming that the distributed multi-view SAR system conducts SAR imaging for the target region from $N$ different routes ($N$ is greater than or equal to 3; non-parallel routes intersect with each other and maintain certain included angles, as shown in Figure 5), the target position coordinates can be determined by the position coordinate of the radar antenna phase center during target imaging and its slant range from the target.

Single-view location analysis is carried out for the aerial carrier at the viewing angle $W_1$. The location model is shown in Figure 6, and the location analysis of other positions is also carried out in the same way.

As shown in Figure 6, an imaging geometric model is established by taking the point of the radar beam center as an imaging reference point and the imaging reference point as the original point of the coordinate system. The aircraft flies at a constant speed $v$, the flying altitude is $h$, and the radar range is $r_0$. The starting and ending positions of synthetic aperture are $A$ and $B$, respectively, and $C$ is the position of the antenna phase center (APC).

Assuming that the squint angle and pitch angle of the aerial carrier at the aperture center point are ${\theta }_0$ and ${\beta }_0$, respectively, the coordinate of APC at the aperture center point is $\left(x_0,y_0,z_0\right)$:

$$\{\begin{array}{l}{x}_0=-\frac{h}{\cos \left({\beta }_0\right)}\times \sin \left({\theta }_0\right)\\ y_0=\sqrt{{\left(h\times \tan \left({\beta }_0\right)\right)}^2-x_0{}^2}\\ z_0=h\end{array}$$

(5)

Assuming that the components of the speed of the aerial carrier along the three coordinate axes are $V_x$, $V_y$ and $V_z$, respectively and the positive direction of the speed is the same as that of the coordinate axis, the coordinate of APC at any time can be given as follows:

$$\{\begin{array}{l}{x}_{apc}\left(t\right)=x_0+V_{x}t\\ y_{apc}\left(t\right)=y_0+V_{y}t\\ z_{apc}\left(t\right)=z_0+V_{z}t\end{array}$$

(6)

The instantaneous coordinate of the APC is $\left(x_{apc}\left(t\right),y_{apc}\left(t\right),z_{apc}\left(t\right)\right)$, and the imaging center point is the original point. If the coordinate of the ground target $P$ is $\left(x_p,y_p,0\right)$, the real-time distance from APC to the ground target can be obtained as follows:

$$r_p\left(t\right)=\sqrt{{\left(x_{apc}\left(t\right)-x_p\right)}^2+{\left(y_{apc}\left(t\right)-y_p\right)}^2+z_{apc}{\left(t\right)}^2}$$

(7)

According to the definition of the Doppler frequency, the Doppler frequency of the target $P$ is:

$$f_p=-\frac{2}{\lambda }\frac{d{r}_p\left(t\right)}{dt}$$

(8)

where $\lambda$ is the electromagnetic wave wavelength. In $z_0=h$, $h$ is the flight altitude of the aerial carrier. The range Doppler equation can be obtained as follows:

$$\{\begin{array}{l}r=\sqrt{{\left(x_0-x_p\right)}^2+{\left(y_0-y_p\right)}^2+h^2}\\ f_p=\frac{-2V_x\left(x_0-x_p\right)-2V_y\left(y_0-y_p\right)-2V_{z}h}{\lambda \sqrt{{\left(x_0-x_p\right)}^2+{\left(y_0-y_p\right)}^2+h^2}}\end{array}$$

(9)

where $r$ is the distance between the APC and the target $P$, and $f_p$ is the Doppler frequency of the target $P$. Assuming that the pixel coordinate of any point $p$ in the SAR image is $\left(n,m\right)$, the corresponding distance and Doppler frequency can be obtained:

$$\{\begin{array}{l}r=r_{ref}+\left(n-N_r/2\right)p_r\\ f_p=f_{dc}+\left(m-N_a/2\right)\Delta f_d\end{array}$$

(10)

where $n$ is the pixel coordinate in the range direction, $m$ is the pixel coordinate in the azimuth direction, $r_{ref}$ is the reference slant range from the aperture center point to the imaging reference point, $f_{dc}$ is the Doppler frequency of the imaging reference point, $N_a$ is the number of sampling points in the azimuth direction, $N_r$ is the number of sampling points in the distance direction, $p_r$ is the distance represented by each pixel unit in the distance direction of SAR image, and $\Delta f_d$ is the Doppler frequency represented by each pixel point in the azimuth direction. Substituting $r$ and $f_p$ of Equation (10) into Equation (9), the corresponding $x_p$ and $y_p$ can be obtained.

The coordinate $\left(x_p,y_p,0\right)$ corresponding to any pixel point $\left(n,m\right)$ in the SAR image can be obtained according to Equations (9) and (10).

2.2.2. Coordinate System Transformation

In the same scene, the pixel point position information of the target $P$ is different in the SAR images at different positions. In fact, the target $P$ is stationary, which means that the position of the target $P$ in the geodetic rectangular coordinate system is fixed. The preliminary registration of SAR images at different positions can be realized through the coordinate system transformation.

Assuming that the coordinates of the target $P$ in the ENU coordinate system and the geodetic rectangular coordinate system are $\left(x_p,y_p,z_p\right)$ and $\left(X_p,Y_p,Z_p\right)$, respectively, the coordinate transformation relationship can be obtained as follows:

$$\begin{array}{l}\left[\begin{array}{l}{X}_p\\ Y_p\\ Z_p\end{array}\right]=\left[\begin{array}{ccc}-\sin L_0& -\cos L_0& 0\\ \cos L_0& -\sin L_0& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& \sin B_0& -\cos B_0\\ 0& \cos B_0& \sin B_0\end{array}\right]\left[\begin{array}{l}{x}_p\\ y_p\\ z_p\end{array}\right]+\left[\begin{array}{l}{X}_0\\ Y_0\\ Z_0\end{array}\right]\\ \hspace{1em}\hspace{1em}=\left[\begin{array}{ccc}-\sin L_0& -\sin B_0\cos L_0& \cos B_0\cos L_0\\ \cos L_0& -\sin B_0\sin L_0& \cos B_0\sin L_0\\ 0& \cos B_0& \sin B_0\end{array}\right]\left[\begin{array}{l}{x}_p\\ y_p\\ z_p\end{array}\right]+\left[\begin{array}{l}{X}_0\\ Y_0\\ Z_0\end{array}\right]\end{array}$$

(11)

where $\left(X_0,Y_0,Z_0\right)$ is the coordinate of the original point $O$ of the ENU coordinate in the geodetic rectangular coordinate system. $\left(L_0,B_0,H_0\right)$ is the geodetic coordinate of the aperture center point, where $L_0$ is longitude, $B_0$ is latitude, and $H_0$ is altitude. The geodetic coordinate $\left(L_0,B_0,H_0\right)$ of the aperture center point is given by the aerial carrier platform system.

Assuming that when the aerial carrier is at the viewing angle $A$, the pixel coordinate of the target $P$ in the SAR image is $\left(m_p^a,n_p^a\right)$, the ENU coordinate $\left(x_p^a,y_p^a,z_p^a\right)$ of the target $P$ at the viewing angle $A$ can be obtained through the location model at a single viewing angle. The absolute position information $\left(X_p,Y_p,Z_p\right)$ of the target $P$ in the geodetic rectangular coordinate system is obtained after substituting it into Equation (11). According to the inertial navigation information of the aerial carrier at the viewing angle $B$, the ENU coordinate $\left(x_p^b,y_p^b,z_p^b\right)$ of the target $P$ is obtained through Equation (11) when the aerial carrier is at the viewing angle $B$. After substituting the coordinate into Equations (9) and (10), the pixel coordinate $\left(m_p^b,n_p^b\right)$ of the target $P$ is obtained when the aerial carrier is at the viewing angle $B$. The operation flow is shown in Figure 7:

Through the above method, a connection is established between the pixel point position in the SAR image when the aerial carrier is at the viewing angle $A$ and the pixel point position in the SAR image when the aerial carrier is at the viewing angle $B$. However, due to the error of the parameters given by the aerial carrier platform, the pixel point coordinate obtained by this method is not completely accurate. The range of target matching can be reduced in this way, which greatly improves the accuracy of target matching and reduces the matching time.

$O_g$, $X_g$, $Y_g$ and $Z_g$ represent the earth rectangular coordinate system, $X_A$, $Y_A$ and $Z_A$ represent the northeast celestial coordinate system centered on carrier $a$, and $X_B$, $Y_B$ and $Z_B$ represent the northeast celestial coordinate system centered on carrier $b$.

2.3. Matching Algorithm Based on Structural Similarity

After the target detection and feature extraction of SAR images, how to match them accurately and quickly has become a problem to be solved [32]. The matching strategy of the proposed ININSC algorithm is to use inertial navigation information for preliminary nearest neighbor screening, and all the feature points in the screening range may be the corresponding matching of the target at another viewing angle [33]. In order to solve the problem of matching between targets and feature points, a matching method based on structural similarity is adopted. The target and its corresponding matching feature points at different viewing angles as well as other corresponding reference feature points in their neighborhoods should maintain a consistent spatial structure relationship [34].

Assuming that the target $P$ has $K$ candidate points within the neighborhood at the viewing angle $B$ after preliminary registration through inertial navigation information, these $K$ candidate points may be correctly matched with the target $P$. Through nearest-neighbor and brute-force matching, the reference points around these $K$ candidate points and the reference points in the neighborhood of the target $P$ at the position $A$ are quickly matched to obtain $K$ matching results $I$. Only one of these $K$ matching results is a correct matching result. The ININSC algorithm solves the matching problem by designing a structural similarity function.

To define the similarity of each match, this paper designs a method to calculate the similarity score. The higher the score, the higher the similarity of this match. As shown in Figure 8, the red point represents the target $P$, the yellow point represents the reference point near the target, the left image represents the original image, and the middle image represents the position of the feature points on the original image on the registration image after the inertial navigation information conversion, and the right is the registration image. Two point sets $O$ and $C$ are defined to store the target and the reference points near the target, respectively. In $O=\left(p_0,q_0\right)$, $p_0$ is the coordinate of the target $P$ at the viewing angle $A$, and $q_0$ is the coordinate of the possible candidate point of the target at the viewing angle $B$. In $C={\{\left(p_i,q_i\right)\}}_{i=1}^N$, $p_i$ and $q_i$ are the coordinates of the paired matching points when the $O$ matching is correct.

This paper measures the similarity from three perspectives: distance, angle and vector.

(1)

Distance similarity

The distance between the target and the reference point is constant in the process of changing the viewing angle. But in the actual matching process, the distance cannot be exactly the same. Define $d(x,y)$ as the distance between two points of $x$, $y$. As shown in Figure 8, $d(p_0,p_1)$ and $d(q_0,q_1)$ are not exactly the same, but the difference between the lengths of the corresponding edges should be approximately the same. The difference between $d(p_0,p_2)$ and $d(q_0,q_2)$ is approximately the same as that between $d(p_0,p_1)$ and $d(q_0,q_1)$. Therefore, the distance similarity function is defined as follows:

$$S\left(O,C\right)={\displaystyle \sum _{i=1}^N\frac{1}{N}}\cdot e^{-\frac{|d\left(p_{0,}{p}_i\right)-d\left(q_{0,}{q}_i\right)|}{max\left(d\left(p_{0,}{p}_i\right),d\left(q_{0,}{q}_i\right)\right)}}$$

(12)

where $\frac{1}{N}$ stands for weight.

(2)

Angle similarity

In the process of changing the viewing angle, the angle between the target and any two reference points is constant. But in the actual matching process, the angles cannot be exactly the same. $\theta (x,y,z)$ is defined as the angle formed by $x$ as the vertex in $x$, $y$, $z$. As shown in Figure 8, $\theta \left(p_0,p_i,p_j\right)$ and $\theta \left(q_0,q_i,q_j\right)$ are not exactly the same, so the ratio is used to form the angle consistency, and the angle similarity function is defined as follows:

$$G\left(I;O,C\right)=S\left(O,C\right)+R\left(O,C\right)+V\left(O,C\right)$$

(13)

where $C_N^2$ represents the number of combinations of two elements taken from $N$ different elements and $\frac{1}{C_N^2}$ stands for weight.

(3)

Vector similarity

As shown in Figure 8, $\overrightarrow{v_0}$ is a vector pointing from $p_0$ to $q_0$, and $\overrightarrow{v_i}$ is a vector pointing from $p_i$ to $q_i$. For correctly matched points, the displacement vector between any two points in its neighborhood should be very close. For mismatched points, the displacement vectors of points in the neighborhood and between points may be different. Therefore, the product of the length ratio and the angle between two displacement vectors is used to describe the consistency between the two vectors, and the vector consistency function is defined as follows:

$$V\left(O,C\right)={\displaystyle \sum _{i=1}^N\frac{1}{N}}\cdot \frac{min\left(|\overrightarrow{v_i}|,|\overrightarrow{v_0}|\right)}{max\left(|\overrightarrow{v_i}|,|\overrightarrow{v_0}|\right)}\cdot \frac{\left(\overrightarrow{v_i},\overrightarrow{v_0}\right)}{|\overrightarrow{v_i}|,|\overrightarrow{v_0}|}$$

(14)

where $\frac{1}{N}$ stands for weight.

The structural similarity function is equal to the sum of the distance, angle and vector consistency functions, as follows:

$$G\left(I;O,C\right)=S\left(O,C\right)+R\left(O,C\right)+V\left(O,C\right)$$

(15)

Therefore, when the structural similarity function reaches the maximum value, the best matching result $I^{\ast }$ can be obtained as follows:

$$I^{*}=argmaxG\left(I;O,C\right)$$

(16)

By using the neighborhood structure consistency of feature points, the correct matching point group is obtained. It is assumed that the transformation between SAR images is the affine transformation in the process of view angle change. The affine transformation matrix between two images is obtained by matching the point group. Through the affine transformation matrix, the registration between two images can be realized, and each part of the image will correspond to each other.

3. Experimental Verification

In order to verify the accuracy and advantages of the ININSC algorithm, the ININSC algorithm is compared with the two traditional SAR registration methods. The first algorithm is SAR-SIFT algorithm, which is improved from the SIFT algorithm. The feature point pair registration method based on gradient calculation has a good effect on SAR image registration. The second algorithm is the PSO-SIFT algorithm, which uses a transformation for the speckle noise of SAR images and has a good registration effect in the field of SAR image registration.

3.1. Data Description and Parameter Settings

The X/Ku-band NUAA MiniSAR system [35] used in the experiment was independently developed by the research team of radar detection and imaging technology of Nanjing University of Aeronautics and Astronautics in 2017. It adopts the frequency modulated continuous wave system with a resolution of 0.1 m and a mass of only 5 kg. The main parameters are shown in

Table 1. System parameters of X-band MiniSAR.

System Parameters	Value
Bandwidth/MHz	1800
Carrier frequency/GHz	9.7
Flight speed/(m/s)	5
Average power/W	2
Sampling rate/MHz	100
Operating range/km	2

. The carrying platform is a UAV (Unmanned Aerial Vehicle) with eight rotary wings, and the aerial flight attitude diagram of the system is shown in Figure 9.

The data source of the experiment is from the field flight test in a national defense park of Nanjing from 2021 to 2022 by using the MiniSAR system of Nanjing University of Aeronautics and Astronautics. In this experiment, the typical ground military targets such as tanks and armored vehicles in the park were observed in all directions, a large amount of echo data were collected, and the SAR target data sets taking the national defense park as a circumference range was preliminarily built. The imaging mode of the MiniSAR system was circular spotlight SAR, and the system parameters were set as flight height (500 m), flight speed (5 m/s), azimuth angle (0°~360°) and observation pitch angle (45°). The field experiment observation image is shown in Figure 10, in which the military operation targets were marked.

The echo data sets obtained from the experiment were imaged by the back project (BP) algorithm, there were totally 100 frame images. The size of the images is 1024 × 1024. The viewing angle interval between the frames was approximately 1.8°, and the images were saved in tif format. The viewing angle difference between the adjacent frames was small. In order to obtain a certain viewing angle difference, some images were selected at a certain interval for experimental verification. As shown in Figure 11, the images of the 1st frame, the 18th frame and the 35th frame were selected for experiment, assuming that the viewing angles of the 1st frame, the 18th frame and the 35th frame were 0°, 30.6° and 61.2°, respectively.

A 64-bit Win10 operation system, 8G memory and Intel (R) Core (TM) i5-9600KF processor were provided in the experiment. During the experiment, GPU was not used for optimization. The experimental software was MatlabR2020a. SAR-SIFT, PSO-SIFT and ININSC algorithms run on the Matlab platform. The parameter settings of SAR-SIFT and PSO-SIFT algorithms were the default settings in related articles, and they used the RANSAC algorithm to eliminate the mismatching points.

The setting of the maximum connected domain threshold parameter was based on the prior conditions, mainly on the actual size of the target to be separated and the resolution of the remote sensing image. The remote sensing imaging resolution of the MiniSAR system used in this paper was 0.1 m. The actual tank was about 7.3 m long and 3.5 m wide, occupying 640 pixel units in the image. To ensure that the target was detected, the threshold of the largest connected domain was set to 500, and the connected domain greater than 500 was regarded as the target. The parameter $K$ of the K-means clustering algorithm is set to 2 to separate the two clusters of the target region and strong scattering region.

3.2. Comparative Experiment Results and Analysis

The SAR images at the viewing angles 0°, 30.6° and 61.2°constituted three pairs of test image groups: 0°and 30.6°; 0°and 61.2°; and 30.6° and 61.2°. The registration results after adopting the ININSC algorithm and other two algorithms are shown in Figure 12, Figure 13 and Figure 14. After registration, the transformation matrix between the SAR images at different viewing angles was calculated through the matching points, and the registration error of SAR images was calculated by the transformation matrix obtained by the three algorithms. As shown in Figure 10, there were many military operation targets in the national defense park, and whether the key operation targets at different viewing angles could be correctly matched was considered as an indicator for accurate matching.

Subjective evaluation and objective evaluation were adopted in the experiment. The method of subjective evaluation was used to judge the connection of registration points and the correctness of such connection. After three pairs of images were registered, the checkers were spliced in Figure 12, Figure 13 and Figure 14. The registration effect was judged subjectively for detailed splicing. The evaluation indicators adopted in the objective evaluation included the root-mean-squared error (RMSE), time, and matching rate of key operation targets.

The RMSE equation is as follows:

$$\mathrm{RMSE}=\sqrt{\frac{{\displaystyle \sum _{i=1}^N{\left(I_1^i-I_2^i\right)}^2}}{N}}$$

(17)

where $I_1$ is the SAR image at the initial viewing angle, $I_2$ is the image of SAR at different viewing angles after the transformation matrix is registered and calculated, $I_1^i$ and $I_2^i$ are the gray values of the corresponding points, $N$ is the total number of pixels in the overlapping part, and RMSE is the image registration accuracy. The smaller the RMSE value, the higher the registration accuracy.

The matching ratio $\eta$ equation of key operation targets is as follows:

$$\eta =\frac{M}{M_0}\times 100\%$$

(18)

where $M$ is the number of key operation targets that are correctly matched after registration, and $M_0$ is the total number of key operation targets. If accurate registration can be realized in the proposed algorithm, the matching rate of key operation targets should be 100%.

The time represents the processing time of a single target.

According to

Table 2. Statistics of the estimation results.

Method	0°, 30.6° Image Groups			30.6°, 61.2° Image Groups			0°, 61.2° Image Groups
	$\mathit{\eta }$	RMSE	Time/s	$\mathit{\eta }$	RMSE	Time/s	$\mathit{\eta }$	RMSE	Time/s
SAR-SIFT	100%	0.383	158.76	100%	0.395	155.37	0	−	−
PSO-SIFT	100%	0.365	49.27	100%	0.358	50.71	0	−	−
ININSC	100%	0.374	5. 003	100%	0.380	5.17	100%	0.552	5.27

, compared with PSO-SIFT and SAR-SIFT, the matching rate of key operation targets was 100% in the first two pairs of test image groups in the ININSC algorithm, the RMSE values were approximate, and the corresponding registration accuracy was equally good. The processing time of the ININSC algorithm was an order of magnitude less than that of the other two algorithms, which objectively verified the effectiveness and timeliness of the ININSC algorithm. In the third group, the matching rate of key operation targets of the first two algorithms was 0, and RMSE could not be calculated. The reason why the RMSE could not be computed is that the feature points match incorrectly, as shown in Figure 14. This indicated that the first two algorithms were no longer applicable to this situation. In this case, the matching rate of key operation targets in the ININSC algorithm was still 100%. The ININSC algorithm was more robust than PSO-SIFT and SAR-SIFT in processing the differences at large viewing angles.

The viewing angle differences between the first pair of the images and the second pair of the images in the test image groups were 30.6°, and there were slight differences in rotation, translation and scaling as well as noise interference. The registration results are shown in Figure 12 and Figure 13. SAR-SIFT, PSO-SIFT and ININSC algorithms achieved accurate matching results in this case. It can be seen from the intuitive checker mosaic images that the SAR images at different viewing angles were connected well in details, and the ground targets overlapped completely after registration. When the viewing angle difference was small, the three algorithms achieved good registration results.

Compared with the first two test groups, the third pair of test groups had a viewing angle difference of 61.2°. The differences caused by rotation, translation, scaling, RCS and noise were even greater, and the geometric distortion of the target increased. The registration result is shown in Figure 14. The SAR-SIFT and PSO-SIFT algorithms lost their accuracy in matching points, the number of matching points was small, and the connection of registration points was wrong. These two algorithms were no longer applicable to the third pair of the test group. In the ININSC algorithm, the registration points were connected accurately, the registered images were connected accurately after checkers were spliced, and the ground targets were overlapped well. The ININSC algorithm still achieved a good registration result, while the other two algorithms could not adapt to the matching at large viewing angles.

The first frame was considered as the image to be registered, and there was only a slight viewing angle difference (less than 9°) among the 2nd to 5th frames in the data sets. Thus, the SAR image after the 6th frame (a viewing angle difference of about 9° between the 6th frame and the 1st frame) was selected to conduct image registration with the 1st frame. The RMSE results of SAR-SIFT, PSO-SIFT and ININSC algorithms are shown in Figure 15, and the average time is shown in Figure 16.

It can be clearly seen from Figure 15 that the registration accuracy of the ININSC algorithm was the same as SAR-SIFT and PSO-SIFT when correct registration was achieved. Whenever a viewing angle difference is greater than 60°, SAR-SIFT and PSO-SIFT could not be registered correctly with the 1st frame image. However, the ININSC algorithm could still achieve accurate registration. The ININSC algorithm could be used to process the matching at large viewing angles, which was hardly processed by traditional algorithms. It can be clearly seen from Figure 16 that the average time of the ININSC algorithm was far less than that of other two traditional algorithms.

In the three pairs of test groups, the connection of registration points in the ININSC algorithm were uniform and accurate, and the results of checker mosaic images after registration showed that the detail connection was accurate and the registration effect was good. Compared with SAR-SIFT and PSO-SIFT algorithms, the ININSC algorithm could still obtain accurate registration results under the great differences in test images. It can be seen from Table 2 that the ININSC algorithm had low RMSE, the registration accuracy was as good as the other two algorithms, and its time consumption was far less than the other two algorithms.

The simulation results of a large amount of data are shown in Figure 15 and Figure 16. The accuracy of the proposed INIANS algorithm could be compared favorably with the SAR-SIFT and PSO-SIFT algorithms, and it could solve the problem of differences at large viewing angles that was difficultly handled by traditional algorithms. Its time consumption was far lower than the other two algorithms. Compared with the other two traditional algorithms, the ININSC algorithm had better robustness and timeliness.

4. Discussion

In the three pairs of test groups, the connection of registration points in the ININSC algorithm were uniform and accurate, and the results of checker mosaic images after registration showed that the detail connection was accurate and the registration effect was good. Compared with SAR-SIFT and PSO-SIFT algorithms, the ININSC algorithm could still obtain accurate registration results under the great differences in test images. It can be seen from Table 2 that ININSC algorithm had low RMSE, the registration accuracy was as good as the other two algorithms, and its time consumption was far less than the other two algorithms.

The simulation results of a large amount of data are shown in Figure 15 and Figure 16. The accuracy of the proposed ININSC algorithm could be compared favorably with SAR-SIFT and PSO-SIFT algorithms, and it could solve the problem of differences at large viewing angles that was difficultly handled by traditional algorithms. Its time consumption was far lower than the other two algorithms. Compared with the other two traditional algorithms, ININSC algorithm had better robustness and timeliness.

The reason why SAR-SIFT and PSO-SIFT fail is that their method based on sift uses artificially designed descriptors for matching. This method depends on the similarity of target regions to a certain extent. In the case of large angle difference, the similarity between SAR images decreases sharply in the face of the complex geometric and radiometric differences of SAR images. As shown in Figure 17, due to the backscattering coefficient, the similarity of the same target area at different angles of SAR image is very poor, and the feature points described based on SIFT algorithm are difficult to match correctly in this case. In this case, the spatial structure relationship between the target and the nearby reference object is invariant, and the ININC algorithm uses this feature to complete the matching. The 61° mentioned in the article is for the experimental data, and the main influence is the imaging conditions. According to the data generated by different equipment and in different positions, the unmatched angle will change.

In terms of extracting feature points, the advantage of ININSC algorithm is that it can detect and separate targets without a large number of sample data sets compared with the deep learning method. Compared with the traditional SAR image registration algorithms SAR-SIFT and PSO-SIFT, the traditional algorithm is based on SIFT algorithm to extract feature points from SAR images. The feature points extracted by SIFT algorithm include the noise speckle signal in SAR image, which will cause difficulties for subsequent matching, increase the matching time, and easily form false matching.

The proposed algorithm has the following limitations: the algorithm has certain requirements for hardware facilities, needs to be equipped with the inertial navigation system, and the cost will be higher than the traditional algorithm. Another disadvantage is that the target and background need to be significantly distinguished, which can make the target accurately extracted.

5. Conclusions

In this paper, a fast matching method for the SAR images with large viewing angles based on inertial navigation information and neighborhood structure consensus was proposed, which consisted of target detection and extraction, coordinate transformation based on inertial navigation information, and a matching algorithm based on neighborhood structure consensus. The experimental results demonstrate the following:

(1)

The ININSC algorithm has the same registration accuracy as the traditional SAR image registration algorithms and can solve the problem of SAR image registration at large viewing angles that the traditional algorithm cannot tackle. It has stronger robustness.

(2)

The ININSC algorithm is far less time-consuming than traditional SAR image registration algorithms.

The above research results have an important reference value in the application field of SAR image registration engineering. Compared with traditional SAR image registration algorithms, the ININSC algorithm does not only meet the requirements of multi-view target location accuracy under the differences at large viewing angles, but also greatly reduces the time consumption of the SAR image matching at large viewing angles.