Multi-Sensor SAR Image Registration Based on Object Shape

Abstract

Owing to significant differences in synthetic aperture radar (SAR) images caused by diverse imaging mechanisms and imaging conditions, inconsistent features and relationship correspondences constitute key problems for traditional image registration algorithms. This study presents a novel SAR image registration method based on the shape information of distinct ground objects, which is obtained via object extraction and morphological operations. We utilize a shape context descriptor to compare the contours of objects and detect invariant control points. The experimental results show that the proposed method can achieve a reliable and stable registration performance for SAR images of different sensors.

1. Introduction

Image registration is an extremely important modern information processing technology, especially in the field of remote sensing, for integrating multi-source and multi-temporal data. It has been widely used in image classification, image fusion, change detection, image mosaicking, multi-temporal analysis, and cartography updating. Synthetic aperture radar (SAR) image registration remains a challenging task due to the inherent speckle noise and significant differences caused by differing imaging mechanisms and conditions (e.g., sensor, viewpoint, temporal, band).

Area-based methods, using correlation coefficient and mutual information (MI), are sensitive to intensity changes and are not suitable for image deformations; they are therefore unable to achieve desirable results when applied directly to multi-source images. Feature-based methods extract salient features (e.g., points, edges, linear structures, and regions) and match them using similarity measures to establish the geometric correspondence between two images [1,2,3]. Feature-based methods have been proven to be effective, however, respective features are difficult to match and the robustness of the methods strongly depends on feature extraction results. To improve reliability and robustness, sophisticated approaches based on multi-features or multi-layer features have been developed [4,5]. Meanwhile, registration methods using both area-based and feature-based approaches have received more attention [3,6,7].

However, most methods were only used for mono-sensor images [4,5,8] and were even limited to the same or similar imaging conditions [7,9,10]. For multi-sensor and multi-temporal imagery, the textures and grey levels at the locations of conjugate positions will not likely be similar. These significant differences will definitely affect the local-feature descriptor, which would generate many mismatched control point (CP) pairs and may lead to low accuracy. In recent years, the scale-invariant feature transform (SIFT) and its modifications have been mostly applied to SAR images [5,8,9]. SIFT-based methods have achieved good results on mono-sensor or approximate images, however, the number of detected feature matches may be small and many outliers exist in feature matches for multi-sensor images [3].

In this paper, we propose a robust registration method based on the object shape that is focused on solving multi-sensor SAR image registration. The challenges of this work lie on two aspects: (1) the correspondence problem between two contours of the same object in different images caused by image differences and segmentation technology; (2) detection of stable and invariant CP pairs. Obvious ground objects are extracted, and two images are registered by means of the shape information of the objects. Morphological processing, curve smoothing, shape matching, and other related technologies are used in the registration process. The proposed method exhibits strong noise resistance ability and can alleviate the effects of image deformation and nonlinear distortion caused by different sensors or different incident angles.

2. Methodology

2.1. Matching Primitives and Process

2.1.1. Matching Primitives

Distinct points, linear features, and homogenous/areal regions are commonly extracted as registration primitives. Point features in multi-sensor images usually have significant feature differences and inconsistent correspondences, which can generate many mismatched CPs or cause the algorithm to fail. Edge and contour features can reflect the structural and textural information of the image and maintain a certain degree of stability under the circumstances of various imaging modes and conditions. Compared with point features, linear and areal features exhibit more stable performances in multi-sensor images. Extracted linear features often appear to be fragmented or incomplete, and their information content is small. In contrast, the information content of contours is large. It is not difficult to solve the correspondence problem and remove false matches. Information content is a well-known criterion for judging feature distinctiveness. Thus, using contours as registration primitives can be a better alternative.

Contours of distinct objects are easy to identify and match, even in different images. Contour-based methods are especially suitable for multi-sensor image registration [11,12,13]. However, the contours in different images are not completely coincidental and have local differences for various reasons, e.g., imaging conditions, object changes, occlusion, and segmentation technology. These differences between contours are so large that conventional shape description and matching algorithms are difficult to apply in most cases. Robust matching between two contours with local deformation or absence, and the invariant control point extraction in the case of significant change or serious distortion are two key problems.

2.1.2. Registration Process

The proposed method is divided into three steps.

(1) Shape extraction. First, distinct ground objects (e.g., lake, vegetation) are extracted from reference and sensed images. Applicable segmentation algorithms and parameters should be used for these different objects respectively. Morphological processing and data vectorization will be used to obtain the contours of the objects (shapes).

(2) Shape matching. Shape processing is essential for shape matching, and contour smoothing can reduce the differences between the shapes of an object in different images. Then, the object shapes in two datasets are compared one by one based on the shape context descriptor. The two shapes that are matched successfully are conjugate object pairs.

(3) Geometric transformation. Control points are detected from unchanged segments on the contours of conjugate object pairs, and transformation estimation is performed using the least squares technique. The two images are geometrically registered based on the transformation parameters.

The flowchart of the proposed method is presented in Figure 1.

2.2. Shape Matching Using Shape Context

Shape is an important visual feature of an object, which provides the easiest and most obvious recognition information. However, shape matching is still a challenging task, especially for objects in sensed images, which often have complex irregular shapes. The key strategy is to develop an appropriate descriptor to characterize the object’s shape and tolerate local deformations simultaneously. Therefore, a robust matching method for varying the shape of objects in remote sensing images is required.

The original contours are rough and contain small-scale variations due to image differences and segmentation technology. Thus, contour smoothing is essential to reduce these variations and details, i.e., to reduce the differences between object shapes, which is useful for shape matching. In this study, a Gaussian filter was applied to the shape contour. The filter is given by

$$\{\begin{array}{c}\mathrm{G}\left(\mathrm{x},\mathsf{\sigma }\right)=\frac{1}{\mathsf{\sigma }\sqrt{2\mathsf{\pi }}}{\mathrm{e}}^{-{\mathrm{x}}^2/2{\mathsf{\sigma }}^2}\\ \mathrm{G}\left(\mathrm{y},\mathsf{\sigma }\right)=\frac{1}{\mathsf{\sigma }\sqrt{2\mathsf{\pi }}}{\mathrm{e}}^{-{\mathrm{y}}^2/2{\mathsf{\sigma }}^2}\end{array}$$

(1)

where, $\left(\mathrm{x},\mathrm{y}\right)$ are the coordinates of the contour and $\mathsf{\sigma }$ is the kernel width parameter. For the images used in this study, slight smoothing ($\mathsf{\sigma }=3$) was appropriate.

The Gaussian filter can effectively protect the extremes from being removed during the process of moving from a coarse to fine scale [14], as shown in Figure 2.

Numerous shape representations and analysis methods based on contour information have been proposed. The most suitable tool for contours is the shape context [15,16]. The shape context at a reference point characterizes the distribution of all other points relative to it, thereby generating a globally discriminative characterization. The shape context uses a log-polar histogram of coordinates and is tolerant of local deformations. The shape context can be used to quickly perform a search for similar shapes via recognition.

For a point ${\mathrm{p}}_{\mathrm{i}}$ on the shape, a histogram ${\mathrm{h}}_{\mathrm{i}}$ of the relative coordinates of the other $\mathrm{n}-1$ points is defined as follows.

$${\mathrm{h}}_{\mathrm{i}}\left(\mathrm{k}\right)=\#\left\{\mathrm{q}\ne {\mathrm{p}}_{\mathrm{i}}:\left(\mathrm{q}-{\mathrm{p}}_{\mathrm{i}}\right)\in \mathrm{bin}\left(\mathrm{k}\right)\right\}$$

(2)

where #{.} is the statistic of the set of points, $\mathrm{k}\in \left\{1,2,\dots ,\mathrm{K}\right\}$ and $\mathrm{K}$ is the number of histogram bins. Five bins are suitable for the polar radius, and 12 bins are suitable for the polar angle [15]. The expression $\left(\mathrm{q}-{\mathrm{p}}_{\mathrm{i}}\right)\in \mathrm{bin}\left(\mathrm{k}\right)$ indicates that the vector originating from ${\mathrm{p}}_{\mathrm{i}}$ to another point, $\mathrm{q}$, belongs to the $\mathrm{k}$th bin.

Note the visual similarities among the shape contexts shown in Figure 3. Corresponding points between two similar shapes share analogous shape contexts, providing a solution for the optimal assignment problem. This method can also be used to measure shape similarity.

The shape context descriptor is based on the characteristics of the histogram, and the χ² test statistics are used to measure the difference between two points. Consider a point ${\mathrm{p}}_{\mathrm{i}}$ on shape A and a point ${\mathrm{q}}_{\mathrm{j}}$ on shape B. The cost of matching the two points is defined as follows.

$${\mathrm{C}}_{\mathrm{ij}}=\mathrm{C}\left({\mathrm{p}}_{\mathrm{i}},{\mathrm{q}}_{\mathrm{j}}\right)=\frac{1}{2}{{\displaystyle \sum }}^{\text{}}\frac{{\left[{\mathrm{h}}_{\mathrm{i}}\left(\mathrm{k}\right)-{\mathrm{h}}_{\mathrm{j}}\left(\mathrm{k}\right)\right]}^2}{{\mathrm{h}}_{\mathrm{i}}\left(\mathrm{k}\right)+{\mathrm{h}}_{\mathrm{j}}\left(\mathrm{k}\right)}$$

(3)

where ${\mathrm{h}}_{\mathrm{i}}\left(\mathrm{k}\right)$ and ${\mathrm{h}}_{\mathrm{j}}\left(\mathrm{k}\right)$ denote the $\mathrm{K}$-bin normalized histograms at ${\mathrm{p}}_{\mathrm{i}}$ and ${\mathrm{q}}_{\mathrm{j}}$, respectively.

For the ${\mathrm{p}}_{\mathrm{i}}$ of shape A, the matched point in shape B is determined by traversing all of the points of shape B, computing the point-to-point costs, and selecting the point with the minimum cost value (which is less than the threshold). Similarly, we can obtain the matched point of ${\mathrm{q}}_{\mathrm{j}}$ in shape A. Two points are called a best point pair if they correspond to one another. The matched point and the best point pair are defined in Equation (4). An example is shown in Figure 4a, b.

$$\begin{gathered} {\mathrm{MP}}_{\mathrm{i}\to \mathrm{j}}:\mathrm{j}=\arg \underset{\mathrm{j}\in \mathrm{B}}{\min }{\mathrm{C}}_{\mathrm{ij}} \\ {\mathrm{MP}}_{\mathrm{j}\to \mathrm{i}}:\mathrm{i}=\arg \underset{\mathrm{i}\in \mathrm{A}}{\min }{\mathrm{C}}_{\mathrm{ij}} \\ {\mathrm{BP}}_{\mathrm{i}\leftrightarrow \mathrm{j}}:{\mathrm{MP}}_{\mathrm{i}\to \mathrm{j}}\mathrm{and}{\mathrm{MP}}_{\mathrm{j}\to \mathrm{i}} \end{gathered}$$

(4)

where ${\mathrm{MP}}_{\mathrm{i}\to \mathrm{j}}$ is a matched point pair and indicates that the matched point of ${\mathrm{p}}_{\mathrm{i}}$ is ${\mathrm{q}}_{\mathrm{j}}$; ${\mathrm{MP}}_{\mathrm{j}\to \mathrm{i}}$ indicates that the matched point of ${\mathrm{q}}_{\mathrm{j}}$ is ${\mathrm{p}}_{\mathrm{i}}$; and ${\mathrm{BP}}_{\mathrm{i}\leftrightarrow \mathrm{j}}$ is a best point pair.

Shape matching aims to find the conjugate object pairs from two datasets. The shape context descriptor can be used to find the matched points between two similar shapes. The percentage of the number of matched points can reflect the shape difference; consequently, a simple similarity measure between two contours was used in this study:

$${\mathrm{S}}_{\mathrm{AB}}=\frac{\mathrm{Total}\left(\mathrm{MP}\right)}{\mathrm{Total}\left(\mathrm{A}\right)}$$

(5)

where ${\mathrm{S}}_{\mathrm{AB}}$ is the shape similarity between shapes A and B, $\mathrm{Total}\left(\mathrm{MP}\right)$ is the total number of matched points, and $\mathrm{Total}\left(\mathrm{A}\right)$ is the total number of points of shape A.

Undoubtedly, control points are located in unchanged segments of two conjugate contours. All of the best point pairs can be deemed to be CP candidates, and several points selected from these point pairs are sufficient for transformation estimation. It should be noted that the CPs remain separated from one another and their distributions should be as even as possible, as shown in Figure 4c.

3. Experiment Results

3.1. Data Introduction

Two SAR images of Beijing, China, are chosen for the evaluation of the proposed method. The Radarsat-2 image was acquired on 22 October 2008, and is shown in Figure 5a. The spatial resolution of the image is 1.5 meters per pixel. The TerraSAR-X image was acquired on 22 April 2011, and is shown in Figure 5c. The spatial resolution is 1.25 meters per pixel. The grey differences between the two images are significant, and the features at the conjugate positions are obviously inconsistent, indicating that the registration will be a challenge.

3.2. Shape Extraction and Shape Matching

In this experiment, we select only the water body as the matching object because there are few consistent distinct objects in the urban area. In SAR images, water surfaces behave as spectrum reflectors at radar wavelengths and thus appear as low intensity areas in contrast to brighter regions, such as the rough surrounding terrain, which has diffuse scattering properties. The proposed method based on shape context uses a log-polar histogram of coordinates and is tolerant of local deformations. The shape detail of the object contour is not the key factor for matching. Shape extraction is not the emphasis of this research, therefore image segmentation based on intensity and texture will be used. For the extraction of smooth water bodies, amplitude thresholding approaches are commonly applied. The local mean and variance can be used to distinguish between water and land.

After image segmentation, the filling of holes, the removal of small objects and contour vectorization are necessary to get applicable object shapes. In these procedures, appropriate morphological processing (e.g., erosion, dilation, opening, closing, and boundary) will be used. The contours extracted from the images are shown in Figure 5b,d. The contours in different images are not completely coincidental and have local differences for various reasons, e.g., imaging conditions, object changes, occlusion, and segmentation technology. The example can be shown in Figure 6.

In this study, we focus on the correspondence problem between two incomplete contours of the same object in different images, meanwhile, we detect stable and invariant CPs. Firstly, contour smoothing is essential before shape matching. The contour smoothing can reduce the variations and details. Because contour smoothing is not a key process for shape matching, a slight smoothing is appropriate and excessive smoothing is avoided. The example of smoothing is shown in Figure 2. Secondly, the shape context descriptor is used in shape matching to overcome the correspondence problem. This descriptor exhibits globally discriminative characterization and local deformation tolerance for varying contours. Shape context matching results are shown in Figure 7. Some object contours have relatively large differences, as shown in Figure 7f. It can be seen that the shape detail of the contour is not the key factor for matching.

In previous publications, shape similarity calculations faced heavy computational burdens, and the thresholds of similarity were difficult to determine. In this study, the similarity between two contours is determined by the percentage of the number of matched points. Two contours can be deemed similar if most points in two contours are matched; the threshold (0.5 to 0.8) is suitable in most cases. The proposed similarity measurement is relatively simple in terms of computation and threshold setting.

3.3. Geometric Transformation and Registration Result

Control point candidates are selected from the best point pairs of matched contours and should be distributed as uniformly as possible on the contours. The number of CPs is determined by the number of points (or length) of the contour. In our experiments, it is appropriate to select one CP for approximately every 160 to 200 points.

A polynomial model can be used to model the transformation between two SAR images. The first- and second-order polynomial models are the most commonly applied models. These two models were tested in the experiment, and a comparison is shown in

Table 1. Quantitative Results of Control Points.

Method	Number of CPs	Polynomial Model	RMSE			Standard Deviation			Max Error
			x	y	xy	x	y	xy	x	y	xy
SC	31	1st-order	1.92	1.88	2.69	1.92	1.88	1.06	3.46	−3.97	4.39
		2nd-order	1.72	1.83	2.51	1.72	1.83	1.01	3.21	−3.57	4.07
SIFT	9	1st-order	2.33	2.73	3.59	2.33	2.73	1.88	4.08	5.34	6.72
		2nd-order	1.57	2.11	2.63	1.57	2.11	1.48	−3.45	−4.60	5.75
SC+SIFT	40	1st-order	2.34	2.43	3.37	2.34	2.43	1.59	−5.28	7.23	7.61
		2nd-order	2.22	2.31	3.21	2.22	2.31	1.33	−4.35	6.74	7.05

and

Table 2. Quantitative Results of Check Points.

Method	Number of Check Points	Polynomial Model	RMSE			Standard Deviation			Max Error
			x	y	xy	x	y	xy	x	y	xy
SC	30	1st-order	2.45	2.33	3.38	2.45	2.27	1.10	−4.36	−4.44	5.16
		2nd-order	3.00	2.43	3.86	2.98	2.34	1.50	−7.34	−4.98	7.35
SIFT		1st-order	4.05	2.51	4.76	4.02	2.48	2.14	8.19	5.22	8.35
		2nd-order	22.52	22.31	31.70	18.16	18.59	24.71	77.41	71.16	105.15
SC+SIFT		1st-order	2.27	1.67	2.82	2.25	1.61	1.07	4.74	−3.31	5.16
		2nd-order	2.85	2.45	3.76	2.65	2.39	1.63	8.26	−5.41	9.06

. The results show that the two models are appropriate for image registration in flat urban areas. The first-order polynomial model (i.e., affine transformation) exhibited a slightly higher precision than the second-order model.

The evaluation of the precision of the image registration involves two aspects: an evaluation of the CPs and an evaluation of the check points. All of the CPs can also be considered to be check points by calculating the residuals using the transformation parameters that are based on these CPs. The residual of the CP is the difference (error) between the reference coordinate (in reference image) and the transformed coordinate from the matching image. The evaluation of the CPs not only detects the precision of the CPs and outliers but also indicates whether the current transformation model is suitable or not. The precision statistics of the CPs are shown in Table 1. The CP residuals based on different methods and different models are shown in Figure 8. In general, the magnitude and distribution of CPs residual can be thought of as an indicator of outliers or bias. It can be seen that all of the residuals based on shape context (SC) are congregated and have a good distribution with no bias.

We manually extracted 30 control points from the pairs of images. These points were regarded as check points to evaluate registration precision. The precision statistics of the check points are shown in Table 2. The check point residuals are shown in Figure 9.

The precision criteria include three indexes: root-mean-square error (RMSE), standard deviation (SD), and max error. Every index was computed in three coordinate directions: horizontal, vertical, and integral.

The experimental results show that the proposed method based on shape context (SC) achieves high registration precision. We compare these results with the results obtained using a SIFT-based method. The SIFT descriptor is affected by significant grey differences, and many outliers exist in feature matches. For this image pair, the number of matches based on SIFT is 19, of which 10 point pairs are mismatched. The mismatch ratio can be reduced by adjusting parameters or applying constraints, however, the number of correct matches may be decreased simultaneously. More importantly, the number of correct matches is usually low and the accuracy of CPs is not high, which is not sufficient to achieve high registration precision. This was concluded from Table 1, Table 2 and Figure 7, Figure 8. As expected, the method combining SC with SIFT exhibited slightly better check point precision than SC (see results in Table 2). This is because SIFT CPs may expand the control range and improve the accuracy.

4. Conclusions

This paper presents a novel multi-sensor SAR image registration method based on the object shape. Compared with traditional feature-based methods, the main advantages of the proposed method are that objects can be matched based on shape information and reliable CPs can be detected from unchanged contours, without the correspondence problem mentioned previously. This method is able to sufficiently select many high-quality CP candidates to achieve high registration precision. The shape context descriptor is used for shape matching between objects with local deformations and exhibits globally discriminative characterization for various objects. Traditional edge- or contour-based methods may heavily rely on the initial segmentation result. In contrast, the proposed method is barely affected by grey differences and segmentation results. Local differences can be easily detected and excluded. The proposed method can also be used for SAR and optical images or even images and maps. The major limitation of the algorithm is that it is suitable to distinct ground objects, but there are few distinct ground objects in some areas. Extending this method to general objects will be the goal of our future work.