Automatic Registration Method for Optical Remote Sensing Images with Large Background Variations Using Line Segments

Abstract

Image registration is an essential step in the process of image fusion, environment surveillance and change detection. Finding correct feature matches during the registration process proves to be difficult, especially for remote sensing images with large background variations (e.g., images taken pre and post an earthquake or flood). Traditional registration methods based on local intensity probably cannot maintain steady performances, as differences are significant in the same area of the corresponding images, and ground control points are not always available in many disaster images. In this paper, an automatic image registration method based on the line segments on the main shape contours (e.g., coastal lines, long roads and mountain ridges) is proposed for remote sensing images with large background variations because the main shape contours can hold relatively more invariant information. First, a line segment detector called EDLines (Edge Drawing Lines), which was proposed by Akinlar et al. in 2011, is used to extract line segments from two corresponding images, and a line validation step is performed to remove meaningless and fragmented line segments. Then, a novel line segment descriptor with a new histogram binning strategy, which is robust to global geometrical distortions, is generated for each line segment based on the geometrical relationships,including both the locations and orientations of theremaining line segments relative to it. As a result of the invariance of the main shape contours, correct line segment matches will have similar descriptors and can be obtained by cross-matching among the descriptors. Finally, a spatial consistency measure is used to remove incorrect matches, and transformation parameters between the reference and sensed images can be figured out. Experiments with images from different types of satellite datasets, such as Landsat7, QuickBird, WorldView, and so on, demonstrate that the proposed algorithm is automatic, fast (4 ms faster than the second fastest method, i.e., the rotation- and scale-invariant shape context) and can achieve a recall of 79.7%, a precision of 89.1% and a root mean square error (RMSE) of 1.0 pixels on average for remote sensing images with large background variations.

1. Introduction

Image registration is the process of geometrically-aligning two or more images of the same scene taken from different viewpoints, at different times or by different sensors, and the main objective of it is to find the most appropriate transformation parameters between the images [1,2,3,4]. It has been widely used in the field of remote sensing, including image fusion, change detection [5,6,7], disaster analysis [8], semantic labelling of images [9] and mapping sciences [10,11,12]. Classical methods generally fall into two categories: area-based methods and feature-based methods [1]. The majority of registration algorithms consist of four steps: feature detection, feature matching, transformation model estimation and image resampling [13,14,15]. Many algorithms thathad been proposed in order to register remote sensing images with geometric and illumination differences mainly focused on the following parts: feature detection, feature description and feature matching [16,17,18,19].

When a disaster, such as a flood or earthquake, takes place, images taken pre and post the disaster should be registered rapidly in order to do the change detection [20]. Registration of this type of remote sensing image pairs turns out to be difficult because significant intensity differences always exist in the same area of the corresponding images. The main characteristics of such image pairs are as follows: local intensity values change significantly; local objects, such as blocks or buildings, are ruined; changes in scale and viewpoint exist in images; the main shape contours, such as mountain ridges, long roads and coastal lines, remain relatively stable. Points, lines and regions are the three most common features used in the feature-based registration method [21,22,23], and the information of intensity and shape structure isusually used to describe the feature [23,24,25]. The most popular local intensity-based feature detector and descriptor is the scale-invariant feature transform (SIFT) [26], which uses the local maximum of the scale space as the point feature and describes it depending on the gradient location and orientation around the point. The SIFT descriptor is highly discriminative, and many other descriptors, such as the speeded-up robust feature (SURF) [27], the gradient location and orientation histogram (GLOH) [28] and the affine-SIFT (ASIFT) [29], have been developed based on it. Bradley [30] presented a complex scale-invariant feature transform (CSIFT) algorithm, which is suitable for complex-valued images. Sedaghat et al. [31] proposed a uniform robust scale-invariant feature matching (UR-SIFT) approach for optical remote sensing images, which had solved the problem of the uneven distribution of the SIFT point features. However, SIFT-based features are hard to detect and match in remote sensing images with large background variations because of significant intensity changes in their neighborhoods. Ghosh et al. [32] introduced a method containing both registration and segmentation based on the advances in sparsity learning. An efficient and accurate set-based registration algorithm for aerial images was proposed by Arandjelović [33]. This algorithm formulated the joint registration problem of a set of image sequences as an optimization scheme, and both robustness and high accuracy could be obtained. A few georeferencing-based methods had also been proposed in recent years [34,35], but ground control points (GCP) are not always available in remote districts and mountainous areas. Generally, the main shape contours, such as coastal lines, long roads and mountain ridges, hold relatively stable information, although global geometrical distortions between two corresponding images are more complex than local ones. The shape context (SC) proposed by Belongie et al. [36] is an effective and rich shape-based descriptor, which is used to describe each point of a shape silhouette based on the geometrical relationships of the remaining points relative to it, and it has been widely used in shape retrieval and trademark recognition. Huang [37] firstly introduced the shape context into remote sensing image registration. The Harris corner detector and the shape context were used as the feature detector and descriptor, respectively, and preferable results can be achieved in images with small background variations. However, its histogram binning strategy is not appropriate for global geometrical distortions, which may produce many outliers. Jiang [38] proposed a rotation and scale invariant shape context (RSISC) using a multi-scale morphological edge detector and an improved shape context descriptor, which was specially designed for global geometrical distortions. The magnitude of Fourier transform in one dimension was applied to achieve rotation invariance, which once was a big obstacle for the widespread use of the shape context. Nonetheless, RSISC cannot cope with high-resolution satellite images, because the gradient orientation of the edge point is too unreliable to be used. Arandjelović [39] proposed an object matching method using boundary descriptors for smooth and untextured images. Boundary curves were segmented from images, and equidistant key points could be obtained by sampling the curves. Then, the boundary normals of a local profile around one key point were utilized to describe this point. Although this algorithm isvery successful when handling the matching problem in many classical object image sets, there still exist two limitations when dealing with images taken pre and post disasters: since the key points are drawn independently from the two non-closed contours, there is inevitable jitter noise due to the differences of the starting points; the local descriptor is more sensitive to local noise and incorrect shape components, which commonly exist in remote sensing images.

Several line segment matching algorithms designed for industrial vision applications have also been reported in recent years. Arandjelović [40] proposed a gradient edge mapping algorithm for frontal face recognition, but the vertical symmetry of faces was utilized in the edge extraction process, which might not be effective in other scenarios. Wang et al. [41] defined a robust descriptor called MSLD (mean-standard deviation line descriptor) for line matching.The neighbor region of each line segment is divided into non-overlapped sub-regions, and the descriptor is generated using a SIFT-like strategy. Zhang et al. [42] presented a line matching method utilizing both the local appearance of line segments and their geometrical attributes. López [43] proposed a novel approach for line detection and matching. A phase-based edge detector and a fusion stage are performed for line detection, and an iterative process utilizing both the structural information and the appearance of each line is performed for line matching. Although the line matching algorithms mentioned above can achieve preferable results in certain situations, they may be non-effective for images taken pre and post disasters as a result of the use of local intensity information.

In order to register remote sensing images with large background variations, in which most local intensity information is unreliable, we proposed an automatic image registration method based on the line segments, which is described using the information of the main shape contour (named RMLSM for short). First, EDLines (Edge Drawing Lines) [44] is used to extract line segments from the reference and sensed images, and a line validation step is performed to remove meaningless and fragmented ones thatdo not belong to the main shape contour. Then, a novel line segment descriptor is generated for each line segment based on the geometrical relationships of the remaining line segments relative to it. As the main shape contours are relatively stable, correct line matches could be obtained by cross-matching among their descriptors. Finally, a spatial consistency measure is utilized to increase matching accuracy, and transformation parameters between the reference and sensed images can be figured out. Experiments on different types of satellite datasets comprising both rough terrains (e.g., mountainous areas or urban regions)and flat areas demonstrate that the proposed algorithm is automatic, fast and can achieve high accuracy (evaluated by recall, precision, correct matches (CM) and root mean square error (RMSE)) for remote sensing images with large background variations.

The rest of this paper is organized as follows. In Section 2, the proposed method is described in detail. Experiments and results are presented in Section 3. The discussion and conclusions parts are given in Section 4 and Section 5, respectively.

2. Methodology

The entire process of the RMLSM algorithm is shown in Figure 1, and it can be divided into the following steps: feature detection, feature matching, transformation model estimation and accuracy evaluation. An image pair taken before and after a tsunami in Sendai city, Japan, is shown in Figure 2a, and there are two types of line segments with red and yellow colors in Figure 2b. We can see that although the backgrounds change significantly between two images, the main shape contours, which are depicted in Figure 2b using red lines, remain relatively invariant. Line segments depicted in yellow are meaningless and fragmented ones that should be discarded during the line segment detection process.

2.1. Feature Detection

As shown in Figure 1, this section can be divided into two parts: line segments’ detection and line segments’ validation. From Figure 2b, we can see that there exist two types of line segments. Red ones mainly consist of so-called meaningful line segments, which constitute the main invariant information (i.e., the main shape contour), while yellow ones mainly consist of meaningless and fragmented line segments, which may perturb the feature descriptor generation. The main characteristics of these meaningful line segments are as follows: large gradient magnitude; long distance; small slope differences between adjacent line segments. EDLines and a line validation step are used in this section in order to obtain as many meaningful line segments as possible from the reference and sensed images.

2.1.1. Line Segments’ Detection Using EDLines

EDLines is a real-time line segment detector with high accuracy and blazing speed, and it is very suitable for extracting the line segment that constitutes the main shape contour. This algorithm is composed of three parts: (1) Given a grayscale image, a fast and novel edge detector called edge drawing (ED) [45], which can produce a set of contiguous, clean chains of pixels, is run. These pixels intuitively correspond to object boundaries. (2) Next, line segments are extracted from the generated pixel chains using the least squares line fitting method. (3) Finally, a line selection step based on the Helmholtz principle is used to eliminate incorrect line segment detections.

Compared toother famous edge detectors, such as Canny, etc., ED produces a set of contiguous, clean and connected chains of edge pixels rather than disjoint, independent and discontinuous entities. The flow chart of ED is shown in Figure 3. The comparative edge detection results of ED, Canny and Prewitt implemented on the left image of Figure 2a are shown in Figure 4. From Figure 4, it can be observed that a set of clean, contiguous, well-localized, one-pixel-wide edge segments isextracted by ED instead of only an isolated binary edge map (Prewitt) or an edge map containing many disjoint, unattended noisy edges (Canny).

Next, the line segment extraction step is performed by splitting an edge segment consisting of a contiguous chain of edge pixels into one or more straight line segments. Walking over the pixels in sequence, lines are fit to pixels by means of the least squares line fitting method until the error exceeds a certain threshold (e.g., 1 pixel error). A new line segment is generated when the error exceeds this threshold. The remaining pixels of the chain are processed recursively until the last pixel of the edge segments and a set of line segments with endpoints expressed by sub-pixel coordinates are obtained.

Table 1. Line segment extraction algorithm.

Algorithm 1: Algorithm to extract line segments

FitLines(Pixel * pixelChain, intpixels_num)

{

doubleError_LineFit=INFINITY; // current line fit error

Line_Equationline_equation; // y=ax+b or x=ay+b

while(pixels_num>MIN_LINE_DISTANCE)

{

LeastSquareFitLine(pixelChain, MIN_LINE_DISTANCE, &line_equation, &Error_LineFit);

if(Error_LineFit<=1.0) break; // An initial line segment detected

pixelChain++;

pixels_num--;

} // end-while

if(Error_LineFit>1.0) return; // no initial line segment

// if an initial line segment has detected

intlineLen= MIN_LINE_DISTANCE;

while(lineLen<pixels_num)

{

double d=ComputePointDistanceToLine(line_equation, pixelChain[lineLen]);

if(d>1.0) break;

lineLen++;

} // end-while

// output current line equation

LeastSquareFitLine(pixelChain, lineLen, &line_equation);

Output “line_equation”

FitLines(pixelChain+lineLen, pixel_num-lineLen); // Extract line segments from the remain pixels

} //end-FitLines

shows the algorithm thatis used to extract line segments from a set of edge segments.

Finally, a line selection step is implemented based on the Helmholtz principle, which basically states that the expectation of one structure by chance must be very low for this structure to be perceptually meaningful [46]. This step can help to eliminate a part of the meaningless and fragmented lines. To make this selection process by the Helmholtz principle concrete, “Number of False Alarms(NFA)” of a line segment is defined as follows: Let X be a line segment of length “ld” with at least “z” points having their directions (thedirection of a unit vector, which is orthogonal to the unit gradient vector of the reference point) aligned with the direction of X in an image of size $N\times N$ pixels. The NFA of X is defined as:

$$NFA(ld,z)=N^4\cdot {\displaystyle \sum _{i=z}^{ld}\left(\begin{array}{l}ld\\ i\end{array}\right)}{q}^i{(1-q)}^{ld-i}$$

(1)

where “q” represents the accuracy of the line direction (q is set to be 0.125, referring to [44]). The line selection step can be realized as follows: we compute the NFA of a line segmentand preserve it if $NFA(ld,z)\le \epsilon$; otherwise, this line segment will be rejected. Desolneux et al. [47] suggest setting $\epsilon$ to 1, which indicates that at most, one false detection is allowed per image. Figure 5a shows EDLines in action on an image pair taken pre and post a tsunami.

2.1.2. Line Segments’ Validation

A preliminary set of line segments is obtained after the implementation of EDLines. We see from Figure 5a that although the main shape contours have been extracted, there still exist many fragmented line segments, which are likely to perturb the feature matching process. As these fragmented line segments are usually discontinuous and shorter than the ones on the main shape contour, we use a length threshold $le{n}_t$ to remove these line segments. In order to distinguish meaningful line segments from fragmented ones, this threshold should be determined appropriately. For the evaluation of this purely empirical threshold, we use $lm$ to represent the median length of all line segments, and a sequence of candidate thresholds from $0.2\times lm$ to $1.8\times lm$ with an interval of $0.2\times lm$ isevaluated on ten real image pairs. Two evaluation criteria, i.e., $recall$ and $precision$, which are defined as the ratio of the number of correct matches obtained after validation to the number of all original correct matches and the ratio of the number of correct matches obtained after validation to the number of all matches after validation, respectively, are utilized in order to test the effectiveness of each candidate threshold. The $recall$ and $precision$ values against the candidate threshold are shown in Figure 6, and each point in the graph represents the average of ten image pairs. According to [21], in order to obtain a better result in the shape-based feature matching algorithm, both the original recall and precision values would be best to be more than 0.7. We see from Figure 6 that in order to get a relatively better recall-precision tradeoff, the most appropriate value of $le{n}_t$ is in $[0.8\times lm,1.2\times lm]$, and $le{n}_t=1.0\times lm$ is chosen in this method for all datasets. Figure 5b shows line segments’ validation in action on Figure 5a.

2.2. Feature Matching

In this section, we assume that the main shape contour of an image is essentially captured by a subset of line segments originated from the edge segments. More concretely, the main shape contour can be represented by a discrete set of line segments produced by Section 2.1. The final line segment matching set can be obtained as follows: (1) a descriptor is proposed for each line segment to find its best match in the other image; (2) a spatial consistency measure is used to remove incorrect matches obtained in Step (1).

2.2.1. Feature Descriptor

Two line segment sets obtained in Section 2.1 can be denoted as $L=\left\{l_i\right\}(i=1,2,3...m)$ and $S=\left\{s_j\right\}(j=1,2,3...n)$; $m$ and $n$ are the numbers of line segments in $L$ and $S$, respectively. For each line segment $l_i$ in the first image, we want to find its “best” matching line segment $s_j$ in the second image. Experience suggests that the richer the information one descriptor has, the easier the matching process will be. As intensity information around the line segment is unreliable, we consider the set of vectors originating from the position of a line segment to that of all other line segments on the main shape contour. These vectors represent the relationship of the entire shape contour relative to the reference line segment. Obviously, a descriptor constructed by this set of vectors is rich enough. In our approach, the geometric attributes of the line segments are represented by their midpoints, which can be denoted as $P=\left\{p_i\right\}=\{x_i,y_i\}(i=1,2,3...m)$ ($p_i$ corresponds to $l_i$) in the first image and the orientations of their normal vectors. For a line segment $l_i$ in the first image, we compute a histogram $h_i(t)$ expressing the relative coordinates of the remaining $m-1$ midpoints to its midpoint $p_i$,

$$h_i(t)=\#\{(p_j-p_i)\in bin(t),(j=1,2,3...m,j\ne i)\}$$

(2)

We use a new histogram binning strategy as shown in Figure 7b, and Figure 7a shows the log-polar space used in the shape context(SC), which is more sensitive to the location of nearby points than to those farther away. However, when log-polar space is applied, bins near the center become too small, which may lead to the sensitivity of the descriptor to the pixel location errors, and bins far from the center become too big to describe the shape silhouette exactly. Consider that the distribution of line segments is not as dense as that of points;the proposed histogram binning strategy with relatively uniform bins seems more suitable. We use identical radial quantization lengths and larger angular quantization numbers for outer circles compared toinner circles. The radial quantization number nr and the angular quantization number $\{M_1,M_2,...,M_{nr}\}$ ($M_1$ corresponds to the smallest circle) will be discussed in Section 3. The advantages of the proposed histogram binning strategy are as follows: (1) robustness to pixel location errors; (2) accurate description of the shape contour far from the reference line segment; (3) low dimension of the descriptor.

As shown in Figure 7c, the same distribution of points may produce different shape contours. Therefore,in each bin, we not only count the number of line segments falling in it, but also consider the orientation of each line segment (the orientation of the normal vector of each line segment) in this bin. A unit vector is attached to each line segment with its direction equal to the orientation of this line segment $l_i$, and it is decomposed into two vectors along the x-axis and the y-axis, respectively. The d-bin histogram $h_i(t)$ for the line segment $l_i$ is then converted to a 2d-bin histogram as follows:

$$h_i\prime (o)=\{h_i{(1)}^x,h_i{(1)}^y,h_i{(2)}^x,h_i{(2)}^y...,h_i{(d)}^x,h_i{(d)}^y\}$$

(3)

It is easy to see that this 2d-bin histogram can provide a more accurate description for each line segment. We normalize radial distances by a factor $\alpha$, which is the mean distance among the $m^2$ midpoint pairs in the image to achieve scale invariance, and orientations relative to that of the reference line segment are used to achieve rotation invariance. Finally, a Gaussian weighting function (see Figure 8b) is used to increase the robustness against the geometric distortions. The complete process of the proposed descriptor generation is shown in Figure 8. Compared to the descriptor in [39] (named OMBD (Object Matching Using Boundary Descriptors) for short), which is presented for smooth and untextured object matching, our proposed descriptor seems more suitable for remote sensing images with large background variations mainly due to the following aspects.

We use a global descriptor instead of a local one in the OMBD, which is more sensitive to local noise and incorrect shape components, which commonly exist in remote sensing images; the OMBD descriptor only considers the local profile of the boundary normal’s direction around a reference key point rather than the geometric relations of them, which may result in an unstable descriptor in case disordered segments exist.

Consider a line segment $l_i$ in the first image and a line segment $s_j$ in the second image. The cost of matching these two line segments isdefined as:

$$C_{ij}=C(l_i,s_j)=\sqrt{\frac{1}{2d}{{\displaystyle \sum _{o=1}^{2d}\left(\frac{h_i\prime (o)-h_j\prime (o)}{h_i\prime (o)+h_j\prime (o)}\right)}}^2}$$

(4)

where $h_i\prime (o)$ and $h_j\prime (o)$ denote the 2d-bin normalized histogram at $l_i$ and $s_j$, respectively.

$l_a$ and $s_b$ are deemed as a correct line segment match if the following cross-matching expressions are satisfied:

$$\{\begin{array}{l}{C}_{ab}==\min \left\{C_{ib}\right\}(i=1,2,3...m)\\ C_{ab}==\min \left\{C_{aj}\right\}(j=1,2,3...n)\end{array}$$

(5)

That is, not only $s_b$ is the best match of $l_a$ in the second image, but $l_a$ is the best match of $s_b$ in the first image, as well. This is a cross-matching process thataims at obtaining accurate results in case fragmented line segments or occlusions exist.

2.2.2. A Spatial Consistency Measure

Two preliminary line segment matching sets of size $Q$ have been obtained after the steps above, which can be denoted as $Lp=\left\{l_i\right\}=\{(k_i,b_i)\}$ and $Sp=\left\{s_i\right\}=\{(k_i\prime ,b_i\prime )\}$ $(i=1,2,3,...,Q,$ $l_i$ matches $s_i$, parameters $k$ and $b$ of the line equation $y=kx+b$ are used to represent a line segment). Although the proposed descriptor is robust and discriminative, there still exist a few outliers in the preliminary matching sets mainly due to the small differences of two main shape contours. In this part, a spatial consistency measure [48] is utilized to remove the outliers. The spatial consistency is a measurement of the closeness of the reference features to the sensed features under the present transformation, and it is usually used for point matching. We roughly use the global affine transformation, of which the parameters are denoted as $T_g(\theta )$, to approximate the present transformation and to evaluate the closeness of the matches one by one. The transformation is formulated as:

$$\left[\begin{array}{l}x\prime \\ y\prime \end{array}\right]=\left[\begin{array}{l}{a}_1 a_2\\ a_3 a_4\end{array}\right]\left[\begin{array}{l}x\\ y\end{array}\right]+\left[\begin{array}{l}{a}_5\\ a_6\end{array}\right]$$

(6)

where $(x\prime ,y\prime )$ and $(x,y)$ represent the coordinates of two corresponding points. In order to calculate $T_g(\theta )$ (i.e., $a_1$ - $a_6$) from $\{(k_i,b_i)\}$ and $\{(k_i\prime ,b_i\prime )\}$, $(x\prime ,y\prime )$ and $(x,y)$ should be replaced by $(k\prime ,b\prime )$ and $(k,b)$ using the equation $y=kx+b$.

In each iteration, first, a group of affine transformation parameters $T_g(\theta )$ between the two corresponding images can be calculated by means of the least square method using $\{(k_i,b_i)\}$ and $\{(k_i\prime ,b_i\prime )\}$ $(i=1,2,3,...,Q)$; then, an error threshold $E_t=\{k_e,b_e\}$ is set for the slopeand intercept independently,and the root mean square error (RMSE) is obtained as follows: $E[T_g(\theta )]=\{\sqrt{[{\displaystyle \sum _{i=1}^{Q}d\_k_i{}^2}]/Q},\sqrt{[{\displaystyle \sum _{i=1}^{Q}d\_b_i{}^2}]/Q}\}$, $d(i)=(T_g(\theta )(l_i)-s_i)=\{d\_k_i,d\_b_i\}$, $i=1,2,3,...Q$. Both $d(i)$ and $E[T_g(\theta )]$ are composed of two components, which relate to the slope and intercept, respectively. Finally, while $E[T_g(\theta )]>E_t$, the match with the maximum of $d(i)$ is removed. The line segment matching sets are renewed, and the transformation parameters are recalculated. The iteration process will not stop until $E[T_g(\theta )]\le E_t$, and two final line segment matching sets can be obtained.

2.3. Transformation Model Estimation

After two matching sets of line segments have been obtained, the exact transformation parameters between the reference and sensed images will be calculated. According to [21], the relationship between two remote sensing images taken from the same area can be approximately modeled by affine transformations. In this section, a global affine transformation model is applied. In general, the affine transformation consists of a translation and linear transformations (rotation, shear or scaling), and it preserves collinearity and the ratio of distance. In the ideal situation, affine transformation parameters can be figured out from three nonparallel corresponding line segment matches. However, in order to increase the accuracy and robustness, all of the line segment matches obtained from Section 2.2 are used to estimate the coefficients by means of the least square method.

2.4. Accuracy Evaluation

The accuracy evaluation of the registration is performed using recall, precision, CM and RMSE, which can be defined as follows:

$$r\mathrm{e}call=\frac{true positives}{total positives}$$

(7)

$$p\mathrm{r}ecision=\frac{true positives}{true positives+false positives}$$

(8)

$$correct matches=true positives$$

(9)

$$RMSE=\sqrt{\frac{1}{np}{\displaystyle \sum _{i=1}^{np}{[\Vert T(p{e}_i\prime ,\theta )-p{e}_i\Vert ]}^2}}$$

(10)

where the $true positives$ represents the number of correct line segment matches in the final matching sets, the $total positives$ represents the number of correct line segment matches existing in two images after feature detection and the $false positives$ represents the number of incorrect line segment matches in the final matching sets. Two completely correct and well-distributed point matching sets of size $np$, which are denoted as $\left\{p{e}_i\right\}$ and $\{p{e}_i\prime \}$ $(i=1,2,3...,np,$$p{e}_i matches p{e}_i\prime )$, are selected manually fromthe two corresponding images by an experienced expert to calculate the RMSE, and $T(\theta )$ represents the transformation parameters obtained in Section 2.3.

3. Experiments and Results

In this part, the performance of the RMLSM is evaluated using several types of satellite images. Experimental results compared tothe UR-SIFT, the RSISC and the MSLD are presented to verify the effectiveness of the proposed method. Image datasets, parameter setting and experimental results are introduced in the following.

3.1. Image Datasets

Two groups of remote sensing images are used to evaluate the performance of the RMLSM, including simulated images and multisensor/multitemporal real images. Because the highlight of the RMLSM is its robustness against large background variations, the image pairs of the first group are simulated by utilizing several patches of cloud in various shapes, of which centers are distributed randomly across the image, instead of the original pixels, and a certain amount of affine transformations are also added to the images. Figure 9 shows several examples of the simulated images. The second group consists of different types of multisensor and multitemporal images, most of which are taken pre and post disasters. The complete information of all image pairs is shown in

Table 2. The complete information of all experimental satellite image pairs.

Group	No.	Location	Image Size	Bits	GSD(m)	Date	Sensor	Events
Simulatedimages	1-1	Hangzhou	2129 × 2373	8	30	2011	Landsat5	Band1
		Hangzhou	2129 × 2373	8	30	2011	Landsat5	Band2
High-resolution real images	2-1	Pakistan	1817 × 1194	11	2	2007	QuickBird	Pre-flood
		Pakistan	1739 × 1056	11	2	2010	WorldView	Post-flood
	2-2	Mississippi	4254 × 3646	11	0.61	2010	QuickBird	Pre-tornado
		Mississippi	4254 × 3647	11	0.61	2010	QuickBird	Post-tornado
	2-3	Natori	940 × 529	11	0.61	2007	QuickBird	Pre-tsunami
		Natori	940 × 529	11	0.61	2011	QuickBird	Post-tsunami
	2-4	Chile	2559 × 2528	11	0.61	2010	QuickBird	Pre-tsunami
		Chile	2566 × 2621	11	0.61	2010	QuickBird	Post-tsunami
Medium- and low-resolution real images	2-5	Myanmar	1800 × 1200	8	231.65	2008	Aqua	Pre-flood
		Myanmar	1687 × 1081	8	231.65	2008	Aqua	Post-flood
	2-6	Alaska	1800 × 1200	8	30	2004	Landsat7	Band 3
		Alaska	1800 × 1200	8	30	2004	Landsat7	Band 4
	2-7	Fukushima	584 × 923	8	10	2009	SPOT 4	Pre-tsunami
		Fukushima	728 × 1064	8	10	2011	SPOT 4	Post-tsunami

. This dataset covers a variety of high-, medium- and low-resolution images with a ground sample distance (GSD) from 0.61 to 231.65 m, and both rough terrains and flat regions are included in this dataset. All real images have significant background variations and a certain amount of scale and viewpoint differences.

3.2. Parameter Setting

In our experiments, the UR-SIFT, the RSISC and the MSLD are implemented based on the codes offered by the authors of the three algorithms.

During the feature detection, the EDLines algorithm is implemented based on the publicly-available codes on their website with the default parameters presented explicitly in [44]. The length threshold $le{n}_t$ in the line segments validation step is set to be $1.0\times lm$, which has already been described in detail in Section 2.1.2.

In the feature matching process, there are three parameters, i.e., radial quantization number $nr$, angular quantization number $\{M_1,M_2,...,M_{nr}\}$ and an error threshold $E_t$. The choices of these three parameters are purely empirical, and two aspects should be considered in the selection process: higher accuracy and lower dimension of the descriptor. Several combinations are tested on ten pairs of real remote sensing images, and the average results are depicted in

Table 3. Average results of the accuracy with different parameters.

Parameter Setting	Recall (%)	Precision (%)	Descriptor Dimension
$nr=3,\{M_1,M_2,M_3\}=\{4,6,8\}$	65.2	73.1	36
$nr=3,\{M_1,M_2,M_3\}=\{4,8,10\}$	69.7	77.3	44
$nr=4,\{M_1,M_2,...,M_4\}=\{4,6,8,10\}$	76.0	82.4	56
$nr=4,\{M_1,M_2,...,M_4\}=\{4,8,10,12\}$	77.4	84.0	68
$nr=5,\{M_1,M_2,...,M_5\}=\{4,6,8,10,12\}$	81.7	86.4	80
$nr=5,\{M_1,M_2,...,M_5\}=\{4,8,10,12,14\}$	80.1	87.6	96
$nr=6,\{M_1,M_2,...,M_6\}=\{4,6,8,10,12,14\}$	82.5	91.1	108
$nr=6,\{M_1,M_2,...,M_6\}=\{4,8,10,12,14,16\}$	83.7	90.9	128
$nr=7,\{M_1,M_2,...,M_7\}=\{4,6,8,10,12,14,16\}$	84.2	91.4	140
$nr=7,\{M_1,M_2,...,M_7\}=\{4,8,10,12,14,16,18\}$	84.7	92.5	164

. The best accuracy-dimension tradeoff is obtained when $nr=6;$ $\{M_1,M_2,...,M_6\}=\{4,6,8,10,12,14\}$, which results in a 108-dimensional descriptor. In the section of the spatial consistency measure, $E_t$ is used to remove false matches. As the slope and intercept of a line segment are two different variables, $E_t=\{k_e,b_e\}=\{0.1,5\}$ is able to distinguish an outlier from an inlier based on experience.

3.3. Experimental Results

EDLines and a line segment validation step are implemented to extract line segments from all image pairs. Then, the proposed descriptor is used for line segment description and matching. Finally, accuracy comparisons with the UR-SIFT, the RSISC and the MSLD are carried out to evaluate the proposed registration method.

Table 4. Comparative matching accuracy between the RMLSM and the shape context (SC).

Recall/Precision	No. 2-1	No. 2-2	No. 2-3	No. 2-4	No. 2-5	No. 2-6	No. 2-7
RMLSM	75.2%/86.9%	80.5%/93.1%	65.6%/81.4%	68.7%/70.4%	81.8%/94.1%	76.7%/93.7%	82.4%/90.2%
SC	71.2%/80.4%	74.6%/83.2%	67.2%/78.4%	67.0%/65.1%	73.2%/82.0%	70.7%/84.4%	77.3%/81.6%

shows the comparative matching accuracy of the proposed descriptor and the traditional shape context using all real remote sensing images in the image datasets. The process of feature detection is identical before the comparison of the two descriptors. The matching results show that the proposed descriptor is more effective than the traditional shape context in terms of both recall and precision.

Figure 10 shows the comparative results of these four algorithms implemented on the simulated image pairs in terms of recall, precision, CM and RMSE. The No.1–No.5 image pairs refer to the (a)(b)–(a)(f) in Figure 9, all of which contain considerable background variations and a certain amount of affine transformations. The results presented in Figure 10 show that the proposed method outperforms the other three techniques in recall, precision and RMSE. In some cases, the CM of the RMLSM is less than that of the RSISC. However, this will have little effect on the computational accuracy of the final transformation model, because a line segment contains more information than a point. As the UR-SIFT purely depends on the intensity information in the feature matching process, it performs worst among the four algorithms for all evaluation criteria.

The matching results of the RMLSM for three sample simulated image pairs are presented in Figure 11. To make the graphs easier to read, the corresponding line segments have the same number and color. Registrations for such images prove to be difficult because the intensity information is unreliable. This figure shows that although there exist significant background variations and affine transformations in the image pairs, almost all of the line segments on the main shape contour have been extracted and correctly matched.

Figure 12 shows the performances of the four algorithms implemented on seven real image pairs in terms of recall, precision, CM and RMSE. All real image pairs are taken pre and post disasters, and rough terrains, such as mountainous regions and building areas, are included. Results demonstrate the effectiveness and robustness of the proposed method. The UR-SIFT achieves the worst result because it purely relies on the intensity information, which is unreliable in these real disaster image pairs. The RSISC can obtain better results than the UR-SIFT mainly due to the use of shape information, but the neglect of the point orientation results in more outliers than the RMLSM. The MSLD is a line matching algorithm, but it also relies too much on the appearance and the local structure of the neighborhood of each line segment, which makes it not perform as wellas the RMLSM. However, it can be observed that the recall and precision are somewhat low for No.4 in the second group as a result of relatively complicated distortions between these two images.

The matching results of the RMLSM implemented on all real image pairs are shown in Figure 13a–d correspond to (2-1)–(2-4) in Table 2, and Figure 14a–c correspond to (2-5)–(2-7) in Table 2, while Figure 13 covers the high-resolution satellite images, and Figure 14 covers the medium- and low-resolution ones. To make the graphs easier to understand, the corresponding line segments are presented using the same number and color. Accurate and well-distributed line segment matches can be obtained as shown in these two figures. Results demonstrate the robustness of the proposed method against large background variations and a certain amount of geometric distortions.

Figure 15 shows the registration results of the UR-SIFT (Figure 15a), the RSISC (Figure 15b)and the MSLD (Figure 15c) on three sample real image pairs respectively and the results of the RMLSM (Figure 15d–f) on all these three image pairs. Six sub-images are enlarged for visual assessment. The sub-images cover a variety of scenes, such as roads, coastal lines and mountain ridges. From the results of the RMLSM, we can see that the sensed image fits well with the reference image in spite of the significant background variations. However, it can be observed that slight mismatches around the mountain ridges (Figure 15f) still exist as a result of the relief displacement of the mountains. This is a challenge for the registration of high-resolution images, and it cannot be settled perfectly through an image-to-image registration.

Because of the blazing speed of EDLines and the low dimension of the proposed descriptor, the computational complexity of the proposed registration method is the lowest among the four algorithms;the average time of the registration process is 65 ms for the UR-SITF, 47 ms for the RSISC, 96 ms for the MSLD and 43 ms for the RMLSM (using an Intel Core (TM) 2 2.93-GHz central processing unit and a memory of 2 GB).

4. Discussion

An automatic image registration method for remote sensing images with large background variations is proposed in this paper, and preferable results are obtained on both simulated and real image pairs.

The average results of the proposed method for both simulated and real image pairs are 79.7%, 89.1% and 1.0 pixels in recall, precision and RMSE, respectively. The RMLSM outperforms the other methods for all three evaluation criteria. The UR-SIFT achieves a recall of 63.1%, a precision of 70.7% and an RMSE of 1.4 pixels on average, which is the worst result among the four algorithms, because it purely depends on the intensity information during the feature matching process. The MSLD can obtain a recall of 73.9%, a precision of 79.4% and an RMSE of 1.2 pixels on average. The accuracy of the MSLD is lower than that of the RMLSM because local structure-based methods are usually more sensitive to local noise and incorrect structure components. The recall of 78.8%, the precision of 85.0%, and the RMSE of 1.1 pixels are achieved on average during the implementation process of the RSISC, of which accuracy is lower than that of the RMLSM mainly due to the unreliability of the gradient orientation of the edge point. In several image pairs, such as No.2 and No.6 in the second group, the CM of the RMLSM is lower than that of the RSISC. However, this will have little effect on the computational accuracy of the final transformation model because a line segment contains more information than a point.

The results from this study show that the shape can be utilized to constitute a feature descriptor independently when the intensity around this feature is unreliable. Most of the previous studies concentrated on local intensity-based registration methods to achieve robustness to geometric and illumination variations between the corresponding images. However, finding the invariant information is the common goal of all methods. The experimental results show that local intensity-based algorithms may fail when dealing with image pairs taken pre and post disasters, and in that case, the main shape structure becomes the most stable information.

Although EDLines and a line segment validation step can effectively extract the main shape contour consisting of line segments on it and the proposed line segment descriptor is invariant to scale, rotation and a certain amount of other affine transformations, there still exist several limitations in our registration algorithm. Global geometric distortions are more complex than the local ones, and non-rigid transformations may exist between the satellite image pairs. However, we are uncertain whether the proposed algorithm can cope with the satellite image pairs with high order non-rigid transformations.

5. Conclusions

In this paper, an automatic image registration method has been proposed for remote sensing images with large background variations (e.g., images taken pre and post an earthquake or flood). First, EDLines is used for line segment extraction, and a validation step is developed to reject the meaningless and fragmented features. Then, a shape-based line segment descriptor is introduced for each line segment. Finally, a spatial consistency measure is used to remove incorrect matches, and transformation parameters can be figured out. The experiment results on both simulated and real satellite image pairs demonstrate that high-quality and well-distributed line segments can be extracted and correctly matched, and a recall of 79.7%, a precision of 89.1% and an RMSE of 1.0 pixels have been achieved on average for all image pairs. All of the comparative algorithms are implemented on the same platform, and the running time of the proposed method is 4ms faster than that of the second fastest method. From this study, the global shape structure information is proven to be reliable and effective in the process of remote sensing image registration, especially when significant background variations exist in the corresponding images.

As future work, a registration method for remote sensing images with both large background variations and complex global distortions, such as non-rigid transformations and thin-plate-spline transformation, will be investigated. Some compound features, like the combination of line segments and points, could also be considered in order to increase matching accuracy.