In this section, we provide the results of two experiments—based on the methodology derived in Section 4—to jointly register and classify a set of remotely sensed images. The first experiment was conducted over a simulated dataset in order for us to investigate many aspects of our proposed algorithm. Next, we examined the performance of our algorithm in an actual remote sensing image. For both examples, the goal was to examine the performance of the algorithm to different degrees of initial registration errors. If our algorithm performed perfectly, it would be able to align images together and produce an LCM from unregistered images as accurate as when images were registered.
5.1. Experiment 1
In the first experiment, we examined the performance of the proposed algorithm in terms of classification performance and registration accuracy by attempting to produce a land cover map from a set of four simulated images. All the simulated images were of an equal size of 512 × 512 pixels (
Figure 3) and contained four land cover classes (Classes 1–4) with intensity values of zero, one, two and three for black, dark gray, light gray and white areas, respectively. Based on the noiseless image, the ground truth image in this example is provided in
Figure 4 where the blue, black, green and red colors correspond to Classes 1–4, respectively. Next, all of the input images were added with the independent and identical Gaussian noise with a zero mean and standard deviation of σ = 1 to examine the performance of our proposed algorithm to image noise.
Figure 5 shows an example of the input image for σ = 1. We perceived that the observed image appeared to be very noisy. We note here that all four noiseless simulated images are identical, and differ only once the noise has been added.
Since our algorithm performed both image registration and land cover mapping simultaneously, the performance of our algorithm could be evaluated in terms of how much the resulting LCM deviated from the reference LCM, and the estimation error between our calculated map parameters and the actual parameters that registered the LCM to the simulated images. If our algorithm performed perfect registration and land cover mapping, the resulting percentages of mis-classified pixels would be zero, and the registration error between the images and LCM would also be zero. In this example, the correct mapping parameters for all observed images were identical and equal to
MPerfect = [1,0,0,1,0,0] which correspond to unit scale, zero skew, and zero displacement. Next, since we wanted to examine the effects of the initial registration errors on the performance of our algorithm, we investigated different scenarios of initial registration errors by varying the initial mapping parameters between the observed images and LCM at different values of displacement, scale and skew parameters. In particular, we investigated three scenarios for only the displacement, only the scale and only the skew errors, respectively.
Table 1 shows the initial mapping parameters for all three scenarios. Here, δ, ρ and η are the initial displacement, scale, and skew parameter errors. Note that the initial mapping parameter errors for Image 1 for all scenarios were zero since we assumed that the first image is registered to the LCM as mentioned in Section 3.1.
Before investigating the performance of our proposed algorithm, we examined the effect of registration errors on the performance of image classification. This value can be viewed as the worst case scenario where the LCM is derived directly from the set of mis-registered images. Here, we employed the maximum likelihood classifier (MLC) [
30] to the set of four remapped images, and the LCM was obtained from
where the subscript
n denotes the
n-th remapped image. We note here that
Equation (37) is the special case of the optimum LCM obtained from
Equation (22) when
β = 0.
Figure 6a–c displays the resulting LCM for
δ = 12 and
σ = 1 for Scenario I,
ρ = 0.05 and
σ = 1 for Scenario II, and
η = 0.05 and
σ = 1 for Scenario III. We chose these values so that all scenarios yielded the averaged registration errors between 12 and 22 pixels or 2%–5% of the image size. The averaged percentages of misclassified pixels after a hundred independent runs are equal to 28.66%, 31.93% and 27.03%, for Scenarios I, II and III given above, respectively.
Next, the proposed algorithm was applied to the above datasets. The whole process was implemented using CUDA on NVIDIA Tesla M2090 with 1 GB memory. Here, we assigned
as the most extreme case where no prior information was given. In different trials, the value of β was set to be 0.00, 0.25, 0.50, and 0.75 (see
Equation (2)). Since our algorithm performed both image classification and registration, the termination criteria had to ensure the convergences in both the estimated posterior probability and mapping parameters. As a result, we defined
to measure changes in the posterior probabilities from two consecutive iterations. We also define
to characterize the movement of coordinates of the remapped image Z
n from two consecutive iterations where
Here,
denotes the mapping parameter m
i from the nth at the tth iteration. In this example, the algorithm terminates when p
changes is less than p
min = 10
−5, and d
movement,n is less than 0.1 pixels for five consecutive iterations for
n = 2,3,4. To create a benchmark for our proposed algorithm, we investigated two extreme cases where LCMs were derived directly from the unregistered image pairs and from a perfect registered image pair. The LCMs from these extreme cases were classified using our proposed algorithm by fixing
Mt =
M*. For perfect registration, we had
M* =
MPerfect whereas, for unregistered image pairs, we set M
* equal to the values given in
Table 1 for the respective scenarios. The first extreme case could be considered as the lower limit on the classification accuracy if we performed the land cover mapping without alignment of images first. The second case was an upper bound on the classification accuracy when we produced a map from a registered image pair. By setting up our experiment in this fashion, we could investigate how much improvement our algorithm could gain by integrating the registration and classification together, and how far the performance of our algorithm was from the upper limit where all uncertainties in registration were removed. To ensure the statistical significance of our experiment, all experiments were repeated ten times.
Table 2 displays the averaged percentages of misclassified pixels (PMP) of the LCMs for different values of
β and for Scenario I with
δ = 12, Scenario II with
ρ = 0.05 and Scenario III with
η = 0.05 when
σ = 1. Note here that, in this example, we employed the percentages of mis-classified pixels as the performance metric to evaluate the classification performance rather than the overall accuracy to highlight small differences in the classification performance between LCMs derived from image datasets without registration error and LCMs obtained from our proposed algorithm. From
Table 2, it is clear that, from all scenarios, the PMPs derived from image datasets without registration errors corrections were always significantly poorer than those derived from registered image datasets. These results support our claims that it is important to consider a lack of alignments in performing image classification. We also observed that, for
β = 0.25, 0.5 and 0.75, our proposed algorithm produced the LCM with an accuracy similar to those obtained from the image dataset without any registration errors. These results imply that our proposed algorithm attained the upper-bound accuracy with proper selection of the MRF parameter.
To ensure the statistical significance, we computed the pairwise t-statistics for unequal variance populations [
29] of the PMPs obtained from LCMs derived from the proposed algorithm for various initial registration errors against those obtained from image dataset with no registration error, The resulting
p-values [
29] of the t-statistics are given in
Table 3. The
p-value represents the probability that there is no difference in PMPs. Hence, a smaller
p-value implies that PMPs from two experiments are different. We also computed the t-statistics comparing LCMs obtained from the image dataset with and without registration errors. The resulting
p-values of these t-statistics are also summarized in
Table 3. It is clear from
Table 3 that there are significant differences in term of PMPs from LCMs obtained from the image dataset with and without registration errors. Furthermore, the
p-values also support our claim that for
β = 0.25, 0.5 and 0.75, our proposed algorithm produced the LCM with an accuracy similar to those obtained from the image dataset without any registration errors. However at
β = 0, our proposed algorithm performed significantly poorer than those of perfect registration. In fact, at
β = 0, our proposed algorithm achieved roughly the same performance as in the situation where there is no registration error correction since, at
β = 0, our proposed algorithm could not correctly estimate the map vectors.
Figure 7 shows examples of the resulting LCMs at
β = 0.75 for all scenarios. We observed that all the LCMs appeared to be more connected than the MLC-based LCMs shown in
Figure 6.
Since at
β = 0.75, our proposed algorithm achieved the highest performance, we examined the effect of the initial registration errors to the performance of our algorithm by varying values of
δ,
ρ, and
η for Scenarios I, II and III, respectively, for
β = 0.75. Again, ten independent runs were performed to ensure the statistical significance and the results are given in
Table 4. We observed that, for all scenarios, the PMPs were roughly the same. In other words, the initial registration errors had little effect on the performance of our algorithm. These results imply the robustness of our proposed algorithm to the initial mis-registration errors if the proper value of
β is chosen.
Another key performance metric in this example is the residual registration errors after processing.
Table 5 displays the means and standard deviations of the root mean square errors (RMSEs) from ten independent runs between each of the simulated images and the reference LCM. The RMSE of the
n-th image is computed from
where (
,
) and (
,
) are the ground truth and estimated coordinates. Here, the ground truth coordinates obtained by letting
Mn =
Mperfect. Clearly, for
β = 0.25, 0.5, and 0.75, our algorithm can successfully register all images with the LCMs. However, at
β = 0, our algorithm could not align these images with the LCM. The results in
Table 5 emphasize the importance of parameter selection. Note here that the RMSE of Image 1 is not shown in
Table 5 since it is assumed to be perfectly aligned (registration error is zero) with the LCM.
Next, we examined the effects of image noise on the registration accuracy by varying the noise variance
σ2 from −30 dB to 0 dB, and the resulting averaged RMSEs for
β = 0.0 and 0.75 are given in
Tables 6 and
7, respectively. We observed here that there were slight performance differences in term of the RMSEs for
σ2 of −30, −20 and −10 dB for both
β = 0.00 and 0.75. However, for the noise variance equal to 0 dB, our algorithm could only correctly align Images 2–4 with the LCM at
β = 0.75. This result emphasizes the importance of the MRF model to the convergence of our algorithm.
For the performance comparison, we compared the registration accuracy of our proposed algorithm for various scenarios and
β = 0.75. with a traditional image-to-image registration technique. Here, we employed the mean square error criteria (MSEC) [
8,
34] since MSEC is suitable for registering images with the same modality and suffering from additive Gaussian noise. For the traditional image-to-image registration, we registered Images 2–4 with Image 1 since Image 1 was assumed to be aligned with the LCM. The averaged RMSEs from ten independent runs for various noise variances are shown in
Table 8. Again, the particle swamp optimization algorithm with eighty particles was employed to ensure global optimality. As expected, the registration accuracy decreased as the noise variance increased. By comparing
Tables 6 and
8, the RMSEs from our proposed algorithm seem to be lower (better) than those obtained from MSEC for noise variances equal to −20, −10 and 0 dBs, respectively.
Next, we again performed the pairwise
t-test to determine whether there were significant differences in RMSEs obtained from our proposed algorithm and MSEC. The resulting
p-values [
29] are shown in
Table 9. From the
p-values, we can conclude that our proposed algorithm achieves significantly better registration accuracies than those obtained from MSEC for the noise variances of −20, −10 and 0 dBs. Note here that, for a noise variance equal to −30 dB, the registration errors from our proposed algorithm and the MSEC were roughly zero and, therefore there was no difference in terms of registration accuracy. Another key performance metric is the processing time. For noise variance equal to 0 dB, the total processing time for image-to-image registration was 170 s whereas our approach with
β = 0.75 took 3,761, 6,063 and 4,620 s for Scenario I with
δ = 12, Scenario II with,
ρ= 0.05 and Scenario III with
η = 0.05, respectively. However, the total processing times for our approach with
β = 0.25 reduced 2,816, 1,039, and 989 s for Scenario I at
δ = 12, Scenario II at
ρ= 0.05 and Scenario III at
η = 0.05, respectively. Clearly, our algorithm demanded more computation than the image-to-image registration based on MSEC. However, the computation time can be significantly reduced if a small value of
β is chosen.
Figure 8 shows the averaged number of iterations that the algorithm required before the convergence criterion was satisfied for different scenarios and
β. For
β = 0.25, 0.5 and 0.75, more iterations were needed as the value of
β increased. However, at
β = 0, our algorithm terminated at the higher numbers of iterations than for
β = 0.25 for Scenarios II and III whereas, for Scenario I, our algorithm terminated at the lower number of iterations. For Scenario I at
β = 0, our algorithm quickly converged to the local optima since the resulting and initial registration errors shown in
Table 5 were almost identical. The identical resulting and initial registration errors also suggested that this local optimum point was very close to the initial point. However, for Scenarios II and III at
β = 0, the local optimum points might be further than in Scenario I, and the algorithm converged slowly due to the small changes in the mapping parameters from one iteration to another and since
β = 0, these small changes in the mapping parameters had a significant influence on the posterior probability.