Skip to Content
You are currently on the new version of our website. Access the old version .
Remote SensingRemote Sensing
  • Technical Note
  • Open Access

26 May 2022

13 Pages

An Optimal Transport Based Global Similarity Index for Remote Sensing Products Comparison

,
,
,
,
,
and
School of Transportation Science and Engineering, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing 100191, China
*
Author to whom correspondence should be addressed.

Abstract

Remote sensing products, such as land cover data products, are essential for a wide range of scientific studies and applications, and their quality evaluation and relative comparison have become a major issue that needs to be studied. Traditional methods, such as error matrices, are not effective in describing spatial distribution because they are based on a pixel-by-pixel comparison. In this paper, the relative quality comparison of two remote sensing products is turned into the difference measurement between the spatial distribution of pixels by proposing a max-sliced Wasserstein distance-based similarity index. According to optimal transport theory, the mathematical expression of the proposed similarity index is firstly clarified, and then its rationality is illustrated, and finally, experiments on three open land cover products (GLCFCS30, FROMGLC, CNLUCC) are conducted. Results show that based on this proposed similarity index-based relative quality comparison method, the spatial difference, including geometric shapes and spatial locations between two different remote sensing products in raster form, can be quantified. The method is particularly useful in cases where there exists misregistration between datasets, while pixel-based methods will lose their robustness.

1. Introduction

With the quick and great development of remote sensing technologies, together with the spread of open science, there are many available similar remote sensing products, and global land cover products are very typical examples, which are typically presented as digital thematic maps in raster. Thus, to facilitate common users’ easy-to-choose appropriate products, the need to compare their accuracy is growing. An ideal accuracy assessment is based on comparing a dataset with its true value; however, usually, it is impossible to obtain ‘ground truth’ in practice, thus accuracy assessments are usually conducted by comparing the dataset with some ‘reference data’. We know that remote sensing products from different sources are different in many aspects, including data source, classification scheme, methodology and resolution, etc. [1]. This is not surprising, given the fact that quantitative analyses of complex land cover types remain an arduous task [2].

3. Methodology

Wasserstein distance provides a way to measure the distance between two non-empty datasets, and a raster dataset can be taken as a set of multiple types of pixel coordinates, so this measurement applies.
Wasserstein-P distance between two pixel-based distributions q 1   and q 2   could be expressed as follows:
W P q 1 , q 2 = i n f γ x , y ∈ Γ q 1 , q 2 E x , y ~ γ x , y [ d x , y ) P 1 P
where x,y are the distribution of pixel coordinates in the two raster datasets, respectively,   Γ q 1 , q 2 is the set of all possible joint distributions on (x,y) with marginals q 1 and q 2 , d(x,y) is the distance metric between x and y , generally, the Euclidean distance is taken. In a two-dimensional case, P = 2.
Figure 2a illustrates the one-dimensional case, that is, using the minimum cost to convert one distribution into another distribution. Figure 2b is a two-dimensional case, using a transmission matrix P to describe the transmission plan.
Figure 2. Optimal transport problems: (a) one-dimensional case; (b) two-dimensional case.
In fact, it is difficult to calculate the Wasserstein distance in two or more dimensions directly through the optimal transport theory. Kantorovich–Rubinstein duality can be used to calculate its one-dimensional case:
W q 1 , q 2 = s u p ‖ f ‖ L ≦ 1 E x ~ q 1 f x − E x ~ q 2 f x
where the supremum is over all the 1-Lipschitz functions f   X   → R . The function f is commonly represented via a deep net and various ways have been suggested to enforce the Lipschitz constraint [25].
Then, a sliced version of the Wasserstein-2 distance, proposed by [26], shows its advantage, which only requires estimating distances of one-dimensional distributions and is more efficient. The “sliced Wasserstein-p distance” between distributions q 1 , q 2 is defined as:
W ˜ p q 1 , q 2 = [   ∫ ω ∈ Ω W P P q 1 ω , q 2 ω d ω ] 1 P
where q 1 ω , q 2 ω denote the projection (i.e., marginal) of q 1 , q 2 onto the direction ω , and   Ω is the set of all possible directions on the unit sphere.
The W ˜ p q 1 , q 2   distance has important practical implications: provided that the projected distributions q 1 ω , q 2 ω can be computed, then for any ω ∈ Ω , the distance W ˜ p q 1 , q 2 , as well as its optimal transport map and the corresponding Kantorovich potential can be analytically computed by using projected measures that are one-dimensional.
For two given datasets D = x   of samples x ∼ q 1 ,     F = y   of samples y ∼ q 2 :
W 2 2 D ω , F ω = 1 D ∑ i = 1 D ‖ D φ D i ω − F φ F i ω ‖ 2 2
where φ D   and φ F are permutations that sort the projected sample sets D ω and F ω , respectively.
D φ D 1 ω ≤ D φ D 2 ω ≤ ⋯ ≤ D φ D D ω
F φ F 1 ω ≤ F φ F 2 ω ≤ ⋯ ≤ F φ F F ω
When the number of elements in the two datasets D , F are different, find the greatest common divisor θ of D and F , then make D * = F θ , F * = D θ , and replace D i with D * elements, together with replacing F i with F * elements. Because D * D = F * F , it is sure that the element numbers in the two new sets are the same.
If we could find the most meaningful projection direction, the 2D max-sliced-Wasserstein distance will be calculated, and it is defined as follows:
m a x − W ˜ 2 q 1 , q 2 = m a x ω ∈ Ω W 2 2 q 1 ω , q 2 ω 1 2
This metric satisfies the properties of non-negativity, the identity of indiscernible, symmetry, and subadditivity [27]. Hence, it is a true metric.
Taking two land cover datasets, A and B, with K categories and the size of Row × Col as an example, after each   m a x − W ˜ 2 is calculated by categories, the similarity between A and B can be defined:
S i m i l a r i t y A , B = ∑ i = 1 K [ 1 − 1 Row 2 + Col 2 2 × m i n ( Row 2 + Col 2 2 , m a x − W 2 ˜ i 1 − N i Row × Col ) ] × N i Row × Col
where m a x − W 2 ˜ i is the max-sliced Wasserstein distance of category   i , and N(i) is the average number of pixels of category i in datasets A and B, that is, N i = 1 2 N A i + N B i .
It could also be deduced that the similarity between two datasets is order independent, so we have:
S i m i l a r i t y A , B = S i m i l a r i t y B , A
This method has the following advantages: (1) It can quantify the difference in both spatial position and shape, then measure it under a unified standard; (2) it can give a continuous transformation process while preserving geometric features; (3) it has symmetry and can still give reasonable measurement results in the case of regional misregistration.

4. Experiment

To demonstrate the rationality of the above-proposed similarity index, we designed several validation experiments both on test datasets and real datasets. For test datasets, we set four cases, and use a gradient descent method to give the continuous transformation process. For real datasets, we calculate the similarity index among three open land cover products.

4.1. Experiments on Test Datasets

(i)
Max-sliced Wasserstein distance between two points
Suppose there are two points on the two-dimensional plane, the coordinates of the two points are (9.0, 9.0) and (13.0, 12.0), respectively.
The results of the max-sliced Wasserstein distance with the number of projections and the percentage difference with the Euclidean distance are shown in Table 2.
Table 2. Distance with the number of projections.
The Euclidean distance between the two points is 5.0. Because the sliced distance is calculated by using projection, there is a deviation from the Euclidean distance. We consider that when the number of projections is not less than 20, the deviation is within the allowable range.
(ii)
Max-sliced Wasserstein distance between areas
The geometric shape and spatial position difference between point sets (usually represented as polygons in a raster dataset) cannot be simply measured by the Euclidean distance between points. Here, we design four cases to illustrate the rationality of the proposed index:
Case 1: Shapes of the polygons in two datasets to be compared are the same, and the spatial position has a translation transformation;
Case 2: The centroid of polygons is the same, but the shapes are different;
Case 3: The polygon shapes and spatial positions are both different;
Case 4: The distribution is not continuous and there are multiple areas.
Then, we used the gradient descent method to construct the continuous transformation process.
The initial states of the four examples are shown in Table 3 and Figure 3.
Table 3. Initial state of four cases.
Figure 3. Initial state of four cases.
The projection number is set to 20, and then the max-sliced Wasserstein distances of each case are shown in Table 4.
Table 4. Max-sliced Wasserstein distance of four cases.
Distance changes during the gradient descent process are shown in Figure 4. As the number of iterations increases, the distance tends to decrease, which means that distribution is shifted to the target distribution.
Figure 4. Distance with steps.
The process of continuous transformation of four cases in the gradient descent process is shown in Figure 5.
Figure 5. Transformation process of source distribution to target.
In Case 1, the shapes of the two polygons are the same, and the max-sliced Wasserstein distance measures their difference in spatial position. When it goes to the 99th step, the initial distribution is transferred to the target distribution (the green circle moves to the same position as the orange circle).
In Case 2, the positions of the two centroids are the same, so the factor that affects the max-sliced Wasserstein distance measurement is only their geometric shapes.
In Case 3, both their geometric shapes and spatial positions affect the final measurement, so compared with that in Case 1 and Case 2, the distance is the largest.
In Case 4, the distribution of the points is discontinuous, and there are multiple parts (blue points). This simulates a more complicated situation in raster datasets. This metric gives a reasonable result and a continuous transformation process of multiple regions.

4.2. Experiments on Real Remote Sensing Products

We choose open-source global land-use raster datasets to verify the proposed index, and the three selected datasets are GLC_FCS30 [28], FROM_GLC [29], CNLUCC [30] in 2015.
The selected area is a rectangle with the coordinates range of (116.18575, 40.00125) and (116.37300, 40.18375) in the coordinate system of WGS84. The three datasets are the land-use data of this area at the same time (2015), created by different researchers, with the same resolution of 30 m.

4.2.1. Dataset Preprocessing

The classification systems of the three datasets are shown in Figure 6:
Figure 6. Detail of GLC_FCS30, FROM_GLC and CNLUCC datasets.
Comparisons cannot be made under the original non-uniform classification system, so we reclassified the pixels and classified them into four types:(1) Cropland; (2) Forest/Grass; (3) Water body; (4) Impervious surfaces. Figure 7 illustrates the classification system and reclassification results.
Figure 7. Classification System and results of reclassification.
Table 5 shows the pixel numbers of four classification types in each dataset.
Table 5. The number of pixels of four types.

4.2.2. Similarity Calculation

Taking water bodies as an example, distributions of water bodies on the three datasets are shown in Figure 8.
Figure 8. Water body distribution. (a) GLC_FCS30; (b) FROM_GLC; (c) CNLUCC.
Compare the three datasets in pairs separately to calculate the max-sliced Wasserstein distance of the water distribution (Figure 9 and Table 6):
Figure 9. Overlay of raster data map frames (a-b. GLC_FCS30-FROM_GLC; a-c. GLC_FCS30-CNLUCC; b-c. FROM_GLC-CNLUCC).
Table 6. Results of water body.
The results show that the comparison between GLC_FCS30 and FROM_GLC products (a-b) has the smallest max-sliced Wasserstein distance and thus, the highest similarity, which is consistent with the visual judgment.
The multi-category comparison results among the three chosen products are listed in Table 7:
Table 7. Results of three datasets.

4.2.3. Comparison in Unregistered Case

We designed an experiment to illustrate a significant advantage of the proposed similarity index: it pays more attention to the shape and structure of spatial distribution than traditional methods, therefore, it can better represent the spatial features of data. We simulated a misregistered case of a water body in CNLUCC data. Cases 1–3 offset 1 to 20 pixels on the x-axis, y-axis, x- and y-axis, respectively (Figure 10), and then we calculated according to the similarity index, kappa coefficient and intersection over union (IoU). The results are shown in Table 8.
Figure 10. Unregistered cases (Case 1: x-axis; Case 2: y-axis; Case 3: both).
Table 8. Similarity, Kappa and IoU results of three unregistered cases.
It could be seen that traditional methods are very sensitive to registration accuracy (shown in Figure 11).
Figure 11. Line charts of three indicators changes with pixel offset.
Both the kappa coefficient and IoU are not robust in three cases, because these methods do not consider spatial features of pixels; therefore, their results are quite different in cases of different pixel offsets. The similarity index proposed in this paper focuses on the structure and characteristics of space, thus, the max-sliced Wasserstein distance is close to the true pixel offset and the change of similarity result is small.

5. Discussion

In the above experiments, test datasets are used to illustrate the rationality of Wasserstein distance for quantitatively calculating the distribution of pixels in two-dimensional space. This index can comprehensively consider spatial location information and geometric shape information, which has practical significance.
After unifying the spatial resolution and classification standards of three chosen land cover products, single-category similarity and overall similarity are both calculated. In the calculation of water body similarity, the distribution of large-area water objects in products GLC_FCS30 and FROM_GLC is relatively consistent, and the difference in the surrounding scattered and small water bodies is the main reason for the generation of the distance measurement. The overall similarity results indicate that the GLC_FCS30 and CNLUCC data have a higher similarity. This is because, in FROM_GLC, the confusion between Cropland and Forest/Grass types appears more frequently, resulting in a large gap in the number of pixels between these two types. Therefore, after calculating the overall similarity with the number of pixels and similarity in the similarity evaluation index, the overall similarity is quite different from the other two products. The most similar type is Impervious surfaces, and the pixels of this type account for a large proportion of the total number of pixels, therefore, it is the main factor that affects the final similarity.
Compared with the error matrix and IoU, the similarity index represents more characteristics of spatial structure. We hope to propose this index to make up for the shortcomings of existing methods. It can better characterize the spatial relationship between datasets and solve the comparison problem under a certain degree of misregistration.

6. Conclusions

With the increasing demand for quantitative evaluation of different remote sensing products, the traditional pixel-based accuracy evaluation system needs to be improved. The proposed similarity index is a promising way to facilitate this need. It is not limited to pixel-by-pixel comparison in the case of complete alignment. In a word, the main idea of this study is to transform the relative accuracy comparison problem of multi-source raster datasets into an optimal transmission problem in two-dimensional space. For the mathematical expression of similarity index, based on max-sliced Wasserstein distance, when the number of projections reaches the threshold, it can be regarded as the true value. Using the gradient descent method to give the continuous transformation process also shows that the index can reasonably quantify the spatial distribution. Based on this similarity index, the spatial difference between multi-source raster datasets can be quantified. Theoretically, this index is more suitable for land-use change monitoring and continuous raster datasets comparison. More theoretical development and practical application of this index are still under work, and we will continue to improve it.

Author Contributions

Conceptualization, Y.T., Y.S. and G.J.; methodology Y.T. and Y.S.; software, Y.S.; validation, Y.T., Y.S. and L.X.; formal analysis, Y.T., Y.S. and X.W.; investigation, Y.T. and Y.S.; resources, Y.T. and Y.S.; data curation, K.Z.; writing—original draft preparation, Y.S. and L.X.; writing—review and editing, Y.S. and L.X.; visualization, Y.T., Y.S. and B.B.; supervision, Y.T.; project administration, Y.T.; funding acquisition, Y.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Key R&D Program, grant number No. 2019YFE0126400.

Data Availability Statement

This statement if the study did not report any data.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Hansen, M.C.; Reed, B. A comparison of the IGBP DISCover and University of Maryland 1 km global land cover products. Int. J. Remote Sens. 2000, 21, 1365–1373. [Google Scholar] [CrossRef] [Scilit]
  2. Zhu, Z.; Waller, E. Global forest cover mapping for the United Nations Food and Agriculture Organization forest resources assessment 2000 program. For. Sci. 2003, 49, 369–380. [Google Scholar]
  3. Stehman, S.V. Sampling designs for accuracy assessment of land cover. Int. J. Remote Sens. 2009, 30, 5243–5272. [Google Scholar] [CrossRef] [Scilit]
  4. Visser, H.; Nijs, T.D. The Map Comparison Kit. Environ. Model. Softw. 2006, 21, 346–358. [Google Scholar] [CrossRef] [Scilit]
  5. Ma, Z.; Redmond, R.L. Tau coefficients for accuracy assessment of classification of remote sensing data. Photogramm. Eng. Remote Sens. 1995, 61, 435–439. [Google Scholar]
  6. Pontius, R., Jr.; Huffaker, D.; Denman, K. Useful techniques of validation for spatially explicit land-change models. Ecol. Model. 2004, 179, 445–461. [Google Scholar] [CrossRef] [Scilit]
  7. Rees, W.G. Comparing the spatial content of thematic maps. Int. J. Remote Sens. 2008, 29, 3833–3844. [Google Scholar] [CrossRef] [Scilit]
  8. Wu, B.; Zhang, L.; Zhao, Y. Feature selection via Cramer’s V-test discretization for remote-sensing image classification. IEEE Trans. Geosci. Remote Sens. 2013, 52, 2593–2606. [Google Scholar] [CrossRef] [Scilit]
  9. Ohana-Levi, N.; Gao, F.; Knipper, K.; Kustas, W.P.; Anderson, M.C.; del Mar Alsina, M.; Sanchez, L.A.; Karnieli, A. Time-series clustering of remote sensing retrievals for defining management zones in a vineyard. Irrig. Sci. 2021, 1–15. [Google Scholar] [CrossRef] [Scilit]
  10. Hagen-Zanker, A. Map comparison methods that simultaneously address overlap and structure. J. Geogr. Syst. 2006, 8, 165–185. [Google Scholar] [CrossRef] [Scilit]
  11. San-Miguel-Ayanz, G.H. Conventional and fuzzy comparisons of large scale land cover products: Application to CORINE, GLC2000, MODIS and GlobCover in Europe. ISPRS J. Photogramm. Remote Sens. 2012, 74, 185–201. [Google Scholar]
  12. Dou, W.; Ren, Y.; Wu, Q.; Ruan, S.; Chen, Y.; Bloyet, D.; Constans, J.M. Fuzzy kappa for the agreement measure of fuzzy classifications. Neurocomputing 2007, 70, 726–734. [Google Scholar] [CrossRef] [Scilit]
  13. Hargrove, W.W.; Hoffman, F.M.; Hessburg, P.F. Mapcurves: A quantitative method for comparing categorical maps. J. Geogr. Syst. 2006, 8, 187. [Google Scholar] [CrossRef] [Scilit]
  14. White, R. Pattern based map comparisons. J. Geogr. Syst. 2006, 8, 145–164. [Google Scholar] [CrossRef] [Scilit]
  15. Zhu, D.; Chen, T.; Wang, Z.; Niu, R. Detecting ecological spatial-temporal changes by remote sensing ecological index with local adaptability. J. Environ. Manag. 2021, 299, 113655. [Google Scholar] [CrossRef] [Scilit]
  16. Giri, C.; Zhu, Z.; Reed, B. A comparative analysis of the Global Land Cover 2000 and MODIS land cover data sets. Remote Sens. Environ. 2005, 94, 123–132. [Google Scholar] [CrossRef] [Scilit]
  17. Strahler, A.H.; Boschetti, L.; Foody, G.M.; Friedl, M.A.; Hansen, M.C.; Herold, M.; Mayaux, P.; Morisette, J.T.; Stehman, S.V.; Woodcock, C.E. Global Land Cover Validation: Recommendations for Evaluation and Accuracy Assessment of Global Land Cover Maps. Eur. Communities Luxemb. 2006, 51, 1–60. [Google Scholar]
  18. Foody, G.M. Assessing the accuracy of land cover change with imperfect ground reference data. Remote Sens. Environ. 2010, 114, 2271–2285. [Google Scholar] [CrossRef] [Scilit]
  19. Foody, G.M. Explaining the unsuitability of the kappa coefficient in the assessment and comparison of the accuracy of thematic maps obtained by image classification. Remote Sens. Environ. 2020, 239, 111630. [Google Scholar] [CrossRef] [Scilit]
  20. Enøe, C.; Georgiadis, M.P.; Johnson, W.O. Estimation of sensitivity and specificity of diagnostic tests and disease prevalence when the true disease state is unknown. Prev. Vet. Med. 2000, 45, 61–81. [Google Scholar] [CrossRef] [Scilit]
  21. Wu, X.; Naegeli, K.; Wunderle, S. Geometric accuracy assessment of coarse-resolution satellite datasets: A study based on AVHRR GAC data at the sub-pixel level. Earth Syst. Sci. Data 2020, 12, 539–553. [Google Scholar] [CrossRef] [Scilit]
  22. Wang, H.; Skau, E.; Krim, H.; Cervone, G. Fusing heterogeneous data: A case for remote sensing and social media. IEEE Trans. Geosci. Remote Sens. 2018, 56, 6956–6968. [Google Scholar] [CrossRef] [Scilit]
  23. Tardy, B.; Inglada, J.; Michel, J. Assessment of optimal transport for operational land-cover mapping using high-resolution satellite images time series without reference data of the mapping period. Remote Sens. 2019, 11, 1047. [Google Scholar] [CrossRef] [Scilit]
  24. Deshpande, I.; Hu, Y.T.; Sun, R.; Pyrros, A.; Schwing, A. Max-Sliced Wasserstein Distance and its use for GANs. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop, Seoul, Korea, 27–28 October 2019; pp. 10648–10656. [Google Scholar]
  25. Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; Courville, A. Improved training of wasserstein gans. arXiv Prepr. 2017, arXiv:1704.00028. [Google Scholar]
  26. Kolouri, S.; Rohde, G.K.; Hoffmann, H. Sliced wasserstein distance for learning gaussian mixture models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3427–3436. [Google Scholar]
  27. Kolouri, S.; Park, S.R.; Rohde, G.K. The radon cumulative distribution transform and its application to image classification. IEEE Trans. Image Processing 2015, 25, 920–934. [Google Scholar] [CrossRef] [Scilit]
  28. Zhang, X.; Liu, L.; Chen, X.; Gao, Y.; Xie, S.; Mi, J. GLC_FCS30: Global land-cover product with fine classification system at 30 m using time-series Landsat imagery. Earth Syst. Sci. Data 2021, 13, 2753–2776. [Google Scholar] [CrossRef] [Scilit]
  29. Gong, P.; Wang, J.; Yu, L.; Zhao, Y.; Chen, J. Finer resolution observation and monitoring of global land cover: First mapping results with Landsat TM and ETM+ data. Int. J. Remote Sens. 2013, 34, 48. [Google Scholar] [CrossRef] [Scilit]
  30. Xu, X.; Liu, J.; Zhang, S.; Li, R.; Yan, C.; Wu, S. China’s Multi-Period Land Use Land Cover Remote Sensing Monitoring Data Set (CNLUCC); Resource and Environment Data Cloud Platform: Beijing, China, 2018. [Google Scholar]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.