Consistency Analysis and Accuracy Assessment of Three Global Ten-Meter Land Cover Products in Rocky Desertification Region—A Case Study of Southwest China

Abstract

Rocky desertification is one of the most critical ecological and environmental problems in areas underlain by carbonate rocks globally. Land cover and land use in the region affects large-scale ecosystem processes on a global scale, and many Earth system models rely on accurate land cover information. Therefore, it is important to evaluate current global land cover products and to understand the differences between them, and the findings of these studies can provide guidance to different researchers when using or making land cover products. Whereas there are many studies on the assessment of coarser resolution land cover products, there are few studies on the assessment of higher resolution land cover products (10 m). In order to provide guidance for users of 10 m data, this paper uses the rock deserted southwest region of China as the experimental area. We analyzed the consistency and accuracy of the FROM-GLC, ESA WorldCover 10 and ESRI products using spatial pattern consistency, absolute accuracy assessment of three validation samples, and analyzed their intrinsic relationships among classification systems, classification methods, and validation samples. The results show that (1) the overall accuracy of the FROM-GLC product is the highest, ranging from 49.47 to 62.42%; followed by the overall accuracy of the ESA product, ranging from 45.13 to 64.50%; and the overall accuracy of the ESRI product is the lowest, between 39.03 and 61.94%. (2) The consistency between FROM-GLC and ESA is higher than the consistency between other products, with an area correlation coefficient of 0.94. Analysis of the spatial consistency of the three products shows that the proportion of perfectly consistent areas is low at 44.89%, mainly in areas with low surface heterogeneity and more homogeneous cover types. (3) Across the study area, the main land cover types such as forest and water bodies were the most consistent across the three product species, while the grassland, shrubland, and bareland were lower. All products showed high accuracy in homogeneous areas, with local accuracy varied in other areas, especially at high altitudes in the central and western regions. Therefore, land cover users cannot use these products directly when conducting relevant studies in rocky desertification areas, as their use may introduce serious errors.

1. Introduction

Land cover products are the fundamental geospatial data products needed by analysts and decision makers in governments, society, industry, and the financial sector to monitor global environmental change and measure risks to sustainable livelihoods and development [1,2,3,4,5]. Currently, large regional land cover products are being developed by major agencies around the world, including the USGS IGBP DISCOVER product [6], 1km UMD land cover products at the University of Maryland, USA [7], GLC2000 products from the EU Joint Research Centre [8], MODIS land cover products from Boston University [9], GLOBCOVER products produced by the European Space Agency [10], China National Basic Geographic Information Centre global 30 m land cover product in 2014 [11], Copernicus Global Land Services Annual 100 m Global Land Cover Product [12], and the NLCD land cover products produced by the USGS [13]. While these coarser resolution land cover products have provided valuable information for many studies [14,15,16], some studies have shown that they have low classification accuracy in transition zones with heterogeneous landscapes, where finer resolution land cover products are needed [17,18].

In recent decades, with the development of satellite remote sensing technology, freely available high-resolution remote sensing imagery has facilitated the development and publication of land cover products [19,20]. In particular, with the first launch of the Sentinel satellite in 2014, fine-grained measurements of land cover products are possible relay on its high spatial, spectral, and temporal resolution [1]. Recently, a number of land cover products with a resolution of up to 10 m have been produced. These include the global-scale FROM-GLC land cover product produced by Tsinghua University, which is based on a random forest approach using training samples for classification [21]. The other two are the ESA WorldCover 2020 landcover products led by the European Space Agency, and the global ESRI 2020 Landcover product produced by the Environmental Systems Research Institute. Both products are global-scale land cover products produced using deep learning methods. These high-resolution land cover products provide potential users with a wealth of spatial detail.

However, they are extracted from multi-source remote sensing images using different classification systems and methods. Well-known accuracy values [22], such as kappa or overall accuracy, give the concept of the overall accuracy of land cover products, but do not convey information about spatial differences in map quality [23,24,25]. Spatial accuracy consistency assessments inform the user of the level of uncertainty in the mapping of land cover across the space, and from the a user’s perspective. These spatially clear accuracy values help to compare different land cover products in order to select the best product for the area of interest [26]. Therefore, quantitative and independent consistency analysis and accuracy assessment is essential for users to select the best product for their specific application [27,28].

At present, some scholars have conducted comparative evaluation and analysis of different land cover products currently published [29,30,31,32]. For example, Gao et al. [27] used the LUCAS dataset for consistency analysis and accuracy assessment of three global 30 m land cover products in the EU, and the results showed that inconsistencies between the three products occurred mainly in heterogeneous areas. Kang et al. [33] conducted a consistency study of 30 m multi-source land cover products through area consistency, spatial consistency, and accuracy evaluation using Indonesia as an example, and the results showed low accuracy for grassland, shrub, bare ground, and wetland types. An evaluation by Liang et al. [34] of the accuracy of four global land cover products in the Arctic region showed that the classification accuracy of shrub types was low. Herold et al. [35] compared four global-scale land cover products with a resolution of 1 km and the experimental results showed a high degree of accuracy and consistency for evergreen broadleaf forest, bare ground, and snow and ice cover types. Tchuente et al. [18] studied four land cover products at the Africa continent scale, aiming to highlight the consistency and difference between the four land cover product classification systems, and the results showed that the consistency of the four land cover products was between 56 and 69%.

However, existing evaluation and analysis studies of different products are mainly focused on medium- and low-resolution products (30–1000 m). A literature search showed that there are few consistent analyses of land cover products with 10 m resolution in current global comparative land cover product studies. Located in eastern Asia and on the west coast of the Pacific Ocean, China has about 3.44 million km² of karst areas, about 36% of its total land area and 15.6% of all the 22 million km² of karst areas in the world [36]. The destruction of ecologically fragile carbonate areas as a result of extensive human activity has led to rock desertification disasters [37]. Among them, the ecological disaster of rocky desertification is particularly serious in southwest China, which has seriously hampered the economic growth of the region and directly affected the livelihood of 1.7 million people in the region [38]. Rocky desertification is used to characterize the processes that transform a karst area covered by vegetation and soil into a rocky landscape almost devoid of soil and vegetation [39]. It has occurred largely in the European Mediterranean basin [40], the Dinaric Karst [41], and in southwest China [39] due to extensive human activities on ecologically fragile carbonate rock formations. Cao et al. [42] showed that the rocky desertification area expanded drastically by 3.76 times from 1970 to 2005 in Guizhou province. There have been numerous studies showing that rocky desertification has a dramatic impact on hydrological, soil, and ecological conditions at different scales, resulting in more geological hazards such as droughts, floods, landslides, and ground subsidence [43,44]. For example, Jiang et al. [45] studied the effects of land use change on groundwater quality in a typical karst watershed in the southwest, concluding that the conversion of forested and unused land to cropland resulted in diffuse groundwater contamination from fertilizer application and building development on newly cultivated land. Liu et al. [46] studied the environmental changes caused by land use changes in the southwest karst region and concluded that changes in forests and grasslands are the main causes of ecological changes and that further deterioration is likely to continue in the coming decades. On a larger scale, it even affects the carbon balance and regional climate conditions [47]. For example, Kalnay et al. [48] used the difference between observed surface temperature trends and the corresponding trends in surface temperatures determined by re-analysis of global weather over the past 50 years to estimate the impact of land use change on surface warming. The results showed that half of the observed daily temperature decrease was due to urban and other land use changes. Song et al. [49] develop datasets for improved modelling of land use change, biogeochemical cycling and vegetation-climate interactions to contribute to our understanding of global environmental change. Further, various environmental issues such as greenhouse gas emissions [50], heat island effect [51], habitat loss [52], and ecosystem degradation [53] due to land cover and land use change have become a global research and concern. Therefore, the spatial structure and change of land cover in the region are of great significance for global ecological change research as well as for sustainable regional economic and social development.

In order to provide guidance to users and producers of high-resolution land cover products, and to provide a reference for ecosystem conservationists studying the impacts of global rocky desertification hazards when selecting base data for land cover products. This paper takes the rocky desertification region of southwest China as the experimental area, using the composition similarity assessment, consistency of spatial pattern distribution and absolute accuracy assessment methods, and the consistency and accuracy of high-resolution land cover products FROM-GLC, ESA WorldCover 2020, and ESRI 2020 Landcover are analyzed in depth. Additionally, the factors that influence the results of its classification are explored. The results of the study can provide guidance for future improvements in the quality of land cover mapping and for different researchers to select the best land cover products for their applications. Further, the results of the study could provide the necessary information for the ecological management of the region, as well as advancing research and engineering practices to combat rocky desertification and assist in sustainable development.

2. Materials and Methods

2.1. Study Area

China is located in east Asia, on the west coast of the Pacific Ocean, with a relatively complex topography, with the terrain being high in the west and low in the east. The study area belongs to the southwest of the seven major geographical divisions of China (Figure 1), located between longitudes 97°21′–110°11′ E and latitudes 21°08′–33°41′ N, with a total area of 2,340,600 square kilometers. Geographically, it includes the southeastern part of the Qinghai-Xizang Plateau, the Sichuan Basin, and most of the Yunnan-Guizhou Plateau, which includes a total of five provinces: Chongqing, Sichuan, Guizhou, Yunnan, and Xizang. It is bordered by Bhutan, Pakistan, Nepal, India, Laos, and Myanmar. The southwest region is located in the subtropics and the terrain is predominantly mountainous, so the subtropical mountain climate with lots of rain and clouds, high humidity and low sunlight is notable. The average annual temperature in the southwest reaches 24 degrees Celsius in the east, while the average annual temperature in the west can be as low as below 0 degrees Celsius [54]. The overall distribution of annual precipitation is “more in the east and less in the west”. Southwest China has about 0.51 million km² of exposed carbonate rock areas, accounting for 5.8% of the total land area, and up to 82% of the rock desertification areas are located in Yunnan, Guizhou, Chongqing, and Sichuan [36].

2.2. Data and Preprocessing

In this paper, three 10 m resolution land cover products were selected from a wide range of current global and regional land cover products for consistency evaluation analysis. The three selected data are the global-scale FROM-GLC land cover data produced by Tsinghua University [21] (http://data.ess.tsinghua.edu.cn/, accessed on 2 October 2021), European Space Agency-led production of ESA WorldCover 2020 land cover data with global coverage (https://doi.org/10.5281/zenodo.5571936, accessed on 8 January 2022) and the global ESRI 2020 Landcover data produced by the Environmental Systems Research Institute [1] (https://www.arcgis.com/index.html, accessed on 12 January 2022). The data selected are all current cover products of relatively high resolution, and although the year of production for the FROM-GLC data differs from the other two by 3 years. as natural ecosystems typically do not change significantly for 10 years or more [55]. The ecological condition of the study area was relatively stable during this period, as verified by Google’s high-resolution image data. Therefore, the data selected can better support the research of this paper. Information on the main parameters of the three data types is shown in

Table 1. Main parameters of different products.

Name	Resolution (m)	Number of Categories	Time	Method	Overall Accuracy (%)	Production Institution	Satellite	Scale
FROM-GLC	10	10	2017	Random forest	72.76	Tsinghua University	Sentinel-2	Global
ESA WorldCover 2020	10	11	2020	Deep learning model	74.40	European Space Agency	Sentinel-1/2	Global
ESRI 2020 Landcover	10	10	2020	Deep learning model	86	Impact Observatory for Esri	Sentinel-2	Global

.

The first task before data consistency analysis is data pre-processing, which mainly includes data cropping, projection transformation, and harmonization of classification systems between different products. Using ArcGIS (v. 10.3), land cover datasets of the study area were cropped with vector boundaries. In order to carry out the area comparison analysis, it was secondly necessary to unify the WGS-84 coordinate system into a UTM projection. In addition, various land cover classification systems have been proposed by different scholars in combination with the ability of remote sensing to acquire the characteristics of surface features. These classification systems are well suited for different application needs, but need to be grouped under the same classification benchmark when performing data consistency analysis between multiple products. If there are no common category correspondence rules, direct comparison analysis will produce a series of problems, such as errors. The classification system of the original classification is reflected in

Table 2. Original classification systems and codes for different land cover products.

Code	FROM_GLC	Code	ESA	Code	ESRI
10	Cropland	10	Tree cover	1	Water
20	Forest	20	Shrubland	2	Trees
30	Grassland	30	Grassland	3	Grass
40	Shrubland	40	Cropland	4	Flooded vegetation
50	Wetland	50	Built-up	5	Crops
60	Water	60	Bare/sparse vegetation	6	Scrub/shrub
70	Tundra	70	Snow and Ice	7	Built Area
80	Impervious surface	80	Permanent water bodies	8	Bare ground
90	Bareland	90	Herbaceous wetland	9	Snow/Ice
100	Snow/Ice	95	Mangroves	10	Clouds
		100	Moss and Lichen

. In recent years, the international community has done a great deal of work on the normalization of multi-source data classification systems, ultimately concluding that the Land Cover Classification System (LCCS) classification system can be used as a reference and conversion standard for future land cover classifications. The system is strictly a classifier that provides a common conversion language that enables conversion between existing classification systems [56]. Therefore, the paper refers to the LCCS classification description language to aggregate the land cover categories of the three data species into nine broad categories (

Table 3. Merged classification system and its correspondence with the original classification system.

Type	FROM_GLC	ESA	ESRI
10 Cropland	10	40	5
20 Forest	20	10	2
30 Grassland	30	30	3
40 Shrubland	40, 70	20, 100	6
50 Wetland	50	90, 95	4
60 Water	60	80	1
80 Built up	80	50	7
90 Bareland	90	60	8
100 Snow/Ice	100	70	9

), and the LCCS category criteria and common classification thresholds are strictly used in the aggregation process. In addition, the number of images of clouds in ESRI products is a negligibly small proportion of the total number of images in the study area. Figure 2 shows the spatial pattern distribution of cover types for the three data products after pre-processing. For ease of presentation, the ESA WorldCover 2020 and ESRI 2020 Landcover product names are hereafter referred to as ESA and ESRI.

2.3. Composition Similarity Assessment

For each land cover product, the area of each land type is aggregated and Pearson correlations are calculated for the corresponding land type area series between the different datasets, thus assessing the similarity of the land composition of each type between the products [33]. The calculation formula is as follows:

$$R_i=\frac{{\sum }_{k=1}^9\left(X_k-\overline{X}\right)\left(Y_k-\overline{Y}\right)}{\sqrt{{\sum }_{k=1}^9{(X_k-\overline{X})}^2{(Y_k-\overline{Y})}^2}}$$

(1)

where $R_i$ is the correlation coefficient of area; $i$ denotes the i-th land cover product mix; $k$ denotes the land cover type; $X_k$ denotes the area of type $K$ in dataset $X$ (km²); $Y_k$ denotes the area of type K in dataset Y (km²); X bar denotes the average of the area of all nine types of land in dataset X (km²); Y bar denotes the average of the area of all nine types of land in dataset Y (km²).

2.4. Evaluation of the Distribution of Spatial Patterns

In order to visually represent the spatially consistent distribution characteristics of the three land cover data products, this paper overlays the three data types spatially based on ArcGIS (v. 10.3) software. First, using the spatial resolution size of 10 × 10 m pixels as the smallest unit. Then, the three land cover products are calculated pixel by pixel in a raster calculator to obtain the spatial correspondence between the different land types of cover data pixels. Finally, the number of different land cover data matching cover types is determined on a pixel-by-pixel basis, and the degree of consistency is divided into three levels, from highest to lowest, as follows: (1) high consistency, where the three data show exactly the same land cover classes in the corresponding image element; (2) moderate consistency, where the three data show only two land cover classes in the corresponding image element; and (3) low consistency, where the three data show completely different land cover classes in the corresponding image element [33]. A schematic diagram of the spatial overlay for the cropland type is shown in Figure 3.

2.5. Sample Accuracy Evaluation

The error matrix is one of the most common methods used to evaluate the accuracy of land cover products [57,58]. The method is based on comparing the type consistency between the reference data and the data to be verified at a specific location, and then establishing an error matrix between the two, from which the producer accuracy (PA), user accuracy (UA), overall accuracy (OA), and Kappa coefficients are calculated to express the accuracy of the product to be verified. The formulae for calculating each indicator are as follows [59]:

$$\mathrm{OA}=\frac{{\sum }_{i=1}^r{x}_{ii}}{n^2}\times 100\%$$

(2)

$$\mathrm{PA}=\frac{x_{ii}}{x_{+i}}\times 100\%$$

(3)

$$\mathrm{UA}=\frac{x_{ii}}{x_{i+}}\times 100\%$$

(4)

$$\mathrm{Kappa}=\frac{N·{\sum }_{i=1}^r{x}_{ii}-{\sum }_{i=1}^r\left(x_{i+}·x_{+i}\right)}{N^2-{\sum }_{i=1}^r\left(x_{i+}·x_{+i}\right)}$$

(5)

where $x_{ii}$ is the correctly classified pixel number of type i; n is the total pixel number in the study area; $x_{i+}$ is the total pixel number of type i in the data to be verified; $x_{+i}$ is the total pixel number of type i in the reference data; r represents the number of rows in the confusion matrix; N is the total number of sample points.

For global land cover products, validation sample data is difficult to obtain through fieldwork, which requires a strong workforce and a sufficient amount of time. However, an accurate and sufficient validation sample is important to evaluate the results. A common global validation dataset, and in particular an adequate, well-described, compatible, and real-time updated dataset, would therefore greatly facilitate better accuracy assessment and comparison (Figure 4). In order to compare the accuracy of the three land cover data, three independent validation sample sets were used: (1) The Geo-Wiki Global Validation Sample Set (2011–2012) are obtained through the Geo-Wiki (http://geo-wiki.org/, accessed on 11 September 2021) crowdsourcing platform from four separate campaigns, which includes 10 surface cover types [60]. The Geo-Wiki global validation sample was processed to obtain 3113 validation samples covering the study area (Figure 4a). (2) The Global Land Cover Validation Sample Set (GLCVSS, 2009–2011) global validation sample data, which followed a random sampling strategy to ensure that all samples were evenly distributed across the globe [61]. It is based on interpreting Landsat TM and ETM+ images and the MODIS enhanced vegetation index (EVI) time series data and other high-resolution imagery on Google Earth was supplemented. The GLCVSS global sample was processed to obtain 488 validation samples covering the study area (Figure 4b). (3) Validation samples based on Google Earth imagery, Sentinel imagery, and visually collected samples from third-party sample sets. Google Earth is one of the main data sources for accuracy evaluation due to its accurate positioning, rich temporal phase, high resolution, easy access, and wide overlay [57,61,62]. In order to reduce the negative impact of positioning and interpretation errors on sample quality, the following principles should be followed when selecting and interpreting samples. ① To reduce the effect of positioning error, the sample points were chosen as the center of a homogeneous area of the size of the error range area. ② In order to reduce the interpretation errors caused by the temporal phase of the data, remote sensing images from 2017 to 2020, which are consistent with the temporal phase of the data to be evaluated, were mainly used, and multi-temporal data from other years were referenced. ③ For some of the more difficult samples to interpret, the interpretation is assisted by combining references to other information. For example, combining other map information to distinguish between bare ground and artificial ground, using the voluntary geo-information platform Geo-wiki [63] to assist in the interpretation, etc. ④ Multiple independent interpretations were used, and the sample was discarded when the interpretation results could not be agreed upon after negotiation. Based on the above principles, a final sample of 4606 validation samples covering the study area was obtained by visual interpretation (Figure 4c). Based on the fact that, within the study area, the decadal total land cover change is less than 8%, even in a rapidly changing part of the world [64], this means that the error in the validation sample is only roughly between 0.32 and 3.2%, even if the sample data varies in time by more than 10 years, so the global sample data selected for this paper is perfectly acceptable. The percentage of each type at different sample points is shown in Figure 5.

3. Results

3.1. Comparative Analysis of Land Cover Composition

Figure 6 shows the land cover type area composition of the three products in southwest China and

Table 4. The area composition Pearson correlation coefficients between different products.

Product	FROM-GLC	ESA	ESRI
FROM-GLC	1.00	0.94	0.37
ESA	0.94	1.00	0.40
ESRI	0.37	0.40	1.00

reflects the type area correlation coefficients between the three products.

In general, the FROM-GLC and ESA land cover products describe essentially the same characteristics of land composition in the southwest region, i.e., the southwest region is dominated by forest, bareland, and grassland. ESRI products show a difference, that is, the southwest region is dominated by shrubland and forest, accounting for more than 67% of the total areas, followed by bareland.

The FROM-GLC product has bareland (34.46%) and forest (32.25%) as the most dominant land type in the region, with grassland (19.13%), cropland (9.19%), water (1.89%), snow/ice (1.40%), built-up (1.08%), shrubland (0.54%), and wetlands (0.06%) decreasing in order of size; the ESA product are clearly dominant by forest (32.30%) and grassland (28.08%), with decreasing areas of bareland (25.35%), cropland (5.68%), shrubland (4.65%), water (1.97%), snow/ice (1.20%), built-up land (0.70%), and wetlands (0.07%) in that order; the ESRI products have decreasing areas of shrubland (37.15%), forest (30.04%) accounted for a heavier area, with decreasing areas of bareland (16.92%), built-up (4.58%), cropland (4.37%), grassland (3.12%), water (2.15%), snow/ice (1.55%), and wetlands (0.12%) in that order.

For each land cover type, there is a high level of agreement between the three products regarding the area of forest cover, yielding 30 to 33% of the total area of forest in the study area. For snow and water types, there is a high degree of consistency in the determination of the area covered by the three products, i.e., the proportion of snow and water area in the southwest is around 1 and 2% respectively. There is a high degree of consistency in the determination of wetland cover area with the FROM-GLC and ESA products, both of which yield a proportion of wetland area in the study area of approximately 0.06 to 0.07%. For the land cover types of cropland, grassland, shrubland, built-up land, and bareland, the estimates of the percentage of area covered vary considerably between products, with FROM-GLC yielding a larger area covered by cropland and bareland in the southwest, ESA yielding a larger area covered by grassland (28.08%), and ESRI yielding a larger area covered by shrubland and built-up in the southwest; while ESRI yielding smaller area covered by cropland, bareland, and grassland in southwest China, FROM-GLC yielded a smaller area covered by shrubland in southwest China (0.54%), and ESA yielded a smaller area covered by built-up land in southwest China (0.70%).

The correlation coefficient between the same satellite remote sensing land cover products for each type of land area has a high correlation between FROM-GLC and ESA (0.94) and a low correlation with ESRI products at 0.37.

3.2. Differences in Spatial Patterns

3.2.1. Consistency of Spatial Distribution

A consistent distribution of the three land cover products was obtained based on the spatial overlay method (Figure 7). The results show that the areas where the three products are in high consistency and are mainly located in the eastern part of the study area. The reason for this is that the surface cover type in the eastern area is mainly forest. The areas where the three products are in moderate consistency are mainly located in the central and north-western part of the study area, where the single surface cover type is mainly grassland and bareland. The areas where the three products are in low consistency are mainly located in the south-western part of the study area, the reason for the low agreement in this area is that it is difficult to distinguish between grassland and shrubland types in this area.

Further statistical analysis showed (Figure 8) that the areas where the three remotely sensed land cover product indication types were highly consistent and accounted for 44.89% of the total area. The areas where they were moderately consistent accounted for 39.53% of the total area, and the areas where they had low consistency accounted for 15.57% of the total area of the study area. This means that if we consider a 65% confidence level (i.e., three product types with more than two products of the same indication type at the same time), the current international mainstream satellite remote sensing products have a high degree of consistency over 84% of the land area in southwest China, and another 16% of the land area, where there is still much room for improvement in the consistency between different remote sensing land cover products.

3.2.2. Comparative Analysis with Google Earth Imagery

Based on Google Earth high-resolution imagery, different surface landscape types were selected in each of the five provinces covered by the study area to verify the accuracy of the three land cover products and to compare the spatial pattern consistency of several products (Figure 9).

Figure 9a shows the overlap pattern in the product when in areas of bareland, grassland, and shrubland. The FROM-GLC divides these areas into a combination of bareland and grassland, with the ESA product greatly overestimating the area of grassland and the ESRI product identifying most of the area as shrubland. Figure 9b shows the confusion of the three products for built-up land, bareland, and water, which may have been misclassified due to the more similar spectral and textural information in the region. Figure 9c shows that in areas where forest, grassland, and shrubland are interspersed, the classification of forest, shrubland, and grassland is more similar for the FROM-GLC and ESA products, while the ESRI product shows a disproportionate classification of shrubland as built-up. Figure 9d shows that the three products diverge in their classification of the vegetation within the built-up area, with the FROM-GLC classifying the vegetation within the built-up area as grassland, while the ESA product classifies the vegetation within this area as bareland and shrubland, and the ESRI product classifies it more crudely, classifying some of the vegetation almost entirely as built-up. The classification of cropland type is shown in Figure 9e, where both FROM-GLC and ESA are able to identify individual agricultural plots, while the ESRI products are classified with a coarser representation of detail.

In summary, the comparative analysis of the three products shows that the inconsistency of the classification results is higher for areas with similar spectral and textural characteristics such as shrubland, grassland, and bareland in terms of the type of classification. In terms of the detailed presentation of the classification, the FROM-GLC and ESA products are better able to describe the details of the land cover features and the ESRI products describe them in less detail.

3.3. Absolute Accuracy Evaluation

The absolute accuracy of the three land cover products was evaluated using the two existing published freely available sample points. The experimental results show that for the Geo-Wiki sample point data (

Table 5. Accuracy evaluation results based on Geo-Wiki samples.

Geo-Wiki
		1	2	3	4	5	6	7	8	9	OA (%)	Kappa
FROM-GLC	PA (%)	32.68	75.44	42.07	2.67	0.00	50.00	65.22	58.52	8.89	49.47	0.35
	UA (%)	53.55	51.45	24.61	35.30	0.00	60.00	40.00	67.59	72.73
ESA	PA (%)	32.68	76.56	51.44	5.33	0.00	52.38	65.22	38.21	6.67	45.13	0.314
	UA (%)	65.37	54.53	20.90	10.17	0.00	48.89	73.17	68.09	85.71
ESRI	PA (%)	20.56	78.06	12.98	45.78	0.00	52.38	93.48	27.73	10.00	39.03	0.27
	UA (%)	62.91	58.67	30.68	9.85	0.00	46.81	18.07	69.40	81.82

Note: 1: Cropland; 2: Forest; 3: Grassland; 4: Shrubland; 5: Wetland; 6: Water; 7: Build up; 8: Bareland; 9: Snow/Ice.

), the overall accuracy and Kappa coefficient of FROM-GLC are the highest, at 49.47% and 0.35, respectively, followed by the ESA product with overall accuracy and Kappa coefficient of 45.13% and 0.314, while the overall accuracy and Kappa coefficient of the ESRI product are the lowest, 39.03% and 0.27, respectively. For the GLCVSS sample point data (

Table 6. Accuracy evaluation results based on GLCVSS samples.

GLCVSS
		1	2	3	4	5	6	7	8	9	OA (%)	Kappa
FROM-GLC	PA (%)	38.33	86.21	34.29	0.00	0.00	100.00	50.00	70.18	11.77	56.56	0.42
	UA (%)	54.76	64.10	28.24	0.00	0.00	42.86	28.57	64.17	100.00
ESA	PA (%)	33.33	83.62	61.43	0.00	0.00	100.00	50.00	57.90	11.77	54.92	0.42
	UA (%)	64.52	66.90	32.09	0.00	0.00	42.86	50.00	72.26	100.00
ESRI	PA (%)	23.33	84.48	2.86	48.28	100.00	100.00	100.00	35.09	14.71	41.19	0.30
	UA (%)	63.64	70.00	14.29	7.29	50.00	50.00	17.39	73.17	71.43

Note: 1: Cropland; 2: Forest; 3: Grassland; 4: Shrubland; 5: Wetland; 6: Water; 7: Build up; 8: Bareland; 9: Snow/Ice.

), the FROM-GLC product had the highest overall accuracy and Kappa coefficient of 56.56% and 0.42, respectively, followed by the ESA product with an overall accuracy and Kappa coefficient of 54.92% and 0.42, while the ESRI product had the lowest overall accuracy and Kappa coefficient of 41.19% and 0.30. Overall, even though different sample point data were used, the results of the calculations all indicate a high level of accuracy for the FROM-GLC product.

For each type of accuracy, the Geo-Wiki sample point data shows a high mapping accuracy of over 75% for all three different land cover products for the forest type and a low mapping accuracy for the wetland type. The GLCVSS sample point data show a high mapping accuracy of over 80% for the three different land cover products for the water body and forest types. It is worth noting that the ESRI product also has a high mapping accuracy of 100% for the wetland and built-up land types.

The absolute accuracy of the three land cover products was evaluated using sample points obtained by manual visual interpretation, and the experimental results (

Table 7. Accuracy evaluation results based on visual interpretation samples.

Visual Interpretation
		1	2	3	4	5	6	7	8	9	OA (%)	Kappa
FROM-GLC	PA (%)	65.73	78.37	66.57	1.69	0.00	83.93	61.65	69.66	54.05	62.42	0.56
	UA (%)	84.49	53.09	23.26	12.50	0.00	97.66	84.93	41.41	99.01
ESA	PA (%)	66.09	75.90	87.98	0.00	3.92	90.53	74.49	41.04	50.81	64.50	0.58
	UA (%)	92.28	57.97	27.52	0.00	92.31	97.98	94.57	29.96	100.00
ESRI	PA (%)	59.48	73.81	29.03	24.72	7.4	91.82	98.89	29.19	55.14	61.94	0.56
	UA (%)	95.94	63.15	55.62	3.96	92.31	97.56	73.59	49.75	99.03

Note: 1: Cropland; 2: Forest; 3: Grassland; 4: Shrubland; 5: Wetland; 6: Water; 7: Build up; 8: Bareland; 9: Snow/Ice.

) showed that the ESA product had the highest overall accuracy and kappa coefficient of 64.50% and 0.58, respectively, followed by the FROM-GLC product with overall accuracy and Kappa coefficient of 62.42% and 0.56, while the ESRI product had the lowest overall accuracy and kappa coefficient of 61.94 and 0.56%, respectively. For each type of accuracy, the FROM-GLC product has a high mapping accuracy of over 75% for the water body and forest types and a low mapping accuracy of less than 2% for the shrubland and wetland types. For ESA products, the mapping accuracy is higher for the water, grassland, and forest types with an accuracy of 75% or more, and lower for the shrubland and wetland types with an accuracy of less than 4%. For ESRI products, the mapping accuracy is higher for the forest, water, and built-up land types, with an accuracy of over 70%, and lower for the wetland types, with an accuracy of 7.4%.

In summary, the evaluation of the accuracy of the three products using different sample data shows that the overall accuracy of the FROM-GLC and ESA is higher than that of the ESRI product. For each of the nine cover types, all three products show a high level of accuracy for forest types and a low level of accuracy for shrubland, grassland, and wetland types.

4. Discussion

4.1. Analysis of the Impact of Typical Land Class Difference on the Study of Rocky Desertification Area

Rocky desertification is caused by the low soil formation rate and high permeability of carbonate strata in this region, which creates a fragile ecological environment and is easily disturbed by human activities. Finally, karst areas covered with vegetation and soil are transformed into rocky landscapes [36]. Therefore, the basic data of built-up, vegetation, bare land, and other land cover types in this region is an important supporting data for monitoring the change of rocky desertification in this region.

Built-up land is an important track of human activities, and its utilization of natural resources will directly or indirectly affect the rocky desertification of undeveloped areas [65]. Therefore, the distribution and development and utilization of construction land in southwest China are of great significance to the monitoring of regional ecosystem. In this study, it was found that the spatial consistency of the construction land types of the three land cover products was relatively low, and the PA of ESRI products was 93.48, 100, and 98.89%, respectively, higher than the other two products. The ESRI has a higher proportion of area than the FROM_GLC and ESA products, and the ESA product has the lowest proportion of area. Therefore, ecological researchers targeting rocky desertification areas may underestimate the impact of rocky desertification area ecosystems if they select the FROM_GLC and ESA product built-up type for their research when selecting land cover products. In terms of details, FROM-GLC and ESA products have a more detailed delineation of built-up types, while ESRI data lacks in detail delineation of construction land types (Figure 10). Therefore, suitable data can be selected as supporting data in the study of built-up types in rocky desertification areas.

Vegetation is an important land cover type that contributes to ecosystem change [44]. The vegetation cover types selected for the land cover products in this paper contain forest, grassland, and shrubland. The consistency of all three products regarding forest types is high, with the PA above 70%. For the grassland type, the results of the evaluation of the three products using different sample points showed that the PA of the ESA product was 51.44, 61.43, and 87.98% respectively, all higher than the other two products (Figure 11). ESRI’s grassland type area ratio of 3.12% is much lower than the area ratio of the other two products. This is noteworthy because once the grassland type of ESRI products is used as an ecological study of rocky desertification areas, the conclusions drawn will contradict those from the use of other land cover products. For the shrubland type, the results of the evaluation of the three products using different sample points show that the ESRI products have a PA of 45.78, 48.28, and 24.72% respectively, which are higher than the other two products. However, by comparing with Google Earth, it is found that these three products all have different degrees of omission and multiplicity in shrubland classification (Figure 12). The 48.28% mapping accuracy for the shrubland type of land cover products may not be sufficient for regional rocky desertification studies, so it is preferable to select additional land cover products with high accuracy of shrubland classification when studying ecosystems in rocky desertification areas. The percentage of area in the FROM_GLC on the shrubland type is only 0.54%, while the percentage of area in the ESRI product on the shrubland type is 37.15%. Such a large difference is of particular attention to ecological researchers when selecting subsequent land cover products. Therefore, in order to provide remote sensing data support to researchers of vegetation change on regional rocky desertification, regional desertification, and other ecosystem conservation, suitable remote sensing support data can be selected for studies with different needs. Ecosystem restoration requires the development or adoption of new technologies in addition to revegetation, the development of more forest reserves, the exploitation of water resources, and the reduction of natural hazards. The use of remote sensing technology for dynamic monitoring of large areas, for example, can identify areas of ecological damage and reduce the formation of rock desertification in a timely manner.

It has been shown that severe rocky desertification is located in areas of bare ground cover types [66]. The ecological protection of stone desertification areas is mostly carried out by reducing the land use of bare land, developing ecological protection forests, or improving grassland, which ultimately contributes to the goal of gradual ecological landscape restoration and continuous improvement of land use efficiency. It was found that the results of the evaluation of the three products using different sample points showed that the PA of the FROM-GLC product was 58.52, 70.18, and 69.66% respectively, all higher than the other two products (Figure 13). Therefore, the FROM-GLC data can be used as supporting data when conducting research on the use of bare ground types in rocky deserts. Given the importance of bareland types for the conservation and restoration of rocky desertification, we expect land cover mappers to improve the accuracy of bareland types in future mapping in order to increase the reliability of data for rocky desertification studies.

Global climate change has slowed the recovery process due to the frequency of droughts and the increase in extreme droughts caused by global climate change, and even expanded lithification in some areas could lead to secondary desertification if not managed effectively [36]. It is hoped that the consistent study of multi-source land cover products in this paper will help developing countries to achieve sustainable development and that the results of the study can be adapted to other areas where restoration efforts are needed.

4.2. Discussion of Inconsistent Factors

A statistical analysis of the consistency of the three 10 m land cover products revealed that there were differences between the three land cover products due to their classification systems, classification methods, and differences between the sample data [58,67]. This difference makes rigorous comparison between maps and the synergistic use of different maps a huge challenge [68,69].

(1) Differences in classification systems are one of the main factors leading to inconsistent classification results [69]. Among them, the spatial distribution consistency and correlation coefficients are higher for FROM-GLC and ESA products, and in terms of the classification system (https://esa-worldcover.org/en/data-access, accessed on 13 January 2022), the type definitions of FROM-GLC and ESA products are more similar, and the reason for this is that the classification maps for these two products are based on the UN Land Cover classification system. The main difference is the shrubland type in FROM-GLC, which is a mixture of shrubland, grassland, and lichen types in the ESA product species. Similarly, greenhouse agriculture is included in the cropland type according to the definition of land cover type, while areas of this type are included in the built-up type in the ESA product (https://esa-worldcover.org/en/data-access, accessed on 18 January 2022). Compared to the validation dataset, globally, accuracy was rated higher for the forest and slightly lower for the shrubland and grassland types, mainly because differences in vegetation canopy thresholds for shrubland and grassland are difficult to define in 10 m satellite imagery. As for the confusion between grassland and built-up land, this is mainly a matter of contradiction between land use and land cover, reflected in the fact that built-up areas contain functions such as grassland and parks, and although the land cover of the area may be grassland, functionally they are part of the built-up land [1]. Therefore, in order to reduce inconsistencies between different land cover products, definitions for shrubland, grassland, and built-up land types need to be considered in a combination of factors.

There are also differences between the ESRI product and the ESA product in terms of type descriptions. The most notable difference is that the ESRI submerged vegetation type includes rice and irrigated/submerged agriculture, which is included in the cropland types in the FROM-GLC and ESA products. This corroborates the fact that ESRI’s share of cropland is relatively low in the results in Section 4.1. Despite the differences between the ESRI product and the FROM-GLC and ESA product classification systems, the ESRI product has a resolution of 10 m, similar in time to the ESA product and identical in resolution to the FROM-GLC product. As a result, the ESRI products were compared for consistency with the FROM-GLC and ESA products. In summary, the accuracy of the classification of land cover products can be significantly improved by improved type definitions.

(2) The different classification methods and the singularity of the data sources also affect the consistency between the different products. Specifically, the FROM-GLC product uses a random forest approach that is optimal in terms of computational efficiency and performance [70], while the ESA and ESRI products use a deep learning approach, where the deep learning algorithms have excellent matching performance with the rich spectral and contextual information [71]. All three products use sentinel data, and the addition of auxiliary data as additional predictors can improve classification accuracy, although FROM-GLC uses space shuttle radar terrain elevation data auxiliary data in its classification, but the data are still relatively homogeneous. Due to the similarity between natural vegetation classes and in order to better differentiate vegetation, taking into account the relationship between climate and vegetation, Sullamenashe et al. [72] used MODIS data to classify the northern Eurasian region and succeeded in increasing the classification accuracy to 73%. Abdi [73] and Zha [74] et al. also agree that agricultural activities related to planting and harvesting times also vary across the region in response to crop and climate changes. In addition, almost all land cover products currently rely on optical remote sensing imagery, whereas SAR and LIDAR techniques have proven to be advantageous in many ways, particularly in distinguishing between shrubland and wetland types [75,76,77,78]. The low statistical consistency results in this paper may be due to the openness of the area and the highly confused shrubland and grassland.

The timing of the original images selected for classification can also greatly affect the accuracy of the product classification. For example, even in the simplest of features, the body of water may change seasonally, i.e., it may be snow and ice in winter and bare ground in summer. The study showed [79] that the accuracy of all images available during the year was lower than the classification accuracy of the seasonal image composites, and in addition, the accuracy of the land cover products generated from the composite images from multi-year data was lower than that of the data from one year. Frantz et al. [80] also point out that trade-offs between different years of imaging and different days within the same year (potential loss of phenological consistency) should be taken into account when producing land cover products. In this case, the land cover product is best classified using the same year’s data source if a high level of accuracy is to be obtained.

(3) The quantity and quality of the validation samples can also have an impact on the results of the consistency evaluation. The published Geo-Wiki and GLCVSS validation samples used in this paper are limited in number in this study area, and the validation sample points obtained based on visual interpretation of Google Earth imagery, although large in number, are subject to greater human intervention and the selection of sample points may be biased towards the categories we are most certain of or consider most important, which may produce an error of 5–10% [61]. When collecting sample data and performing image classification, a broad assumption is that classes are mutually exclusive and have strong, well-defined boundaries [24], which is rare in natural environments such as natural vegetation or wetlands. So, the method of Millard et al. can be used in the future for selecting sample point data [81], where polygons are drawn around regions where a category is known to exist, then highly aggregated training sample points with inherently high spatial autocorrelation are generated, and finally the grid cell values for each input derivative are extracted from the locations of the training data points to generate the sample data. Therefore, the design of obtaining reasonable validation sample points for a more subjective and reasonable evaluation of land cover products is a worthwhile in-depth study. In particular, for the classification and updating of land cover products worldwide, the establishment of a shared and updated sample database is important for both the production of early remote sensing products and the analysis of later geographical results.

4.3. Suggestions on Land Cover Mapping

The spatial consistency demonstration and accuracy evaluation analysis presented in this paper shows that the FROM-GLC, ESA, and ESRI products use Sentinel-1 and Sentinel-2 data at 10 m resolution to characterize land cover globally with higher quality and good spatial detail. However, it was found during the study that there were still some limiting factors between the different products that could be addressed in future mapping exercises.

(1) The issue of edging between different land cover types. As shown in Figure 14, there is a clear demarcation between forest and grassland in the FROM-GLC product, while a similar anomaly can be found between shrubland and grassland classes in the ESA product, and similarly between shrubland and bareland in the ESRI product. In order to avoid such a problem, in the future, we can consider increasing the training accuracy of the model when classifying, and using a block stacking strategy, so that the blocks are stacked with each other in a certain number of steps, thus avoiding the problem of edge-joining.

(2) For feature categories with small differences in spectral and textural characteristics, it is more difficult to accurately classify optical remote sensing images, so some auxiliary data can be considered in the classification. For example, to accurately distinguish between vegetation such as grassland, cropland, and shrubland, consider using Sentinel-1 Radiation Corrected Ground Distance Detection (GRD) data. In addition, the inclusion of time-series features, such as measurements of vegetation health over a year, can improve the classification accuracy of grassland, cropland, and shrubland. Since the NDVI and EVI values of vegetation differ between seasons or growth stages, it is possible to find the implicit phenological information in vegetation to accurately distinguish between vegetation types, and this method has also received widespread attention in recent years [82,83]. In addition, the accuracy of the classification was influenced by the different features selected for the three land cover products during pre-processing. For example, the center of each sample location is used in the pre-processing of the FROM-GLC product to match the nearest locations of the sentinel data to extract and construct spectral features. The slope and aspect data extracted from the SRTM elevation data are included in the feature set. The ESA product uses long range averaged timestamps as features along with quartiles, introduces time dynamics information in the classification, and extracts height and slope features from the Copernicus Global 30m Digital Elevation Model (DEM), and includes a range of spatial positioning features. The ESRI product was trained using over 5 billion hand-labelled Sentinel-2 pixels, and the underlying deep learning model uses six bands of Sentinel-2 surface reflectance data: visible blue, green, red, near infrared, and two shortwave infrared bands. To create the final map, the model will be run on images from multiple dates throughout the year and the output will be synthesized into a final representative map for 2020. Therefore, how feature data is selected during pre-processing has an impact on areas with strongly seasonal surface cover types. For the wetland type, in areas with high vegetation cover, the use of radar data should be focused on because optical and thermal systems are limited because they cannot penetrate the vegetation canopy [84,85].

(3) Built-up land faces great challenges in classification due to the small size of the patches and the high internal heterogeneity. It has been shown that construction sites can be effectively extracted using DMSP OLS nighttime lighting data [86]. Therefore, nighttime light data could be introduced into the production of global-scale land cover products in the future. Conflicts between land use and land cover when distinguishing between grassland and built-up land could be addressed by extending the land cover classification products to secondary categories to better understand specific land uses in different areas (e.g., plantations versus natural forests, residential versus commercial built-up areas).

5. Conclusions

This paper compares and analyzes the consistency between three of the current mainstream global land cover products at 10 m resolution, taking southwest China as the study area, in order to provide a reference for the selection of suitable land cover data for many studies in rocky desertification areas. The satellite remote sensing land cover products assessed include: FROM-GLC, ESA, and ESRI. The methods of consistency analysis include three main methods: similarity of type composition, spatial consistency, and absolute accuracy assessment. The main findings of the study are as follows.

The FROM-GLC product had the highest overall accuracy of between 49.47 and 62.42%, followed by the ESA product with an overall accuracy of between 45.13 and 64.50%, and the ESRI product with the lowest overall accuracy of between 39.03 and 61.94%. Analysis of the spatial consistency of the three products shows that the proportion of perfectly consistent areas is low at 44.89%, mainly in areas with low surface heterogeneity and more homogeneous cover types. Differences in the discrimination of some vegetation types, such as grassland, shrubland, and bareland, are the main reason for the low consistency of the three products. In general, the accuracy of these three high-resolution global land cover products in southwest China is not ideal, and the accuracy of some vegetation types such as grassland, shrubland, and bareland needs to be further improved in the future.

Overall, the production of a global 10 m high-resolution land cover product based on Sentinel 1 and Sentinel 2 data shows great potential. However, as different land cover products express the fineness of different land types differently, the user needs to select the appropriate land cover product in the best way based on a statistical accuracy analysis and an assessment of spatial accuracy.