Comparison and Validation of Multiple Medium- and High-Resolution Land Cover Products in Southwest China

Abstract

The rapid advancement of remote sensing technology has given rise to numerous global- and regional-scale medium- to high-resolution land cover (LC) datasets, making significant contributions to the exploration of worldwide environmental shifts and the sustainable governance of natural resources. Nonetheless, owing to the inherent uncertainties embedded within remote sensing imagery, LC datasets inevitably exhibit inaccuracies. In this study, a local accuracy assessment of LC datasets in Southwest China was conducted. The datasets utilized in our analysis include ESA WorldCover, CLCD, Esri Land Cover, CRLC, FROM-GLC10, GLC_FCS30, GlobeLand30, and SinoLC-1. This study employed a sampling approach that combines proportional allocation and stratified random sampling (SRS) to gather sample points and compute confusion matrices to validate eight LC products. The local accuracy of the eight LC maps differs significantly from the overall accuracy provided by the original authors in Southwest China. ESA WorldCover and CLCD demonstrate higher local accuracy than other products in Southwest China, with their overall accuracy (OA) values being 87.1% and 85.48%, respectively. Simultaneously, we computed the area for each LC map based on categories, quantifying uncertainty through the reporting of confidence intervals for both accuracy and area parameters. This study aims to validate and compare eight LC datasets and assess precision and area of diverse spatial resolution datasets for mapping and monitoring across Southwest China.

1. Introduction

The impacts of the greenhouse effect, such as global climate warming [1], extreme oceanic events [2], precipitation extremes [3,4], and glacial retreat [5], are significantly affecting the future survival of humanity. The emissions of greenhouse gases such as carbon dioxide (CO₂) are detected as the primary drivers of global warming [6,7,8]. With the continuous advancement of urbanization and industrialization, a substantial amount of carbon dioxide is released into the atmosphere due to human activities [9]. Among these, land cover change (LCC) emerges as a pivotal factor in carbon emissions, comprising roughly one-third of the carbon emissions attributed to human activities throughout the Industrial Revolution period [10]. The carbon emission coefficients differ across various land types. For instance, cropland stands at approximately 0.04, whereas forest is approximately −0.06 [11]. To quantify the carbon emissions, LCC reconstructions are needed and utilized in relevant carbon models [12,13,14]. But the carbon emissions it causes are also one of the most uncertain factors in the global carbon budget. Hence, the information acquiring accurate land cover (LC) data about the current state of the Earth’s land surface holds theoretical significance for studying the interrelationship between LCC and carbon emissions [15], contributing to the realization of the objective of carbon neutrality [16].

With the rapid development of Earth observation technology, remote sensing (RS) monitoring offers a more efficient and effective approach to generate LC products. Scholars are making active efforts on both domestic and international levels to develop region-specific LC maps with varying resolutions, and the spatial resolution has been increased from 1 km [17] to 500 m [18], 30 m [19,20,21], 10 m [22,23,24,25], and even as fine as 1 m [26]. These LC data have been widely utilized to analyze the carbon stocks of terrestrial ecosystems [27,28,29]. However, the differences in the acquisition time of remote sensing data, LC classification techniques, classification systems, etc., used in existing LC products, result in inconsistent classification standards and accuracy [30]. For example, inaccurate data of LCC in China led to a serious underestimation of the terrestrial carbon sink [31]. It is a challenge for us to choose suitable products for related research. With the growing demand for more detailed and accurate LCC information, it becomes essential to document the accuracy of these LC products [32]. Commonly employed techniques for validating LC maps include classic methods like Multiple-Resolution Cross Tabulation, Cross-Tabulation Matrix, Soft Classification Maps, and Pattern Analysis (Map Curves), etc. [33]. Researchers and various organizations have actively utilized these diverse methods to validate LC maps. For example, Chaaban [34] used stratified random sampling (SRS) and equalized stratified random (ESR) to assess the accuracy of existing 10 m spatial resolution LC products: ESA WorldCover and Esri Land Cover. Gao [35] evaluated three 30 m LC products concerning areal and spatial consistency using the Land Use/Cover Area Frame Statistical Survey (LUCAS) reference dataset over the European Union (EU). Each approach possesses unique strengths and applicability [36,37,38,39,40]. However, it is important to note that limitations still exist.

Such “accuracy” generated by existing research mainly involves national and intercontinental scales (such as global accuracy), and further verification is needed to determine whether it meets the requirements in the specified research area. Bai found that even for the same LC product, GLCD-2005, there are significant differences in accuracy among different regions in China [36]. Most LC products primarily rely on RS images as their main data source. During production, they are often affected by issues related to the accuracy of these images itself, such as cloud cover and mixed pixels [41]. This impact is particularly important in areas with complex and fragmented LC distributions, leading to a noticeable decrease in accuracy. Consequently, traditional per-pixel sampling designs are ill-suited for these environments, as they fail to adequately characterize the complex spatial variability and relationships within fragmented area [42]. This gap highlights the need for optimized sampling techniques to enhance the representation of spatial variability in accuracy assessments of heterogeneous regions. Area estimation, especially for forested areas, can be utilized to quantify global forest carbon losses through remote sensing estimates [43]. Keenan believes that furnishing reliable information on global forest area trends is invaluable for all people in decision-making on policies and investments [44]. However, estimates of LCC, particularly those related to critical factors like forest change in carbon accounting [45], are often susceptible to significant errors [46]. The key to addressing certain global environmental science issues lies in the accurate measurement of LCC areas. Traditional methods of obtaining this involve directly extracting information from categorized LC products. However, it is possible that the area computed from the LC map could be badly biased even when the two classifications are both highly accurate [47,48,49].

Southwest China has a robust carbon sequestration capacity, becoming the largest terrestrial carbon sink in China over the past 30 years [50]. Its carbon absorption constitutes slightly over 35% of Chinese total absorption [51]. On the other hand, Southwest China has experienced a “once-in-a-century” prolonged severe drought, triggering severe impacts on vegetation growth and thereby suppressing this carbon sink [52]. Therefore, LCC in Southwest China is undoubtedly necessary to pay high attention to prevent further degradation of the existing environment. However, it poses significant challenges for obtaining sufficient high-quality and high-resolution remote sensing images in this place, because of its frequently cloudy and foggy weather [53]. At the same time, the geomorphological structure of this area is complex, and the distribution of land classes is extremely fragmented. Driven by such factors, the challenges in classifying LC in the southwestern region have consequently increased. Therefore, it is quite important to obtain an accurate assessment for LC products in this region.

In this paper, the primary objective is to validate the accuracy of medium- and high- resolution LC data in Southwest China, an area with diverse and fragmented landscapes, by comparing it with independent LC samples obtained by optimized sampling methods. Secondly, we will produce scientifically rigorous and transparent estimates of area for each LC category based on the sampled points. Through this analysis, our objective is to improve the estimates of the carbon budget in Southwest China, where forests are widespread, representing an extensive carbon sink and contributing to the global carbon balance.

2. Materials and Methods

2.1. Study Area

Southwest China is located between 21°04′~34°23′N and 97°26′~112°06′E, including Sichuan Province, Yunnan Province, Guizhou Province, Chongqing City, and Guangxi Zhuang Autonomous Region (Figure 1). The region is often covered by extensive cloud cover (Figure 1a), making it hard to obtain high-quality satellite images [53]. Moreover, the subtropical humid monsoon climate results in extremely hot and abundant rainfall in summer. Rainfall has led to the southwestern mountainous regions becoming one of the most severely affected areas by water-eroded desertification in China [54]. The topography in this area is varied and complex, featuring deeply incised valleys and expansive karst areas, with a relative elevation difference exceeding 7000 m from the northwest to the southeast [55]. These combined influences have intensified land fragmentation in this region.

2.2. Medium- and High-Resolution LC Products

Eight LC maps with widely used and recent production dates, featuring different spatial resolutions, were sourced from various sensors, including Landsat and Sentinel-2 (

Table 1. Parameters of LC products used in the study.

Product	Sensor	Production Organization	Time	Classification Number	Overall Accuracy	Source
CLCD	Landsat	Wuhan University	1985–2022	9	79.31%	https://zenodo.org/records/5816591#.ZAWM3BVBy5c (accessed on 16 March 2024)
GLC_FCS30	Landsat	AIR	1985–2020 *	29	82.50%	https://data.casearth.cn/ (accessed on 16 March 2024)
GlobeLand30	Landsat	NGCC	2000, 2010, 2020	10	80.33%	https://www.webmap.cn/mapDataAction.do?method=globalLandCover (accessed on 16 March 2024)
ESA WorldCover	Sentinel-1, Sentinel-2	ESA	2020, 2021	11	74.40%	https://viewer.esa-worldcover.org/worldcover/ (accessed on 16 March 2024)
Esri Land Cover	Sentinel-2	ESRI	2017–2022	9	85.96%	https://www.arcgis.com/apps/mapviewer/index.html (accessed on 16 March 2024)
CRLC	Sentinel-2	Wuhan University	2020	8	84.87%	https://github.com/LiuGalaxy/CRLC (accessed on 16 March 2024)
FROM_GLC10	Sentinel-2	Tsinghua University	2017	10	72.76%	https://data-starcloud.pcl.ac.cn/zh/resource/1 (accessed on 16 March 2024)
SinoLC-1	Sentinel-2	Wuhan University	2020	11	73.61%	https://zenodo.org/records/8214871 (accessed on 16 March 2024)

* GLC_FCS30 dataset was generated every five years from 1985 to 2020.

). The products selected for this study are all from 2020. Since the latest production year for FROM_GLC extends only up to 2017, considering the proximity of dataset phases, it is still included in this study.

(1)

CLCD [19] land cover map with 30 m spatial resolution from Wuhan University. It is the annual long-term Landsat-derived land cover dataset, established by utilizing visually interpreted samples from satellite time-series data and a random forest classifier.

(2)

GLC_FCS30 [20] land cover map with 30 m spatial resolution from Aerospace Information Research Institute, Chinese academy of science (AIR). It offered the most refined classification system in the dataset used for this study (encompassing 16 global LCCS land cover types along with 14 intricate regional land cover types). The data adopted a local adaptive random forest model.

(3)

GlobeLand30 [21] (Global land cover dataset with 30 m spatial resolution) land cover map with 30 m spatial resolution based on the integration of pixel- and object-based methods with knowledge (POK-based) from National Geomatics Center of China (NGCC).

(4)

ESA WorldCover [22] land cover map with 10 m spatial resolution from European Space Agency (ESA) from the WorldCover project. This product was based on both the Sentinel-1 and Sentinel-2 satellite data and used the pixel-based strategy.

(5)

Esri Land Cover [23] land cover map with 10 m spatial resolution from ESRI and Microsoft’s Planetary Computer. It was used for building the global map based on a deep learning artificial intelligence (AI) land classification model.

(6)

CRLC [24] land cover map with 10 m spatial resolution from Wuhan University. They have introduced a cross-resolution land cover mapping framework that incorporates the principles of noisy label learning.

(7)

FROM_GLC [25] (Finer Resolution Observation and Monitoring of Global Land Cover) land cover map with 10 m and 30 m spatial resolution from the team of Professor Gong Peng of Tsinghua University. It was developed on Google Earth Engine (GEE) using a supervised random forest classifier with FROM-GLC Plus (FGP).

(8)

SinoLC-1 [26] land cover map established by Wuhan University, is the first land cover dataset which has 1 m spatial resolution of China, utilizing a deep learning-based framework and open-access data.

The above eight LC products used in this study have undergone independent accuracy assessments. Nonetheless, the results of these independent accuracy evaluations cannot be directly compared due to the use of different verification methods and reference data. In this research, these LC products were validated in Southwest China to obtain local accuracy.

2.3. Methods

Since obtaining the actual LC information for the whole study area is both expensive and challenging, conducting accuracy assessments based on statistical sampling is a more common practice. Given the significant spatial heterogeneity of the land surface in the Southwest China, this study integrates stratified random sampling and proportional allocation while considering the rare classes, building upon traditional probability statistical models.

Figure 2 presents the validation flowchart for LC products in Southwest China.

Step 1: Data pre-processing. Main contents of data pre-processing include projection transformation, mosaic, extraction, reclassification, etc. This step was performed in the ArcGIS 10.8 platforms, a desktop Geographic Information System (GIS) software developed by ESRI (www.esri.com, accessed on 16 March 2024).

Step 2: Sample collection using SRS. Olofsson proposed that regardless of which map is used to define the strata when assessing multiple maps, the data from the sample references remain available for evaluating any map [56]. Considering both prior knowledge and computational costs, this study chose CLCD products to define the strata. Strata were delineated based on eight LC types. Based on SRS, calculate the total sample size and determine sample allocation based on the area of each LC types. The samples obtained through this method serve for accuracy estimation, area calculation, and quantifying the associated uncertainty.

Step 3: Integration of data sources for sample points interpretation. In this step, a combination of field survey data, multiple remote sensing images from diverse sources and periods, and high-resolution Google Earth Images was employed. Each sample points underwent interpretation, leading to the identification of its corresponding LC type.

Step 4: Validate the accuracy of the LC data using the chosen sampling points, encompassing both accuracy validation and area estimation. Sample points were applied to all LC products to calculate validation indicators and the area of each category. In order to comprehensively characterize the LC patterns in the southwestern region of China, an analysis emphasizing the spatial distribution differences among various LC datasets was conducted.

2.3.1. Reclassification

We contrasted the diverse classification systems of various products and converted it into a consistent classification system. Additionally, as both the CLRC and Esri Land Cover merge shrubland and grassland into a single category named “Grassland/Shrubland”, we have also unified these two classes. This study simplified the classification system to eight classes: Forest, Grassland (including Shrubland), Cropland, Impervious, Barren, Ice, Water, and Wetland (

Table 2. Correspondence between target classification system and original classification system.

Product	Forest	Grassland	Cropland	Impervious	Barren	Ice	Water	Wetland
GLC_FCS30	12/51/52,61/62/71/72/81/82/91/92 *	11/120/121/122/130/140/150/152/153	10/20	190	200/201/202	220	210	180
GlobeLand30	20	30/40	10	80	90	100	60	50
CLCD	2	3/4	1	8	7	6	5	9
FROM_GLC10	20	30/40	10	80	90	100	60	50
ESA WorldCover	10	20/30	40	50	60	70	80	90
Esri Land Cover	2/6	11	5	7	8	9	1	4
CRLC	2	3	1	8	9	10	6	5
SinoLC-1	2	3/4	5	1/6	7	8	9	10

* The numbers presented in the table denote the category codes of the original classification system for different remote sensing product.

).

2.3.2. Stratified Random Sampling Method

Sampling design is the protocol for selecting a subset of spatial units, which will constitute the foundation for accuracy assessment. The crucial recommendation is that the sampling design should be a probability sampling design. Multiple probability sampling designs are applicable for precision assessment and area estimation, with the most common designs being simple random, stratified random, and systematic designs [57]. The implementation of stratification has two advantages: The first purpose is if there are strata of interest for the reported results, such as the accuracy of various categories divided by LC type. In the stratified random sampling method, the partitioning of each stratum must include certain probabilities, and these probabilities form the basis for accuracy and area estimation. Because of the diverse LC types and significant area variations in our study area (Figure 1b), aiming to obtain adequate samples for each LC type in the classification system, we collected reference samples for all LC classes using stratified random sampling method (SRS) following good practices as recommended by Olofsson [56].

For SRS, assuming equal sampling costs for each stratum, the formula for calculating the sample size (n) is as follows:

$$n=\frac{{({\displaystyle \sum W_i{S}_i})}^2}{{[S(\widehat{O})]}^2+(1/N){\displaystyle \sum W_i{S}_i^2}}\approx {(\frac{{\displaystyle \sum W_i{S}_i}}{S(\widehat{O})})}^2$$

(1)

where $S(\widehat{O})$ is the standard error of the estimated overall accuracy we aim to achieve, W_i is the mapped proportion of area of class i, $S_i$ is the standard deviation of stratum i, calculated as $S_i=\sqrt{U_i(1-U_i)}$ ($U_i$ is the user accuracy), and N is the number of units in the study area. Because N is typically large, the second term in the formula always can be ignored. Based on prior knowledge, we assume a user’s accuracy of 0.6 and the standard error of 0.005.

When using SRS for sampling, it is necessary to determine the total number of samples and the distribution method of the samples required for each stratum. The sample allocation methods commonly used include random allocation, proportional allocation, Neyman allocation, etc. [58].

Proportional allocation ensures all strata are represented proportionally, relative to the size of the landscape areas within it, preventing under-sampling of some strata. We choose the proportional allocation which is in stratified sampling, where the sample size n_i in each stratum is proportional to the stratum size $N_i$:

$$n_i=n{W}_i$$

(2)

Finally, the sample size for rarer categories is appropriately increased, catering to samples from all categories to ensure a more equitable distribution of samples for each stratum compared to proportional allocation results.

2.3.3. Sample Labeling

In practice, it is not always possible to visit the sample locations we selected, as they may be located in excessively high mountains or in overly dangerous ravines. The cost of accessing all sample points is excessively high. Therefore, we use the following multi-source remote sensing data as reference data to identify the LC types of validation points (

Table 3. Feature classification and description of imagery.

Class Name	Images	Google Earth	Feature Description
Forest			Dark green in RGB, red in false color composite, dark and coarse texture.
Grassland			Thin lines of fine textured vegetation in moss green (on the bands 8, 4, 3)
Cropland			With geometric shapes of the rules and finely, uneven color
Impervious			Regular or irregular distribution, violet in false color composite
Barren			Objects (in bands 8, 4, 3) are pink to dark red area sometimes brown.
Ice			Light blue in false color composite, white in RGB
Water			Light blue, whitish blue or black (on bands 8, 4, 3), shape is linear or block
Wetland			Mainly distributed in rivers and lakes around, rivers and lakes as the center

):

(1)

GF-2 imagery from National Major Projects on High-Resolution Earth Observation System. Southwest China has unique weather, with an exceptionally short duration of effective sunlight (Figure 1a). Therefore, we have chosen multiple time-series data from 2019 to 2020.

(2)

Google Earth Images.

Another issue to consider is that the size represented by the sampling points should be appropriate for the pixel size of the LC map to be evaluated. This study aims to select sample points from homogeneous areas of at least 60 m × 60 m to ensure the representativeness of the sample points and mitigate the scale effect on verification accuracy. If the representative size of the sample points equals only a single pixel, the validation results can be unduly influenced by inherent location errors in satellite imaging and the impact of mixed pixels.

We employed various independent interpretation methods by different individuals, involving three experts with extensive remote sensing image interpretation experience and one quality controller in this process. The three experts independently interpreted sample land classes, recording interpretation confidence levels (high, medium, low). These independently interpreted results were passed on to another expert for cross-verification. After removing obvious errors, the consistency among the results from the three experts now exceeds 95%. The cross-verified interpretation results are submitted to the quality control personnel for final inspection. In cases where a consensus on interpretation results cannot be reached through discussion, those particular samples are discarded.

A total of 9579 samples were collected, encompassing eight categories: Forest, Grassland, Cropland, Impervious, Barren, Ice, Water, and Wetland (Figure 3). In Figure 3d, the distribution of sampling points is depicted, with a predominant focus on Forest, Cropland, and Grassland, showcasing the quantity of each category. In Figure 3f, the distribution of sampling points in latitude can be observed. Forests are most widely distributed, with lower latitudes corresponding to the Impervious. Conversely, regions with higher latitudes demonstrate the distribution of snow and ice, grassland, and wetlands.

Typical regions to showcase the sampling results are presented in Figure 3a–c. Figure 3a depicts Chongqing city on the edge of the Sichuan Basin, characterized by persistent cloud cover and fog, restricting access to high-quality remote sensing data. Due to intensive human activities and complex karst topography in the region, land cover is highly fragmented. Figure 3b shows the southwestern region of Mount Siguniang in Sichuan Province, characterized by diverse LC types due to significant terrain variations. Figure 3c illustrates Kunming city in Yunnan Province, where the terrain, landforms, and Land Use and Land Cover (LUCC) are complex and variable.

2.3.4. Validation Indicators

The collected sample points generated error matrices, offering detailed measurements, including overall accuracy (OA), producer accuracy (PA), user accuracy (UA), and the Kappa coefficient (Kappa), to assess accuracy variations among different products [58].

$$OA=\frac{{\displaystyle {\sum }_{i=1}^q{n}_{ii}}}{N}$$

(3)

$$PA=\frac{n_{ii}}{n_{\cdot i}}$$

(4)

$$UA=\frac{n_{ii}}{n_{i\cdot }}$$

(5)

$$Kappa=\frac{N{\displaystyle {\sum }_{i=1}^q{n}_{ii}-{\displaystyle {\sum }_{i=1}^q(n_{i+}{n}_{+i})}}}{N^2-{\displaystyle {\sum }_{i=1}^q(n_{i+}{n}_{+i})}}$$

(6)

where i is the LC type of the product; n_ii is the number of pixels correctly classified in type i; n_+i is the number of pixels of type i in the reference product; n_i+ is the number of pixels of type i in the product to be evaluated number.

Differences in area and spatial distribution are also crucial metrics for evaluating the accuracy of LC datasets. Because of classification error, relying solely on area weighting and total area to calculate the areas of each category often introduces biases. The sample size for each map class is proportional to the relative area of the map class, allowing us to estimate the area using the sample points. Equation (7) serves as an area estimator, relying on the reference classification of each sample unit rather than directly obtaining the area from the map classification [56].

$${\widehat{A}}_j=A_{tot}{\displaystyle {\sum }_i{W}_i}\frac{n_{ij}}{n_{i\cdot }}$$

(7)

where ${\widehat{A}}_j$ is an estimate of the area for category j; and $A_{tot}$ is the total area.

Articles published between 2005 and 2010 in several mainstream remote sensing journals with a track record of publishing articles on land change were surveyed by Olofsson. He found that these articles rarely presented accuracy information for calculating the error-adjusted estimated area and confidence intervals [59]. Once the product has been validated, and an error matrix is created, estimating these parameters is not difficult but can yield a wealth of valuable information.

The estimated standard error of the estimated area proportion is

$$S({\widehat{p}}_{\cdot j})=\sqrt{{\displaystyle {\sum }_{i=1}^q{W}_i^2\frac{\frac{n_{ij}}{n_{i\cdot }}(1-\frac{n_{ij}}{n_{i\cdot }})}{n_{i\cdot }-1}}}$$

(8)

The standard error of the error-adjusted estimated area is

$$S({\widehat{A}}_j)=A_{tot}\times S({\widehat{p}}_{\cdot j})$$

(9)

An approximate 95% confidence interval (for 95% confidence, z = 1.96) for $A_j$ is ${\widehat{A}}_{\dot{J}} \pm 1.96\times S\left({\widehat{A}}_j\right)$.

2.3.5. Spatial Distribution Consistency Analysis

It is imperative to include an analysis that highlights the disparities in spatial distribution among various land cover datasets to comprehensively characterize the land cover patterns in southwestern China. LC products with different spatial resolutions cannot be directly compared for consistency. It is necessary to resample the product with a 10 m spatial resolution to 30 m (due to computational costs, the experiment did not utilize data with a 1 m spatial resolution). The resampling method employed the maximum area resampling approach.

Through spatial analysis overlay, a Boolean comparison was employed to generate a consistency map. Pixels with the same LC type were labeled as consistent, while those with different LC types were labeled as inconsistent, further categorized into full agreement, high agreement, and low agreement.

3. Results

3.1. Validation of LC Products

A summary of the calculated results for OA and Kappa between each map pair and LC samples are given in Figure 4 and

Table 4. Summary of OA and Kappa of eight LC maps in Southwest China.

Name	CLCD	GLCFCS	Global30	FROM_GLC	ESRI	ESA	CRLC	SINOLC
OA	85.48%	75.47%	72.78%	80.89%	84.43%	87.10%	81.85%	77.03%
KAPPA	0.74	0.58	0.54	0.65	0.72	0.76	0.68	0.51

separately. OA of the eight LC maps across Southwest China are ESA WorldCover (87.10%), CLCD (85.48%), Esri Land Cover (84.43%), CRLC (81.85%), FROM-GLC10 (80.89%), SinoLC-1 (77.03%), GLC_FCS30 (75.47%), and GlobeLand30 (72.78%) in the descending order. Among the 30 m resolution datasets, the CLCD data achieved the highest overall accuracy and Kappa coefficient, with values of 85.48% and 0.74, respectively. For the 10 m resolution datasets, the ESA WorldCover data had the highest overall accuracy and Kappa coefficient, with values of 87.10% and 0.76, respectively. As for the 1 m resolution LC data, the overall accuracy was 77.03%, and the Kappa coefficient was 0.51. In addition, we compared OA of our results with the OA reported by the source data authors. The OA for CLCD, FROM_GLC10, ESA WorldCover, and SinoLC-1 is higher than original data OA, while GLC_FCS30, GlobeLand30, Esri Land Cover, and CRLC have OA lower than that.

Subsequently, we calculated the UA and PA of the eight classes in the eight LC maps (Figure 5). Neither of the eight maps can achieve high accuracy for both UA and PA across all categories. The Forest and Water categories demonstrate high accuracy, displaying strong PA and UA. In the Forest category ESA WorldCover data achieves the highest accuracy, with both PA and UA exceeding 85%. Water is effectively classified in most datasets, with ESA WorldCover, CRLC, and GLC_FCS30 data showing notably high accuracy. Impervious performs well with GLC_FCS30 data, with both PA and UA reaching 80%. For Barren, Ice, and Wetland categories, the accuracy is relatively poor. Most datasets for Barren and Ice have significant differences between PA and UA. Wetland category accuracy is low, with GLC_FCS30 data even lacking wetland classes in Southwest China.

3.2. Area Comparison of the Eight Classes

The area of eight classes of all LC maps was calculated, excluding SinoLC-1, due to its significantly higher computational cost compared to other datasets (

Table 5. Area and error range (95% CI) for each land cover category in the southwestern region land cover product. All area estimates are provided in units of ten thousand hectares.

Class	CLCD	GLC_FCS30	Globaland30	FROM_GLC	Esri Land Cover	ESA World Cover	CRLC
Forest	8953 ± 256	9379 ± 305	8945 ± 337	8408 ± 285	8719 ± 269	8697 ± 264	9025 ± 336
Grassland	2089 ± 187	2046 ± 247	2098 ± 254	1931 ± 218	2264 ± 235	2322 ± 231	2441 ± 262
Cropland	2095 ± 228	1725 ± 241	1989 ± 251	2068 ± 243	2018 ± 222	1888 ± 191	2532 ± 255
Impervious	280 ± 105	255 ± 80	272 ± 82	252 ± 102	328 ± 120	265 ± 80	295 ± 97
Barren	113 ± 64	109 ± 74	123 ± 75	138 ± 77	163 ± 69	132 ± 81	119 ± 75
Ice	22 ± 10	13 ± 7	31 ± 11	37 ± 18	42 ± 13	23 ± 15	40 ± 15
Water	112 ± 19	103 ± 17	157 ± 23	122 ± 29	146 ± 29	131 ± 21	159 ± 30
Wetland	11 ± 11	11 ± 13	27 ± 8	30 ± 19	50 ± 26	19 ± 11	35 ± 17

and Figure 6).

In Southwest China, the areas of various LC categories show notable inconsistencies. Among all categories, the area of the Forest category is the closest, with GLC_FCS30 having the largest coverage. The Grassland and Cropland categories exhibit relatively small differences among all categories. In Grassland categories, CRLC has the largest area, while FROM_GLC10 has the smallest area. In Cropland categories, CRLC has the largest area, and GLC_FCS30 has the smallest area. However, there is quite a difference in the Ice and Wetland categories, even though they cover the smallest proportion, collectively making up less than 1%. E.g., the area of Ice in Esri Land Cover is 3.16 times than that of GLC_FCS30, while the area of Wetland in Esri Land Cover is 4.43 times than that of GLC_FCS30. GLC_FCS30 depicts more areas in Forest categories than the other datasets, resulting in lower calculated areas for other categories. In terms of standard errors for total area among various products, CLCD has the smallest, followed by ESA WorldCover. This indicates better performance of these land cover products.

3.3. Spatial Distribution Difference Comparison

Figure 7 illustrates the spatial consistency of different LC products in Southwest China. Generally, high spatial consistency is more prevalent in typical homogeneous regions, such as vast grasslands in the Western Sichuan Plateau and cropland and impervious in the Sichuan Basin. In contrast, low spatial consistency is more widely distributed in the complex terrain of Southwest China. For example, for the Yunnan–Guizhou Plateau, intense crustal movements result in its rugged topography. The Guangxi Zhuang Autonomous Region, as the region with the largest area of artificial forests in the Southwest, has frequent human activities, and the distribution of land categories is extremely scattered, resulting in highly intense LUCC. In the northeastern and southeastern parts of Chongqing, the dual impact of complex topography and frequent cloud cover poses challenges to LC map production. To provide a more intuitive understanding of spatial consistency, we selected Caoshang in Chongqing city for an in-depth display of land cover patterns in this typical region. In this region, characterized by a mixture of Forest, Impervious, and Cropland, CLCD, GLC_FCS30, and CRLC exhibit similar LC patterns. Esri Land Cover shows a greater extent of Impervious, while GlobeLand30 shows a higher proportion of Cropland in this area.

4. Discussion

4.1. Comparison and Analysis of Accuracy

Generally, the accuracy of the eight LC datasets is good, the value of OA are large than 70%, and the accuracy metrics show consistency. In the 30 m resolution data, CLCD data have the highest overall accuracy and Kappa coefficient. In the 10 m resolution data, ESA World Cover data have the highest overall accuracy and Kappa coefficient. Additionally, the producer accuracy and user accuracy of these two datasets also reach high levels. Liu [60] examined five datasets in the South China region in 2020, including CCI-LC, MCD12Q1, GlobeLand30, GlobCover, and CGLS-LC. The five products were resampled to a resolution of 1 km, yielding consistency results similar to this study. During area comparison, relying solely on area weighting and total area to calculate the area of each category often introduces biases. In this study, error-adjusted estimates of area and confidence intervals were calculated, providing more robust results. Notably, our assessment result, focused specifically on Southwest China, partially revealed lower accuracy compared to the national-scale assessment provided by the authors (Figure 4). This highlights the importance of validating global products at regional scales. This warrants caution when utilizing the product for localized assessments in these areas where LC is fragmented, given higher challenges in classification.

There are various reasons for the discrepancies among different LC data, which may include the difficulty in different LC categories, classification algorithms, as well as considerations related to the spectral, spatial, and temporal resolutions of the datasets. In terms of LC categories, the classification accuracy for Forest and Water bodies is higher. This may be attributed to the distinct spectral characteristics of these two LC types, coupled with their continuous and extensive distribution in the study area, making their boundaries easily identifiable in remote sensing imagery. On the other hand, the accuracy of the classification results for bare land, ice/snow, and wetland is lower. The similarity of these land cover types to other categories with confusing spectral and textural features may give rise to this discrepancy. Particularly in regions characterized by complex land classes and heterogeneity, accurately identifying these types becomes more challenging. Regarding the method for generating LC products, machine learning remains the mainstream approach. FROM-GLC, GLC_FCS30, CLCD, and ESA WorldCover employ a machine learning method known as random forest method. GlobeLand30 utilizes the POK strategy. Due to the different classification methods used, GlobeLand30 exhibits a significantly higher classification of cultivated land compared to other LC products (Figure 7). In terms of data sources, the OA of LC data derived from time-series remote sensing images surpasses that obtained from single-period images. LC data produced using Sentinel data as the data source demonstrate high accuracy. For example, ESA WorldCover integrates data from the Sentinel-1 radar satellite and the Sentinel-2 optical satellite. Radar Imagery is less affected by clouds and rain, performing well in regions with frequent cloud cover, such as the Southwest China.

Each LC map should take various factors into account early in the generation process, including not only accuracy but also classification categories, mapping time, mapping area, and more. Researchers should pay more attention to choosing LC data that suit their specific needs.

4.2. The Advantage and Limitation of Sampling Method

Compared to simple random sampling, sampling methods that combine proportional allocation and stratified random sampling better cater to the quantity of rare LC samples, taking into account the spatial heterogeneity of the land surface and the scale differences among various products. At the end, we obtained 9579 sample points. The number of sample points satisfy the minimum sample size requirement. For regions with highly unbalanced LC category areas, it is worth considering improvements to the sampling method. In terms of sample location selection, additional reference data can be utilized to reduce the sample size while maintaining relatively objective results.

In addressing potential sources of error, despite using multi-source remote sensing data as references, the confidence in obtaining sample points is low due to the scarcity of high-quality images of the Sichuan Basin (Figure 1a). Although the research team conducted field observations, the obtained sample quantity constitutes a relatively small proportion of the total samples. Future efforts should incorporate additional measures to bolster confidence in the samples, such as collecting more reference data or conducting further field investigations. While adhering to globally recognized sampling methods, there are still some shortcomings in our sample point selection. For example, the impact of sample location poses a challenging problem that is not easily quantifiable. In the sampling of LUCC, attention is focused not only on the accuracy of classification but also on the rationality of the classification boundaries. Due to spatial heterogeneity, these boundaries are inherently fuzzy. Designing evaluation metrics becomes a crucial consideration. In this study, the approach used involves sampling from sufficiently large and uniform land surfaces. However, based on the literature review, there are alternative approaches, such as using semivariogram functions to assess the spatial representativeness of measurement points, which is a consideration that precedes our research.

4.3. Reference of Regional-Scale LC Products

Accurate monitoring of LCC is pivotal in developing science-based strategies for managing conservation needs and promoting sustainable development. Many environmental problems in Southwest China, such as the continuous expansion of rocky desertification, have hindered the increase in carbon sinks and severely limited the sustainable development of the region [61]. In recent years, China has implemented various forest protection policies in Southwest China, effectively preserving the existing forest ecosystem. This ecosystem plays a crucial role in supporting our carbon-neutral policy. These policies’ development and maintenance rely on precise LC data for guidance. Specifically, understanding the accuracy of land overlay data in Southwest China greatly helps us to obtain LCC signal. We advocate that these effects should be considered when developing land-use policies. For example, the huge impact of these bias corrections on China’s carbon dynamic assessment since 1900 proves [31]. Therefore, emphasizing accurate descriptions of LCC becomes paramount in carbon budget accounting, biodiversity assessment, and ecosystem services evaluation.

This study validates the accuracy of existing medium- to high-resolution LC data in Southwest China, with the goal of aiding scholars in selecting appropriate data for their research. Each LC map should take various factors into account early in the generation process, including not only accuracy but also classification categories, mapping time, mapping area, and more. Researchers should consider not only the accuracy of products but also comprehensively choose LC data that suit their specific needs.

5. Conclusions

In this study, we synthesized the findings from validating and comparing eight LC datasets. By employing a stratified random sampling method, the research offers a detailed assessment of the accuracy and area estimates of these LC products.

The result shows the SRS method to obtain sampling points for validating existing medium- to high-resolution LC data. Based on the map classes, we stratified the classification, allocate samples to each stratum through proportional assignment, aiming to minimize the standard errors associated with accuracy estimates for each class. The accuracy of the eight LC datasets was relatively high, with ESA WorldCover being the highest among them. There was also a significant difference between UA and PA. Among them, the “Forest” and “Water” categories performed well among all classes, while the “Barren”, “Ice”, and “Wetland” categories exhibited relatively lower accuracy. The area of each category was determined based on classification estimates derived from reference data, using unbiased or consistent accuracy and area estimators to determine the area for each category.

In this research, the different spatial resolutions of land cover datasets are evaluated by different indicators such as OA, PA, UA, and Kappa coefficient. Our results provide a guide for users to selected the most suitable land cover dataset based on various research topics in Southwest China. Moreover, fully leveraging the results of accuracy assessment and consistency evaluation, combining the strengths of multiple sources of LC data products for data fusion is crucial in providing a comprehensive and more accurate LC data product for regional studies. This is an important aspect of future work.