Accuracy Assessment of GlobeLand30 2010 Land Cover over China Based on Geographically and Categorically Stratified Validation Sample Data

Abstract

Land cover information is vital for research and applications concerning natural resources and environmental modeling. Accuracy assessment is an important dimension in use and production of land cover information. GlobeLand30 is a relatively new global land cover information product with a fine spatial resolution of 30 m and is potentially useful for many applications. This paper describes the methods for and results from the first country-wide and statistically based accuracy assessment of GlobeLand30 2010 land cover dataset over China. For this, a total of 8400 validation sample pixels were collected based on a sampling design featuring two levels of stratification (ten geographical regions, each with nine or eight land-cover classes). Validation sample data with reference class labels were acquired from visual interpretation based on Google Earth high-resolution satellite images. Error matrices for individual regions and entire China were estimated properly based on the sampling design adopted, with the former aggregated to get the latter through suitable weighting. Results were obtained, with agreement at a sample pixel defined both as a match between the map (class) label and either the primary or alternate reference label therein and, more strictly, as a match between the map label and the primary reference label only. Based on the former definition of agreement, the overall accuracy of GlobeLand30 2010 land cover for China was assessed to be 84.2%. User’s accuracy and producer’s accuracy were both greater than 80% for cultivated land, forest, permanent snow and ice, and bareland, with user’s accuracy for water bodies estimated 94.2% (82.1% for wetland, 79.8% for artificial surface) and producer’s accuracy for grassland estimated 89.0%. These indicate that GlobeLand30 2010 depicts land cover circa 2010 in China quite accurately, although estimates of accuracy indicators based on the latter definition of agreement were lower as expected with an estimated national overall accuracy of 81.0%. Regional and class variations in accuracy were revealed and examined in the light of their associations with land cover distributions and patterns. Implications for use and production of GlobeLand30 land cover information were discussed, so were commonality and lack of it between GlobeLand30 and other fine-resolution land cover products.

1. Introduction

As Earth observation technologies have evolved, huge amount of remote-sensing image data covering the whole globe has been acquired. Multi-scale land cover datasets are widely used in the Earth surface processes research, ecosystem assessments, environmental modeling, and sustainable development planning [1,2,3]. Several global land cover products have been developed by national and international organizations during the past decades. These include, for example, the International Geosphere-Biosphere Program Data and Information System’s land cover (IGBP DISCover) data product (1 km spatial resolution) [4], the University of Maryland (UMD) land cover maps (1 km resolution) [5], the Global Land Cover 2000 (GLC2000) maps from the European Commission’s Joint Research Center (JRC) (1 km resolution) [6], the Moderate Resolution Imaging Spectroradiometer (MODIS) land cover maps (500/1000 m resolution) [7], the GlobCover land cover maps from European Space Agency (ESA), 2005–2006/2009 (300 m resolution) [8,9], the Climate Change Initiative Land Cover maps (CCI-LC) from ESA, 2000/2005/2010 (300m resolution) [10], and the Global Land Cover by National Mapping Organizations (GLCNMO) maps, 2003/2008/2013 (500/1000 m resolution) [11]. Recently, GlobeLand30 land cover datasets have been developed by the National Geomatics Center of China, based on integration of pixel- and object-based classification followed by knowledge-based interactive verification of classification results (shortly known as the POK-based approach) [12]. This product has been available worldwide since September 2014 [12]. In comparison with global land cover products previously developed, GlobeLand30 has the finest spatial resolution of 30 m and thus can provide information, in greater details, about land cover status and dynamics.

The importance of accuracy assessment is increasingly recognized. Accuracy assessment helps map users to evaluate the utility of land cover maps for their intended applications [13]. It also informs map productions so that map quality may be improved through adopting more robust classifiers, exploring more informative class-discriminant features, incorporating extra image data or ancillary data, or a combination of these.

To verify GlobeLand30’s accuracy globally, a preliminary assessment was conducted based on a two-rank sampling strategy. In first-rank (first-stage) sampling, map sheets were selected globally, while in second-rank (second-stage) sampling sample data for each land cover type within each of the selected map sheets were collected [14]. Eighty sample map sheets were selected from a total of 847 map sheets in the first-rank sampling. A total number of 159,874 sample pixels were selected for the assessment of GlobeLand30 2010. The overall accuracy of the product was estimated about 80.3 ± 0.2% [15].

Regional and country-wide accuracy assessments for GlobeLand30 have also been carried out by researchers from many countries. Overall accuracies were estimated about 86% for Thessaly, Greece [16], 87% for Siberia [17], 90% for Kyiv Oblast, Ukraine [18], 78% for Iran [19], 59% for Kenya [20], 46% for Central Asia [21], 80% for Nepal [22], 92% for land surface water in Greece [23], and 80% for cropland in three continents with high risk of food insecurity [24]. Validation of GlobeLand30 was done also through comparisons with existing land cover products in Germany [25], Italy [26], Portugal [27] and East Africa [28], with rates of agreement ranging from 74% to 93%.

However, there have been few accuracy assessment efforts for GlobeLand30 in China. Below, we reviewed relevant results from country-wide, thematic (e.g., cropland), and regional assessments, respectively. To compare and assess seven global land cover datasets (including GlobeLand30) in China, Yang et al. [29] manually collected five sets of multi-scale validation sample units (VSUs), with spatial resolution ranging from 600 m × 600 m to 2 km × 2 km. It was found that GlobeLand30 2010 was the most accurate datasets examined, with an overall accuracy of 82.4% reported based on 1063 validation sample units (with 600 m resolution, implying coarsening for GlobeLand30 and exclusion of sample units at heterogeneous locations), while the overall accuracies of the other datasets ranged from 33.9% to 67.2% [29].

Lu et al. [30] compared five global cropland datasets in China for the year 2010. Cropland census data at the provincial and regional scales were used to evaluate the cropland areas derived from these datasets, while 5704 validation sample units (including reference data at 2130 test sample units originally acquired for validating the Finer Resolution Observation and Monitoring of Global Land Cover (FROM-GLC) dataset by Gong et al. [31]) were utilized to verify locations of cropland parcels. The results showed that GlobeLand30 datasets are the most accurate for cropland area estimation, with an overall accuracy of 79.6%.

Regional accuracy assessments of GlobeLand30 were also undertaken in China. For example, accuracy assessments of GlobeLand30 2010 in Shaanxi Province and Henan Province indicated overall accuracies of 80.0% and 81.5%, respectively [32,33].

Clearly, validation results in [30] and [32,33] need to be extended in terms of thematic and geographic coverages, respectively. For the country-wide assessment reported by Yang et al. [29], sampling was not probability-based, as admitted. Besides, the assessment was not done in its nominal (spatial) resolution of 30 m but in a coarsened resolution of 600 m. This country-wide assessment would be usefully strengthened with respect to statistical soundness and resolution refinement to enable inter-comparisons between classification accuracies of products with comparable resolution.

We should look for theoretical guidance and good practice examples regarding accuracy assessment from the literature. For example, the National Land Cover Database (NLCD) series products (also of 30 m resolution), which were developed by the MultiResolution Land Characteristics (MRLC) Consortium (www.mrlc.gov), provide consistent land cover information in the United States from decadal Landsat satellite imagery and other supplementary datasets [34]. Accuracy assessments were carried out for NLCD land cover following statistical framework and procedures well established as in [35,36,37,38,39]. To facilitate sustainable research and developments concerning use and production of GlobeLand30 datasets globally and in China, in particular, country-wide assessments of GlobeLand30 over China should be pursued based on well established procedures of sampling design, response design, and analyses. This is what this paper seeks to contribute to.

The remainder of the paper is organized as follows. Section 2 describes the methods for accuracy assessment of GlobeLand30 land cover in China. Section 3 reports accuracy assessment results obtained for different regions and for entire China. In Section 4, the main results are discussed, with patterns of misclassifications summarized, followed by conclusion in Section 5.

2. Methods

In GlobeLand30 maps, ten land-cover classes (as defined in

Table 1. Classification, codes, and definition of each land cover type of GlobeLand30.

Code	Type	Definition
10	Cultivated land	Land used for agriculture, horticulture and gardens, including paddy fields, irrigated and dry farmland, vegetable and fruit gardens, etc.
20	Forest	Land covered by trees, vegetation covers over 30%, including deciduous and coniferous forests, and sparse woodland with cover 10–30%, etc.
30	Grassland	Land covered by natural grass with cover over 10%, etc.
40	Shrubland	Land covered by shrubs with cover over 30%, including deciduous and evergreen shrubs, and desert steppe with cover over 10%, etc.
50	Wetland	Land covered by wetland plants and water bodies, including inland marsh, lake marsh, river floodplain wetland, forest/shrub wetland, peat bogs, mangrove and salt marsh, etc.
60	Water bodies	Water bodies in land area, including river, lake, reservoir, fish pond, etc.
70	Tundra	Land covered by lichen, moss, hardy perennial herb and shrubs in the polar regions, including shrub tundra, herbaceous tundra, wet tundra, and barren tundra, etc.
80	Artificial Surfaces	Land modified by human activities, including all kinds of habitation, industrial and mining area, transportation facilities, and interior urban green zones and water bodies, etc.
90	Bareland	Land with vegetation cover lower than 10%, including desert, sandy fields, Gobi, bare rocks, saline and alkaline land, etc.
100	Permanent snow and ice	Lands covered by permanent snow, glacier and icecap.

) are depicted across the world at a nominal pixel resolution of 30 m, in accordance with a hierarchical classification method featuring pixel classification, object-based post-processing, and knowledge-based interactive verification [15]. In this paper, the accuracy assessment of GlobeLand30 2010 was performed by following the guidance suggested by leading scientists in the field, such as Stehman et al. [40] and Olofsson et al. [41]. According to these authors, three major components of the accuracy assessment are sampling design, response design, and analysis, which are described in this section.

In this paper, the spatial units for sampling and analysis concerning accuracy assessment are pixels. Although there is debate over the most appropriate sample unit for accuracy assessment, Stehman et al. [42] suggested that using pixels as sample units could avoid many of the complications that might arise when other spatial support units (e.g., blocks of pixels and polygons) were used. Use of pixels as units for sampling and analysis is a well-established practice in the validation of NLCD land cover information [35,36,37,38,39].

2.1. Sampling Design

It may not be feasible to compare GlobeLand30 maps with reference data of complete coverages even for a county let alone whole China (with 34 provincial administrative regions) pixel by pixel. Sampling design determines both the cost and statistical rigor of accuracy assessment [43]. By sampling design, we select certain number of locations at which reference data will be collected. In this paper, two-level stratified random sampling is adopted. The first level of stratification is based on partitioning Chinese administrative regions (including the regions of Hong Kong, Macao, and Taiwan, but with small islands and the seas excluded) into 10 geographical regions (Figure 1). The regional stratification helps to reduce standard errors in accuracy estimates, facilitate regional reporting of accuracy, and provide an indication of how accuracy varies spatially across China. The second level of stratification is by GlobeLand30 map classes (Table 1 and Figure 1).

In terms of sample size, each class in each region is allocated 100 sample pixels. The statistical explanation is that 100 sample pixels per stratum results in an expected standard error of 0.05 for simple random sampling within a stratum, if the true user’s accuracy is 50% with a confidence level of 95% [35]. It should be noted that the tundra area in China is almost zero and it was not included in the validation. In addition, permanent snow and ice is only distributed in R1–R4. Thus, in total, 8400 sample pixels were collected for the accuracy assessment, as shown in

Table 2. The regional distributions of the mapped land-cover type (percent of area) for GlobeLand30 2010 over China. The sample size is 100 for all land-cover types in all regions with the exception that permanent snow and ice is not sampled in R5–R10.

Type	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10
Cultivated land	5.42	0.36	13.09	28.51	12.01	27.19	45.01	57.69	39.97	43.71
Forest	1.15	8.45	10.46	45.65	11.55	61.22	42.27	16.81	44.63	37.62
Grassland	21.95	64.28	45.96	20.82	47.75	5.79	4.95	15.52	4.42	12.03
Shrubland	0.83	1.95	1.06	2.87	0.83	0.95	0.06	0.07	0.11	0.00
Wetland	0.29	0.18	0.40	0.39	0.52	0.02	0.33	0.31	0.45	1.36
Water bodies	0.64	2.59	1.34	0.60	0.46	2.32	2.79	1.73	4.18	1.51
Artificial Surfaces	0.29	0.03	0.55	0.59	0.77	2.49	4.56	7.81	6.18	3.09
Bareland	67.0	17.75	26.61	0.33	26.17	0.01	0.02	0.06	0.07	0.67
Permanent snow and ice	2.43	4.4	0.50	0.24	0.0	0.0	0.0	0.0	0.0	0.0
Country-wide proportion	17.2	12.68	14.56	11.89	12.08	4.71	5.94	5.55	7.08	8.31

.

2.2. Response Design

Reference land-cover labels for sample pixels are usually obtained from images of finer resolution. Online access to reference images is greatly enhanced thanks to Google Earth. GlobeLand30 2010 datasets for China are available in raster format with 54 tiles; the datasets are provided in WGS84 (World Geodetic System 1984) reference system and UTM (Universal Transverse Mercator) projection. Before selecting sample locations, the datasets need to be re-projected into the Web Mercator projection used for Google Earth. Other available datasets (e.g., Bing maps, Yahoo maps and color composites of Landsat images) were used when Google historical images of good quality were not available.

Three experienced interpreters carried out reference data acquisition. All reference data for each of individual regions were collected by a single interpreter. Interpreters had no a priori knowledge of map land-cover labels to avoid interpreter bias [36]. In addition to the primary reference land-cover labels, alternate land-cover labels should also be recorded for sample locations which it would be more appropriately labeled with both reference labels than any one alone. A nominal level of confidence was assigned to reference land-cover labels, namely “confident”, “somewhat confident”, and ”not confident”, as in Wickham et al. [36]. The acquisition dates of Google historical images (used for determining the reference land-cover classes) and satellite images (used for map production, acquired by Landsat satellites and the Chinese Environmental and Disaster satellite) were also recorded, since the satellite images used for GlobeLand30 2010 were not restricted to the year 2010 alone.

After the first round of visual image interpretation, verification of “somewhat confident” and ”not confident” reference class labels was performed by the project manager, who was typically the most experienced photo-interpreter in the team, to minimize errors in reference data. Consistency in reference label assignments within the team is also important, but precedence is given to the project manager’s assignments when label assignments disagree. Especially, great attention should be paid to “not confident” labels, which are allowed to be re-assigned to “somewhat confident” or “confident” labels. In this step, ancillary data, such as DEM data, local photos, and Landsat images could be used to assist image interpretation.

With consideration for positional uncertainty and thematic ambiguity between land-cover map and reference data, agreement is registered for a sample pixel if the map class matches either the primary or alternate reference label therein [39]. To examine the effects of surrounding pixels on accuracy and to increase the reliability of interpretation, reference land-cover labels in a 3 × 3-pixel neighborhood centered on each sample pixel were also collected.

2.3. Analysis

By cross-tabulation between map and reference classes at sample pixels, error matrix can be constructed, allowing for estimating accuracy indicators, such as overall accuracy, user’s accuracy, and producer’s accuracy [44]. An example error matrix, as shown in

Table 3. Error matrix of sample counts, $n_{ij}$.

Class	1	2	…	k	Total
1	$n_{11}$	$n_{12}$	…	$n_{1k}$	$n_1$
2	$n_{21}$	$n_{22}$	…	$n_{2k}$	$n_2$
…	…	…	…	…	…
k	$n_{k1}$	$n_{k2}$	…	$n_{kk}$	$n_k$
Total	$n_{·1}$	$n_{·2}$	…	$n_{·k}$	n

, is constructed from sample counts, where an element $n_{ij}$ represents the number of sample pixels classified as map category i (i = 1, 2, …, k; k being the total number of candidate classes) but actually verified to be reference category j (j = 1, 2, …, k). In Table 3 (and

Table 4. Error matrix of estimated area proportions, ${\widehat{P}}_{ij}$ (Equation (1)).

Class	1	2	…	k	Total
1	${\widehat{P}}_{11}$	${\widehat{P}}_{12}$	…	${\widehat{P}}_{1k}$	${\widehat{P}}_1$
2	${\widehat{P}}_{21}$	${\widehat{P}}_{22}$	…	${\widehat{P}}_{2k}$	${\widehat{P}}_2$
…	…	…	…	…	…
k	${\widehat{P}}_{k1}$	${\widehat{P}}_{k2}$	…	${\widehat{P}}_{kk}$	${\widehat{P}}_k$
Total	${\widehat{P}}_{·1}$	${\widehat{P}}_{·2}$	…	${\widehat{P}}_{·k}$	1

below), the rows indicate map classes while the columns represent reference classes.

Olofsson et al. [45] described a more informative presentation of the error matrix. It is in terms of unbiased estimators of area proportions in cells (i, j) corresponding to map-reference class label pairs i and j:

$${\widehat{P}}_{ij}=W_i\frac{n_{ij}}{n_{i·}}$$

(1)

where $W_i$ represents the proportion of the area mapped as category i (in the region under study) and is calculated as $W_i$ = $A_{m,i}$÷$A_{tot}$. $A_{tot}$ is the total area of the region under study and $A_{m,i}$ (subscript m denotes “mapped”) is the mapped area of category i. An example error matrix populated with estimated area proportions is shown in Table 4, from which accuracy measures can be computed (see Equations (2)‒(4)).

Estimating the proportion of area in each cell of the error matrix using Equation (1) takes into account the inclusion probabilities of the stratified design. An inclusion probability is defined as the probability that a particular pixel is included in the sample. Unlike in simple random and systematic sampling where inclusion probability of each selected pixel is the same (so that accuracy measures may be computed directly based on Table 3), in stratified random sampling, sample units from different strata usually have different weights, as sample units from different strata likely have different inclusion probabilities. Thus, area proportions of the map classes ($W_i$) must be incorporated in the stratified estimators of overall and producer’s accuracies to account for different sampling intensities in different strata.

Once the area proportions are estimated (Equation (1)) as in Table 4, user’s accuracy (${\widehat{U}}_i$) and producer’s accuracy (${\widehat{P}}_j$) for any category and overall map accuracy ($\widehat{O}$) are estimated directly from area proportions (Table 4). The estimators are:

$${\widehat{U}}_i=\frac{{\widehat{P}}_{ii}}{{\widehat{P}}_{i·}}$$

(2)

$${\widehat{P}}_j=\frac{{\widehat{P}}_{jj}}{{\widehat{P}}_{·j}}$$

(3)

$$\widehat{O}={\displaystyle \sum }_{j=1}^k{\widehat{P}}_{jj}$$

(4)

Variance estimators are also necessary for accuracy assessment, which could be applied to calculate confidence intervals [43]. For stratified random sampling, the estimated variance of overall accuracy is

$$V(\widehat{O})={\displaystyle \sum }_{i=1}^k{W}_i^2{\widehat{U}}_i(1-{\widehat{U}}_i)/(n_{i·}-1)$$

(5)

As for the estimated variance of user’s and producer’s accuracy, the method described by Wickham et al. [38] is usually applied. Further detail about variance estimation is provided in Appendix A.

Equations (1)‒(5) are applied for estimating accuracies on the assumption that stratified random sample data are used. For the study in this paper, a two-level stratified sampling design was adopted (Section 2.1). Thus, Equations (1)‒(5) are not direct applicable for estimating national accuracies, although they are perfectly suitable for estimating accuracies in individual regions where map-class-stratified random sampling was applied independently.

To evaluate nationwide accuracies, regional error matrices need to be aggregated to a national error matrix at first. This is done by summing up corresponding cell values at (i, j) (i.e., area proportions for map-reference class pair (i, j)) in regional error matrices via proper weights that are individual regions’ areal proportions in the entire study area. Consider the study here as an example. The ten regional error matrices were estimated using Equations (1)–(4), as listed in Appendix B. The weights ($W_h$ in Equation (6)) were the ten regions’ areal proportion relative to the whole study area (Table 2, the last row, Row “Country-wide proportion”). In terms of a formula, the aggregated proportion for a particular class pair (i, j) (${\widehat{P}}_{i,j}$) is computed as a weighted sum of corresponding proportions in regional error matrices:

$${\widehat{P}}_{i,j}={\displaystyle \sum }_{h=1}^H{W}_h{\widehat{P}}_{i,j|h}$$

(6)

where H is the total number of regional strata in the study area (H = 10 for this paper), $W_h$ is region h’s areal proportion in the whole study area ($W_h$ = $N_h$/N, with $N_h$ being the population size of region h, and N being the population size for the whole study area), and ${\widehat{P}}_{i,j|h}$ represents cell (i, j) in region h’s error matrix. These properly calculated cell values (${\widehat{P}}_{i,j}$) constitute the national error matrix. Based on it, national accuracy indicators are computed using Equations (2)–(4) by inserting properly estimated ${\widehat{P}}_{i,j}$ values (Equation (6)), as was implemented in this study.

Alternatively, country-wide accuracy estimates can be computed by following the method described in Wickham et al. [37,38]. It is the so-called combined ratio estimator based on suitably formulated indicator functions. This method is highly recommendable for computing accuracy estimates in situations where stratifications are more than one level as in this case study here and where strata are different from land cover classes for which accuracy indicators need to be estimated properly [37,46].

In Appendix A, we provide further detail about how national accuracy indicators are estimated based on aggregation of regional error matrices and the combined ratio estimator. We also show the equivalence of these two kinds of methods with explanations and using data from the study reported in this paper.

3. Results

In this section, we mainly report results of accuracy assessment based on the less strict definition of agreement (at a sample pixel) as a match between the map label and either the primary or alternate reference label. Assessment results obtained using the stricter definition of agreement (as a match between the map label and the primary reference label only) are reported briefly at the end section.

The ten regional error matrices were estimated (using the less strict definition of agreement unless stated otherwise in the remainder of this section), as shown in Appendix B (

Table A1. The error matrix of estimated area proportions for R1: Overall accuracy is 90.3% (2.0%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	4.9	0	0.2	0.1	0	0	0.1	0.1	0	5.4	91(3)
Forest	0	0.8	0.1	0.2	0	0	0	0	0	1.2	68(5)
Grassland	0	0.4	19.3	1.3	0.2	0	0	0.4	0.2	21.9	88(3)
Shrubland	0	0	0.2	0.5	0	0	0	0.1	0	0.8	57(5)
Wetland	0	0	0	0	0.3	0	0	0	0	0.3	86(3)
Water	0	0	0	0	0	0.6	0	0	0	0.6	92(3)
ArS	0	0	0	0	0	0	0.2	0	0	0.3	76(4)
Bareland	0	0	4	1.3	0	0	0	61.6	0	67	92(3)
PSI	0	0	0.1	0	0	0	0	0.1	2.1	2.4	87(3)
Total	5	1.3	24.1	3.5	0.6	0.6	0.3	62.3	2.4	100
PA	99(0)	63(16)	80(5)	14(4)	44(17)	100(0)	67(15)	99(1)	90(8)

,

Table A2. The error matrix of estimated area proportions for R2: Overall accuracy is 82.0% (2.7%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	0.2	0	0	0	0	0	0	0	0	0.4	64(5)
Forest	0	7.5	0.5	0.4	0	0	0	0	0	8.5	89(3)
Grassland	0	0.6	52.1	7.1	1.9	0	0	2.6	0	64.3	81(4)
Shrubland	0	0.1	0.4	1.3	0.1	0	0	0	0	2	68(5)
Wetland	0	0	0	0	0.2	0	0	0	0	0.2	85(4)
Water	0	0	0	0	0.1	2.5	0	0	0	2.6	97(2)
ArS	0	0	0	0	0	0	0	0	0	0	59(5)
Bareland	0	0.2	2.7	0.2	0.2	0	0	14.4	0.2	17.8	81(4)
PSI	0	0.1	0.3	0.1	0	0	0	0.1	3.8	4.4	87(3)
Total	0.2	8.5	55.9	9.2	2.4	2.5	0	17.1	4	100
PA	99(0)	88(7)	93(1)	14(3)	6(3)	99(0)	46(11)	84(6)	95(4)

,

Table A3. The error matrix of estimated area proportions for R3: Overall accuracy is 85.9% (2.0%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	11.3	0.4	0.7	0.4	0	0	0.4	0	0	13.1	86(3)
Forest	0.1	8.7	0.6	1	0	0	0	0	0	10.5	83(4)
Grassland	0	0.5	40	2.8	0.9	0	0	1.8	0	46	87(3)
Shrubland	0	0	0.2	0.7	0	0	0	0	0	1.1	70(5)
Wetland	0	0	0	0	0.4	0	0	0	0	0.4	90(3)
Water	0	0	0	0	0	1.3	0	0	0	1.3	97(2)
ArS	0.1	0	0	0	0	0	0.4	0	0	0.6	78(4)
Bareland	0	0	2.9	0.5	0.5	0	0	22.6	0	26.6	85(4)
PSI	0	0	0	0	0	0	0	0	0.5	0.5	98(1)
Total	11.5	9.6	44.4	5.5	1.9	1.3	0.8	24.5	0.5	100
PA	98(1)	91(5)	90(2)	14(3)	20(8)	100(0)	52(14)	92(3)	96(3)

,

Table A4. The error matrix of estimated area proportions for R4: Overall accuracy is 76.0% (2.3%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	22.8	2.6	0	2	0.3	0.3	0.6	0	0	28.5	80(4)
Forest	0.5	37.9	0.9	5.5	0	0	0	0.9	0	45.7	83(4)
Grassland	1.2	1.9	11.2	5	0.4	0	0.2	0.8	0	20.8	54(5)
Shrubland	0.1	0.3	0.1	2.3	0	0	0	0	0	2.9	80(4)
Wetland	0	0	0	0	0.4	0	0	0	0	0.4	90(3)
Water	0	0	0	0	0	0.6	0	0	0	0.6	95(2)
ArS	0.1	0	0	0	0	0	0.4	0	0	0.6	75(4)
Bareland	0	0	0	0	0	0	0	0.2	0	0.3	67(5)
PSI	0	0	0	0	0	0	0	0	0.2	0.2	81(4)
Total	24.7	42.7	12.4	14.8	1.1	0.9	1.3	2	0.2	100
PA	92(3)	89(2)	91(5)	15(2)	33(13)	64(21)	35(13)	11(4)	92(3)

,

Table A5. The error matrix of estimated area proportions for R5: Overall accuracy is 87.7% (2.0%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	11.2	0	0.2	0.2	0	0	0.2	0.1	–	12	93(3)
Forest	0	11.1	0.1	0.3	0	0	0	0	–	11.5	96(2)
Grassland	2.9	0.5	38.7	2.4	0	0.5	0	2.9	–	47.8	81(4)
Shrubland	0.1	0	0.2	0.5	0	0	0	0	–	0.8	59(5)
Wetland	0	0	0	0	0.5	0	0	0	–	0.5	90(3)
Water	0	0	0	0	0	0.4	0	0	–	0.5	93(3)
ArS	0.1	0	0	0	0	0	0.6	0	–	0.8	80(4)
Bareland	0	0	0.8	0.5	0	0	0	24.8	–	26.1	95(2)
PSI	–	–	–	–	–	–	–	–	–	–	–
Total	14.2	11.6	40.1	4	0.5	0.9	0.9	27.9	–	100
PA	79(6)	96(4)	97(1)	12(4)	97(2)	47(25)	72(14)	89(4)	–

,

Table A6. The error matrix of estimated area proportions for R6: Overall accuracy is 77.4% (2.6%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	19	3.8	0.5	1.4	0.8	0.5	0.5	0.5	–	27.2	70(5)
Forest	1.2	51.4	1.2	6.1	0	0.6	0.6	0	–	61.2	84(4)
Grassland	0.5	1.3	2	1.6	0	0	0.3	0	–	5.8	35(5)
Shrubland	0	0.1	0	0.7	0	0	0	0	–	1	78(4)
Wetland	0	0	0	0	0	0	0	0	–	0	67(5)
Water	0	0	0.1	0	0.1	2	0	0	–	2.3	87(3)
ArS	0.1	0.1	0	0.1	0	0	2.2	0	–	2.5	87(3)
Bareland	0	0	0	0	0	0	0	0	–	0	56(5)
PSI	–	–	–	–	–	–	–	–	–	–	–
Total	21	56.7	3.9	9.9	0.9	3.2	3.7	0.5	–	100
PA	91(4)	91(2)	52(13)	7(2)	2(1)	63(14)	58(12)	1(1)	–

,

Table A7. The error matrix of estimated area proportions for R7: Overall accuracy is 78.1% (2.5%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	34.2	1.8	0	2.3	1.4	0.9	4.5	0	–	45	76(4)
Forest	1.3	35.9	1.7	3.4	0	0	0	0	–	42.3	85(4)
Grassland	0.7	0.8	2	1	0	0	0.2	0	–	5	41(5)
Shrubland	0	0	0	0	0	0	0	0	–	0.1	78(4)
Wetland	0	0	0	0	0.3	0.1	0	0	–	0.3	81(4)
Water	0.1	0	0	0	0.1	2.6	0	0	–	2.8	92(3)
ArS	1.2	0.2	0	0	0	0	3.1	0	–	4.6	67(5)
Bareland	0	0	0	0	0	0	0	0	–	0	54(5)
PSI	–	–	–	–	–	–	–	–	–	–	–
Total	37.6	38.8	3.7	6.7	1.7	3.6	7.8	0.1	–	100
PA	91(2)	93(2)	54(13)	1(0)	15(7)	72(13)	39(7)	21(17)	–

,

Table A8. The error matrix of estimated area proportions for R8: Overall accuracy is 82.4% (2.2%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	50.8	1.7	2.3	0	0	0.6	1.7	0.6	–	57.7	88(3)
Forest	0.7	13.8	0.7	1.7	0	0	0	0	–	16.8	82(4)
Grassland	1.7	0.6	9.6	3.4	0	0	0.2	0	–	15.5	62(5)
Shrubland	0	0	0	0	0	0	0	0	–	0.1	67(5)
Wetland	0	0	0	0	0.3	0	0	0	–	0.3	85(4)
Water	0	0	0	0	0.1	1.6	0	0	–	1.7	92(3)
ArS	0.5	0.4	0.2	0.2	0.1	0	6.3	0	–	7.8	81(4)
Bareland	0	0	0	0	0	0	0	0	–	0.1	37(5)
PSI	–	–	–	–	–	–	–	–	–	–	–
Total	53.7	16.5	12.8	5.4	0.4	2.2	8.2	0.6	–	100
PA	95(1)	83(5)	75(7)	1(0)	60(12)	73(19)	77(9)	4(4)	–

,

Table A9. The error matrix of estimated area proportions for R9: Overall accuracy is 84.0% (2.1%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	32.8	3.2	1.2	0.4	0	1.2	1.2	0	–	40	82(4)
Forest	0.4	40.2	0.4	3.1	0	0.4	0	0	–	44.6	90(3)
Grassland	0.8	0.8	1.5	0.7	0.1	0.2	0.2	0	–	4.4	35(5)
Shrubland	0	0	0	0.1	0	0	0	0	–	0.1	53(5)
Wetland	0	0	0	0	0.3	0.1	0	0	–	0.5	63(5)
Water	0.1	0	0	0	0.1	4	0	0	–	4.2	95(2)
ArS	0.7	0.2	0	0	0	0.1	5.2	0.1	–	6.2	84(4)
Bareland	0	0	0	0	0	0	0	0	–	0.1	42(5)
PSI	–	–	–	–	–	–	–	–	–	–	–
Total	34.9	44.4	3.2	4.3	0.5	6	6.6	0.1	–	100
PA	94(1)	90(2)	48(13)	1(0)	56(11)	66(9)	78(8)	31(22)	–

and

Table A10. The error matrix of estimated area proportions for R10: Overall accuracy is 88.0% (1.6%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	40.6	0.4	0.9	0.9	0	0	0.9	0	–	43.7	93(3)
Forest	0	35.4	0.4	1.9	0	0	0	0	–	37.6	94(2)
Grassland	1.9	3.2	6.5	0	0	0	0.4	0	–	12	54(5)
Shrubland	0	0	0	0	0	0	0	0	–	0	69(5)
Wetland	0.3	0	0	0	1	0.1	0	0	–	1.4	73(4)
Water	0	0	0	0	0.1	1.4	0	0	–	1.5	94(2)
ArS	0.4	0.1	0	0	0	0	2.6	0	–	3.1	84(4)
Bareland	0	0	0.1	0	0	0	0	0.5	–	0.7	72(5)
PSI	–	–	–	–	–	–	–	–	–	–	–
Total	43.3	39.1	7.9	2.8	1.1	1.5	3.9	0.5	–	100
PA	94(1)	90(2)	82(8)	0(0)	91(3)	93(3)	67(11)	100(0)	–

). Overall accuracies based on estimated population class area proportions (Equation (4)) are shown in Figure 2 (also shown in

Table 5. Regional user’s accuracies for nine land cover classes with standard errors (SE) in parentheses. Agreement is defined as a match between the map class and either the primary or alternate reference class (Cultivated land is abbreviated as CuL, Artificial Surfaces is abbreviated as ArS, Permanent snow and ice is abbreviated as PSI; – means there is no permanent snow and ice in a given region; Column “National” represents countrywide user’s accuracies).

	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	National
CuL	91(3)	64(5)	86(3)	80(4)	93(3)	70(5)	76(4)	88(3)	82(4)	93(3)	84(1)
Forest	68(5)	89(3)	83(4)	83(4)	96(2)	84(4)	85(4)	82(4)	90(3)	94(2)	87(1)
Grassland	88(3)	81(4)	87(3)	54(5)	81(4)	35(5)	41(5)	62(5)	35(5)	54(5)	78(2)
Shrubland	57(5)	68(5)	70(5)	80(4)	59(5)	78(4)	78(4)	67(5)	53(5)	69(5)	70(2)
Wetland	86(3)	85(4)	90(3)	90(3)	90(3)	67(5)	81(4)	85(4)	63(5)	73(4)	82(2)
Water bodies	92(3)	97(2)	97(2)	95(2)	93(3)	87(3)	92(3)	92(3)	95(2)	94(2)	94(1)
ArS	76(4)	59(5)	78(4)	75(4)	80(4)	87(3)	67(5)	81(4)	84(4)	84(4)	80(2)
Bareland	92(3)	81(4)	85(4)	67(5)	95(2)	56(5)	54(5)	37(5)	42(5)	72(5)	90(2)
PSI	87(3)	87(3)	98(1)	81(4)	–	–	–	–	–	–	88(2)
Overall	90(2)	82(3)	86(2)	76(2)	88(2)	77(3)	78(3)	82(2)	84(2)	88(2)

, the last row, Row “Overall”).

The overall accuracies of ten regions range from 76.0% to 90.3%, while standard errors for estimated overall accuracy range between 1.6% and 2.6%, as shown in Figure 2. Most regions have overall accuracies over 80%, except for three regions (R4, R6 and R7). Overall accuracies in these three regions are about 8.6% lower than that of the other seven regions. Geographically, overall accuracies of northern regions (i.e., R1, R3, R5, and R10) rank the top four, with the three southern regions (R4, R6, and R7) ranking bottom three, while the rest (R2, R8, and R9) are in the middle. These differences are related to various factors, such as class composition, classification system, and landscape heterogeneity. In terms of land cover compositions, land cover classes are unequally distributed across China, as shown in Table 2. Based on GlobeLand30 2010 maps, R1–R3 and R5 are dominated by grassland and bareland, accounting for 89%, 82%, 73%, and 74% of their total areas, respectively. R4 and R6–R10 are dominated by cultivated land and forest, accounting for 74%, 88%, 87%, 74%, 85%, and 81% of their total areas, respectively. Among them, bareland in R1 occupies as much as 67% of its whole area. The top three classes with the largest areas in each of the ten regions account for at least 85% of the corresponding region’s total areas.

Cultivated land, forest, wetland, water bodies, and permanent snow and ice register user’s accuracies greater than 82%, as shown in Table 5. Grassland and bareland have higher user’s accuracy in regions where these classes are dominant (R1–R3 and R5), while their user’s accuracies are much decreased in the other six regions because of class rarity. Shrubland and artificial surfaces, which account for small area proportions of China (1.0% and 1.8%, respectively), have medium-level user’s accuracies (Table 5).

As for producer’s accuracy, cultivated land, forest, water bodies, permanent snow and ice are usually classified with good accuracy, as shown in

Table 6. Regional producer’s accuracies for nine land cover classes with standard errors (SE) in parentheses. Agreement is defined as a match between the map class and either the primary or alternate reference class (see Table 5 for meanings of CuL, ArS, PSI, and –, and Column “National” represents countrywide producer’s accuracies).

	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	National
CuL	99(0)	99(0)	98(1)	92(3)	79(6)	91(4)	91(2)	95(1)	94(1)	94(1)	93(1)
Forest	63(16)	88(7)	91(5)	89(2)	96(4)	91(2)	93(2)	83(5)	90(2)	90(2)	90(1)
Grassland	80(5)	93(1)	90(2)	91(5)	97(1)	52(13)	54(13)	75(7)	48(13)	82(8)	89(1)
Shrubland	14(4)	14(3)	14(3)	15(2)	12(4)	7(2)	1(0)	1(0)	1(0)	0(0)	11(1)
Wetland	44(17)	6(3)	20(8)	33(13)	97(2)	2(1)	15(7)	60(12)	56(11)	91(3)	30(5)
Water bodies	100(0)	99(0)	100(0)	64(21)	47(25)	63(14)	72(13)	73(19)	66(9)	93(3)	79(5)
ArS	67(15)	46(11)	52(14)	35(13)	72(14)	58(12)	39(7)	77(9)	78(8)	67(11)	62(4)
Bareland	99(1)	84(6)	92(3)	11(4)	89(4)	1(1)	21(17)	4(4)	31(22)	100(0)	93(1)
PSI	90(8)	95(4)	96(3)	92(3)	–	–	–	–	–	–	93(4)

. For regions R1 through R5, grassland is classified quite accurately as grassland areas occupy more than 20.8% of these regions’ total areas, while producer’s accuracies for grassland in the other five regions are decreased with reduced proportions of grassland areas in these regions. Bareland has higher producer’s accuracy in regions dominant by it (i.e., R1–R3 and R5), while it is more likely misclassified in the other six regions because of class rarity. Shrubland, wetland, and artificial surfaces, accounting for 3.3% of the total area together, are poorly classified, with shrubland registering extremely low producer’s accuracy.

Table 5 and Table 6 reveal an evident geographic pattern in classification errors related to class abundance. User’s and producer’s accuracies of grassland and bareland tend to decrease from west to east. These classes are abundant in the west (R1–R3 and R5) but generally rare in the east (R4 and R6–R10). Conversely, accuracy for forest tends to decrease from east to west, and this is correlated with the proportions of forest in the regions concerned. Permanent snow and ice of R1–R4 is well classified since it has unique spectral signature.

To evaluate the nationwide thematic accuracy, a country-wide error matrix of estimated area proportions was constructed by aggregating the ten regional error matrices using methods described in Section 2.3, with results shown in

Table 7. The country-wide error matrix, cell entries are expressed as percent of area (see Table 5 for meanings of CuL, ArS, and PSI). Agreement is defined as a match between the map class and either the primary or alternate reference class. User’s accuracy (UA) and producer’s accuracy (PA) are reported with standard errors (SE) in parentheses. Overall accuracy is 84.2% (0.7%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	18	1	0.5	0.6	0.2	0.2	0.7	0.1	0	21.3	84(1)
Forest	0.3	19.3	0.6	1.9	0	0.1	0	0.1	0	22.2	87(1)
Grassland	0.9	1	23.2	2.8	0.5	0.1	0.1	1.1	0	29.6	78(2)
Shrubland	0	0.1	0.2	0.7	0	0	0	0	0	1	70(2)
Wetland	0	0	0	0	0.4	0	0	0	0	0.4	82(2)
Water	0	0	0	0	0.1	1.5	0	0	0	1.6	94(1)
ArS	0.2	0.1	0	0	0	0	1.4	0	0	1.8	80(2)
Bareland	0	0	1.6	0.4	0.1	0	0	18.8	0	20.9	90(2)
PSI	0	0	0.1	0	0	0	0	0	0.9	1.1	88(2)
Total	19.5	21.5	26	6.5	1.2	1.9	2.3	20.2	1	100
PA	93(1)	90(1)	89(1)	11(1)	30(5)	79(5)	62(4)	93(1)	93(4)

. At the country level, the overall accuracy for GlobeLand30 2010 over China is 84.2%, as shown in Table 7. This estimate of overall accuracy is slightly higher than the results reported in [29].

Below, we report results of accuracy evaluation obtained following the procedures above but using the stricter definition of agreement at a sample pixel (as a match between the map label and the primary reference label only). The results are shown in Appendix C. Specifically, regional overall accuracies and user’s accuracies are shown in

Table A11. Regional user’s accuracies for nine land cover classes with standard errors (SE) in parentheses when agreement is defined as a match with the primary label.

	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	National
CuL	86(3)	58(5)	77(4)	69(5)	90(3)	63(5)	70(5)	87(3)	72(5)	91(3)	78(1)
Forest	65(5)	86(3)	76(4)	77(4)	94(2)	81(4)	82(4)	78(4)	86(3)	91(3)	83(1)
Grassland	88(3)	80(4)	86(3)	51(5)	77(4)	25(4)	32(5)	55(5)	29(5)	45(5)	76(2)
Shrubland	56(5)	64(5)	67(5)	69(5)	54(5)	69(5)	70(5)	57(5)	43(5)	57(5)	64(2)
Wetland	85(4)	83(4)	86(3)	90(3)	88(3)	58(5)	80(4)	79(4)	58(5)	70(5)	79(2)
Water bodies	90(3)	97(2)	92(3)	87(3)	89(3)	86(3)	88(3)	91(3)	93(3)	91(3)	92(1)
ArS	63(5)	50(5)	63(5)	67(5)	67(5)	75(4)	58(5)	78(4)	73(4)	71(5)	70(2)
Bareland	92(3)	81(4)	84(4)	59(5)	95(2)	50(5)	53(5)	29(5)	41(5)	67(5)	90(2)
PSI	87(3)	87(3)	98(1)	80(4)	–	–	–	–	–	–	88(2)
Overall	90(2)	81(3)	83(2)	69(3)	85(2)	73(3)	73(3)	80(2)	77(2)	84(2)

(Appendix C), while regional producer’s accuracies in

Table A12. Regional producer’s accuracies for nine land cover classes with standard errors (SE) in parentheses when agreement is defined as a match with the primary label.

	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	National
CuL	98(0)	98(0)	96(1)	86(4)	75(6)	89(4)	87(2)	94(1)	91(2)	91(2)	89(1)
Forest	52(12)	87(7)	88(5)	83(3)	87(6)	89(2)	90(3)	79(6)	88(2)	87(2)	86(1)
Grassland	80(5)	93(1)	88(2)	83(6)	95(1)	31(9)	48(12)	73(8)	33(9)	79(9)	87(1)
Shrubland	13(4)	13(3)	11(2)	13(2)	10(3)	6(1)	1(0)	1(0)	1(0)	0(0)	9(1)
Wetland	42(16)	6(3)	18(7)	32(12)	93(2)	2(1)	15(6)	58(12)	45(9)	89(3)	28(5)
Water bodies	99(1)	77(15)	98(1)	62(21)	46(24)	63(14)	63(12)	72(19)	58(8)	72(14)	70(5)
ArS	62(16)	38(10)	31(9)	19(6)	68(15)	46(10)	34(6)	76(10)	71(9)	64(12)	53(4)
Bareland	99(1)	84(6)	92(3)	9(3)	89(4)	1(1)	30(21)	3(3)	5(4)	100(0)	93(1)
PSI	90(8)	95(4)	96(3)	84(4)	–	–	–	–	–	–	93(4)

. The national error matrix is shown in

Table A13. The country-wide error matrix; cell entries are expressed as percent of area (see Table 5 for meanings of CuL, ArS, and PSI). Agreement is defined as a match between the map label and the primary reference label. User’s accuracy (UA) and producer’s accuracy (PA) are reported with standard errors (SE) in parentheses. Overall accuracy is 81.0% (0.8%).

	CuL	Forest	Grassland	Shrubland	Wetland	Water	ArS	Bareland	PSI	Total	UA
CuL	16.7	1.5	0.6	1	0.2	0.3	1	0.1	0	21.3	78(1)
Forest	0.6	18.4	0.8	2.2	0	0.1	0	0.1	0	22.2	83(1)
Grassland	1	1.2	22.5	3	0.5	0.2	0.1	1.1	0	29.6	76(2)
Shrubland	0	0.1	0.2	0.7	0	0	0	0	0	1	64(2)
Wetland	0	0	0	0	0.3	0	0	0	0	0.4	79(2)
Water	0	0	0	0	0.1	1.4	0	0	0	1.6	92(1)
ArS	0.3	0.1	0	0	0	0	1.3	0	0	1.8	70(2)
Bareland	0	0	1.6	0.4	0.1	0	0	18.7	0	20.9	90(2)
PSI	0	0	0.1	0	0	0	0	0	0.9	1.1	88(2)
Total	18.7	21.3	25.8	7.3	1.2	2	2.4	20.2	1	100
PA	89(1)	86(1)	87(1)	9(1)	28(5)	70(5)	53(4)	93(1)	93(4)

. Table A11, Table A12 and Table A13 are similar in format to Table 5, Table 6 and Table 7, respectively, for convenience of comparisons between them.

Consider national accuracy assessment results. As expected, with the stricter definition of agreement, country-wide overall accuracy is estimated lower at 81.0% (Table A13), reduced by about 3% as opposed to the estimated overall accuracy of 84.2% (Table 7) based on the less strict definition of agreement. User’s and producer’s accuracies are decreased to differing extents depending on the specific classes concerned, although there are no differences between either user’s accuracies or producer’s accuracies for bareland and permanent snow and ice. The most obvious decreases in accuracies are observed for the class of artificial surface, which registers a user’s accuracy of 70% (Table A13) (as opposed to 80%, Table 7) and a producer’s accuracy of 53% (Table A13) (as opposed to 62%, Table 7). Further detail about decreases in accuracies (using the stricter definition of agreement as opposed to the less strict definition) and their variations in regional, categorical, and national terms is shown in Table A11, Table A12 and Table A13 in Appendix C.

4. Discussion

In this section, discussion is based on assessment results obtained with the less strict definition of agreement (as a match between the map label and either the primary or the alternate reference label at a sample pixel). It is, however, sensible to be aware of the implications of using a stricter definition vs. a less strict definition of agreement for accuracy assessment and to appreciate the relevance of the latter definition of agreement for accuracy validation in a landscape of complexity.

4.1. Patterns of Misclassification Errors

Error patterns for GlobeLand30 2010 over China are summarized below (in the remainder of the paper, without causing ambiguity, reference to China will not be made unless necessary): (1) Some dominant classes are overestimated where there are inclusions, such as the forest and cultivated land included in artificial surfaces, artificial surfaces and grassland in cultivated land, shrubland and grassland in forest, grassland and shrubland in bareland. (2) Grassland, shrubland and forest are difficult to distinguish as their spectral characteristics are often similar. These three classes are usually mixed in natural environment, making it “look like its surroundings”. (3) Fragmented patches, such as scattered villages and small blocks of cultivated land in hilly/mountainous areas, are likely to be omitted and classified as dominant classes in neighborhoods. (4) “Salt and pepper” noise is common in GlobeLand30 maps. GlobeLand30 has been prescribed minimum mapping unit (MMU) for each land-cover class and allowable minimum error of omission or commission per scene for each class. However, there is no restriction to small blocks with size smaller than specified MMU. There are still many small blocks of shrubland, grassland, bareland, and forest in GlobeLand30 maps. When sample pixels are located in these small blocks, they are likely to be classified as dominant classes in neighborhoods. (5) Time lags between map image acquisition dates and the dates for image interpretation also have effects on reported accuracies, because of the possibility of land-cover change. The dates of Landsat images used for GlobeLand30 2010 over China range from January to December (2010), while 89.4% of sample pixels were based on fine resolution images flown from May to November. Although Land-cover change within the year is relatively rare, misclassifications caused by time lags are observed (e.g., bareland in summer is misclassified as permanent snow and ice, cultivated land in non-growing season is misclassified as bareland). (6) Map heterogeneity has the expected effect of reducing reported agreements between map and reference classes. Land-cover heterogeneity is defined as the number of land-cover classes occurring in a 3 × 3-pixel window centered on the sample pixel [47]. A heterogeneity value equal to one is defined as homogeneous (interior pixels), otherwise heterogeneous (edge pixels) in this paper. Nationwide, 72.6% of the sample pixels are interior, while the probability of disagreements for edge pixels is 2.3 times that of interior pixels.

4.2. Information for User Community and Product Improvement

Accuracy assessment is a standard component of GlobeLand30 mapping protocol. The overall accuracy for GlobeLand30 2010 dataset was estimated to be 84.2%, while user’s accuracies for individual classes (except for shrubland) exceeded 78% (Table 7). This indicates that the GlobeLand30 2010 dataset depicts spatial distributions of different land cover types in China with relatively great accuracy, given its fine resolution. Regional accuracy assessment results showed variations in accuracies due to regional differences in landscape patterns, as summarized at the end of Section 3.

Accuracy assessment can also inform and guide future map production. The results of GlobeLand30 2010 accuracy assessment show greater than 79% producer’s accuracies for all classes except for shrubland (only 11%), wetland, and artificial surface (Table 7). This suggests that mapping protocols need to be further developed and refined to better distinguish scattered grassland, shrubland, and small mixed grass-shrub patches, which are usually misclassified as other classes nearby. In China, especially southwestern China, spectrally mixed pixels are common in complicated and fragmented landscape. Although image segmentation and object-based classification techniques developed in GlobeLand30 data production can suppress “pepper-and-salt” effects in resultant classifications to some extent, determination of suitable segmentation parameters that are globally applicable is extremely difficult. Therefore, omission errors (of shrubland) are inevitable, especially in areas with a complex landscape. As shown in [48,49], scattered classes of small extents are likely misclassified as dominant classes in neighborhoods. This was similarly observed for wetland and artificial surface (e.g., human settlements in rural areas dominated by cultivated land). Further work is required to solve these problems in future global land cover mapping at fine spatial resolution. Regions and classes prone to misclassifications (in particular, shrubland in all regions, wetland in R2, R6 and R7, and artificial surface in R2, R4 and R7; see Table 6) should be given special attention.

4.3. Comparison of GlobeLand30 with Other Related Data Products of 30 m Resolution

In this subsection, we compare GlobeLand30 with two fine-resolution land cover products, whose Level I thematic resolution (i.e., number of classes) is similar to that of GlobeLand30. One is a global product, known as FROM-GLC [31], the other is a US national product, NLCD [50], as mentioned in Section 1.

FROM-GLC contains two levels of land cover classes: 10 Level I classes (i.e., cropland, forest, grassland, shrubland, water bodies, impervious areas, bare lands, snow and ice, clouds and unclassified) and 29 Level II classes. It utilized thousands of Landsat images flown from 1981 to 2011 and was generated from fully automated image classification. The overall accuracies (Level I) range from 54% (maximum likelihood classifier) to 65% (support vector machine, SVM), while its best overall accuracy (Level II) is 53%. Yu et al. [51] improved FROM-GLC classification results using MODIS time series image and other auxiliary data, achieving an overall classification accuracy of 67%. In comparison, FROM-GLC is of lower accuracy than GlobeLand30 2010 as the former resulted from fully automated classification while the latter was enhanced with object-based classification and knowledge utilization.

As mentioned in Section 1, US NLCD time series land cover products are also of 30 m spatial resolution and based on Landsat TM images. NLCD 1992 is based primarily on unsupervised image classification [50], while NLCD 2001, NLCD 2006, and NLCD 2011 are based on more advanced decision-tree classifiers [52,53,54]. According to Wickham et al. [35,36,37,38] and Stehman et al. [39], overall accuracies for NLCD 1992, 2001, 2006, and 2011 land cover (Level I, 8 classes, see below) were 80% (unweighted averages of 10 US regional estimates), 85%, 84%, and 88%, respectively, while those for level II were 55% (unweighted averages of 10 US regional estimates), 79% , 78%, and 82%, respectively. In general, user’s accuracies for water, forest, shrubland, agriculture, and high intensity developed exceeded 80% for most NLCD products. For NLCD 2011 (which is temporally close to GlobeLand30 2010), in particular, producer’s accuracies for all 8 Level I classes (i.e., water, developed, barren, forest, shrubland, grassland, agriculture, and wetland) were estimated to be greater than 80%, while user’s accuracies for all Level I classes (except for barren and wetland) were estimated to be greater than 81% [38]. Misclassifications usually occurred in regions with heterogeneous landscape, as also observed with GlobeLand30 2010.

Clearly, in terms of overall accuracy, GlobeLand30 2010 is slightly inferior to NLCD level I products (except for NLCD 1992 and NLCD 2006). We take the liberty in assuming approximate semantic equivalence between the following GlobeLand30 and NLCD classes: “cultivated land” = “agriculture”, “bareland” = “barren”, and “artificial surface” = “developed”. As for user’s accuracies, shrubland, grassland, and artificial surface are less accurately classified in GlobeLand30 than in NLCD 2011, while barren and wetland are less accurately classified in NLCD 2011 than in GlobeLand30. With respect to producer’s accuracies, NLCD 2011 is apparently superior to GlobeLand30, as there were large omission errors for shrubland, wetland, and artificial surface in the latter.

However, it should be noted that comparisons with FROM-GLC and NLCD were made for Level I classes above. A total of 15 land cover classes will be mapped in GlobeLand30 2015 which is under development [55]. Class ambiguity tends to increase as thematic detail of classification increases (e.g., from 10 classes to 15 classes), and this increasing ambiguity may have a negative impact on accuracy. For instance, overall accuracies decrease from 84% to 78% for NLCD 2006, when thematic resolution increases from Level I (8 classes) to Level II (16 classes) [37]. Users and producers should be aware of the implications of increased thematic resolution for GlobeLand30 2015.

5. Conclusions

Accurate assessment in land cover information is of great importance to research and applications concerning natural resources and the environment. This paper provides detailed information about regional vs. national accuracies (overall, user’s, and producer’s) of GlobeLand30 2010 land cover over China by adopting a two-level stratified random sampling design and furnishing suitable methods for aggregating regional error matrices to a national one. The national overall accuracy for GlobeLand30 2010 was estimated as 84.2% (with agreement at a sample pixel defined as a match between the map label and either the primary or the alternate reference label), indicating GlobeLand30’s relatively high accuracy. The national overall accuracy was estimated 81.0% when defining agreement at a sample pixel more strictly as a match between the map label and the primary reference label only. However, areas with heterogeneous landscapes and scattered small patches, in particular, need to be mapped with improved classification methods in future endeavors, as such areas tend to be labeled with low accuracies, as revealed in this study. GlobeLand30 production and validation teams will benefit from the research reported in this paper.