Assessing Consistency of Five Global Land Cover Data Sets in China

Global land cover mapping with high accuracy is essential to downstream researches. Five global land cover data sets derived from moderate-resolution satellites, i.e., Global Land Cover Characterization (GLCC), University of Maryland land cover product (UMd), Global Land Cover 2000 project data (GLC2000), MODIS Land Cover product (MODIS LC), and GLOBCOVER land cover product (GlobCover), have been widely used in many researches. However, these data sets were produced using different data sources and class definitions, which led to high uncertainty and inconsistency when using them. This study looked into the consistencies and discrepancies among the five data sets in China. All of the compared data sets were aggregated to consistent spatial resolution and extent, along with a 12-class thematic classification schema; intercomparisons among five datasets and each with reference data GLCD-2005 were performed. Results show reasonable agreement across the five data sets over China in terms of the dominating land cover types like Grassland and Cropland; while discrepancies of Forest classes, particularly Shrubland and Wetland among them are great. Additionally, GLC2000 has the highest agreement with GLCD-2005; MODIS LC gets the highest map-specific consistency compared with others; whereas UMd has the lowest agreement with GLCD-2005, but also has the lowest map-specific consistency. OPEN ACCESS Remote Sens. 2014, 6 8740


Introduction
Land cover, the observed (bio)physical cover on the earth's surface [1], is important ecosystem-based information.It is essential for scientific studies, sustainable management of land resources, and political purposes [2].Issues such as biogeochemical and climate cycling, terrestrial modeling including Land Surface Model (LSM) [3], Simple Biosphere Model Version 2.0 (SiB2) [4,5], Common Land Model (CLM) [6] and CoLM [7], environmental studies such as soil erosion [8], desertification and biological diversity [9], all benefit from the availability of land cover information [10,11].Therefore, it is very necessary to produce accurate land cover datasets at global and regional scales.
One of the major data sources for producing land cover datasets is satellite data, which provides a synoptic view of earth surface at various spatial scales and regular time intervals [12], and facilitates mapping and monitoring areas where are difficult or unable to access as well.In the past decades, land cover maps at different scales have been generated from satellite data [13][14][15][16][17][18][19][20][21][22][23][24], of which five available global land cover data sets have been widely used in a variety of applications: (i) Global Land Cover Characterization Database (GLCC) [19,25], (ii) University of Maryland land cover product (UMd) [17], (iii) Global Land Cover 2000 project data (GLC2000) [26], (iv) Moderate Resolution Imaging Spectro-radiometer annual Land Cover product (MODIS LC) [15], and (v) GLOBCOVER land cover product (GlobCover) [13].All of these data sets have been produced from moderate-resolution satellites, and thematically focused on characterizing the various vegetation types worldwide [27].
Despite the multiplicities of available information for the five land cover data sets, both data producers and potential users are troubled with the lack of interoperability among them and the deficiency of sufficient information on accuracy evaluation [28,29].Firstly, users have little guidance on which dataset to use and why, since each of them was produced using different data inputs and algorithms, which led to the fact that they were not designed to be comparable and basically exist as independent data sets.Secondly, the nature of the differences, the spatial agreements/disagreements and the relative qualities of data sets are rewarding to both data producers and end users.From the producer's perspective, areas of spatial agreement can be used effectively as one of the ancillary data sets during training areas' selection; whereas areas of disagreement could identify the limits deserving for further improvement in future land cover characterization and mapping.On the other hand, in-depth understanding of similarities and differences will help users make informed decisions regarding the selection of land cover data needed for their specific applications [30].Areas of agreement could be utilized conveniently by users as accurate and reliable information, while areas of disagreement might inspire users to verify the information in these areas by consulting additional information.In these regards, it is desirable to perform a comparison between the five global land cover data sets, and to highlight their individual strengths and weaknesses before using them for a variety of studies at regional to global scales.
In general, the quality of a satellite-derived land cover data can be identified with reference to an absolute scale that can be measured independently of the land cover map using in situ data, or to a relative scale that can be assessed using another land cover reference map, or both combined [31].Various assessments of the global land cover data sets were carried out in early studies [27,29,30,[32][33][34][35][36][37][38][39][40][41].However, the assessments were not evenly distributed among the global, and China has not been given enough attention regarding accuracy assessment of land cover dataset.China has a large territory, the largest population, and long history of human activities.Its land cover is featured with high diversity and high degree of fragmentation, making it very challenge to produce an accurate land cover map; existing global land cover data sets are likely to have high uncertainties over China.In order to better understand the degree and distribution of the uncertainties of the five global land cover data sets (i.e., GLCC, UMd, GLC2000, MODIS LC, and GlobCover) over China, this study will have a closer look on their consistencies and discrepancies, involving thematic similarities, spatial agreements and general patterns of map-specific consistency, and try to give users an insight into different land cover data sets when making wiser choices for their applications.

Global Land Cover Data Sets
In this study, three US and two European global land cover data sets were chosen for consistency assessments over China: (1) GLCC with the International Geosphere-Biosphere Programme (IGBP) Land Cover Legend produced by the United States Geological Survey (USGS) [19]; (2) UMd with the Simplified IGBP Land Cover Classification System developed by the University of Maryland [17]; (3) GLC2000 with the Land Cover Classification System (LCCS) of Food and Agricultural Organizations (FAO) generated by European Commission's Joint Research Center (EC-JRC) [26]; (4) MODIS LC (Collection 5) with the IGBP land cover classification scheme produced by Boston University [42], and (5) GlobCover with the FAO LCCS created by European Space Agency (ESA) [43].
Selection of the five global land cover data sets takes into account several considerations: GLCC and UMd were the first generated 1 km global land cover maps; MODIS LC was standard MODIS land products; GLC2000 was the first global maps produced by the collaboration of regional experts, and GlobCover had a finer spatial resolution at the global scale.The characteristics of these data sets are summarized in Table 1, which indicates the major differences between them are related to: (i) sensor capabilities (i.e., spatial and spectral properties and resolution), (ii) raw data processing (e.g., algorithms for cloud detection, corrections for atmospheric distortions), (iii) acquisition year of data set, (iv) selection of input data for classification, (v) land cover legend, (vi) classification algorithms, and (vii) validation of the product.

Reference Data
The Geodata Land Cover Dataset for year 2005 (GLCD-2005) at a scale of 1-250,000, funded by Ministry of Science and Technology (MOST) of China and developed by the Chinese Academy of Sciences (CAS), is the first dataset that presents the general land cover landscape of China after 2000 with high accuracy.In this study, GLCD-2005 was used as reference data for comparison with the five global data sets in China.A hierarchical land cover classification system of remote sensing considering terrestrial ecosystem characteristic was applied to the database covering 25 land cover classes, grouped into six aggregated classes (i.e., forest, grassland, cropland, built-up, wetland/water, and bare ground).
In order to combine the accuracy of interpretation and speediness of self-classification, the GLCD was mapped mainly by supervised decision tree combined with visual interpretation based on expert knowledge, under the control of polygon's position and borderline provided by the National Land Use Data at a scale of 1:100,000 in China [18], with the data source of MODIS vegetation index products at 250 m resolution, including 16-day MODIS Normalized Difference Vegetation Index (NDVI) and Enhanced Vegetation Index (EVI) available at Land Process Distributed Active Archive Center (LP DAAC) [44].Also, a series of ancillary data, such as Grass and Soil thematic maps of China, and Digital Elevation Model (DEM) were used to support the classification procedure.Accuracy assessment of GLCD-2005 was implemented using field survey datasets in 2005, and the result revealed that overall accuracy of this dataset was ranging from 80% to 91% [44][45][46].

Data Pre-Processing
Five compared global land cover data sets were distributed in different geographic reference systems, spatial resolutions, and coverage (Table 2).To facilitate comparison, these data sets were co-registered and reprojected to the Sinusoidal projection [47] with map datum WGS_84, which is a pseudocylindrical equal-area map projection and has been widely used in the MODIS products; and then were rescaled to 1 km spatial resolution with the same coverage extent (180°W-180°E, 55°S-90°N).Because Antarctica is excluded by several global land cover data sets (e.g., GLC2000), it is not included in data processing.As the data sets have coarser resolutions, directly resampling using nearest neighbor method may cause non-ignorable disagreement between the original and the rescaled data sets.Maximum area method has been proved to be a much robust approach for aggregating discrete land cover data [48,49].In order to minimize the impact from rescaling, the five global land cover data sets were resampled to a grid with 250 m resolution, which is much finer than the source data, and the 250 m grid was aggregated to a 1 km resolution grid using maximum area method to pick the dominated land cover type for each 1 km × 1 km pixel.Both the 250 m and the 1 km resolution grids were designed to be in Sinusoidal projection and share the same extent to minimize some influential issues during the aggregation processes.Reference data GLCD-2005 was also aggregated to the 1 km grid using maximum area method to facilitate comparisons.

Classification Scheme Conversion
All of the global data sets and GLCD-2005 were dedicated to providing a spatial distribution of diverse land cover types over the Earth's surface.However, differences in their land cover legends are conspicuous (Table 3).Therefore, reconciling the map legends is the second crucial precondition for facilitating comparisons of different land cover data sets.In this study, we defined the target legend with 12 major classes upon several parameters of classification standard for Plant Functional Types (PFTs) [50][51][52], i.e., the occurrence of life forms and leaf attributes (leaf type/leaf longevity), which are two of the common classifiers defined by LCCS [37].A legend conversion table for the original legends of compared land cover data sets and the target legend is given in Table 4. Recognizing that translating numerous land cover types into a small number may lead to a misrepresentation of source dataset on account of the transferability of classes from one to another [35], this arbitrary trade-off seemed to preserve the nature of the land cover landscape of China.
Regarding the visual requirement for maps across China, the reclassified data sets of (a) GLCC, (b) UMd, (c) GLC2000, (d) MODIS LC, and (e) GlobCover in China, as well as (f) GLCD-2005 are displayed with Albers Equal Area projection as shown in Figure 1 (relative figures over China in the following sections are visualized in the same way).

Areal and Spatial Comparison
Two essential components of the comparative assessments between different land cover data sets are comparisons in terms of the quantity and the location of each class, respectively [53].Comparison in terms of quantity, mainly referring to areal comparison, considers whether the area proportion of each class on the classification data set is similar to the proportion of the corresponding category on the reference data.Comparison in terms of location concerns the issue on spatial (dis)agreement of each category pixel by pixel usually.Typical per-pixel comparisons between different datasets can be divided into two sets, one is the fuzzy approach [54][55][56][57], and another is the crisp or Boolean approach [27,32,33,36].As the fuzzy method requires expert knowledge to quantify uncertainty in classification and transition zones of boundaries [29], the crisp or Boolean approach, which is based on cross-walking between classes, is preferred here.After being assigned to the target legend according to Table 4, the five global data sets were then overlaid to produce an agreement map, which indicates the level of agreement among these data sets.Consulting to McCallum et al. [36], five levels of agreement were distinguished in this pixel-based comparison process: Similarly, we overlaid each global land cover data set with the reclassified GLCD-2005, respectively.Pixels with the same land cover class in both data sets were considered as areas of "agreement", whereas pixels with different land cover classes were labelled as areas of "disagreement".

Consistency Evaluation
To investigate the degree of agreement in terms of aggregated classes among the five global land cover data sets across China on a quantitative basis, we calculate the consistency between each land cover combination pair upon pixel-based confusion matrices rather than accuracy, given that validation against ground "true" was not provided.
From the confusion matrices between land cover pairs we derive overall consistency, which is defined as the percentage of pixels where both data sets agree on the aggregated class.Then, the calculated overall consistency estimates where the considered data set was a comparison partner are averaged as an estimate of the map-specific consistency of the land cover data set (Equation ( 1)).
Mean C a = (C ab +C ac +C ad +C ae ) 4 ( 1) Equation ( 1): calculation of map-specific consistency for data set a over China.
C a* separately denote the overall consistency between pairs of data set a and another data set (i.e., b, c, d, or e) in China.
Indices a-e are the global land cover data sets (GLCC, UMd, GLC2000, MODIS LC, GlobCover) in China.

Thematic Similarities
Initially, a total percent area comparison of GLCC, UMd, GLC2000, MODIS LC, and GlobCover that assigned to each with twelve target classes was performed in China (Figure 2), and the result suggests that there is reasonable agreement across the five products for Grassland, Cropland, and Water and Others, particularly.However, substantial disagreements exist for Evergreen/Deciduous Forests classes, Shrubland, Urban, especially Mixed Forest and Wetland (except UMd).Visualizations of the distribution pattern and percent area composite of various assigned land cover categories across five data sets compared with GLCD-2005 are presented in Figure 3.The dominated land cover types of China land mass like Grassland, Cropland and Others, the sum of which account for about 70%, are similarly identified across the five data sets and the reference data.Besides, most of the large patches are distributed around the Northwestern and Eastern China.However, compared with GLCD-2005, major differences appear in Shrubland across the five global land cover data sets except GLC2000 and GlobCover; the discrepancy of Mixed Forest is conspicuous across all of the five data sets.

Spatial (Dis)Agreement
The per-pixel comparison of the five land cover data sets, reclassified to the aggregated classes defined at Table 4, is presented in Figure 4.It is visible that among these data sets over China, there is an understandable agreement on the distributional pattern of different generalized land cover types, especially for large, homogenous patches, such as Grassland and Cropland, while disagreement can also be found primarily for small, heterogeneous patches along edges and transition zones.Besides, areas of full agreement are often adjacent to areas of high agreement, areas of medium agreement are adjacent to areas of low agreement, and areas of no agreement are also adjacent to areas of low agreement.Statistically, no agreement takes up less than 2.5%, high to full agreement amount to about 38%, despite 31% are in medium agreement (3 of 5 agree) and 29% are in low agreement (only 2 of 5 agree).Furthermore, maps of agreement between each compared land cover data set and the reference data were created (Figure 5), and the levels of agreement were calculated (Table 5).Among all map pairs, GLC2000 agrees best with GLCD-2005 in that the "agreement" takes up about 52.2%, followed by GlobCover, MODIS LC, and GLCC in a descending order of agreement; UMd has the lowest agreement to GLCD-2005 with about 67% "disagreement".Spatially, regions of agreement between these data set pairs are primarily appear in Southern Xinjiang and Qinghai, Northwest of Inner Mongolia, Southeast of Tibet and Sichuan, some regions of Jiangsu, Shanghai and Shandong provinces, central Jilin, and Southwest of Heilongjiang as well.

Consistency Analysis
Figure 6 illustrates the calculated overall consistency between any two of the five global data sets for the aggregated land cover classes in China, of which the map-specific consistency of each data set is presented along the diagonal.This figure indicates that MODIS LC has the highest map-specific consistency (50.8%) among all of the compared data sets, while UMd gets the lowest one (42.6%).Among all data set pairs, MODIS LC agrees best with GLCC (overall consistency is about 55.3%), whereas UMd agrees worst with GlobCover (overall consistency is 32.3%).Moreover, the overall consistencies between all data set pairs across China are less than 50% except MOIDS LC and GLCC (55.3%), accompanied with GlobCover and GLC2000 pair (52.0%).

Discussions
From above assessments and analyses, we can find that discrepancies among the compared data sets could either be real or simply be due to differences in their sensors, temporal periods, original classification algorithms, or classification schemes.However, even if land cover changes between the acquisition dates has an effect on this comparison [28,29], it cannot be the primary factor since natural ecosystems typically vary on a decade or longer.This can be demonstrated in this study through the comparison result between GlobCover and GLCD-2005 that, the agreement between these two data sets is not the highest among the compared pairs, although there is no difference between their temporal periods.In terms of classification of land cover mapping, the first important issue is class separability [28], which has already been viewed as a general problem of optical remote sensing in discriminating certain categories with multi-temporal spectral signatures that overlap with other categories, such as Shrubland or Wetland.Classification schemes and class definitions are also problematic in several ways.Mixed classes lack clear definitions; more or less arbitrary thresholds are applied to distinguish between classes, which pose significant challenges for users of land cover data sets that classification schemes may be not flexible or suitable enough for their applications.The third noticeable issue refers to cartographic standards as mapping a continuum transition with a discrete classification scheme.Especially for areas of mixed classes, different maps may give distinct estimates, all of which may be right or wrong to some extent, since final land cover maps are definitely very sensitive to classification algorithms and representation of mixed cartographic units.
Aside from above internal factors, a number of external factors are also the sources of discrepancies between these data sets, including map projections, resolution unifications and classification scheme conversion.External factors mentioned above may attribute to the preparations and modifications done to the different data sets to meet the need of this study.As in the legend conversion, substantial differences of identification criteria for the same or similar categories are undeniable [32].For instance, the GLCC/MODIS LC-IGBP height threshold for dividing trees and shrubs is 2 m, while in the UMd-simplified IGBP and GlobCover-LCCS it is 5 m, and GLC2000-LCCS is 3 m.For coverage thresholds, the IGBP separates closed coverage (>60%) and open coverage (10-60%), while in the simplified IGBP and LCCS, closed coverage is defined as >40%, open coverage is 10-40% and 15-40%, respectively.When classification scheme transformation was performed, differences and uncertainties were already introduced.

Conclusions
To our knowledge, this effort has been the first investigation of the consistencies and differences of all the five global land cover data sets (i.e., GLCC, UMd, GLC2000, MODIS LC, and GlobCover) for China land mass.Although initiatives of these examined data sets were implemented with the same purpose of providing accurate land cover information, different satellite sensors and classification approaches were used.It is no wonder that all these data sets produced different results, and it would be too arbitrary to say which data set is more suitable for a specific application, but the study provides a reference for users to understand the extent and degree of uncertainty or potential errors within these data sets as well as a guideline for further validation of quality assessment.
Based on the intermap comparison of five global land cover data sets and reference data regarding 12 aggregated classes, (1) the thematic similarities, (2) spatial agreement and (3) map-specific consistency between them are identified.Generally, identified from the comparison results of the thematic similarities, the Forest classes, and especially Shrubland, Mixed Forest and Wetland are very problematic across China; while the dominated land cover types like Grassland and Cropland are similarly identified across the five data sets.Spatially, regions of disagreement among these data sets are primarily related to transitional zones with mixed classes, such as regions of Southwest China, although the sum of no agreement and low agreement takes up less than 32.0%, while high to full agreement amount to about 38.0%.Compared with the reference data set GLCD-2005, areas of agreement across the five data sets are distributed around the Northwestern and Eastern China; major differences appear in Shrubland across GLCC, UMd and MODIS LC, and the discrepancy of Mixed Forest is conspicuous across all of the compared data sets.In addition, among all map pairs, GLC2000 has the highest agreement (52.2%) with GLCD-2005, while UMd has the highest disagreement (66.9%) with GLCD-2005.Further, regarding consistencies among the five data sets, MODIS LC has the highest map-specific consistency (50.8%) whereas UMd gets the lowest one (42.6%).
For the next step, a rigorous validation of the five global land cover data sets against test samples could be carried out for a larger area, such as continental or global, providing a guide line for users to choose and use these global land cover data sets in their researches.Furthermore, there is also a need to exploit the synergies and harmonization of different land cover data sets based on examining the agreements and disagreements between them, towards creating a consensus land cover data with combining information provided from other relative global ancillary data.Such a data set can be used to extend and enrich the initiatives fostered by Global Observation of Forest and Land Cover Dynamics (GOFC-GOLD) in conjunction with FAO and Global Terrestrial Observing Systems (GTOS) [58].
(i) No agreement for pixels with a unique aggregated class in each data set.(ii) Low agreement for pixels where only two of the five data sets are in agreement.(iii) Medium agreement for pixels where three of the five data sets are in agreement.(iv) High agreement when four of the five data sets agree for the same pixel.(v) Full agreement for where all the five data sets within a pixel are in agreement.

Figure 2 .
Figure 2. The percent area comparison of the five global land cover data sets regarding target classes over China.

Figure 4 .
Figure 4. Spatial agreement among the five land cover data sets over China according to the aggregated legends.The number in brackets refers to the statistical percentage of the area coverage for corresponding agreement levels.

Figure 6 .
Figure 6.Overall consistencies between GLCC, UMd, GLC2000, MODIS LC, and GlobCover over China.Map-specific consistencies of the compared data sets in China are given along the diagonal.

Table 1 .
Characteristics of the five global land cover data sets assessed in this study.

Table 2 .
Original projections of the compared land cover data sets.

Table 3 .
The original classification schemes of compared land cover data sets.

Table 4 .
Conversion table of map legends.Refer to Table 3 for the definition of each class number of original data sets.
* Wetland is absent in UMd.

Table 5 .
Percent agreement (%) of the five land cover data sets and the reference data across China.Maximum percent agreement and disagreement are displayed in bold text and underlined.