Comparison of Various Annual Land Cover Datasets in the Yellow River Basin

Abstract

Accurate land cover (LC) datasets are the basis for global environmental and climate change studies. Recently, numerous open-source annual LC datasets have been created due to advances in remote sensing technology. However, the agreements and sources of error that affect the accuracy of current annual LC datasets are not well understood, which limits the widespread use of these datasets. We compared four annual LC datasets, namely the CLCD, MCD12Q1, CCI-LC, and GLASS-LC, in the Yellow River Basin (YRB) to identify their spatial and temporal agreement for nine LC classes and to analyze their sources of error. The Mann–Kendall test, Sen’s slope analysis, Taylor diagram, and error decomposition analysis were used in this study. Our results showed that the main LC classes in the four datasets were grassland and cropland (total area percentage > 80%), but their trends in area of change were different. For the main LC classes, the temporal agreement was the highest between the CCI-LC and CLCD (0.85), followed by the MCD12Q1 (0.21), while the lowest was between the GLASS-LC and CLCD (−0.11). The spatial distribution of area for the main LC classes was largely similar between the four datasets, but the spatial agreement in their trends in area of change varied considerably. The spatial variation in the trends in area of change for the cropland, forest, grassland, barren, and impervious LC classes were mainly located in the upstream area region (UA) and the midstream area region (MA) of the YRB, where the percentage of systematic error was high (>68.55%). This indicated that the spatial variation between the four datasets was mainly caused by systematic errors. Between the four datasets, the total error increased along with landscape heterogeneity. These results not only improve our understanding of the spatial and temporal agreement and sources of error between the various current annual LC datasets, but also provide support for land policy making in the YRB.

1. Introduction

Land cover (LC) change plays a key role in understanding global environmental changes, such as the carbon cycle [1,2], local climate [3,4], water quality and its distribution [5,6], biodiversity conservation [7,8], ecosystem services, and human well-being [9,10]. Concerns about LC changes have increased dramatically in recent years with the development of high-level political agreements, such as the UN Sustainable Development Goals (SDGs) and the Convention on Biological Diversity (CBD) [11,12].

Advances in remote sensing technology have contributed significantly to the development of LC change studies [13,14,15]. The international scientific community has initiated and organized numerous projects to produce LC datasets using satellite remote sensing images [16,17,18,19,20], achieving substantial results in land change monitoring and diagnosis [21,22], land change mechanisms [23,24], regional and global land change models [25,26], and global environmental change [27,28]. In addition, the production and application of annual LC datasets has significantly improved our understanding of the land change process [13,17,29]. However, the process of annual land change differs substantially from that identified by snapshots [30], which may underestimate the intensity of LC change and fail to determine whether the trend of LC change is gradual or abrupt [31].

The availability of numerous LC datasets has also raised uncertainties due to the use of various data sources, classification schemes, and methodologies. Consequently, numerous studies have been conducted to evaluate the accuracy of LC datasets from various sources, as well as to assess the agreement between them [32,33,34,35,36,37]. Most studies have focused on assessing the classification accuracy and comparing the spatial agreement of different LC datasets at a single assessment date, or on an assessment of change involving only one or two change periods [33,37,38], but few studies have investigated the spatial and temporal agreement and the source of error between the annual LC datasets. It is crucial to identify the weaknesses and limitations of current coarse land change products (e.g., burned area products) by understanding the sources of error that affect the accuracy of these products [39]. Studies comparing climate indices (e.g., temperature and precipitation) have reported [40,41] that analyzing the annual spatial and temporal agreement and source of error between various datasets is crucial for explaining biases in climate model results, as well as for the data fusion of various datasets.

Landscape heterogeneity has been frequently identified as a driver in determining the spatial agreement and variation between various LC datasets. Previous studies [33,34,42] have revealed that LC datasets in regions with high landscape heterogeneity generally have lower classification accuracy, as well as lower spatial agreement between various datasets. However, it is unclear how landscape heterogeneity affects the annual spatial and temporal agreement and source of error between various LC datasets. The LC of the Yellow River Basin (YRB) is highly heterogeneous and dynamic. Over the past 40 years, the spatial distribution pattern of LC in the YRB has changed dramatically [43,44] due to climate change [43,45,46] and human activities [47,48,49,50]. In particular, the implementation of ecological projects such as “Grain to Green” has led to a large-scale land system shift between cropland and natural vegetation on the Loess Plateau (located in the midstream area of the YRB). Previous studies [51] have reported that the area of change for the main LC classes in the YRB varied considerably between various datasets, but the spatial and temporal agreement between them was unclear. This makes the YRB an ideal region in which to conduct a study comparing the agreement and the source of error between various LC datasets.

The objective of this study is to investigate the spatial and temporal agreement and the source of error for nine LC classes between four annual datasets. The spatial and temporal agreement was assessed using the Mann–Kendall test, Sen’s slope analysis, and Taylor diagram methods, and the source of error was assessed using the error decomposition analysis method. The rest of this article is organized as follows. Section 2 presents the study area, datasets, and the method of data analysis. Section 3 presents the results of spatial and temporal agreement and the source of error between the four datasets. The relationship between landscape heterogeneity and the agreement and total errors between the four datasets is also presented in this section. Section 4 discusses the results in detail, the implications for land policy, and the limitations of this study. Section 5 concludes this study.

This study will provide new insights into the spatial and temporal agreement between the four annual LC datasets, in particular the source of error. Through this study, we will realize that further improvement of the annual LC datasets is possible through appropriate data fusion algorithms that integrate landscape heterogeneity. Furthermore, the analysis methods presented in this study will provide valuable information for future research investigating the comparison of specific land change processes, such as urban development and cropland expansion.

2. Materials and Methods

2.1. Study Area

The Yellow River Basin (YRB) is located between 96°E–119°E and 32°N–42°N (Figure 1). The YRB covers an area of 79.5 million ha. The topography of the YRB is high in the west and low in the east (Figure 1) and includes four geomorphic units from west to east: the Qinghai–Tibet Plateau, the Inner Mongolia Plateau, the Loess Plateau, and the North China Plain. The YRB is the main source of fresh water for approximately 107 million people. It is also an important grain-producing region in northern China, accounting for 18% of China’s grain production [51]. In brief, the YRB is an important ecological security barrier for northern China [52]. Typically, the YRB can be divided into four watersheds: the source area region (SA), the upstream area region (UA), the midstream area region (MA), and the downstream area region (DA) (Figure 1).

2.2. Data Sources and Pre-Processing

The main free and open-access annual LC datasets available for the China region include the Moderate Resolution Imaging Spectroradiometer (MODIS) LC dataset [53,54,55], the ESA Climate Change Initiative LC (CCI-LC) dataset [56], the Global Land Surface Satellite Global Land Cover (GLASS-GLC) dataset [57], and the China Landsat-derived Annual Land Cover Dataset (CLCD) [58]. The four LC datasets mentioned above were selected for use in this study (see

Table 1. Characteristics of the four LC datasets.

LC Dataset	GLASS-GLC	MCD12Q1	CCI-LC	CLCD
Satellite sensor	NOAA AVHRR	Terra and Aqua MODIS	ENVISAT MERIS SPOT VGT	Landsat TM/ETM+, HJ-1
Input data	41 metrics derived from NDVI and 5 spectral bands	16-day nadir BRDF adjusted reflectance, 7 spectral bands, EVI	MERIS 7-day composite surface reflectance, SPOT VGT time series data	Optical bands in TM/ETM + and HJ-1, some auxiliary, e.g., MODIS NDVI time series data
Spatial resolution	4 km	500 m	300 m	30 m
Temporal resolution	1981–2015	2001–2019	1992–2018	1990–2019
Classification scheme	7 classes	IGBP (17 classes)	LCCS (22 classes)	9 classes
Classification strategy and method	The Earth was divided into equal map sheets and classified separately by the combination of pixel- and object-based methods [57].	The Earth was viewed as an entirety and was classified using the decision tree method [53,54,55].	The Earth was divided into 22 climatic regions and each region was classified separately by the combined use of the supervised and unsupervised methods [56].	The Earth was divided into equal map sheets and classified separately by the combination of pixel- and object-based methods [58].

for details), and the CLCD was used as the reference dataset as it has the highest spatial resolution and classification accuracy [58]. The boundaries of the YRB were obtained from RESDC (http://www.resdc.cn, accessed on 15 January 2023). The DEM data were obtained from the Geospatial Data Cloud Platform of the Computer Network Information Center of the Chinese Academy of Sciences (http://www.gscloud.cn, accessed on 15 January 2023) with a spatial resolution of 90 m × 90 m. All data were reprojected in Mercator projection and uniformly clipped by the boundary of the YRB for subsequent processing. For direct comparison of the four annual LC datasets with different classification strategies, all the datasets were reclassified into nine LC classes: cropland, forest, grassland, shrubland, water, snow/ice, impervious, barren, and wetland. (Tables S1–S3).

2.3. Methods

In this study, we compared the four annual LC datasets to determine their temporal and spatial agreement and to analyze their sources of error for nine LC classes in the YRB. As the spatial resolution of these datasets varies from 300 m to 4 km, the spatial agreement and source of errors between these datasets cannot be directly compared. A widely used method has been to resample the LC datasets to the same spatial resolution using nearest neighbor or majority interpolation techniques. However, the resampling method could miss the detailed information of the datasets and propagate some errors into the newly resampled datasets. To overcome this problem, we compared the area estimates in the grid (0.5° × 0.5°) for each LC class instead of resampling these datasets to the same spatial resolution. In this context, the area estimates were calculated by the pixel counting method and the agreement was assessed by the correlation (with a p-value less than the significance level of 0.05) between the reference and the other three datasets.

The main flowchart of the methodology used in this study was as follows (Figure 2). First, we calculated the area estimates in the whole YRB for each LC class and compared the temporal agreement between the four LC datasets. Second, we calculated the area estimates in the grid (0.5° × 0.5°) and compared the spatial agreement between the four LC datasets. Third, the area estimates in the grid (0.5° × 0.5°) were used to calculate the sources of error for each LC class. Finally, we calculated the landscape heterogeneity in the grid (0.5° × 0.5°) of the reference dataset and analyzed the relationship between landscape heterogeneity and agreement, and between landscape heterogeneity and error. Landscape heterogeneity was calculated using the landscape diversity index.

2.3.1. Estimation of Landscape Heterogeneity

Landscape heterogeneity was estimated using the landscape diversity index [59]. The landscape diversity index (H) refers to the diversity of landscape classes in each of the 0.5° × 0.5° grids in this study and was calculated as follows (Equation (1)):

$$H=-{\displaystyle {\sum }_{i=1}^R{p}_i\ln }(p_i)$$

(1)

where H is the landscape diversity index and p_i is the proportion of each landscape class within the 0.5° × 0.5° grid.

2.3.2. Mann–Kendall Test and Sen’s Slope Estimate

The Mann–Kendall test and Sen’s slope estimate (β) were used to analyze the trend in area of change for the nine LC classes. The Mann–Kendall statistical test [60,61] and Sen’s slope estimate [62] are both non-parametric methods of trend detection and are commonly used for time-series data [63,64]. Sen’s slope estimate refers to the slope of the trend in the sample of N pairs of data and is estimated as follows (Equation (2)):

$$\beta =\begin{array}{cc}median(\frac{x_j-x_i}{j-i}),& \forall j>i\end{array}$$

(2)

where x_j and x_i are the values at time j and i (j > i), respectively. β > 0 indicates an increasing trend in the time series data, while β < 0 indicates a decreasing trend. The Mann–Kendall test statistic S is calculated as follows (Equation (3)):

$$S={\displaystyle \sum _{i=1}^{N-1}{\displaystyle \sum _{j=i+1}^N\mathrm{sgn}(x_j-x_i)}}$$

(3)

where N is the number of data points, x_i and x_j are the values in time series i and j (j > i), respectively, and sgn(x_j − x_i) is the sign function, as follows (Equation (4)):

$$\mathrm{sgn}(x_j-x_i)=\{\begin{array}{r}\begin{array}{cc}+1,& (x_j-x_i)>0\end{array}\\ \begin{array}{cc}0,& (x_j-x_i)=0\end{array}\\ \begin{array}{cc}-1,& (x_j-x_i)<0\end{array}\end{array}$$

(4)

The variance is calculated as follows (Equation (5)):

$$Var(S)=\frac{N(N-1)(2N+5)}{18}$$

(5)

For a sample size N > 10, the standard normality test statistic Z_S is calculated as follows (Equation (6)):

$$Z_S=\{\begin{array}{r}\begin{array}{cc}\frac{S-1}{\sqrt{Var(S)}},& \begin{array}{cc}if& S>0\end{array}\end{array}\\ \begin{array}{cc}0,& \begin{array}{cc}if& S=0\end{array}\end{array}\\ \begin{array}{cc}\frac{S+1}{\sqrt{Var(S)}},& \begin{array}{cc}if& S<0\end{array}\end{array}\end{array}$$

(6)

The trend test is performed at a specific α significance level. If |Z_S| > Z_1−α/2, the null hypothesis is rejected and there is a significant trend in the time series data. Z_1−α/2 is obtained from the standard normal distribution table. In this study, the significance level α = 0.05 was used. If |Z_S| > 1.96, the null hypothesis of no trend was rejected at the 5% significance level.

2.3.3. Taylor Diagram Approach

Taylor diagrams [65] were used to evaluate the difference in the area of changes for the nine LC classes between the four datasets. Taylor diagrams provide a way of graphically summarizing how closely an observation dataset matches the reference dataset. The agreement between two datasets is quantified in terms of the correlation coefficient (R), the centered root-mean-square-error (E’), and the amplitude of the standard deviation (σ). These statistics are related by the following equation (Equation (7)):

$$E{\prime }^2={\sigma }_f^2+{\sigma }_r^2-2{\sigma }_f{\sigma }_{r}R$$

(7)

where R is the correlation coefficient between the observation and reference datasets, E’ is the centered RMSE between the observation and reference datasets, and σ_f² and σ_r² are the variances of the observation and reference datasets, respectively. The equations for calculating the correlation coefficient (R), the centered RMSE (E’), and the standard deviations of the test dataset (σ_f) and the reference dataset (σ_r) are as follows (Equations (8)–(11)):

$$R=\frac{\frac{1}{N}{\displaystyle \sum _{n=1}^N(f_n-\overline{f})(r_n-\overline{r})}}{{\sigma }_f{\sigma }_r}$$

(8)

$$E{\prime }^2=\frac{1}{N}{\displaystyle \sum _{n=1}^N{\left[(f_n-\overline{f})-(r_n-\overline{r})\right]}^2}$$

(9)

$${\sigma }_f^2=\frac{1}{N}{\displaystyle \sum _{n=1}^N{(f_n-\overline{f})}^2}$$

(10)

$${\sigma }_r^2=\frac{1}{N}{\displaystyle \sum _{n=1}^N{(r_n-\overline{r})}^2}$$

(11)

where f_n and r_n are the areas for the nine LC classes, $\overline{f}$ and $\overline{r}$ are the mean areas for the nine LC classes of the observation and reference datasets, respectively, and N is the number of values in the time series or space.

In a Taylor diagram, the distance from the data point to the origin is the σ value, reflecting the magnitude of change. The position of the data points on the axes represents the R, reflecting the degree of agreement between the observation and the reference datasets. The distance of the data points from the reference points is the E’, reflecting the deviation between the observation and the reference datasets. It is generally accepted that the smaller the distance between the observation and reference dataset, the better the agreement. The diagram is particularly useful for evaluating multiple dimensions of complex models or datasets [66,67] and is widely used in scientific climate research.

2.3.4. Error Decomposition Approach

Understanding the sources of error is the basis for future improvements in the accuracy of LC classification and the development of uncertainty estimation and bias calibration techniques [68]. The Willmott decomposition technique [69] was used to determine the percentage of systematic and random error. In this approach, the systematic error is defined as the part of the error that can be fitted linearly [69,70]. The errors in the datasets can therefore be divided into percentages of systematic and random errors as follows (Equations (12) and (13)):

$$\begin{array}{ll}\frac{1}{N}\left[{\displaystyle \sum _{n=1}^N{(f_n-r_n)}^2}\right]=& \frac{1}{N}\left[{\displaystyle \sum _{n=1}^N{(f_n^{*}-r_n)}^2}\right]\\ & +\frac{1}{N}\left[{\displaystyle \sum _{n=1}^N{(f_n-f_n^{*})}^2}\right]\end{array}$$

(12)

$$f_n^{*}=a\times r_n+b$$

(13)

where f_n* is the fitted areas for the nine LC classes through a linear function, and a and b are the slope and intercept of the fitting linear function, respectively. In Equation (11), the total mean squared error is decomposed into two components (the terms on the left): (a) the systematic error (first term on the right) and (b) the random error (second term on the right).

3. Results

3.1. Comparison on a Temporal Scale

Figure 3 showed that the LC classes of the YRB are mainly grassland and cropland, but there were differences in their areas between the four datasets. For cropland, the CCI-LC had the largest area (Figure 3a) and the MCD12Q1 had the smallest area; the 15-year (2001–2015) mean area from the CCI-LC was 4.17 times larger than that from the MCD12Q1, and the 15-year (2001–2015) mean areas from the CLCD and GLASS-GLC were comparable. Regarding grassland (Figure 3d), the area from the CLCD was the largest, and the area from the MCD12Q1 was the smallest, with the 15-year mean area from the CLCD being 1.99 times that from the MCD12Q1; the 15-year mean areas from the CCI-LC and GLASS-GLC were comparable. The results of the Mann–Kendall test (S) and Sen’s slope (β) showed that the gradual change process was significant for most LC classes of the YRB from 2001 to 2015. Further analysis revealed that there was a similar temporal trend in area of change for forest and impervious areas between the four datasets (Figure 3c,h), but not for the other LC classes. The shrubland, water, snow/ice, barren, and impervious areas, which cover a small percentage of the total area, reached their highest value in the CLCD. This suggests that the CLCD, which has the highest spatial resolution, may reflect more detail in terms of the temporal change of LC than other datasets.

As shown in Figure 4, the ranking of σ values for the nine LC classes between the four datasets were GLASS-LC > CLCD > MCD12Q1 > CCI-LC, indicating that the GLASS-LC is more variable than the CLCD, while the MCD12Q1 and CCI-LC are less variable than the CLCD. The R between the CCI-LC and CLCD was positive for the seven LC classes (except for snow/ice and wetland), while the R between the MCD12Q1 and CLCD was positive for five LC classes (forest, water, barren, impervious and wetland), and the R between the GLASS-LC and CLCD was positive for only two LC classes (forest and grassland). The ranking of E’ between the three datasets and the CLCD was GLASS-LC > MCD12Q1 > CCI-LC. This suggests that the CCI-LC deviates the least from the CLCD, while the GLASS-LC deviates the most from the CLCD. These results suggest that the CCI-LC and CLCD had the highest temporal agreement, followed by the MCD12Q1, while the GLASS-LC and CLCD had the lowest temporal agreement.

3.2. Comparison on a Spatial Scale

To compare the spatial variation in area for the nine LC classes between the four datasets, the area estimates for each LC class in the grid (0.5° × 0.5°) were calculated. In general, the spatial distribution of the 15-year mean areas for the nine LC classes was relatively comparable between the four datasets, with spatial variation mainly located in the UA and MA of the YRB (Figure S1). As shown in Figure S1, the spatial distribution of cropland, forest, and grassland areas was mainly located in the UA and MA of the YRB, the spatial distribution of the barren area was in the UA of the YRB, and the spatial distribution of the impervious area was in the MA of the YRB. The spatial extent of shrubland, water, snow/ice, and wetland areas were the largest in the CLCD compared to the other three datasets. Regarding the area for the main LC classes (Figure S2), the variation between the GLASS-LC and CLCD was mainly located in the UA of the YRB, the variation between the CCI-LC and CLCD was located in the UA of the YRB, and the variation between the MCD12Q1 and CLCD was located in the UA and MA of the YRB. Compared to the CLCD, the GLASS-LC had 7.65 M ha less grassland area and 16.20 M ha more barren area; the CCI-LC had 9.02 M ha more cropland area and 6.71 M ha less grassland area; and the MCD12Q1 had 9.24 M ha more grassland area and 3.08 M ha less cropland area. These spatial variations in area may be due to the different classification strategies for LC classes such as cropland, forest, and grassland between the four datasets.

The spatial variation of the trends in area of change (β) for the main LC classes between the four datasets were mainly located in the UA and MA of the YRB (Figure 5), with a significantly large difference between the GLASS-LC and CLCD and a relatively small difference between the CCI-LC and CLCD. Further analysis revealed that the CLCD had the highest rate of grid change in terms of LC change compared to the other three datasets (Figure S3). This suggests that the spatial variation of β between the four datasets may be driven by the spatial resolution of the datasets. For the forest, water, barren, and impervious LC classes, the spatial extent of β was similar between the four datasets, varying only in magnitude (Figure S3). For example, for the forest LC class, the β between the four datasets was mainly located in the MA of the YRB; compared to the CLCD, the MCD12Q1 had a higher β in the SA and UA and a lower β in the MA and DA; the CCI-LC had a lower β in the SA, UA, MA, and DA; and the GLASS-LC had a lower β in the SA and a higher β in the MA. However, for the cropland, shrubland, grassland, snow/ice and wetland LC classes, the spatial variation of β between the four datasets was significant in both spatial extent and magnitude.

Correlation maps (R > 0, p < 0.05) were used to illustrate the spatial agreement in the area of change for nine LC classes between the four datasets in the YRB from 2001 to 2015 (Figure 6). In general, the correlation between the four datasets was higher in the UA and MA of the YRB than that of other watersheds, but there were still differences in their values. Figure 6 also showed that the spatial distribution of the correlation for the main LC classes was highest between the CCI-LC and CLCD, followed by the MCD12Q1, and lowest between the GLASS-LC and CLCD. These results suggest that the CCI-LC and CLCD had the highest spatial agreement, followed by the MCD12Q1, while the GLASS-LC and CLCD had the lowest spatial agreement.

3.3. Source of Errors

Error maps were used to illustrate the spatial variation of the total errors for nine LC classes between the four datasets in the YRB from 2001 to 2015 (Figure S4). In general, the spatial distribution of the total errors for the main LC classes between the four datasets was similar, mainly located in the UA and MA of the YRB. With the CLCD as the reference dataset (Figure S4), the total errors for the nine LC classes between the CCI-LC and MCD12Q1 were relatively small (0.0060 and 0.0069, respectively), while the GLASS-LC had the largest total error (0.0454). For the cropland, forest, grassland, barren, and impervious LC classes, the spatial extent of the total error was similar between the CCI-LC, MCD12Q1, and CLCD, while the total error between the GLASS-LC and CLCD was higher in the SA and UA. For the shrubland, water, snow/ice, and wetland LC classes, the spatial variation of the total error was probably due to the difference in classification strategies between the CCI-LC, MCD12Q1, and CLCD.

Pairs of error maps were used to illustrate the spatial variation of the percentage of random and systematic error for nine LC classes between the four datasets in the YRB from 2001 to 2015 (Figure 7). In general, the percentage of systematic error was the source of error for the main LC classes between the MCD12Q1, CCI-LC, and the GLASS-LC reference CLCD, with the CCI-LC reference CLCD having the highest value and the GLASS-LC reference CLCD having the lowest. For cropland, forest, grassland, and barren areas, with the CLCD as the reference dataset, the percentage of random error for the CCI-LC was high in the UA, while for the MCD12Q1 it was high in the DA and for the GLASS-LC it was high in the SA. For shrubland, water, snow/ice, impervious, and wetland areas, the percentage of random error for both the CCI-LC and MCD12Q1 was high in the SA and UA with the CLCD as the reference dataset.

3.4. Landscape Diversity vs. Agreement and Total Errors

The spatial distribution of landscape diversity (H) in the YRB was shown in Figure S5. In terms of elevation (Figure S5a), H was high in the lower (0.5–1.0 km) and middle (1.0–3.5 km) mountain regions of the YRB. In terms of watershed (Figure S5b), the highest H was found in the MA of the YRB, followed by the UA, while the lowest H was found in the SA and DA. In terms of landscape, the highest H was found in the Loess Plateau region (roughly in the MA and UA of the YRB).

A boxplot was used to show the variation in spatial agreement (R > 0, p < 0.05) and the total errors in landscape diversity (H) for nine LC classes between the four datasets in the YRB (Figure 8). Regarding spatial agreement, it was notable that the spatial agreement (median value) increased with H for the cropland, forest, grassland, barren, and impervious LC classes between the four datasets (Figure 8, left panel). This suggests that landscape heterogeneity may influence the spatial agreement between the four datasets. However, the relationships between spatial agreement and H were not available for shrubland, water, snow/ice, and wetland because these LC classes were found only in specific regions and covered small areas. It should be noted that the spatial agreement for the forest LC class first tended to decrease and then to increase with H between the four datasets, which may be explained by the implementation of ecological projects such as the “Grain to Green” project. Regarding the total errors, it was evident that the total error (median) tended to increase with H for the cropland, forest, grassland, barren, and impervious LC classes between the four datasets (Figure 8, right panel). This suggests that landscape heterogeneity clearly influences the spatial variation of the total errors between the four LC datasets, and thus the increase in landscape heterogeneity could lead to more uncertainty for the main LC classes. However, the relationships between the spatial distribution of the total error and H were not available for shrubland, water, snow/ice, and wetland because these LC classes were only located in specific regions and covered small areas.

4. Discussion

4.1. Comparison of Various Datasets

All the LC datasets are simulations and characterizations of LC, but the magnitude and trends in area of change estimated by different LC datasets vary considerably from study to study [33,34,35,71]. One of the main reasons for this is that the accuracy of the datasets used in previous studies has not been carefully assessed. Therefore, comparing LC datasets from various sources helps to assess the applicability of LC datasets and provides further implications for improving the assessment accuracy of LC datasets [35].

In this study, the annual LC change process was gradual in the YRB from 2001 to 2015 (Figure 3), and the temporal agreement in area estimates of change was significant between the four datasets for the forest, water, impervious, and barren LC classes. The most striking finding was that the area of cropland estimated in the CLCD and CCI-LC continued to decrease from 2001 to 2015, while that estimated in the MCD12Q1 and GLASS-LC continued to increase. Since the implementation of ecological projects, such as “Grain to Green” in 1999 [47,49,50,72], the land system in the YRB has shifted from cropland to natural ecosystems such as forest and grassland, and the provision of ecosystem services has increased. However, the trend in area of change estimated in the MCD12Q1 and GLASS-LC for cropland contradicts the “Grain to Green” process in the YRB. With the CLCD as the reference dataset (Figure 4), the area of change estimated in the MCD12Q1 showed the highest temporal agreement for specific forest, water, barren, impervious, and wetland areas, but that estimated in the CCI-LC showed the highest temporal agreement when all nine LC classes were averaged. However, Yang et al. [58] assessed the accuracy of classification between the CLCD, MCD12Q1, and CCI-LC based on the reference dataset from the Geo-Wiki test samples in China, and showed that the overall accuracy of classification in the MCD12Q1 is second only to that in the CLCD and is better than that in the CCI-LC. These results suggest that it is not acceptable to assess the accuracy of the annual LC dataset for a single year or for only a part of the time series period.

The spatial distribution of area estimates in the grid (0.5° × 0.5°) was similar for the main LC classes (i.e., cropland, forest, grassland, barren, and impervious) between the four datasets (Figure S1), but there was significant spatial variation of the trend in area estimates of change (Figure S2). Further analysis showed that the spatial variation of the trend in area estimates of change for the main LC classes between the four datasets was mainly located in the UA and MA. This agreed with the spatial distribution of the total errors between the four datasets (Figure S4). In addition, Figure 7 showed that systematic error was dominant for the main LC classes in the YRB. These results suggest that the spatial variations of the trend in area estimates of change for the main classes between the four datasets may be related to a model (e.g., a linear regression model) and can be corrected [73]. However, for the classes covering only small areas (such as shrubland, water, snow/ice, and wetland), the coarse resolution LC datasets were not suitable for further analysis.

4.2. Potential Source of Errors

Different satellite platforms, sensors, data resolutions, classification schemes, and landscape heterogeneity [34,35,38] can introduce errors into the LC dataset and propagate to affect the area estimation and simulation of the global environment [38,74,75,76]. In this study, the potential source of errors may stem from the classification schemes, pixel counting method, the landscape heterogeneity, the reference dataset, the spatial resolution of the dataset, and the spatial assessment unit.

4.2.1. Classification Scheme

To compare various LC datasets with different classification schemes, datasets with detailed classes are often transformed into a generalized scheme with fewer classes [33,34]. In this study, we used the classification scheme of the CLCD as the target classification scheme (Table 1), and reclassified the MCD12Q1 (IGBP, 17 classes), CCI-LC (LCCS, 22 classes) and GLASS-LC (7 classes) into nine classes (Tables S1–S3). As both the MCD12Q1 and CCI-LC used detailed forest and mosaic classifications, the effect of misclassification and missing data from the reclassification cannot be ignored [34,35,36]. For example, the significant spatial difference in the area estimates for forest, cropland, and grassland areas between the four datasets was partly due to the effect of misclassification and missing data. As there are 19 LC classification schemes worldwide [77] and the class definitions are not continuous, efforts are needed to derive a universal classification scheme with a unique class definition [34].

4.2.2. Pixel Counting Method

The pixel counting method is a simple way of calculating an area by determining the number of pixels assigned to an LC class and multiplying this number by the area of a pixel to obtain the area of the LC class. This method has been widely used in remote sensing studies in its simplest form, but it is also known to be biased due to the effects of commission and the omission errors [73,76,78,79]. In this study, the trend in area of change for cropland estimated in the CLCD and CCI-LC was different from that estimated in the MCD12Q1 and GLASS-LC (Figure 3 and Figure S6), and the total errors between the four datasets were significant in the YRB (Figure S7). This confirmed that area estimation by the simple pixel counting method was biased. As shown in Figure 7, the dominant systematic error components suggested that the errors could be corrected and reduced. Calibration using confusion matrices or regression was the recommended approach to reduce the errors in area estimation [38,73,76,79]. Therefore, the pixel counting method does introduce errors into the area estimates of the LC class, and consideration should be given to reducing or eliminating the bias.

4.2.3. Landscape Heterogeneity

Landscape heterogeneity can affect map accuracy and area estimation and introduce errors into LC datasets [73,80,81]. In general, the landscape heterogeneity increases the complexity of the ground, making it difficult for the satellite to distinguish the ground spectral information [80]. This results in a decrease in the map accuracy of LC and an increase in the errors of LC with landscape heterogeneity (or fragmentation), especially for the coarse resolution datasets [79,81]. In the YRB, due to the combined effects of climate change, geomorphology, and human activity [72,82], the landscape heterogeneity is high in the Loess Plateau region (roughly in the UA and MA of the YRB; Figure S5). The total error was also high for the main LC classes in the UA and MA (Figure S4) and increased with landscape heterogeneity (Figure 8). This confirmed that the hilly and gully landscape of the Loess Plateau, which increased landscape heterogeneity [36], influenced the spatial variation of the total error (especially the percentage of systematic error) between various annual LC datasets. It is worth noting that the spatial agreement of the trend in area estimates of change for the main LC classes between the four datasets was also high in the UA and MA (Figure 6). This suggested that the area estimates of change for the main LC classes were highly correlated, and that a calibration method incorporating landscape heterogeneity held the promise of reducing errors.

4.2.4. Reference Dataset

The quality of the reference dataset was another important factor affecting the assessment accuracy and area estimation of LC datasets [38,76,83,84,85,86]. As the accuracy of the current annual LC datasets provided by the data producers were based on different reference datasets, it is difficult to determine which LC datasets have the highest accuracy and are suitable for use in a specific study area in China [34]. In addition, the lack of reference datasets with annual temporal resolution may also reduce confidence in the accuracy of the annual LC datasets [84]. In this study, since a high-quality annual reference dataset of the YRB was not available, the CLCD dataset was selected as the reference dataset (Table 1). According to previous studies [32,38], it is acceptable that the CLCD could provide more accurate information about the LC class, as it had the highest spatial resolution (30 m) of the four annual LC datasets. However, there are problems to be noted with this choice of reference dataset. On the one hand, the CLCD dataset had a classification scheme of only nine classes which means that detailed information about the LC dataset is lost [34]. Several classes need to be expanded such as (1) irrigated croplands and rainfed croplands, rather than cropland, and (2) evergreen needleleaf forests, evergreen broadleaf forests, deciduous needleleaf forests, deciduous broadleaf forests, and mixed forests, rather than forest. Furthermore, the overall accuracy of the CLCD was 79.31%, according to the data producer [58], which is lower than the accuracy of 85% recommended by previous studies [38,76,83,84]. This may reduce confidence in the results derived from this study.

4.2.5. Spatial Resolution of Dataset

The spatial resolution of the LC dataset can also affect the assessment accuracy and area estimation of LC datasets [38,73]. On the one hand, the spatial resolution of a dataset can affect the spatial errors of a dataset. In general, it is accepted that a high-resolution dataset provides fewer spatial errors compared to a coarse resolution dataset [34,39]. This was confirmed by the results showing higher total errors in the GLASS-GLC compared to the CCI-LC and MCD12Q1 (with the CLCD as the reference dataset) (Figure 8 and Figure S4). On the other hand, the spatial resolution of a dataset affects the occurrence of mixed pixels, which affects the area estimates [73,80]. Generally, the number of mixed pixels was considered to result from the combined effect of the spatial resolution and spatial patterns of a class in the landscape, and the number of mixed pixels was expected to decrease as the spatial resolution increased [80]. In addition, the area estimate was expected to be close to the true area value. This was confirmed by the results showing that the area estimates of the CCI-LC and MCD12Q1 were close to that of the CLCD compared to that of the GLASS-LC (Figure 3 and Figure 4). It should be noted that, as a trade-off between the within-class variability and the boundary effect, assessment accuracy does not always increase with spatial resolution [80,87]. A finer resolution could lead to higher within-class variances, resulting in higher classification errors and variance in the LC class area estimates. Therefore, the errors associated with any LC dataset should be carefully evaluated to ensure that the quality of the LC dataset meets a specific user-defined target.

4.2.6. Spatial Assessment Unit

The spatial unit that serves as the basis for the location-specific comparison of the reference classification and the map classification can be a pixel, a polygon, or a block [38,76,88]. In this study, since the spatial resolution of these datasets varies from 300 m to 4 km, we had to resample the dataset before performing a pixel-to-pixel comparison. To avoid any misclassification introduced by the resampling method, we used a 0.5° × 0.5° grid (roughly equivalent to a “block”) instead of a pixel as the basic spatial assessment unit (Section 2), and calculated the area estimates of the LC classes in each grid (0.5° × 0.5°) using the pixel counting method. We also calculated the area estimates in the watershed and in the entire YRB (roughly equivalent to a “polygon”). The choice of spatial assessment unit can affect the calculation of the metrics used in this study, such as landscape heterogeneity (Section 2). As mentioned in Section 4.2.3, an increase in landscape heterogeneity can lead to a decrease in the map accuracy of LC and an increase in the errors of LC [79,81], so the relationship between the spatial assessment unit and landscape heterogeneity was essential to reveal the effect of the spatial assessment unit on the map accuracy and the area estimates of the LC classes. In this study, the landscape diversity of the YRB was 1.1343, and that of the grids in the YRB was 0.6223 ± 0.3381 (mean ± std.) (Figure S5). The landscape heterogeneity decreased as the size of the spatial assessment unit changed from a “polygon” (large) to a “block” (small). This was similar to the results of a previous study [88]. This suggests that the map accuracy is expected to increase as the size of the spatial assessment unit decreases, possibly due to the change in within-class variances. However, as mentioned in Section 4.2.5, the trade-off between the within-class variability and the boundary effect together determines the map accuracy. We could not assert that the map accuracy always increases as the size of the spatial assessment unit decreases. The relationship between spatial assessment unit, landscape heterogeneity, and classification accuracy needs further investigation.

4.3. Implications for Ecosystem Services Valuation

Understanding land change processes has been shown to be critical in addressing the current policy challenges related to land use and sustainability solutions [89]. The accurate valuation of ecosystem service values (ESV) based on LC is a tool for raising public awareness of natural resource scarcity and informing policy decisions [90]. A recent global valuation of ESV based on various LC datasets showed that different inputs of LC data resulted in an ESV ranging from USD 35.0 to 56.5 trillion/a [91]. This suggests that the inputs of LC data have a significant impact on the valuation of ESV. In this study, we found significant temporal and spatial variation in trends in area of change for the main LC classes between various LC datasets in the YRB (Figure 3 and Figure S4). Ignoring such variation will inevitably lead to divergent results in ESV, which in turn will hamper the land policy-making process. Therefore, policy makers need to fully consider the impact of the input of LC data from various datasets on the valuation of ESV in land policy making.

5. Conclusions

The purpose of this study is to investigate the spatial and temporal agreement and the source of error for nine LC classes between four annual datasets, i.e., the CLCD, MCD12Q1, CCI-LC, and GLASS-LC, in the YRB from 2001 to 2015. We compared the spatial and temporal agreement between the four annual LC datasets. The results showed that in the YRB, the main LC classes were grassland and cropland among the four datasets, and that the temporal trend in area of change for the main LC classes was mainly gradual. However, the temporal trends in area of change for the main LC classes differed between the four datasets. The highest temporal agreement was found in the CCI-LC reference CLCD, followed by the MCD12Q1, while the lowest temporal agreement was found in the GLASS-LC reference CLCD. Although the spatial distribution in area for the main LC classes was largely similar between the four datasets, there was significant spatial variation in the trends in area of change in the YRB. Spatial variation in area for the cropland, forest, grassland, barren, and impervious LC classes was mainly located in the UA and MA of YRB. High percentages of systematic error for the main LC classes were also located in the UA and MA, suggesting that the spatial variation in area between the four datasets may be related to some function, and that such spatial variation can be reduced mathematically. The relationship between landscape heterogeneity and the total error suggests that the spatial agreement between the annual LC datasets can be expected to improve as landscape heterogeneity is incorporated into further calibration algorithms. Future research should focus not only on the development of annual LC validation sample collections, but also on comparative studies between various annual LC datasets, taking into account scale effects on spatial and temporal agreement and sources of error. These results not only improve our understanding of the spatial and temporal agreement and sources of error between the various current annual LC datasets, but also provide support for land policy making in the YRB.