Validation of Multi-Temporal Land-Cover Products Considering Classification Error Propagation

Abstract

Reducing the lag in the accuracy assessment of multi-temporal land-cover products has been a hot research topic. By identifying the changed strata, the annual accuracy in multi-temporal products can be quickly evaluated. However, there are still two limitations in the accuracy assessment of multi-temporal products. Firstly, the setting of the parameters (e.g., the total sample size, allocation of samples in the changed strata, etc.) in the fundamental sampling design is not based on specific setting criteria. Therefore, this evaluation method is not always applicable when the product or research area changes. Secondly, the accuracy evaluation of multi-temporal products does not consider the influence of misclassification. This can lead to an overestimation of the accuracy of changed strata in single-year evaluations. In this paper, we describe how the total sample and the assignment of samples in every stratum can be adjusted according to the characteristics of the land-cover product, which improves the applicability of the evaluation. The samples in the changed strata that propagate misclassification are essentially pixels that have not undergone any land-cover change. Therefore, in order to eliminate the propagation of this inter-annual classification error, the misclassified samples are reclassified as unchanged strata. This method was used in the multi-temporal ESA CCI land-cover product. The experimental results indicate that the single-year accuracy, considering classification error, is closer to the traditional evaluation accuracy of single-temporal data. For the categories with a small ratio of unchanged strata samples to changed strata samples, the accuracy improvement, after eliminating the classification errors, is more obvious. For the urban class, in particular, the misclassification affects its estimated accuracy by 9.72%.

1. Introduction

Land cover is the basis of human existence, and therefore, research on this subject has always been topical [1,2,3,4]. Land-cover change presents a clear picture of the processes of interaction between humanity and nature, helping us to understand and analyze the range of globalization issues caused by human social processes [5,6,7,8,9,10]. For example, the identification of land-cover change can be used to monitor the impact of climate change [11], population increase [12], and ecological change [13], and allows us to take timely and appropriate conservation measures [14].

The accuracy assessment of land-cover data is the path to their practical applications [15]. However, the early land-cover data products cannot be strictly evaluated due to the lack of reference data, limiting the practical applications of these land-cover products [16,17]. With the emergence of more and more large-scale and even global land-cover products, many studies have stressed the precedence of estimating the accuracy assessment of land-cover data products [18,19]. Sampling is an indispensable means of accuracy estimation, which involves selecting representative samples and inferring the accuracy of the data product through probabilistic statistics [20]. At present, there are more than 20 different global land-cover data [21], for example, IGBP DISCover [22], GLC2000 [23,24], GLC-SHARE [25], GlobCover [26], FROM-GLC [27], GlobeLand30 [28,29], and so on. However, these products cannot provide short-term results on the change of land cover data. While moving ahead with medium- and high-resolution remote sensing technology, it is now possible to produce multi-temporal data of land cover at a global scale. The multi-temporal global land-cover data products have become the key data source for many related applications. Multi-temporal data of land cover can be used to evaluate short-term and continuous change information of feature attributes [30,31]. The current state of research on the accuracy evaluation of multi-temporal land cover products is as follows: ➊ Reducing the evaluation lag in multi-temporal data of land cover [32,33,34,35,36]. To quickly estimate accuracy for all years in a multi-temporal land-cover data product, Wickham et al. (2013) created changed strata using change information from the product. This approach estimates single-year accuracy when map labels and stratum labels cannot be matched for all pixels [33]. Tsendbazar et al. (2021) built a continuous verification framework for map accuracy evaluation, which was applied to the verification of the Copernicus Global Land Service GLC products [36]. Despite the availability of many time-series land cover products, timely validation is challenging because it requires the latest updates to a large number of samples. This framework regularly updates the entire validation dataset by making small partial updates to invariant regions and adding new samples to changed regions. This approach improves the dataset’s utilization and significantly reduces the lag in product evaluation. ➋ Monitoring the change of a single class in different periods [36,37,38]. For example, Tang et al. (2019) conducted near real-time monitoring of tropical forest disturbance from 2013 to 2015 and evaluated the results using stratified random sampling [37]. Potapov et al. (2021) considered the global variations in farmland area from the previous 20 years of the 21st century based on satellite data, estimated the area, and calculated the accuracy according to the stable and changed regions of farmland [38]. Druce et al. (2021) promoted the satellite-based dynamic monitoring of surface water by fusing optical and synthetic aperture radar (SAR) data. This was then verified by stratified random sampling and the construction of permanent water, seasonal water, and non-water layers [39]. ➌ Improving the stratification efficiency of land-cover changes [34,35,40,41,42,43]. Arévalo et al. (2020) created six stable layers and five change layers to monitor and estimate the land change activity areas. They also added buffer layers to reduce the impact of omission errors in area estimation [42]. Gong et al. (2023) analyzed a stratified sampling method that considers the frequency of change and conversion types in a stratified sampling of multi-temporal data of land cover. Moreover, binary coding was used to reduce the complexity of the stratification [43].

All these methods mentioned above have their own characteristics and advantages in multi-temporal accuracy assessment, but there are still some problems. Firstly, strict accuracy assessment should be considered as an important component of land-cover products [44]. Therefore, sampling design is needed to guide accuracy assessment practices. Repeatability is an important criterion that cannot be ignored in sampling design [45,46]. Some methods do not provide clear parameter settings for the total sample size and specific allocation of the sample size (i.e., the allocation methods for changed and unchanged strata), which greatly reduces their adaptability. Secondly, in multi-temporal land-cover accuracy evaluation, land cover often changes, causing inconsistencies between the constructed strata and map classes. As a result, traditional single-temporal accuracy evaluation methods cannot be applied. Although corresponding multi-temporal accuracy evaluation methods are available [32,33,34,35,36], they ignore the propagation of the classification error between years. Therefore, in this paper, in view of the above problems, we clearly provide the specific settings of the parameters required for sample design. The total sample size and the assignment of the samples in every stratum can be adjusted according to the characteristics of the land-cover product. Classification error, in essence, refers to the samples in the changed strata that have not, in fact, changed. Current evaluation methods overestimate the precision of change strata by not accounting for the classification errors [47,48], which would affect the authenticity of the multi-temporal single-year accuracy. In order to eliminate the spread of misclassification in the changed strata, we reclassify the misclassified samples into the unchanged strata.

In this study, based on a continuous verification framework, we considered the propagation of misclassification in the changed strata and evaluated the quality accuracy of the ESA CCI land cover (LC) data from 2010 to 2015. This process involved (1) the evaluation of single-year accuracy from 2010 to 2015, (2) the quantitative analysis of the impact of misclassification on the single-year accuracy in the multi-temporal product, and (3) the evaluation of the stability of the time-series product.

2. Methodology

We optimized an operational verification framework for annual time-series products, considering the need to prevent misclassification propagation and to quickly calculate a more accurate product accuracy for each year. This method is called time-series sampling. It includes two important parts: (1) a sampling design for determining the sample size, allocating samples (unchanged and changed strata), and conducting the evaluation analysis; and (2) an evaluation of the accuracy of the remaining years of the time-series land-cover product by partially updating the total samples for the base year and supplementing the samples in the changed regions of the land-cover product. Moreover, the misclassifications in the changed strata samples are eliminated. The specific flowchart of time-series sampling is illustrated in Figure 1 (the land-cover product is updated annually, and the range of the years is from year n to year n + k).

Figure 1. Flowchart of time-series sampling.

2.1. Sampling Design

The design of the complete sampling scheme includes the determination of the target population, the formulation of the sampling method, and the consideration of the requirements of accuracy and reliability.

2.1.1. Determination of the Total Samples

Increasing the sample size will provide more data for estimating the parameters and, therefore, increase the reliability of the estimates. This is because a larger sample size can better reflect the characteristics of the population and reduce the impact of random errors on the estimation results. Under the same conditions, large sample sizes provide more accurate precision than small samples [49,50]. However, with the increase in sample size, the cost will increase at the same time. Therefore, the sample size should be minimized while accurately assessing the population, with the aim to balance inspection costs and accuracy. Xie et al. (2015) developed an improved sample size estimation model based on the above principles [51], as follows:

$$n=({\displaystyle \frac{{\mu }_{1-\alpha /2}^2\times q_e}{r^2(1-q_e)}})/(1+{\displaystyle \frac{1}{N}}({\displaystyle \frac{{\mu }_{1-\alpha /2}^2\times q_e}{r^2(1-q_e)}}-1))$$

(1)

$$q_e={\displaystyle \sum _{i=1}^m{\omega }_i}·q_{ei}$$

(2)

where $q_e$ refers to the expected classification accuracy, $m$ denotes the number of map categories, ${\omega }_i$ represents the area ratio of the $i$th map category,  refers to the maximum relative error between the estimated and true values, $q_{ei}$ refers to the expected classification accuracy of the $i$th map category, $q_{ei}$ is obtained empirically from pre-experiments, ${\mu }_{1-\alpha /2}^2$ signifies the boundary value of the standard normal distribution at the confidence level of $1-\alpha /2$, and $n$ refers to the total sample.

2.1.2. Selection of Sampling Method

To select representative samples and better adjust the sample size when updating the samples in the time series, the stratified sampling method is selected, and stratification is conducted according to the map categories. To improve the accuracy for the rare classes [52], the following sample distribution method is adopted:

$$n_i=[p_i\times ({\displaystyle \frac{n}{2}})+({\displaystyle \frac{1}{k}})\times ({\displaystyle \frac{n}{2}})]$$

(3)

where $n_i$ is the number of samples to be allocated in a stratum $i$, $p_i$ is the percentage of area in the map category $i$, and $k$ is the number of map categories.

This model adaptively determines the total samples and allocates the samples according to the map categories and the area ratio of each category of the products to be evaluated. Therefore, it can be applied to different multi-temporal land-cover products that may have different map categories.

2.1.3. Determination of the Samples for the Changed Strata

In addition to partial revisits of the base year between adjacent years, it is necessary to extract some samples in the changed areas of two images to ensure an accurate assessment of the land-cover change. Before confirming the samples in the changed strata, the minimum number of samples required for every stratum of the changed strata needs to be determined [53,54,55]. The specific calculation is shown in Equation (4). Because of the flexibility of stratified sampling for sample adjustment, Equations (4) and (5) are used in the stratified sampling as follows:

$$n_c={\displaystyle \frac{p_c(1-p_c)}{{\sigma }_c^2}},c=1,\dots ,L$$

(4)

where $n_c$ refers to the sample size required for the stratum $c$ of the changed strata, $p_c$ refers to the expected user precision error rate for the stratum $c$, $L$ refers to the quantity of changed strata, and ${\sigma }_c$ refers to the accepted standard error of the error of commission for category c.

As shown in Figure 2, for an accepted standard error of the error of a commission (${\sigma }_c$) of 0.05, a sample size of at least 100 is required to ensure the expected user accuracy.

Figure 2. Minimum sample size calculated by combining the predicted user precision error rate and significance level.

However, such a sample distribution would undermine the statistical analysis of the unchanged area due to the scarcity of land-cover changes in most applications. Therefore, the sample allocation for every stratum in the stratified sample needs to be combined with the unchanged strata, as shown in Equation (5) [53,54]:

$$n_{Tc}={\displaystyle \frac{n{N}_c{p}_c}{{\displaystyle {\sum }_{k=1}^L{N}_k{p}_k}}}$$

(5)

where $p_c$ represents the estimated error rate of the changed strata, $N_c$ represents the total number of changed strata, ${\displaystyle {\sum }_{k=1}^L{N}_k}{p}_k$ refers to the sum of the unchanged and changed strata, and $n_{Tc}$ refers to the total sample size allocated to the changed strata when considering the unchanged region. After obtaining the total sample size for the changed strata, if $n_{Tc}$ is less than $L_c*100$, the $n_{Tc}=L_c*100$; otherwise, the value of $n_{Tc}$ remains unchanged. $L_c$ denotes the number of changed strata.

The construction of changed strata in this paper is consistent with the evaluation of NLCD accuracy [34], including urban gain, forest loss, forest gain, agriculture loss, agriculture gain, and catch-all (other changed strata, besides urban, forest, and crop, are all in the catch-all stratum). The pixels in the stratum are not repeated. For example, if a pixel appears in the urban gain stratum, it is impossible to appear in other loss strata.

2.1.4. Evaluation of the Accuracy

With the emergence of multi-temporal products, there will be inconsistency between the strata and map classes (Figure 3). The traditional accuracy evaluation approaches for single-temporal products are not suitable for the single-year accuracy evaluation of multi-temporal products.

Figure 3. Relationship between strata and map classes.

In view of the inconsistency between strata and categories, Stehman (2014) proposed calculating the accuracy for a single year by combining the unchanged strata and changed strata [56]. When combining the changed strata to calculate the accuracy of two adjacent land-cover data, the accuracy of the map class for the previous year is calculated from the previous year’s class plus the loss stratum for that class; the latter year’s accuracy is calculated from the previous year’s class combined with the gain stratum for that class.

$$OA={\displaystyle \sum _{i=1}^k\frac{N_i}{N}}\times \overline{y_i}$$

(6)

where $\overline{y_i}={\displaystyle \sum _{u\in i}{y}_u}/n_{i•}$ is the correct proportion of sample classification in the stratum $i$. The range of $\overline{y_i}$ is [0, 1]. If sample pixel $u$ is classified correctly, then $y_u=1$; otherwise, $y_u=0$. $u\in i$ indicates that the sample pixel $u$ is selected from the stratum $i$, and $k$ denotes the number of strata. $N$ is the total number of pixels in the region of interest, whereas $N_i$ is the number of pixels in the stratum $i$.

The user’s accuracy and the producer’s accuracy are calculated using the following equations:

$$R={\displaystyle \frac{{\displaystyle \sum _{i=1}^k{N}_i\times \overline{y_i}}}{{\displaystyle \sum _{i=1}^k{N}_i\times \overline{x_i}}}}$$

(7)

$$y_u=\{\begin{array}{l}1,u\in conditionA\\ 0,u\notin conditionA\end{array}$$

(8)

$$x_u=\{\begin{array}{l}1,u\in conditionB\\ 0,u\notin conditionB\end{array}$$

(9)

For the accuracy of the user, A denotes that the pixel $u$ is classified correctly and has a map class $i$, and B indicates that the pixel $u$ has map class $i$. For the accuracy of the producer, A denotes that the pixel $u$ is classified correctly and has a reference class $i$, and B indicates that the pixel $u$ has reference class $i$. $\overline{x_i}$ is the mean of $x_u$ for the stratum $i$, and $\overline{y_i}$ is the mean of $y_u$ for the stratum $i$.

However, the accuracy evaluation method does not consider the propagation of classification errors. As shown in Figure 4, the two situations in which the map is inconsistent with the reference data in the changed strata are ② and ③, respectively. Situation ② is the inconsistency between the map class and the reference in the year (T + 1). Situation ③ is the inconsistency of year T. Situation ① indicates that the map and reference data in the changed strata are consistent in the two years. According to Equation (6), in order to calculate the single-year overall accuracy of a multi-temporal product, it is necessary to count the total sample size correctly interpreted in year T + 1 in all the unchanged strata and gain strata (taking T + 1 year as an example). However, as can be seen from Figure 4, although pixel 7 is correctly interpreted in the (T + 1) year, the sample pixel will spread the misclassification in 2010. These samples that propagate the misclassification are actually pixels where the reference data remain constant for two years (samples in the unchanged strata). Therefore, in the proposed approach, first, the samples that propagate the misclassification in the changed strata are screened out. These samples are then merged into the unchanged strata to obtain a more accurate single-year accuracy.

Figure 4. Misclassification propagation of pixels in the changed strata. The unchanged strata include pixels 1 to 6. The changed strata include pixels 7 to 9. ① indicates that the map and reference data in the changed strata are consistent in the two years; ② is the inconsistency between the map class and the reference in the year (T + 1); Light green means grass; Dark green means forest.

The revised equation is as follows:

$$OA={\displaystyle \frac{{\displaystyle \sum _{i=1}^k\begin{array}{l}\{[(N_{unchange\_i}+\frac{n_{m\_change\_i}}{n_{original\_change\_i}}\times N_{change\_i})\times {\overline{y}}_{(unchange\_i+m\_change\_i)}]\\ +[\frac{n_{nm\_change\_i}}{n_{original\_change\_i}}\times N_{change\_i}\times {\overline{y}}_{nm\_change\_i}]\}\end{array}}}{N}}$$

(10)

where $n_{original\_change}$ refers to the total samples of the changed strata, $n_{nm\_change}$ refers to the samples of the changed strata without misclassification, $n_{m\_change}$ refers to the sample size in the changed strata that propagate the misclassification, $N_{change}$ refers to the total quantity of pixels in the map of the changed strata, and $N_{unchange}$ refers to the total number of pixels in the unchanged strata.

2.2. Process of Updating and Supplementing Time-Series Samples

The specific steps of the sample update and supplementation (the land-cover product is updated annually, and the range of years is from year n to year n + k) are as follows:

(1)

Starting from the n^th year (base year) of the time series, the samples of the n^th year are obtained by stratified sampling (see Section 2.1.1 and Section 2.1.2 for the specific sample size and sample distribution of each stratum);

(2)

Determining the sample of the changed strata: The changes of the multi-temporal product pixel by pixel according to the sequence of years are detected. The samples from the changed area to supplement the samples in the corresponding “changed stratum” are selected (see Section 2.1.3 for the number of samples to be distributed in each changed stratum);

(3)

Supplementing samples: The samples of the changed strata are screened out, and the pixels that spread misclassification are eliminated;

(4)

Updating samples: In order to achieve rapid map verification after the release of land cover products and avoid too much time lag between the verification results and the release time of products, the updating of verification data sets needs to be cost-effective without affecting the statistical rigor. Therefore, some revisits are used to update the verification of the data set. The update is divided into targeted and random revisits. Targeted revisit: if the samples in the base year fall within the detected change area, the samples in the base year are removed and added to the corresponding changed stratum. Random revisit: starting from each year after the base year, (100/k)% non-repetitive random sampling inspection on the sample set selected in the base year is conducted, and it is ensured that all the sample sets in the first step are sampled once in year n + k.

Additionally, compared to the existing classical multi-temporal land-cover product evaluation methods (Table 1), this paper accounts for and eliminates the transmission of classification errors in accuracy evaluation. Furthermore, instead of empirically determining the sampling parameters, this study proved a sampling design that adaptively determines the sample size and allocates samples for different land cover types. Taking the validation of ESA CCI LC as an example, the sampling design includes the following procedures: (1) determining the total samples required for global scale based on Equations (1) and (2) in Section 2.1.1, (2) deriving the sample size for each feature category of the product according to Equation (3) in Section 2.1.2, and (3) computing the number of samples for each changed stratum based on Equations (4) and (5). Since the framework for verifying multi-temporal land-cover products requires continuous updating of samples for the changed strata, the number of samples for each changed stratum is calculated as outlined in Section 2.1.3. Finally, Section 4.1 provides the detailed calculation results.

Table 1. Summary of existing methods.

Methods	Advantages	Sampling Parameters	Considering Classification Error
Wickham et al. (2013) [32]	Reducing the evaluation lag in multi-temporal data of land-cover	Empirically determined sample size	No
Tsendbazar et al. (2021) [36]		Empirically determined sampling parameters	No
Tang et al. (2019) [37]	Monitoring the change of a single class in different periods	Cochran sampling model; fixed sample size for the rare land type	No
Potapov et al. (2021) [38]		Empirically determined sample size	No
Druce et al. (2021) [39]		Empirically determined sample size	No
Arévalo et al. (2020) [42]	Improving the stratification efficiency of land-cover change	Cochran sampling model; fixed sample size for the rare land type	No
Gong et al. (2023) [43]		Probabilistic statistical sampling model; proportional distribution; fixed sample size for the rare land type	No

3. Materials

3.1. Land-Cover Data

The ESA CCI LC product is an annual land-cover data with a 300 m resolution continuously updated by the CCI-LC project of the European Space Agency (ESA). The important feature of this product is that it can ensure global consistency on the basis of the high temporal resolution [57,58,59,60,61,62]. In this study, the sample set of the base year was supplemented and updated by the time-series sampling, as described in Section 2. Finally, the accuracy of the ESA CCI LC product was validated under a global scope for 2010–2015. The 37 land cover classes of the product were reclassified into nine classes [63,64,65], as shown in Table 2:

Table 2. Nine classes after reclassification.

Classes Considered in This Study	Land-Use Types Used in the CCI LC Maps	Description
Agriculture	10, 11, 12	Rainfed cropland
	20	Irrigated cropland
	30	Mosaic cropland (>50%)/natural vegetation (tree, shrub, herbaceous cover) (<50%)
	40	Mosaic natural vegetation (tree, shrub, herbaceous cover) (>50%)/cropland (<50%)
Forest	50	Tree cover, broad-leaved, evergreen, closed to open (>15%)
	60, 61, 62	Tree cover, broad-leaved, deciduous, closed to open (>15%)
	70, 71, 72	Tree cover, needle-leaved, evergreen, closed to open (>15%)
	80, 81, 82	Tree cover, needle-leaved, deciduous, closed to open (>15%)
	90	Tree cover, mixed leaf type (broad-leaved and needle-leaved)
	100	Mosaic tree and shrub (>50%)/herbaceous cover (<50%)
	110	Mosaic herbaceous cover (>50%)/tree and shrub (<50%)
	160	Tree cover, flooded, fresh, or brackish water
	170	Tree cover, flooded, saline water
Grass	130	Grassland
	140	Lichens and mosses
	180	Shrub or herbaceous cover, flooded, fresh-saline, or brackish water
Shrub	120, 121, 122	Shrubland
Water	210	Water
Bare land	200, 201, 202	Bare areas
Urban	190	Urban areas
Ice/snow	220	Permanent snow and ice
Sparse vegetation	150, 151, 152, 153	Sparse vegetation (tree, shrub, herbaceous cover)

3.2. Reference Data

The reference data were obtained by manual visual interpretations of the selected samples through Google Earth, and the interpretation results were divided into three confidence levels. Confidence level 1: completely correct/wrong (the image is clear, and the pixels are homogeneous objects); Confidence level 2: the judgment is fuzzy (the image is clear, but there are many objects in the pixels); Confidence level 3: unable to judge (blurred image or no image). In order to compare with the accuracy of the samples screened according to the homogeneity and complete interpretation standard of pixel areas in an official document, only the samples with confidence level 1 were considered when evaluating the accuracy. Of course, points of other grades do not participate in the evaluation. Figure 5 shows the judgment results for the ground objects in the ESA CCI LC data with a confidence level of 1 in Google Earth.

Figure 5. Interpretation results for the ESA CCI LC data in Google Earth. The red frame refers to the area range of pixel.

4. Results

4.1. Analysis of the Results of the Sampling Design

4.1.1. Selection of Samples in the Base Year

The total sample size required for precision evaluation in 2010 (base year) for the ESA CCI LC product was obtained as described in Section 2.1.1 (Table 3). The predicted classification accuracy was determined according to the pre-experiment and the user’s guide for the ESA CCI LC data [60]. According to Equations (1) and (2), 14,645 samples were needed to evaluate the quality accuracy of the ESA CCI LC data for 2010 at a global scale.

Table 3. Total samples to be selected in the base year.

No.	Surface Features	Number of Pixels	Predicted Classification Accuracy	Area Ratio (Number of Pixels of Surface Feature/Total Pixels)	Total Sample Size
1	Crop	332019546	80%	3.95%	14,645
2	Forest	668301893	85%	7.96%
3	Grass	253197417	50%	3.01%
4	Shrub	175707096	60%	2.09%
5	Water	5674946201	90%	67.57%
6	Bare	253115927	80%	3.01%
7	Urban	8842764	85%	0.11%
8	Ice	871449834	90%	10.38%
9	Sparse Veg	160499322	30%	1.91%

According to the ESA CCI LC data user guide [60] and the official accuracy evaluation report for GlobCover 2009 [66], the reference data used in the evaluation of the accuracy of CCI’s 2015 product were from the validation dataset of the GlobCover 2009 product. Only inland water (extracted using the CCI Global Map of Open Water Bodies) and ice areas other than Antarctic ice were considered. As many of the classes in the GlobCover 2009 validation dataset have few validation samples (e.g., water, ice, or urban), they were area-weighted for accuracy evaluation. In this study, stratified random sampling was chosen for allocating the total sample, as Neyman allocation and proportional allocation do not address the problem of under-allocation of samples for rare categories. The most commonly used sample distribution methods in stratified sampling are proportional distribution and Neiman distribution. Proportional distribution has the advantages of simple analysis and operation because it is an equal probability sampling design. Neyman distribution needs to consider the standard deviation of each feature category. The total amount and standard deviation of features determine the sample size of this feature stratum. That is, the greater the total amount and the greater the internal differences, the more samples that should be allocated to the feature type. However, Neyman distribution and proportional distribution cannot solve the problem of insufficient sample distribution in rare categories. The standard deviations of features in ESA CCI LC products are all small and close to each other (range: 0.30–0.50), so the area ratio is more important to the sample distribution. For the urban class, its area ratio is about 0.44%, and the standard deviation is about 0.36. Therefore, the number of samples obtained using the Neyman distribution method is 57, and the proportion distribution sample is 65. On a global scale, these allocation numbers cannot accurately evaluate the accuracy of the products. For a global scale, these allocations do not allow for accurate precision evaluation. The allocation technique described in Section 2.1.2 was, therefore, used to obtain the sample allocations for each class (Table 4). The sample selection for 2010 can be seen in Figure 6.

Table 4. Results of the sample distribution.

No.	Surface Features	Number of Pixels	Allocation	Area Ratio
1	Crop	332019546	2082	16.57%
2	Forest	668301893	3345	33.35%
3	Grass	253197417	1786	12.64%
4	Shrub	175707096	1496	8.77%
5	Water	66389169	1085	3.31%
6	Bare	253115927	1786	12.63%
7	Urban	8842764	870	0.44%
8	Ice	85822331	1158	4.28%
9	Sparse Veg	160499322	1438	8.01%

Figure 6. Spatial layout of samples in the base year.

4.1.2. Determination of Supplementary Samples

According to the authors of [34], changed strata can be divided into the urban gain stratum, forest loss stratum, forest gain stratum, agriculture loss stratum, agriculture gain stratum, and catch-all stratum (other changed strata, besides urban, forest and crop, are all in the catch-all stratum), according to the types of ground object transformation. Previous accuracy evaluations of the ESA CCI LC data have reported that the overall accuracy of the unchanged strata is 71% [60,67,68], but the accuracy of the changed strata is unknown. According to Equation (5), with the increase in $p_c$ of the changed strata, more samples are allocated to the changed strata. Therefore, by assuming that the value of $p_k$ in the changed and unchanged strata is 99% and 29%, respectively, then the maximum total samples to be allocated to the changed strata can be obtained, as shown in Table 5. It is worth emphasizing that the total pixels of the unchanged strata and the samples of the unchanged strata are not always the same and need to be constantly adjusted according to the changed areas and targeted revisits.

Table 5. Total samples to be distributed in the changed strata between two adjacent years.

Adjacent Years	The Total Samples in the Changed Strata
2010–2011	65
2011–2012	49
2012–2013	41
2013–2014	72
2014–2015	10

Therefore, the total samples required for the changed strata can be determined, and the allocation method in Equation (3) can be conducted to determine the samples required for every changed stratum (Figure 2 shows that the number of samples in each stratum is at least 100, so less than 100 is made up to 100). The specific allocation is shown in Table 6. By using the above method, the sample size required by the changed strata can be reasonably calculated while maintaining the accuracy of the unchanged strata. Therefore, compared with other studies, the sampling scheme in this paper not only expands the applicability of parameters but also improves the repeatability of the sampling design.

Table 6. Sample size to be allocated for each changed stratum.

	2010–2011	2011–2012	2012–2013	2013–2014	2014–2015
Urban gain	100	100	109	100	111
Forest loss	100	100	112	161	138
Forest gain	167	160	133	140	-
Agriculture loss	100	100	100	100	-
Agriculture gain	100	100	100	100	-
Catch-all	135	136	108	100	100

4.2. Evaluation of the Advantages of Sample Revisiting

In the process of sample updating/random revisiting, the revisited samples for each year from 2011 to 2015 were obtained by randomly selecting 20% of the total samples from 2010 without repetition. By 2014–2015, all the samples had been randomly revisited. According to the randomly revisited samples in these years, the proportion of categories in the corresponding years was counted. Figure 7 shows that the proportion of categories in the other years is basically the same as that in the base year.

Figure 7. Proportion of feature categories between the randomly revisited samples and total samples.

Figure 8 demonstrates the spatial distribution of the total samples in 2010 and the samples updated by random revisits from 2011 to 2015, where (a) refers to the spatial positions of all the samples (including nine classes) in 2010. The upper and right curves are the probability density function curves of these samples, which show the distribution of these samples in longitude (upper curve) and latitude (right curve). The areas under each probability density curve sum to one. The fluctuations in the skewness and kurtosis of the features in these years are very small. It can be seen that the distribution of the feature types in the randomly revisited samples for the other years aligns well with the distribution of the total samples for the base year.

Figure 8. Spatial distribution of the samples. (a) Samples in 2010. (b–f) Samples randomly revisited in the years from 2011 to 2015.

Combined with Figure 7, it can be seen that the randomly revisited samples for the other years, although only accounting for 20% of the total samples, effectively reflect the spatial distribution of the total samples. The proportion of feature types in the randomly revisited samples is also consistent with that in the total samples. Therefore, the randomly revisited samples are selected with good uniformity and reasonableness. The targeted revisit shifts the samples where feature changes occur in the base year and places these changed base samples in the changed stratum. Land cover shifts over time are rare events [69], so sample revisiting can save considerable labor and time compared to the traditional accuracy of multi-temporal data while allowing the base year samples to be updated.

4.3. Analysis of the Accuracy Evaluation Results

As can be seen from the forest gain stratum marked in red in Figure 4, although the sample was correctly interpreted in 2011, the sample pixel will spread the misclassification in 2010. As shown in Table 7, $\overline{y_{\mathrm{c}i}}$, considering the misclassification is always lower than or equal to $\overline{y_{ci}}$, does not consider the misclassification in any year. $\overline{y_{ci}}$ refers to the correct proportion of sample classification in the changed strata. Especially for the ESA CCI LC product in 2013, the difference in $\overline{y_{ci}}$, between considering and not considering the classification errors reached, was 0.9412 ($\overline{y_{ci}}\in [0,1]$). The accuracy evaluation method by Stehman (2014) allows for a year-by-year evaluation of the ESA CCI LC data from 2010 to 2015, but the propagation of misclassification is not taken into account in this evaluation method. The accuracy evaluation described above will be affected by misclassification, so this classification error is eliminated in the assessment approach proposed in this paper.

Table 7. Correct proportion of sample classification in the changed stratum ($\overline{y_{ci}}$). CIM refers to $\overline{y_{ci}}$ which contains misclassification, and CEM refers to $\overline{y_{ci}}$ which eliminates misclassification.

Year 2010			Year 2011			Year 2012
Strata	CIM	CEM	Strata	CIM	CEM	Strata	CIM	CEM
Crop loss	0.2750	0	Crop gain	0.6154	0	Crop gain	0.5833	0.1111
Forest loss	0.6667	0	Forest gain	0.7692	0	Forest gain	0.5417	0.1667
Grass loss	0	0	Grass gain	0	0	Grass gain	0	0
Shrub loss	0	-	Shrub gain	-	-	Shrub gain	-	-
Water loss	1	-	Water gain	0.5	0	Water gain	-	-
Bare loss	0.6842	0	Bare gain	0.5	0	Bare gain	0.6667	0
Urban loss	-	-	Urban gain	0.9216	0.2	Urban gain	0.9333	0.4
Snow loss	-	-	Snow gain	-	-	Snow gain	-	-
Sparse loss	0.4286	0	Sparse gain	0.3750	0	Sparse gain	0.1429	0
Year 2013			Year 2014			Year 2015
Strata	CIM	CEM	Strata	CIM	CEM	Strata	CIM	CEM
Crop gain	0.9048	-	Crop gain	0.4667	0	Crop gain	-	-
Forest gain	0.3684	0	Forest gain	0.5750	0	Forest gain	-	-
Grass gain	0	0	Grass gain	0	0	Grass gain	-	-
Shrub gain	-	-	Shrub gain	-	-	Shrub gain	-	-
Water gain	-	-	Water gain	1	-	Water gain	-	-
Bare gain	0	-	Bare gain	0.4	0	Bare gain	0	0
Urban gain	0.9412	0	Urban gain	0.8966	0	Urban gain	0.9079	0
Snow gain	-	-	Snow gain	-	-	Snow gain	-	-
Sparse gain	0.25	0	Sparse gain	0	0	Sparse gain	-	-

The overall accuracy of ESA CCI LC for 2015, given by the official documents, is 71.1%. However, the reference data used in the official document was the verification dataset of GlobCover 2009. Because the classification systems of GlobCover 2009 and the ESA CCI LC product are inconsistent, only some samples in the verification dataset can be used [66]. This practice leads to a very small sample size for some features, which increases the interference of regional heterogeneity on the overall accuracy. Therefore, in the proposed approach, we kept the total sample size (2329) the same as the official document. According to the sampling design scheme proposed in this paper, the samples of the ESA CCI LC data in 2015 were selected, and the reference data were collected. Finally, the overall accuracy for 2015 was determined to be 66.89% (Table A1). The samples in Table A1, Table A2 and Table A3 are all sample points with a confidence level of 1 (in order to better compare with the overall accuracy obtained by homogeneous pixels in the official document [60]).

In order to objectively compare the single-temporal accuracy with the accuracy for a single year calculated by multi-temporal calculation, for 2015, 14645*20% = 2929 samples were randomly revisited in 2014−2015, and 2329 recollected samples were merged into the total samples for 2015. According to the 5258 total samples, the single-temporal overall accuracy for 2015 was 66.71% (Table A2). We then randomly selected 20% of the samples of each feature (repeated 10 times) and performed targeted revisiting as the samples for the 2014–2015 changed strata. Since the total pixels of the changed strata/the total pixels of the unchanged strata were approximately 0.0009, the samples of the changed strata to be allocated were the same as that of Section 4.1.2 in combination with Equation (5), so the samples of the changed strata for 2014–2015 in Table 6 were directly adopted. The accuracy without considering the misclassification and considering the misclassification can be calculated by using Equations (6) and (10), respectively. The result after averaging is demonstrated in Figure 9. Figure 9a shows that the impact of misclassification has a significant impact on the urban class (9.72%), while the other features remain unchanged. Because the main type of feature that changed in 2015 was urban (Table 7), although there were also some samples in the bare land gain strata, regardless of whether the misclassification was considered or not, $\overline{y_{ci}}$ of this stratum is always zero. Therefore, the accuracy of bare land remains unchanged. In addition, the accuracy of urban without misclassification is closer to the accuracy for a single-temporal obtained from the total samples in 2015.

Figure 9. Single-year precision results for the average of 10 times extracted from the single-temporal samples in 2015. (a) Difference between the accuracy when eliminating the misclassification and including the misclassification. (b) Difference between the single-year accuracy and single-temporal accuracy. WMC represents the difference between the accuracy when eliminating misclassification and the single-temporal accuracy (reference value). IMC represents the difference between the accuracy when including the misclassification and the single-temporal accuracy.

The same procedure was carried out for 2010, where we randomly selected 20% of the total sample (14,645) of each feature (repeated 10 times). The mean value of ten precision results for this year is demonstrated in Figure 10. Figure 10 demonstrates that the WMC of almost all the features is smaller than the IMC, indicating that the accuracy after eliminating misclassification is closer to that of single-temporal accuracy (Table A3). Due to the small difference between eliminating misclassification and including misclassification for each feature, the overall accuracy difference is also very small.

Figure 10. Single-year precision results of the average of ten times extracted from single-temporal samples in 2010. (a) Difference between the accuracy of eliminating the misclassification and including the misclassification. (b) Difference between the single-year accuracy and single-temporal accuracy.

Compared with the other features, there are more times when there is a single land-cover class in the urban pixels. Therefore, after judging confidence, more samples are left in the changed strata of urban. In addition, urban accounts for only 0.44% of the total, so there are very few samples of unchanged strata allocated to urban. Therefore, compared to the other features, the ratio of unchanged strata samples to changed strata samples will be smaller. The elimination of misclassification has a particularly significant impact on urban. Therefore, for the forest in 2010 (this feature has the largest share of the total), 20% to 5% of randomly selected samples from the total samples of the feature and the samples in the forest loss stratum were increased to reduce the ratio of the unchanged strata to the changed strata. The values in Figure 10 were retained for the other features, and Figure 11 was ultimately obtained. Compared to Figure 10b, the forest in Figure 11 is more obviously affected by misclassification, and the overall accuracy is significantly closer to the single-temporal accuracy for 2010. The standard deviation of the accuracy of the single-year forest after eliminating the influence of misclassification is 2.53%. The standard deviation, including misclassification, is 6.61%, so the stability when eliminating misclassification is also better. This method is more effective in improving the accuracy of features with a small sample ratio of unchanged strata/changed strata.

Figure 11. Single-year precision results of the average of ten times extracted from single-temporal samples in 2010.

Figure 11 shows that the accuracy is better after removing the effect of misclassification, and the steps in Section 2.2 allow an evaluation of the global accuracy for 2010–2015, as shown in Figure 12. The accuracy of the CCI product was more stable in 2010–2015, with a range of 67.69–67.82%.

Figure 12. The accuracy of the ESA CCI LC for 2010–2015.

4.4. Stability Evaluation of Land-Cover Products

In this paper, we have described how to evaluate the product accuracy of a global annual time-series land-cover product on a year-by-year basis through a time-series sampling process framework. The inter-annual stability of the global time-series land-cover product can be evaluated in conjunction with Equation (11):

$$SI={\displaystyle \frac{|c{a}_{n+1}-c{a}_n|}{c{a}_n}}\ast 100$$

(11)

where $c{a}_n$ refers to the category accuracy for the year $n$, and $c{a}_{n+1}$ refers to the category accuracy for the year $n+1$.

Figure 13 shows the user’s accuracy and the producer’s accuracy of each feature over the years. These two accuracies of grasslands, shrubs, and sparse vegetation are all low, while the accuracies for water bodies are high.

Figure 13. User’s accuracy and producer’s accuracy of each class. (a) refers to Crop; (b) refers to Forest; (c) refers to Grass; (d) refers to Shrub; (e) refers to Water; (f) refers to Bareland; (g) refers to Urban; (h) refers to Snow; (i) refers to Sparse Veg.

Figure 14 and Figure 15 represent the stability of the ESA CCI LC data in terms of the producer’s accuracy and the user’s accuracy for 2010–2015. According to the authors of [36], a stability index below 15% indicates a stable product. Figure 14 and Figure 15 clearly show that the ESA CCI LC product not only provides annual global land-cover data but also maintains a high level of stability. Except for the urban class, the other features are below 5% for both the user’s and the producer’s accuracy. The class of snow, in particular, has maintained a constant stability of the user’s accuracy over the period of 2010–2015.

Figure 14. Stability of the producer’s accuracy for the ESA CCI LC product.

Figure 15. Stability of the user’s accuracy for the ESA CCI LC product.

5. Discussion

In this paper, the problem of classification error propagation in a homogeneous pixel is considered and eliminated. In order to ensure the evaluation efficiency of products, the verification framework updates some sample points, so there will be some error propagation in reference data [36]. This is also a problem that needs to be studied and solved in the future. However, the samples of the changed strata are always re-selected, and the results obtained by combining the method in this paper do not have the cumulative propagation of classification errors in the changed strata. In addition, the main changed strata types in this paper are determined based on the evaluation method of NLCD. Therefore, for time series products with time intervals greater than 5 years, further research is needed on the main changed strata types. This method is suitable for products with a relatively long time span and supports change detection. Nonetheless, products with different resolutions will require further verification and explanation in the future.

6. Conclusions

In this paper, we clearly described the setting of the parameters, including the sample size and the allocation of samples in the changed/unchanged strata, which can be adjusted according to the characteristics of the land-cover product. Then, the influence of misclassification was considered in the evaluation method.

Especially for the ESA CCI LC product in 2013, the difference in $\overline{y_{ci}}$ between considering and not considering classification errors reached was 0.9412 ($\overline{y_{ci}}\in [0,1]$). In contrast with the previous precision evaluation of multi-temporal products, we have described how to consider the influence of the propagation of misclassification in the changed strata on the single-year precision. The experimental results showed that the overall accuracy and the accuracy of individual categories, after eliminating misclassification, are closer to single-temporal accuracy with more samples than when misclassification is not considered. For the urban category, which typically had few mixed pixels and a small proportion in the classification map, the ratio of samples in the unchanged strata to changed strata was low. After eliminating misclassification, the accuracy improved by 9.72%. The forest class (the largest proportion of land cover) showed an improved accuracy after considering the classification error when the ratio was reduced. Therefore, in a specific area with a large proportion of changed areas, the method proposed in this paper will improve the authenticity of the evaluation more obviously. In addition, all features of the ESA CCI LC product were below 15% for both the user’s and the producer’s accuracy. Thus, the time-series product was shown to have good stability.