Spatial Stratification Method for the Sampling Design of LULC Classification Accuracy Assessment: A Case Study in Beijing, China

Abstract

Spatial sampling design is important for accurately assessing land use and land cover (LULC) classification results from remote sensing data. Spatial stratification can dramatically improve spatial sampling efficiency by dividing the study area into several strata when classification correctness is spatially stratified heterogeneous. By integrating the LULC classification results from different sources and spatial resolutions, a spatial stratification method for spatial sampling of accuracy assessment is presented in this paper. Its efficiency is demonstrated in the case study using LULC data of Beijing, China, in the following steps. First, we standardized and reclassified multiresolution remote sensing data, including China’s land use/cover datasets (CLUDs) from 2017 (resolution: 30 m), 500 m MCD12Q1, and 10 m FROM-GLC10 data, into six classes. Second, we customized stratification rules, formulated a technical specification to realize 11 strata using CLUDs and MCD12Q1, and employed FROM-GLC10 as the reference data for accuracy assessment. Furthermore, six sample sets with sizes of 16,417; 1821; 652; 337; 198; and 142 were drawn using different methods, and their overall accuracy (OA), deviation accuracy (DA), root-mean-square error (RMSE), and standard deviation (STDEV) values were also evaluated to demonstrate the efficiency brought by spatial stratification. Compared with the spatial even sampling method, the OAs of the stratified even sampling method adopting the proposed stratification method was much closer to the true OA, and the corresponding RMSE and STDEV results decreased from 2.097% and 2.127% to 0.914% and 0.713%, respectively, due to the contribution of spatial stratification in the sampling scheme. The method can be used to distinguish the differences and improve the representativeness of samples, and it can be employed to select validation samples for LULC classification.

1. Introduction

Land use and land cover (LULC) information is fundamental for cropland protection, ecological and environmental change studies, and sustainable development [1,2,3], and LULC has changed markedly due to frequent human activities at global and regional scales [4]. The classification accuracy of the LULC maps is the key to their applications. To assess the accuracy of LULC maps, sampling sites need to be established to obtain the true ground classes and compare with the classified results of LULC maps. Classification accuracy assessment not only describes the quality of a map [5], but also provides a means to enhance its usefulness [6]. However, due to the high cost of field sampling, only a limited number of sites can be sampled. Therefore, the validated sites for LULC classification should be distributed efficiently [7]. Moreover, the representatives of both feature space and geographical space should be considered [8,9]. Geographical space consists of latitude, longitude, and elevation, or plane coordinates after map projection [10]. The feature space, or attribute space, is a virtual space with each attribute as an axis [11,12]. Stratification, which divides the study area into several more homogeneous subregions, is a useful tool to improve the representativeness of samples in feature space and can simplify spatial heterogeneity [13]. Stratification strategies can be grouped into two main types: direct and indirect stratification strategies. The direct stratification strategy directly divides the spatial coverage using existing subregional units, e.g., LULC type, ecological zonation, and administrative unit. For example, the stratified even sampling method utilizes the LULC type of the product, whose classification accuracy needs to be assessed, as stratification and then adopts spatial simulated annealing (SSA) to distribute sampling sites evenly in the strata [14]. Existing subregional units, however, are not sufficiently precise in representing the spatial distribution of misclassification. The indirect stratification strategy employs certain clustering methods [15] to achieve spatial stratification using auxiliary factors [16], e.g., prior knowledge, historical data, and ancillary data.

The target population for the spatial sampling design of LULC classification accuracy assessments is an image consisting of pixels with a value that is true (1 if the classification is correct) or false (0 if the classification is wrong) [14]. The stratification should reflect the spatial heterogeneity of those true/false values. The consistency and inconsistency of LULC classification products from different sources and spatial resolutions can reflect the probability of misclassification to some extent, although they are produced using different spatial information, prior knowledge, and classification methods [17,18,19]. The parts where the classification results of different products with different sources and spatial resolutions are consistent, suggest easy-to-identify and low probability of misclassification. In contrast, the parts where the classification results of different products are inconsistent suggest hard-to-identify and high probability of misclassification. Furthermore, the misclassification probability is the basis of spatial stratification of the target population for accuracy assessments. Therefore, this study proposes a spatial stratification method by integrating the consistency and inconsistency of different products for spatial sampling to evaluate the LULC classification accuracy assessment. The performance of the proposed method is demonstrated using the LULC classification products of Beijing, China, as a case study. In the case study, we first standardized and reclassified multiresolution remote sensing data into six classes, then applied the stratification method to obtain several strata, and finally carried out the stratified even sampling method to assess the classification accuracy and compared its performance with spatial even sampling with different sample sizes.

2. Spatial Stratification Method

The proposed spatial stratification method aims to divide the target population into several homogeneous subregions with a close probability of misclassification. Given that the true misclassification is unknown and needs to be inferenced, the spatial stratification method tries to derive information of spatial heterogeneity from the classifications of other LULC products with different sources and spatial resolutions. It is different from the stratified even sampling method, which directly utilizes the LULC types of the product to be assessed as strata. As the LULC type cannot directly represent the spatial distribution of the misclassification probability, the spatial stratification proposed in this study improves it by integrating the consistency and inconsistency of different data products with different sources and spatial resolutions to represent the misclassification probability. In this new spatial stratification method, by comparing the classification results of target data, whose classification accuracy needs to be assessed, using other ancillary LULC products, both consistent and inconsistent classifications can be obtained. Those pixels that all LULC products give out the same judgment could have low probability of misclassification, while pixels that all products give out different judgment may have high probability of misclassification. According to the consistency and inconsistency of the comparison, e.g., three different LULC classification products shown in Figure 1, the target population can be stratified.

The spatial stratification method developed in this study is composed of three steps, as illustrated in Figure 2.

Step 1: Data standardization and reclassification. Prepare the target data and obtain K different LULC products of the same area and time period. Then standardize and reclassify those LULC products to make them comparable.

Step 2: Define the stratification rules and basic stratification units. The spatial overlay analysis of the K + 1 reclassification results was used for spatial stratification by the customized stratification rules. Based on the results of spatial overlay analysis, basic stratification units of each LULC class were obtained for the study area according to the different attribute values of the pixels.

Step 3: Spatial stratification results. Through a spatial combination of basic stratification units, spatial stratification results of each LULC class were achieved accordingly. The strata of all classes were integrated as the spatial stratification results of the study area.

For K + 1 LULC classification data with different sources and spatial resolutions, the target data whose classification accuracy needs to be assessed and K ancillary data with different resolutions should be standardized and reclassified to make them comparable. After the reclassification, the class of one pixel in the target data and k ancillary data are donated with P_o and P_i (i = 1,···, k), respectively.

For each class of the target data, we can stratify it by comparing P_o with P_i, i.e., P_i = [P₁, P₂, ···, P_k], using the spatial overlay analysis. At each pixel, a vector with k binary values can be obtained, for example [1, 0, …, 1], with 1 donating that P_o = P_i, and 0 donating P_o ≠ P_i. There are at most 2^k possible combinations, and the rules for stratification could be designed based on the vector. In practice there will be less than 2^k combinations because some combinations may not exist. When k is small, the vector can be directly used as stratification rules. When k is large, the rules can be designed based on the summary of the vector, for example, using the sum of the vector as stratification rules.

Then, the final spatial stratification results of the study area can be obtained by iterating all classes of the target data and stratifying them as above steps.

3. Case Study

3.1. Data Sources and Experiment Roadmap

CLUDs (China’s land-use/cover datasets) MCD12Q1 and FROM-GLC10 data for Beijing (115°25′ E–117°30′ E, 39°28′ N–41°05′ N) were used in this study. The 30 m CLUDs for 2017 were provided by the Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences [20]. The 500 m MCD12Q1 data for 2017 were developed by Boston University [21], and the 10 m FROM-GLC10 data for 2017 can be freely downloaded from http://data.ess.tsinghua.edu.cn (accessed on 23 September 2019) [22]. The datasets were first converted by file formatting and then mosaicked and reprojected into the UTM zone 50N projection with the WGS 84 datum using nearest-neighbor resampling. The number of classes of MCD12Q1, CLUDs, and FROM-GLC10 data for Beijing was 11, 19, and 8, respectively, and corresponding three datasets are shown in Figure 3a,c,e respectively. To make them comparable, MCD12Q1, CLUDs, and FROM-GLC10 data for Beijing were transformed to have six classes, i.e., cropland, woodland, grassland, water body, built-up land, and unused land. The classification system and corresponding relationships are shown in

Table 1. Classification system and corresponding relationships of different datasets in Beijing.

Classes	MCD12Q1	CLUDs	FROM-GLC10
Cropland	Croplands	Paddy Land Areas; Dry Land Areas	Cropland
Woodland	Deciduous Broadleaf Forests; Mixed Forests; Closed Shrublands; Woody Savannas; Savannas	Forests; Shrublands; Woodlands; Other Woodlands	Forest; Shrubland
Grassland	Grasslands	Dense Grasslands; Moderate Grasslands; Sparse Grasslands	Grassland
Water Body	Permanent Wetlands; Water Bodies	Streams and Rivers; Lakes; Reservoirs and Ponds; Bottomlands	Wetland; Waterbody
Built-up Land	Urban and Built-up Lands	Urban Built-up Land Areas; Rural Settlements; Other Built-up Lands	Impervious Area
Unused Land	Barren	Swampland; Bare Soil Areas; Bare Rock Areas	Bare Land

, and the LULC reclassification results of the three datasets are shown in Figure 3b,d,f respectively. Data processing was conducted using ENVI 5.1 (ITT Visual Information Solutions, Circle Boulder, CO, USA) and ArcGIS 10.5 (Environmental Systems Research Institute, Inc., Redlands, CA, USA) software.

In this case study, the 2017 30 m CLUDs was treated as target data, and 500 m MCD12Q1 was used as ancillary data to help carry out the spatial stratification. Given the true land class was unknown, the 2017 10 m FROM-GLC10 with the higher spatial resolution was adopted as reference data to act as the true ground land class. The experiment roadmap is illustrated in Figure 4 following the spatial stratification method in the previous section.

3.2. Spatial Stratification

In the experiment, the CLUDs and MCD12Q1 were standardized and reclassified according to Table 1, and overlaid to obtain spatial strata. For each class of CLUDs, two possible strata can be obtained: one includes pixels that CLUDs and MCD12Q1 have the same class, the other includes pixels where CLUDs and MCD12Q1 have different classes. Specifically, the class of one pixel in the CLUDs and MCD12Q1 are donated with P_o, and P_l, respectively.

For each class of the CLUDs, we can stratify it by comparing P_o with P_l through the stratification rules illustrated in Figure 4. Based upon the comparison results using spatial overlay analysis, two possible stratifications labeled as stratum Ⅰ and stratum Ⅱ can be obtained. Stratum Ⅰ is composed of those pixels s that belong to class P_o in the CLUDs and P_o(s) = P_l(s), and those with P_o(s) ≠ P_l(s) are divided into stratum Ⅱ. For the CLUDs and MCD12Q1, each LULC class can be divided into two strata at most, as depicted in

Table 2. The strata of each LULC class based on the CLUDs and MCD12Q1.

ID	LULC	CLUDs (30 m)	MCD12Q1 (500 m)	Strata
1	Cropland	√	√	Cropland Ⅰ
2		√	×	Cropland Ⅱ
3	Woodland	√	√	Woodland Ⅰ
4		√	×	Woodland Ⅱ
5	Grassland	√	√	Grassland Ⅰ
6		√	×	Grassland Ⅱ
7	Water Body	√	√	Water Body Ⅰ
8		√	×	Water Body Ⅱ
9	Built-up Land	√	√	Built-up Land Ⅰ
10		√	×	Built-up Land Ⅱ
11	Unused Land	√	√	Unused Land Ⅰ
12		√	×	Unused Land Ⅱ

. By iterating six classes of the CLUDs and stratifying them according to the steps given above, the spatial stratification results were achieved.

3.3. Sampling Optimization

The values of sampling sites were obtained by comparing the land class of CLUDs with that of the reference data. If they are consistent, the value is set to one, otherwise zero. The corresponding estimation methods were then used to assess the overall accuracy (OA) of the whole study area, whose true value was the mean of the target population.

The target population was obtained through a pixel-by-pixel comparison of CLUDs and reference data, and true accuracy OA₀ is calculated as:

$${\mathrm{OA}}_0=\left({\displaystyle \sum }_{i=1}^n{P}_{ii}\right)/N_{total}$$

(1)

where P_ii refers to the number of sampling sites that were correctly classified, n is the number of LULC types, and N_total is the total number of pixels.

To evaluate the efficiency of the stratification, a stratified even sampling method following the divided strata was used to draw samples with different sizes to estimate the OA of CLUDs. The stratified even sampling method included spatial stratification and sample allocation in feature space and sampling optimization in geographical space [14], which was developed based upon compound sampling strategy [23] and polygonal declustering estimation [24]. After the spatial stratification only using the LULC type of the product to be asessed, the SSA and the minimization of the mean of the shortest distances (MMSD) criteria were employed to optimize sampling sites in geographical space [25,26,27]. The OA of the stratified even sampling method is defined as:

$$\mathrm{OA}={\displaystyle \sum }_{i=1}^k\left({\mathrm{OA}}_i\times W_i\right)$$

(2)

where OA_i is the overall accuracy of the i_th stratum and the calculation of OA_i is specifically introduced in [14]; k is the number of strata; and W_i is the weight of the i_th stratum, which is estimated by the ratio of the area of this stratum to the total area.

In addition, the frequently used spatial even sampling method, which was realized by designing a series of kilometer grids to cover the study area and taking the center of a grid as a sampling site [28], was used as a comparison. The OA for this method can be estimated by:

$$\mathrm{OA}={\displaystyle \sum }_{j=1}^n\left(V_j\times w_j\right)$$

(3)

where V_j refers to the value of the j_th sampling site, and V_j is set to 1 if the classification is correct, otherwise zero; n is the total number of sampling sites; and w_j is the weight of the area surrounding the j_th sampling site by polygonal method.

The total sample size of this case study was determined by Foody [29], and the required sample size needed to estimate the population proportion of correctly allocated cases in classification can be calculated by:

$$M=\frac{z_{\alpha /2}^{2}P\left(1-P\right)}{h^2}$$

(4)

where M is the sample size,$z_{\alpha /2}$ is the critical value of the normal distribution for the two-tailed significance level α, P is a planning value for the correctly allocated case population proportion, and h is the half-width of the desired confidence interval.

The area-weighted proportion method was adopted to allocate sampling sites in the case study [30]. The sample size of the k_th stratum N_k is defined as:

$$N_k=N\times \frac{S_k}{S}$$

(5)

where S_k is the area of the k_th stratum, S is the total area, and N is the total sample.

3.4. Comparative Metrics for Accuracy Assessment

The estimated OA was compared with the true OA and the deviation accuracy (DA), root-mean-square error (RMSE), and standard deviation (STDEV) were adopted to measure the assessment accuracy. The DA and RMSE measure the deviation between the observed and true values, while the STDEV measures the discrete range of a given dataset. The spatial stratification method was more effective when the values of DA, RMSE, and STDEV were low. The DA, RMSE, and STDEV are calculated, respectively, as follows:

$$\mathrm{DA}=\left|{\mathrm{OA}}_j-{\mathrm{OA}}_0\right|$$

(6)

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}{\displaystyle \sum }_{j=1}^n{\left({\mathrm{OA}}_j-{\mathrm{OA}}_0\right)}^2}$$

(7)

$$\mathrm{STDEV}=\sqrt{\frac{1}{n-1}{\displaystyle \sum }_{j=1}^n{\left({\mathrm{OA}}_j-\overline{\mathrm{OA}}\right)}^2}$$

(8)

where OA_j is the overall accuracy estimated by the j_th sample, OA₀ is the true accuracy of the classification data, $\overline{\mathrm{OA}}$ is the mean overall accuracy of all the samples, and n is the number of samples.

In addition to OA, producer accuracy (PA) and user accuracy (UA) were also calculated to reveal the classification accuracy in detail.

4. Results

4.1. Spatial Stratification of CLUDs and MCD12Q1 for Beijing

According to the experiment roadmap and spatial stratification rules in Figure 4 and Table 2, the 2017 30 m CLUDs of Beijing was stratified into 11 strata as shown in Figure 5. The unused land Ⅰ is absent in the results because CLUDs and the ancillary data (MCD12Q1) do not match each other at any pixel in judging the unused land.

4.2. Sampling Optimization and Sample Allocation

To obtain a stable performance, we set different sample sizes. The minimum sample size was determined with Equation (4). A 0.05 significance level is adopted by convention and $z_{\alpha /2}$ equal to 1.96 [29]. One hundred thirty-nine sampling sites were required to obtain a target accuracy of 90%. The sample size needs to be larger than 139; thus, grids of different sizes were adopted to generate sampling designs for covering the study area. Six sets of grids (in km), i.e., 1 × 1; 3 × 3; 5 × 5; 7 × 7; 9 × 9; and 11 × 11, were designed, and their corresponding sample size was 16,417; 1821; 652; 337; 198; and 142, which all satisfied the minimum requirement of the sample size.

For the spatial even sampling method, the center of a grid was generated as a sampling site. As an example, the configuration of the samples with a size of 142 for the spatial even sampling method in Beijing is shown in Figure 6a.

Although the stratified even sampling method can draw any sample size and not be restricted by the number of grids divided, the same sample sizes were set for it to make the two sampling methods comparable. In applying the stratified even sampling method, 11 strata in Figure 5 were employed, and Equation (5) was used to allocate the sample size, as shown in

Table 3. The area weights and sample sizes of different strata.

Strata	Area Weights (%)	Sample Size
Cropland Ⅰ	12.970	18	26	44	85	236	2129
Cropland Ⅱ	9.774	14	19	33	64	178	1605
Woodland Ⅰ	34.425	49	68	116	224	627	5652
Woodland Ⅱ	11.257	16	22	38	73	205	1848
Grassland Ⅰ	2.687	4	5	9	18	49	441
Grassland Ⅱ	5.095	7	10	17	33	93	836
Water Body Ⅰ	0.647	1	1	2	4	12	106
Water Body Ⅱ	1.780	3	4	6	12	32	292
Built-up Land Ⅰ	14.128	20	28	48	92	257	2319
Built-up Land Ⅱ	6.608	9	13	22	43	120	1085
Unused Land Ⅱ	0.629	1	1	2	4	11	103
Total	100	142	198	337	652	1821	16,417

. As an example, the configuration of the samples with a size of 142 for the stratified even sampling method in Beijing is shown in Figure 6b.

4.3. Accuracy Assessment of CLUDs Using FROM-GLC10 and Comparative Analysis

Referring to the 10 m FROM-GLC10 data of Beijing, accuracy assessments of the 30 m CLUDs were conducted using samples drawn from the stratified even sampling method and the spatial even sampling method, respectively, to evaluate the contribution of spatial stratification in the sampling scheme. Using Equation (1), we assessed CLUDs in Beijing using FROM-GLC10 data through a pixel-by-pixel comparison, and the wall-to-wall OA result of CLUDs in Beijing was 71.083%, which was the mean of the target population, i.e., the true accuracy.

For the stratified even sampling method proposed in this study, the OAs and DAs were estimated using samples based on Equations (2) and (6), respectively, and the results are shown in

Table 4. The OA and DA of the CLUDs data using the stratified even sampling method (%).

Samples (km × km)	142 (11 × 11)	198 (9 × 9)	337 (7 × 7)	652 (5 × 5)	1821 (3 × 3)	16,417 (1 × 1)	Total
OA	71.278	71.783	72.926	72.123	71.127	71.110	71.083
DA	0.195	0.700	1.843	1.040	0.044	0.027	0.000

. The results suggest that the OA and DA of the CLUDs data for Beijing based on the stratified even sampling method were 71.110–72.926% and 0.027–1.843%, respectively. The UA and PA of each LULC class were also calculated and the results are shown in

Table 5. The UA and PA of CLUDs using the stratified even sampling method for Beijing (%).

Samples (km × km)	Indices	Cropland	Woodland	Grassland	Water Body	Built-Up Land	Unused Land
142 (11 × 11)	UA	75.000	84.615	18.182	25.000	72.414	0.000
	PA	70.588	79.710	18.182	100.000	80.769	0.000
198 (9 × 9)	UA	71.739	78.889	26.667	20.000	80.488	0.000
	PA	63.462	80.682	22.222	100.000	84.615	0.000
337 (7 × 7)	UA	75.325	84.416	26.923	37.500	68.571	0.000
	PA	68.235	83.333	28.000	60.000	73.846	0.000
652 (5 × 5)	UA	71.141	85.522	25.490	56.250	65.185	0.000
	PA	61.272	83.007	26.531	75.000	80.734	0.000
1821 (3 × 3)	UA	67.633	85.577	24.648	50.000	65.252	0.000
	PA	61.947	81.839	25.362	75.862	76.875	0.000
16,417 (1 × 1)	UA	68.595	85.637	19.367	45.067	64.803	0.000
	PA	61.970	81.786	20.324	71.308	78.188	0.000

. Specifically, cropland, woodland, and built-up land were well classified with a high accuracy whilst grassland and water body had low accuracy. Misclassification of unused land may be mainly attributed to this class that covered a very limited area in Beijing (area proportion, 0.01%).

For the spatial even sampling method, the OAs and DAs were estimated using samples based on Equations (3) and (6), respectively, and the results are shown in

Table 6. The OA and DA of the CLUDs data using the spatial even sampling method (%).

Samples (km × km)	142 (11 × 11)	198 (9 × 9)	337 (7 × 7)	652 (5 × 5)	1821 (3 × 3)	16,417 (1 × 1)	Total
OA	70.423	73.232	66.766	69.479	70.730	71.115	71.083
DA	0.660	2.149	4.317	1.604	0.353	0.032	0.000

. The results suggested that the OA and DA of the CLUDs data for Beijing based on the spatial even sampling method were 66.766–73.232% and 0.032–4.317%, respectively. The UA and PA of each LULC class were also calculated and the results are shown in

Table 7. The UA and PA of CLUDs using the spatial even sampling method for Beijing (%).

Samples (km × km)	Indices	Cropland	Woodland	Grassland	Water Body	Built-Up Land	Unused Land
142 (11 × 11)	UA	70.732	83.871	27.273	0.000	3.846	0.000
	PA	67.442	82.540	21.429	0.000	76.190	0.000
198 (9 × 9)	UA	63.158	91.111	11.111	57.143	76.744	0.000
	PA	61.538	79.612	20.000	57.143	84.615	0.000
337 (7 × 7)	UA	62.667	88.194	3.846	44.444	56.098	0.000
	PA	58.025	78.395	5.000	80.000	68.657	0.000
652 (5 × 5)	UA	71.329	84.590	18.750	35.000	56.618	0.000
	PA	59.302	83.226	16.071	87.500	75.490	0.000
1821 (3 × 3)	UA	68.127	86.617	21.818	43.590	64.810	0.000
	PA	60.870	79.704	26.471	68.000	80.000	0.000
16,417 (1 × 1)	UA	68.595	85.612	19.367	45.067	64.803	0.000
	PA	61.970	81.773	20.324	71.308	78.188	0.000

. Compared with the stratified even sampling method, the estimated UA and PA of CLUDs using the spatial even sampling method were generally classified with a lower accuracy except for unused land. Meanwhile, the accuracy for specific LULC types demonstrated notable differences, as depicted in Table 5 and Table 7. Based on Table 4 and Table 6, the results suggest that the OAs estimated by the stratified even sampling method were much closer to the true OA than those by the spatial even sampling method, as depicted in Figure 7.

The RMSE and STDEV were also selected to evaluate the performance of spatial stratification on the accuracy assessment in this study. Using Equations (7) and (8), the RMSE and STDEV of the stratified even sampling and spatial even sampling methods were computed, respectively, and the calculated results are shown in Figure 8. Compared with the spatial even sampling method, the RMSE and STDEV results decreased from 2.097% and 2.127% to 0.914% and 0.713% for the stratified even sampling method, respectively, due to the contribution of spatial stratification.

5. Discussion

Given the true land class was unknown, we selected 30 m CLUDs as target data, used 500 m MCD12Q1 as ancillary data to carry out the spatial stratification, and employed 10-m FROM-GLC10, which had a higher spatial resolution and more reliable global/regional accuracy [31,32], as reference data to measure the sampling efficiency in the case study. However, in application of the proposed spatial stratification method, products with different resolutions should be used in the stratification as much as possible, and reference data, e.g., FROM-GLC10, can be also used for spatial stratification.

Taking three LULC reclassifications of MCD12Q1, CLUDs, and FROM-GLC10 in Figure 3 for Beijing as an example, CLUDs was treated as target data, and MCD12Q1 and FROM-GLC10 were employed as ancillary data to achieve spatial stratification. The class of one pixel in the CLUDs, FROM-GLC10, and MCD12Q1 is donated with P_o, P_h, and P_l, respectively. Based upon the spatial stratification method developed in this study, for each class of CLUDs, the stratification rules of three LULC classification data are illustrated in Figure 9, and four stratification units, labeled as stratum Ⅰ, stratum Ⅱ, stratum Ⅲ, and stratum Ⅳ, were obtained accordingly. Stratum Ⅰ is composed of those pixels s that belong to class P_o in the target data and P_o(s) = P_h(s) and P_o(s) = P_l(s); those with P_o(s) = P_h(s) and P_o(s) ≠ P_l(s) are divided into stratum Ⅱ; those with P_o(s) ≠ P_h(s) and P_o(s) = P_l(s) are divided into stratum Ⅲ; and those with P_o(s) ≠ P_h(s) and P_o(s) ≠ P_l(s) are divided into stratum Ⅳ. In addition, the four stratification units can be grouped, for example, stratum Ⅱ and stratum Ⅲ can be grouped into one stratum to represent partly consistency. Therefore, for three different data, including MCD12Q1, CLUDs, and FROM-GLC10, each LULC class can be divided into one stratum, two strata, three strata and four strata, respectively, as depicted in Figure 9.

Through different combinations of stratum Ⅰ, stratum Ⅱ, stratum Ⅲ, and stratum Ⅳ, the possible strata can be grouped into one stratum, two strata (stratum Ⅰ; stratum Ⅱ/stratum Ⅲ/stratum Ⅳ), three strata (stratum Ⅰ; stratum Ⅳ; stratum Ⅱ/stratum Ⅲ), and four strata (stratum Ⅰ; stratum Ⅱ; stratum Ⅲ; stratum Ⅳ), as shown in Figure 9. We iterated six classes of the CLUDs data and stratified them according to the spatial stratification rules depicted in Figure 9. Accordingly, four spatial stratification results for Beijing were integrated by the strata of six classes, as illustrated in Figure 10. The results suggested that the coverage of Beijing can be divided into 6, 11, 17, and 22 strata using the spatial stratification rules developed in this study. Stratum Ⅰ and stratum Ⅲ for unused land in Beijing did not exist because CLUDs and MCD12Q1 are not consistent in predicting unused land in the whole study area. Additionally, the strata obtained by the spatial stratification method can be used for spatial sampling in the future.

By integrating the LULC classification products of different sources and spatial resolutions, the spatial stratification method developed in this study can achieve spatial stratification and generate better estimations than the commonly used method for accuracy assessment. The main contribution of spatial stratification was to distinguish the differences of the probability of misclassification and improve the representativeness of samples [33]. The spatial stratification can be used in two important scenes: one is to draw a sample with more representativeness from each stratum to ensure that the accuracy assessment results of the LULC classification are much closer to the true accuracy; the other is to select much more representative training samples from different strata to improve the classification accuracy. For future implementation, we formulated a technical specification to describe the appropriate approaches and procedures for design, response, and analysis for data stratification in this study, as depicted in Figure 1, and a case study on accuracy assessment in Beijing, China, as illustrated in Figure 4. It is suggested that spatial stratification should be integrated together with different spatial sampling schemes for LULC classification in the future, for example, the stratified even sampling method in this study.

6. Conclusions

We presented a spatial stratification method to improve the efficiency of spatial sampling for classification accuracy assessment of LULC results of remote sensing data. Its performance was demonstrated in a case study using FROM-GLC10 data as the reference data to evaluate CLUDs for Beijing, and selecting MCD12Q1 as ancillary data for spatial stratification. The results suggested that the coverage of Beijing can be divided into 11 strata using the spatial stratification rules developed in this study. Compared with the spatial even sampling method, the OAs of the stratified even sampling method adopting the proposed spatial stratification method were much closer to the true OA, and the estimated UA and PA of CLUDs were generally classified with a higher accuracy except for unused land. Meanwhile, the corresponding RMSE and STDEV results decreased from 2.097% and 2.127% to 0.914% and 0.713%, respectively, due to the contribution of spatial stratification to sample selection. Therefore, the method proposed has promising performance and great potential to be widely employed to select the samples for the accuracy assessment of LULC classification products.