Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey

Abstract

The U.S. Geological Survey (USGS) has begun the development of operational, 30-m resolution annual thematic land cover data to meet the needs of a variety of land cover data users. The Continuous Change Detection and Classification (CCDC) algorithm is being evaluated as the likely methodology following early trials. Data for training and testing of CCDC thematic maps have been provided by the USGS Land Cover Trends (LC Trends) project, which offers sample-based, manually classified thematic land cover data at 2755 probabilistically located sample blocks across the conterminous United States. These samples represent a high quality, well distributed source of data to train the Random Forest classifier invoked by CCDC. We evaluated the suitability of LC Trends data to train the classifier by assessing the agreement of annual land cover maps output from CCDC with output from the LC Trends project within 14 Landsat path/row locations across the conterminous United States. We used a small subset of circa 2000 data from the LC Trends project to train the classifier, reserving the remaining Trends data from 2000, and incorporating LC Trends data from 1992, to evaluate measures of agreement across time, space, and thematic classes, and to characterize disagreement. Overall agreement ranged from 75% to 98% across the path/rows, and results were largely consistent across time. Land cover types that were well represented in the training data tended to have higher rates of agreement between LC Trends and CCDC outputs. Characteristics of disagreement are being used to improve the use of LC Trends data as a continued source of training information for operational production of annual land cover maps.

1. Introduction

Mapping land cover and monitoring land cover change are important for a variety of societal and scientific purposes, including land management, natural resource management, ecological studies, sustainable development, climate modeling, urban planning, habitat monitoring, and many others [1,2,3,4,5]. The U.S Geological Survey (USGS) is moving forward with a Land Change Monitoring, Assessment, and Projection (LCMAP) initiative to develop an expanded operational capacity for land cover mapping and monitoring to support these needs. One goal of LCMAP is to provide high temporal and moderate spatial resolution land cover and land change products, including annual thematic land cover at 30 m resolution (30 × 30 m pixels). The Continuous Change Detection and Classification (CCDC) algorithm [6] was developed to support continuous monitoring with Landsat data to take advantage of the multi-decadal Landsat archive housed by the USGS and is expected to play a central role in LCMAP mapping and monitoring activities. CCDC will be used to generate annual thematic land cover maps, with a class legend generally based on Anderson Level 1 categories of land cover previously adopted by the USGS Land Cover Trends project [7].

The USGS Land Cover LC Trends project plays an integral role in the development of the current capability for continuous monitoring by providing a reliable, consistent land cover product and related change assessments [8,9,10]. LC Trends data were generated through manual interpretation and were developed for the nominal years of 1973, 1980, 1986, 1992, and 2000 [7]. These data offer a basis for both training and initial testing of CCDC land cover classification.

Our objective was to conduct a comparison between output from the CCDC algorithm and LC Trends data [11] to determine whether CCDC can produce comparable maps. We selected 14 Landsat path/row locations within the conterminous United States to capture a wide range of land cover types and mapping challenges. CCDC currently operates on Landsat data collected since the operation of the Thematic Mapper sensor [6], launched in 1982, so we compared CCDC map output with LC Trends data for 1992 and 2000. Our goal was to determine if results were consistent across time, space, and thematic classes and gain insight into the use of LC Trends data for eventual operational training of annual land cover maps with CCDC. The results we present do not provide a statistical description of error in the CCDC Land Cover maps; they provide levels of agreement with maps generated from the LC Trends project and characterize features associated with common categories of disagreement between LC Trends and CCDC maps. The aim of our assessment is to inform refinement of the CCDC algorithm’s approach for deriving annual land cover maps and provide internal information on data inputs and work flows.

2. Materials and Methods

2.1. Data

2.1.1. Land Cover Trends

The LC Trends project used analyst interpretation of Landsat data for sample blocks of 10 km × 10 km (for most parts of the United States) or 20 km × 20 km (for a few areas) to characterize land cover and land cover change [7]. Land cover map dates were nominally 1973, 1980, 1986, 1992 and 2000, with actual dates of imagery varying for some samples because of clouds, poor data quality, or availability. The classification system used 11 classes representing a mix of land cover and land use types that “paralleled” the Anderson Level I classification system (

Table 1. Land use/land cover classes and descriptions used by the U.S. Geological Survey, Land Cover Trends project.

Land Cover Class	Description
Open water	Areas persistently covered with water, such as streams, canals, lakes, reservoirs, bays, and oceans.
Developed (urban or otherwise built-up)	Areas of intensive use where much of the land is covered with structures or anthropogenic impervious surfaces (residential, commercial, industrial, roads, etc.) or less-intensive use where the land cover matrix includes both vegetation and structures (low-density residential, recreational facilities, cemeteries, utility corridors, etc.), including any land functionally related to urban or built-up environments (parks, golf courses, etc.).
Agriculture (cropland and pasture)	Land in either a vegetated or an unvegetated state used for the production of food and fiber, including cultivated and uncultivated croplands, haylands, pasture, orchards, vineyards, and confined livestock operations. Note that forest plantations are considered forests regardless of their use for wood products.
Forest and woodland	Non-developed land where the tree cover density is >10%. Note that cleared forest land (i.e., clearcuts) is mapped according to current cover (e.g., mechanically disturbed or grassland/shrubland).
Grassland/shrubland	Non-developed land where cover by grasses, forbs, and/or shrubs predominates and tree-cover density is <10%.
Wetland	Land where water saturation is the determining factor in soil characteristics, vegetation types, and animal communities. Wetlands can contain both water and vegetated cover.
Mines and quarries	Areas with extractive mining activities that have a significant surface expression, including mining buildings, quarry pits, overburden, leach, evaporative features, tailings, or other related components.
Barren	Land composed of soils, sand, or rocks where <10% of the area is vegetated. Does not include land in transition recently cleared by disturbance.
Mechanically disturbed	Land in an altered, often unvegetated transitional state caused by disturbance from mechanical means, as by forest clearcutting, earthmoving, scraping, chaining, reservoir drawdown, and other similar human-induced changes.
Non-mechanically disturbed	Land in an altered, often unvegetated transitional state caused by disturbance from non-mechanical means, as by fire, wind, flood, animals, and other similar phenomena.
Snow and ice	Land where the accumulation of snow and ice does not completely melt during summer (e.g., alpine glaciers and snowfields).

) [12,13]. A national set of 2755 blocks was selected using a probabilistic sampling design with stratification based on 1999-era Level III ecoregions defined by the U.S. Environmental Protection Agency [8,14,15].

The LC Trends project mapped land use/land cover for 1992 as a baseline from which successive dates of land cover were mapped both forward and backward in time [16]. This initial baseline was created by starting with the National Land Cover Dataset [17] for 1992, collapsing the more detailed classes to the LC Trends class legend, then manually editing to improve local accuracy [16]. Changes from the 1992 baseline were identified and delineated manually using Landsat data, aided by aerial photographs and other ancillary data, to produce the land cover for successive dates. A minimum mapping unit of 60 × 60 m was used. Analysts conducted group reviews of each other’s work for every sample [7,16].

Whereas LC Trends maps provided an established USGS land cover product, we were not aware of any quantitative assessment of its accuracy. We conducted an accuracy assessment for LC Trends blocks in 5 of our 14 path/rows that overlapped high numbers of blocks (112 of 186 blocks) and represented a range of landscape settings. Included in the assessment were path/rows 23/37, 27/27, 28/33, 43/34 and 46/27. Based on an assumed accuracy rate of 95% with a target standard error of 0.025, a 300 point sample size met criteria for estimating overall accuracy [18]. Sample locations were randomly selected from within all LC Trends pixels in the five path/rows. Analysts manually interpreted high resolution imagery available in Google Earth™, assigning a primary land cover label based on LC Trends class definitions. Secondary labels were assigned for pixels where mixed cover types made class assignment ambiguous. These reference data showed LC Trends to have an overall accuracy of 91% based on the primary label, and 99% accuracy when both the primary and secondary labels were considered. This confirmed expectations that LC Trends data were of high accuracy and appropriate for use as a source of training data for CCDC.

2.1.2. CCDC Annual Land Cover

We acquired time series Landsat data processed to surface reflectance [19,20] from the USGS Earth Resources Observation and Science (EROS) Center’s Science Processing Architecture (ESPA) data system [21]. We included all archived data from Landsat 4, Landsat 5, Landsat 7, and Landsat 8 scenes with processing to the L1T standard [22] that had more than 20% clear observations (no cloud, cloud shadow, or snow). Clouds, cloud shadows, and snow were screened initially by ESPA with the Fmask algorithm (specifically, the CFmask implementation) [23,24], then further screened via a multitemporal cloud, cloud shadow, and snow detection algorithm called Tmask [25].

The CCDC algorithm uses all available Landsat data to estimate time series models and applies the models to predict future observations [6,26]. If the values of new observations are not within the predicted range for six consecutive observations, a break in the time series is flagged and a new time series model will be estimated when sufficient observations are available. The time series models are composed of harmonic models [27,28] that capture annual cycles, seasonality, and a slope component. The breaks found in the time series provide change information, such as caused by land cover conversion. The coefficients that define the time series cycles and slope, along with the root mean square errors (RMSE), are used as inputs to a land cover classifier (Figure 1). CCDC uses the Random Forest classifier [29] to derive decision tree models that are used to generate land cover maps. The time-series approach used by CCDC means that model trajectories can be “consulted” at any given time within the time series period to generate a map of land cover.

As the CCDC algorithm is capable of providing land cover maps at any given date, we selected a fixed day of the year (1 July) to provide annual CCDC land cover maps for our assessment. Note that following a break in the time series (six consecutive observations not within the predicted range), observations may fluctuate such that CCDC is unable to initiate a new time series model. During this time period we label the pixel as “Disturbed” (Figure 1). Thus, the annual land cover at the 14 path/row locations was labeled as Disturbed whenever a pixel was unable to initiate a time series model over the 1 July anniversary date of the map.

We used LC Trends data from 2000 as the pool to extract training data for the classifier. We extracted training data from LC Trends blocks based on criteria developed from an analysis of best practices [30]. That analysis found that a total of 20,000 pixels distributed across classes in proportion to the LC Trends class distribution was optimal, with a minimum of 600 pixels and a maximum of 8000 pixels required for each class (note, if the total number of pixels for a given class was less than 600, we extracted all available pixels). Selection of training pixels within each class was random from all available pixels in that class. We also incorporated eight ancillary datasets for training and classification: digital elevation data and derivatives (aspect, slope, and position index), a Wetland Potential Index (an index map generated for the 2006 U.S. National Land Cover Database [31], National Wetlands Inventory [32], and Soil Survey Geographic [SSURGO] hydric soils maps [33]), and probability of cloud, snow, and water occurrence. The latter probabilities were derived from Fmask statistics and represented the percent of cloud (or snow or water) observations from all available historical observations in the Landsat archive.

The CCDC workflow creates multiple products, but the thematic land cover product was the only type we used for the current analysis.

2.2. Methods of Land Cover Product Comparison

The area of comparison was defined by the footprint of LC Trends blocks in each of the 14 path/rows (Figure 2). We re-projected the data from the LC Trends blocks from Albers Equal Area, NAD83, to the Universal Transverse Mercator system, WGS84, to correspond with the Landsat time series data. It was likely that the re-projection resulted in some degradation in spatial fidelity; however, visual examination showed this to be of minor concern, given the 60 × 60 m minimum mapping unit applied by the LC Trends project and the level of spatial generalization inherent in the (manually delineated) LC Trends data.

We created a map of per-pixel agreement for the area covered by LC Trends blocks within each of the 14 test path/rows, matching the year of CCDC land cover output to the year of source data used for the LC Trends samples. Output layers were used to associate categories of disagreement with specific locations and to characterize the conditions (land cover characteristics and data characteristics) typical of the main categories of confusion. We constructed a set of confusion matrices for all LC Trends blocks in each of the 14 path/rows and for each date of comparison (Tables S1–S30). These were aggregated across path/rows into error matrices representing all pixels in all path/rows for each of the nominal dates of comparison (Tables S31 and S32). These summary error matrices, primarily the one for the nominal 2000 data comparison (

Table 2. Confusion matrix for all pixels from all path/rows for the 2000 period.

		Circa 2000 Land Cover Trends
		Water	Developed	Disturbed	Mining	Barren	Forest	Grass/Shrub	Agriculture	Wetlands	Ice & Snow	Total	Agreement
Circa 2000 CCDC	Water	462,323	4832	1301	615	6992	11,638	8910	11,907	11,395	0	519,913	89%
	Developed	4652	705,417	22,002	4237	2245	113,665	35,972	128,682	5009	0	1,021,881	69%
	Disturbed	3137	9223	17,088	1298	969	30,110	162,546	107,305	7734	52	339,462	5%
	Mining	747	14,105	2829	26,515	1423	5251	4611	13,700	595	0	69,776	38%
	Barren	4274	833	51	20	120,469	11,994	24,645	2495	2176	9701	176,658	68%
	Forest	19,043	118,687	169,391	2278	14,835	5,781,165	273,434	161,377	133,544	227	6,673,981	87%
	Grass/Shrub	7402	28,907	45,205	883	19,596	205,283	3,958,903	292,673	19,373	2451	4,580,676	86%
	Agriculture	15,242	98,464	9168	1801	437	116,467	161,455	5,891,790	55,527	0	6,350,351	93%
	Wetlands	13,611	2102	11,446	155	1522	112,575	25,212	28,568	554,469	0	749,660	74%
	Ice & Snow	5	0	0	0	5863	106	136	0	0	60,573	66,683	91%
		530,436	982,570	278,481	37,802	174,351	6,388,254	4,655,824	6,638,497	789,822	73,004	85.5%	overall agreement
		87%	72%	6%	70%	69%	90%	85%	89%	70%	83%

), were the basis for identifying categories of disagreement covering the largest aerial extent as a fraction of the entire area of study and/or covering the largest area as a fraction of the respective LC Trends and CCDC classes to which the pixels belonged.

For each category of disagreement, we ranked the degree of concentration within the path/row locations and the relative rate of occurrence per the area covered by LC Trends blocks. We developed vector layers delineating the areas of disagreement for the path/rows with the largest concentrations or highest rates per area of comparison. We displayed the vector layers in Google Earth™ and overlaid them on Landsat time series images and the thematic classifications of CCDC and LC Trends. We evaluated the land cover associated with each category of disagreement to identify patterns of occurrence that might be used to improve the CCDC annual land cover accuracy.

3. Results

3.1. Class Distribution

The distribution of classes was very similar for LC Trends and CCDC output when aggregated across the 14 path/rows (Figure 3). The largest differences in absolute area were in the Agriculture and Forest Classes. The LC Trends data resulted in just over 1.4% more of the map labeled as Agriculture and just under 1.4% less of the map labeled as Forest than did CCDC. The area mapped per class by CCDC and LC Trends differed substantially more for most classes when compared within individual path/rows (Figure 4). This difference was most obvious for the Disturbed class in Arizona (36/38) and California (43/34) and for smaller classes such as Mining and Barren. However, some large differences in area mapped for more common classes occurred as well. For example, CCDC mapped 6.7% more Forest in Washington (46/27) and 6.4% less Forest in Minnesota (27/27) than did LC Trends. LC Trends mapped 7.9% more Agriculture than CCDC in California (43/34) and 5.8% more in Kansas (28/33).

3.2. Per-Pixel Agreement

3.2.1. Summary of Per-Pixel Agreement

The summary confusion matrix for the circa 2000 data comparison showed 85.5% overall agreement between CCDC and LC Trends land cover (Table 2). Agreement between the largest classes (Forest, Agriculture, and Grass/Shrub) ranged from a low of 85% producer’s agreement for the Grass/Shrub class to a high of 93% user’s agreement for the Agriculture class. The smaller and generally more fragmented Developed and Wetland classes showed 69% user’s and 72% producers agreement for the Developed class and 74% user’s and 70% producer’s agreement for the Wetland class. The Barren class, which accounted for approximately 1% of the mapped area in both CCDC and LC Trends maps, had 68% user’s and 69% producer’s agreement. Mining, by far the smallest class in either classification (0.3% of area with CCDC and 0.2% of area with LC Trends), had low user’s agreement at 38%, but 70% producer’s agreement. Class agreement for Disturbed area was extremely low (6.1% producer’s agreement, 5.0% user’s agreement).

3.2.2. Per-Pixel Agreement by Path/Row

Overall agreement at individual path/row test locations across all years of comparison ranged from 75% to 98% (

Table 3. Overall agreement per path/row for 1992 and 2000 time periods. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington.

Location	13/29 NH/VT	16/40 FL	22/33 IL/IN	23/37 AR/MS	27/27 MN	28/33 KS	31/27 ND	33/29 SD	34/33 CO	35/32 CO	36/38 AZ	39/26 MT	43/34 CA	46/27 WA
Overall agreement 2000	93.5%	75.1%	87.8%	86.4%	79.4%	86.2%	89.0%	93.9%	89.9%	89.8%	82.1%	91.7%	84.8%	80.4%
Overall agreement 1992	93.8%	76.9%	87.7%	87.4%	79.2%	85.8%	87.7%	89.9%	90.0%	89.3%	97.6%	89.5%	78.9%	80.3%

). For the circa 2000 period only two of the 14 path/rows had overall agreement below 80%, 27/27 in Northern Minnesota (79.4%) and 16/40 in Florida (75.1%). Agreement was very similar for the 1992 and 2000 periods at most path/row locations. The two path/rows with the largest difference in agreement between the 1992 and 2000 periods were 36/38 in Arizona (97.6% overall agreement in 1992 dropped to 82.0% in 2000) and 43/34 in California (78.9% agreement in 1992 rose to 84.8% in 2000). Most of the difference was accounted for by disagreement in the Disturbed class; in both cases CCDC classified very large areas of Disturbed land in one of the two periods under comparison, but LC Trends did not.

Error matrices for the four path/rows with the highest numbers of LC Trends blocks (Table 4) account for 53% of the total area across all 14 path/rows and provide representative examples of the variation of class distribution, overall accuracy, and class accuracy across all test sites.

3.2.3. Per-Pixel Agreement by Land Cover Class

Forest Class Agreement

The CCDC and LC Trends Forest classes had 90.5% producer’s and 86.6% user’s agreement overall. CCDC mapped 1.4% more forest area than did LC Trends. CCDC mapped less forest area than LC Trends in only two locations, Kansas and Arizona (Table 5), the two path/rows with the lowest Forest class producer’s agreement, 60.5% and 54.7%, respectively.

Forest class agreement varied widely across the test path/rows (Table 5). The two locations with the highest Forest class agreement for both 1992 and 2000 were New Hampshire/Vermont (13/29) and South Dakota (33/29). Tree cover occurred as largely unbroken expanses of mixed forest species (New Hampshire/Vermont) or managed, predominantly conifer, national forest (South Dakota). The eight LC Trends blocks in New Hampshire/Vermont had 97.4% producer’s and 96.2% user’s agreement between CCDC Forest and LC Trends Forest classes in 2000, and 97.7% producer’s and 96.4% user’s agreement in 1992. With just a single forested LC Trends block, South Dakota results showed 98.8% producer’s and 93.3% user’s agreement in 2000 and 98.8% producer’s and 93.0% user’s agreement in 1992.

LC Trends blocks in the Washington path/row covered 18.7% of the total study area and 35% of the LC Trends Forest class. The LC Trends and CCDC Forest classes in this path/row exhibited 91.9% producer’s and 86.2% user’s agreement. The majority of LC Trends Forest pixels with which CCDC disagreed were classified as Developed by CCDC, accounting for 55.5% of the Forest class producer’s disagreement. In most cases, high resolution images showed these areas to be associated with low-intensity development, where tree cover mixed with some houses and roads had been generalized to the Forest class by LC Trends interpreters.

Of the roughly 14% of CCDC Forest pixels in Washington that disagreed with LC Trends pixels, 30.6% had been classified as Grass/Shrub and 27.9% had been classified as Developed by LC Trends. Most cases of the former occurred in forest harvest footprints, where early stages of forest regeneration were classified as Grass/Shrub by LC Trends. Cases of the latter were where land cover generally occurred as fragmented clusters of tree cover within a larger context of low-intensity development. The fragmentation of land cover created a high proportion of mixed pixels and edge pixels, where minor misregistration was likely a contributing factor to disagreement. Nevertheless, the majority of disagreement pixels were actually covered by trees, but often were generalized to the Developed class by LC Trends interpreters and mapped as Forest by CCDC.

The Minnesota path/row (27/27) was dominated by tree cover, with much of it in woody wetland. The LC Trends and CCDC Forest classes here had 88.0% producer’s agreement and 82.7% user’s agreement. CCDC disagreement with LC Trends Forest pixels predominantly (82.6% of the time) occurred where CCDC classified the pixels as Wetland. LC Trends disagreed with CCDC Forest pixels approximately 17% of the time, often (15.6% of the time) labeling those pixels as Grass/Shrub or Wetland (40.2%).

Forest class agreement was less strong in Kansas (28/33; 60.5% producer’s and 70.7% user’s agreement for circa 2000) and Illinois/Indiana (22/33; 83.0% producer’s and 75.9% user’s agreement for circa 2000). In both locations forest occurrence was more fragmented, resulting in a much higher proportion of edge pixels relative to interior area in forest stands (Figure 5). Almost all of the Forest class disagreement in these two path/rows occurred along boundaries between Forest and Agriculture or Forest and Grass/Shrub. Some of the disagreement in these cases was from mixed pixels, containing some fraction of Agriculture or Grass/Shrub land cover. This was especially common in the Kansas path/row, where forest occurred as long linear features coinciding with the moister, fire-protected topography along stream courses. The large fraction of edge pixels would have increased the impact from image misregistration—and we observed some apparent minor image misregistration. Frequently, the 60 m minimum mapping unit and greater generalization in the LC Trends data resulted in pixels classified as Agriculture or Grass/Shrub where available high-resolution imagery revealed the actual land cover to be tree cover. In some cases, disagreement appeared to result from misclassification in the LC Trends blocks.

Generally, even path/rows with only limited forest area provided meaningful information regarding the performance of the CCDC classification process in different landscapes. Forest disagreement in Arizona (36/38), for example, occurred primarily as confusion between Grass/Shrub and Forest classes in the drier oak and juniper forested areas in the southernmost LC Trends block. Tree density and forest structure there varied, often along gradients of topography and elevation and along drainages. The spatial transitions between Forest and Grass/Shrub classes are often gradual, and it was in these areas that disagreement with LC Trends classes tended to be concentrated.

Forest class agreement was high at the sites in Colorado (34/33 and 35/32) and California (43/34), with 93.5% producer’s and 93.3% user’s agreement for 34/33, 89.8% producer’s and 89.6% user’s agreement for 35/32, and 88.3% producer’s and 87.0% user’s agreement for 43/34. In both Colorado path/rows over 90% of the disagreement was with the CCDC Grass/Shrub class, where disagreement tended to be in areas where gradual transitions from denser to more diffuse tree cover created ambiguous spatial boundaries between classes. This often occurred in locations of high relief, often where the difference between tree height and shrub height was slight.

In California (43/34) 86.2% of the LC Trends Forest confusion was with CCDC Grass/Shrub class, and in these cases the circumstances were generally similar to those observed for the Colorado sites; the coarser minimum mapping unit and tendency toward generalization in the LC Trends data contributed to the disagreement with the CCDC results. Likewise, when LC Trends pixels disagreed with CCDC Forest pixels it was because LC Trends interpreters had mapped those pixels as Grass/Shrub 65.7% of the time; in these cases, the LC Trends minimum mapping unit and areal generalization accounted for more of the confusion than did the spatially transitional nature of forest cover in 43/34.

Agriculture Class Agreement

The CCDC and LC Trends Agriculture classes had good overall agreement (88.8% producer’s and 92.8% user’s agreement), with the main categories of disagreement being: (1) LC Trends Agriculture confused with CCDC Grass/Shrub; (2) CCDC Agriculture confused with LC Trends Grass/Shrub; and (3) LC Trends Agriculture confused with CCDC Forest. Five of the 14 path/row locations contained 86.4% of the area LC Trends interpreters had classified as Agriculture. Results for these five locations ranged from a low of 85.9% producer’s and 91.2% user’s agreement in the Kansas path/row (28/33) to a high of 93.2% producer’s and 94.1% user’s agreement in North Dakota (31/27) (Table 6a,b). Across all 14 path/rows, 23.7% of the area classified as agriculture by LC Trends was located in California (43/34), where we observed 87.8% producer’s and 95.3% user’s agreement.

The single largest category of confusion was where LC Trends interpreters mapped pixels as Agriculture and CCDC mapped them as Grass/Shrub (Table 6a), accounting for 1.4% of the entire study area. The inverse case, where LC Trends interpreters mapped pixels as Grass/Shrub and CCDC mapped them as Agriculture, made up 0.8% of the study area (Table 6b). Both directions of confusion were concentrated in a few path/rows and were heavily concentrated in the Kansas location (28/33), which represented 54.6% of the confusion between LC Trends Agriculture and CCDC Grass/Shrub and 51.1% of the confusion between LC Trends Grass/Shrub and CCDC Agriculture. For the former case, examination of high resolution imagery indicated the actual land cover was Grass/Shrub approximately 80% of the time and ranged in use from rangeland to lightly managed pasture/hayland, with many patches being difficult or impossible to distinguish from rangeland (Figure 6). Other areas where LC Trends labeled pixels as Agriculture and CCDC labeled them as Grass/Shrub occurred in California (23.7%), North Dakota (11.6%), and Montana (8.2%). The overwhelming majority of the confusion across all locations occurred in areas of Grass/Shrub, based on high resolution imagery in Google Earth^TM, with some cases showing evidence of haying.

A lesser factor contributing to cases of low producer’s agreement in the Agriculture category was where CCDC labeled pixels as Forest that had been classified as Agriculture by LC Trends (see previous details in second paragraph under “Forest Class Agreement”). This confusion was heavily concentrated in a few path/rows, including Illinois/Indiana (22/33), Arkansas/Mississippi (23/37), and Kansas (28/33).

Confusion between the LC Trends Agriculture class and CCDC Developed class, as well as the inverse, made up 1.1% of the total area of the study, with the latter representing the smaller fraction. Most of this disagreement occurred in the suburban fringes, where developed and agricultural lands were fragmented and intermingled with low-intensity development. These two categories of confusion were heavily concentrated in Washington (46/27: 39.2% of LC Trends Agriculture confused with CCDC Developed and 33.0% of the inverse case) and California (43/34: 34.4% of LC Trends Agriculture confused with CCDC Developed and 32.2% of the inverse case). LC Trends map generalization in these settings often labeled fields on the order of 250–500 m × 250–500 m, which are used for hay and pasture, as Developed. Alternatively, LC Trends generalized buildings and roads in agricultural settings to be labeled as Agriculture, whereas CCDC separated those buildings and roads into the Developed class.

Grass/Shrub Class Agreement

The Grass/Shrub class covered the third largest extent across the study area after Forest and Agriculture, accounting for 22.7% of the LC Trends classification and 22.3% of the CCDC classification. The Grass/Shrub class had slightly lower agreement than either Forest or Agriculture at 85.0% producer’s and 86.4% user’s agreement. For eight of the 10 path/row test areas where more than 1% of the area was classified as Grass/Shrub by LC Trends, we observed 82.0% to 94.5% producer’s agreement and 74.3% to 98.4% user’s agreement (Table 7a,b). The remaining two path/rows exhibited low Grass/Shrub class agreement, including 12.9% producer’s and 50.6% user’s agreement in Minnesota (27/27) and 44.3% user’s and 48.7% producer’s agreement in Washington (46/27). Both LC Trends and CCDC Grass/Shrub classes were most often confused with the counterpart classification’s Agriculture class (refer to the second paragraph under “Agriculture Class Agreement”).

Confusion between the Grass/Shrub and Forest classes accounted for 2.3% of the entire area of comparison (1.3% LC Trends Grass/Shrub confused with CCDC Forest and 1.0% LC Trends Forest confused with CCDC Grass/Shrub). In Washington (46/27) and Minnesota (27/27), 83.5% and 53.4%, respectively, of LC Trends Grass/Shrub class disagreement was where CCDC had classified pixels as Forest. In both locations this disagreement was associated with regenerating forest following timber harvest. These patches were classified as Forest by CCDC typically within a year of harvest, but were considered Grass/Shrub by LC Trends, typically for seven or more years following harvest.

Grass/Shrub class agreement was higher in the two Colorado path/rows, including 88.7% producer’s and 86.8% user’s agreement in 34/33 and 91.2% producer’s and 86.8% user’s agreement in 35/32. As before, the largest fraction of disagreement was confusion between the Forest and Grass/Shrub classes. Areas where LC Trends interpreters classified pixels as Grass/Shrub and CCDC classified them as Forest accounted for 74.7% (34/33) and 82.9% (35/32) of the producer’s disagreement. User’s disagreement was also predominantly the result of confusion between Grass/Shrub and Forest; areas where CCDC had classified pixels as Grass/Shrub and LC Trends interpreters classified them as Forest accounted for 67.3% of the disagreement in 34/33 and 84.6% of the disagreement in 35/32 (see second-to-last paragraph under “Forest Class Agreement”).

Developed Class Agreement

The Developed class covered 4.8% of the LC Trends pixels and 5.0% of the CCDC pixels in the study area. The Developed classes generally occurred in more complex, fragmented land cover mosaics and had 71.8% producer’s agreement and 69.0% user’s agreement. The majority of confusion was with the Forest and Agriculture classes and was concentrated in a few path/row locations (Table 8).

Confusion of Developed land with Forest was heavily concentrated in the Washington location (46/27), with 77.2% of cases of LC Trends Developed pixels classified as Forest by CCDC and 87.5% of cases of the CCDC Developed pixels classified as Forest by LC Trends. The generalization of land cover features for the LC Trends classification often included pixels of pure tree canopy that occurred in the very complex land cover mosaic of low-intensity development around the Puget Sound area of 46/27. CCDC generally classified these pixels as Forest. The inverse disagreement, where CCDC Developed pixels were classified as Forest by LC Trends, likewise was often associated with the generalization of the LC Trends classification, which had included areas of developed land within larger tracts of Forest. A similar fraction of the Developed/Forest confusion was not caused by the LC Trends generalization, but from mixed pixels in the very fragmented land cover around the Puget Sound. The fragmented land cover mosaic also increased opportunities for image misregistration to contribute to class confusion.

The Agriculture class also accounted for a large fraction of the confusion between the CCDC and LC Trends Developed classes. CCDC classified 10.0% of LC Trends Developed pixels as Agriculture; conversely, LC Trends interpreters classified 12.6% of the CCDC Developed pixels as Agriculture. In both cases, the vast majority of that confusion was distributed across the Washington (46/27), California (43/34), and Illinois/Indiana (22/33) locations. Pixels where LC Trends interpreters had classified the land as Agriculture and CCDC had classified it as Developed were most often associated with land cover that, if defined strictly by cover as opposed to land use or a mixed definition, was Developed, such as roads, clusters of farm buildings, low intensity residential development and a few commercial/industrial sites. In the California location there were also some cases of bare farmland being classified as Developed by CCDC.

Wetland Class Agreement

LC Trends and CCDC mapped 3.8% and 3.6% of the entire study extent as Wetland, respectively; with 70.2% producer’s and 74.0% user’s class agreement. CCDC classified 16.9% of LC Trends Wetland pixels as Forest, 7.0% as Agriculture, 2.5% as Grass/Shrub, and 1.4% as water (Table 9a). Of the pixels classified as Wetland by CCDC, LC Trends interpreters classified 15.0% as Forest, 3.8% as Agriculture, and 3.4% as Grass/Shrub (Table 9b). Where CCDC disagreed with the LC Trends Wetland class, it labeled those pixels Forest 57% of the time. Where LC Trends disagreed with the CCDC Wetland class it labeled those pixels Forest 58% of the time. Most Wetland confusion occurred in the path/rows with the most Wetland area (Florida, Illinois/Indiana, Arkansas/Mississippi, Minnesota, and Washington).

Minnesota (27/27) had 44.3% of LC Trends and 51.7% of the CCDC Wetland pixels for the entire study extent. The vast majority of wetlands within this path/row were forested and, consequently, most of the disagreement was between Forest and Wetland classes. CCDC mapped 19% of LC Trends Wetland pixels in 27/27 to other classes, primarily forest (87.8%). LC Trends pixels disagreed with 27% of the CCDC Wetland pixels in 27/27, mapping 74.7% of them as Forest.

Much of the confusion between LC Trends Wetland and CCDC Forest pixels occurred along river channels and other elongated features, as well as boundaries between wetland and forest classes where mixed pixels and minor misregistration may have been factors. This was less often the case with CCDC Wetland confusion with LC Trends Forest pixels, where areas of disagreement were clearly associated with a different interpretation of patches of land cover. In many cases it was difficult to distinguish conclusively between forest and forested wetland using visual interpretation of Landsat and high resolution imagery in Google Earth™. Consequently, it was difficult to determine with certainty what the true land cover class should have been. We consulted data from the National Wetlands Inventory (NWI) [32,34] for additional information on the occurrence of wetlands in the landscape (Figure 7). For pixels labeled by LC Trends as Wetland and by CCDC as Forest, the NWI favored the CCDC Forest classification 58.4% of the time (i.e., NWI did not classify these pixels as wetland). For pixels classified by LC Trends as Forest and by CCDC as Wetland, NWI again favored the CCDC interpretation, 60.3% of the time.

The other concentration of wetlands and Wetland class disagreement was in Arkansas/Mississippi (23/37), accounting for 16.5% of the LC Trends Wetland area and 12.4% of the CCDC Wetland area. Producer’s agreement was only 65.2%, with CCDC labeling 18.7% of LC Trends Wetland as Forest and 13.6% of LC Trends Wetland as Agriculture (Table 9). The Wetland class user’s agreement was 91.5%, with LC Trends labeling 4.1% of CCDC Wetland pixels as Forest and 3.1% as Agriculture. NWI data favored the CCDC interpretation 72.4% of the time in pixels identified by LC Trends as Wetland and by CCDC as Forest. NWI favored the CCDC interpretation 67.8% of the time in pixels labeled by LC Trends as Wetland and by CCDC as Agriculture.

Water Class Agreement

The Water class had 87.2% producer’s and 88.9% user’s agreement. This class accounted for only 2.6% of the LC Trends mapped area and 2.5% of the CCDC mapped area. Five path/rows had less than 1% of area mapped as water by either CCDC or LC Trends. In path/rows with greater than 1% water, low agreement was concentrated in Illinois/Indiana (22/33), Arkansas (23/37), and Kansas (28/33) (Table 10).

The Washington location (46/27) had 41% of the total water mapped by LC Trends or CCDC and had the largest concentration (25%) of all disagreement, despite a producer’s agreement of 92.1% and a user’s agreement of 93.3% (Table 10). CCDC disagreement with LC Trends Water was split primarily among Forest (44.6%), Wetland (20.8%), and Agriculture (11.0%). All three types of disagreement were concentrated along stream courses (Figure 8) and, to a lesser extent, along small inland water bodies and coastal margins of the Puget Sound. Image registration was likely a contributing factor, as well as mixed pixels, minimum mapping unit, changing water levels, and, perhaps, shifts in water courses in some streams.

LC Trends interpreters classified only 6.7% of CCDC Water pixels as other land cover types in the Washington path/row, most often as Barren. This confusion occurred almost exclusively at edges of water bodies, often within the intertidal zone of the Puget Sound itself, and along stream channels where water level variation and changing sandbars and shorelines were likely contributing factors.

The Illinois/Indiana path/row (22/33) had less than 5% of the total area mapped in water by both LC Trends and CCDC and had only 65.8% producer’s agreement and 62.8% user’s agreement. The Kansas location (28/33) had slightly more water (5.5% of CCDC total water and 6.1% of LC Trends total water), with a producer’s agreement of 68.7% and a user’s agreement of 77.9%. In both locations the vast majority of disagreement occurred along stream courses and shorelines of small water bodies. In these cases, LC Trends generalization and larger minimum mapping unit appeared to account for much of the confusion, and image registration may have contributed as well. In a few cases, LC Trends interpreters had classified agricultural fields as Water. In Kansas (28/33), confusion between LC Trends Water and CCDC Barren pixels occurred along the channel of the Kansas River, where some areas mapped as exposed sediment by CCDC were included in the LC Trends water class. The closest date of high resolution data in Google Earth™ (16 February 2002) agreed roughly as often with CCDC as with LC Trends. Different dates of high resolution data confirmed the variability of the exposed sediment with water levels, erosion, and deposition.

Barren Class Agreement

The Barren class covered less than 1% of the entire mapped area in either classification and had only 69.1% producer’s agreement and 68.2% user’s agreement—among the lowest across classes. The disagreement was heavily concentrated in Washington (46/27), South Dakota (33/29), and Colorado (35/32) (Table 11).

Confusion between Barren and Forest was heavily concentrated in the Washington path/row, which had 54.9% of all LC Trends Forest pixels that had been identified as Barren by CCDC and 85.5% of all LC Trends Barren pixels that had been identified as Forest by CCDC. Pixels that LC Trends interpreters classified as Barren, but CCDC called Forest, occurred mostly in extreme terrain at or near treeline and snowline in the LC Trends blocks falling in the northern Cascades. In about half these cases, the pixels represented a mix of cover types, with some combination of bare rock, bare soil, terrain shadow, ground vegetation, and trees. Roughly a third of these pixels were where LC Trends interpreters had misclassified or generalized tree cover as Barren. At slightly lower elevations, there were cases of pixels identified by LC Trends as Barren and by CCDC as Forest that occurred along streams where mixed pixels were likely and where LC Trends interpreters had classified the bare soil and rock of the streams quite liberally, in some instances overlapping obvious tree cover.

The inverse confusion, pixels classified by LC Trends interpreters as Forest and by CCDC as Barren, was highly concentrated in four sample blocks located at high elevations. Most of this confusion occurred in lightly to moderately vegetated rocky areas with few, if any, trees. Confusion between LC Trends Barren and CCDC Grass/Shrub classes occurred almost exclusively in the high-elevation terrain in just five sample blocks. Most of these pixels were lightly vegetated, with varying mixes of rock and soil in the pixel in most cases.

Some additional Barren class disagreement occurred in South Dakota (33/29) and Colorado (34/33). In South Dakota the confusion was almost entirely between the Barren and Grass/Shrub classes. Most pixels were lightly to moderately vegetated, with components of exposed soil and rock. The generalization of LC Trends data appeared to have been a factor in many cases of confusion. In Colorado the majority of the disagreement was between the LC Trends Barren and CCDC Grass/Shrub classes. In most cases where LC Trends interpreters classified pixels as Barren and CCDC classified them as Grass/Shrub, the land cover was lightly vegetated, sometimes with scattered trees and rocky understory. Many of the disagreement pixels were at the boundaries of CCDC class patches.

Mining and Ice-Snow Classes Agreement

The Mining class accounted for only 0.2% of the LC Trends mapped area and 0.3% of the CCDC mapped area. Mining class producer’s agreement across all path/rows was 70.1%. However, CCDC mapped 85% more mining pixels than did LC Trends, and user’s agreement across all path/rows was only 38.0%. Confusion between developed and mining classes was the largest category of disagreement. Over 20% of the CCDC Mining pixels were classified as Developed by LC Trends, and 11.2 % of the LC Trends Mining pixels were classified as Developed by CCDC. Another 19.6% of the CCDC Mining pixels were mapped as Agriculture by LC Trends.

The lowest rate of producer’s agreement (56.2%) was in Washington (46/27) (Table 12a), which accounted for 83% of the confusion between LC Trends Mining pixels and CCDC Developed pixels. The Mining class user’s agreement for 46/27 was only 31.5% (Table 12b), with over 60% of all confusion between CCDC Mining pixels and LC Trends Developed pixels occurring in that path/row. User’s agreement for Mining was below 50% for all path/rows having more than 125 pixels of the CCDC Mining class, with the exceptions of California (43/34) (60.6%) and Minnesota (27/27) (86.1%).

The Ice and Snow class accounted for only 0.4% of the LC Trends mapped area and 0.3% of the CCDC mapped area and only occurred in Washington (46/27). Producer’s agreement was 83% and user’s agreement was 90.8%, with confusion between the Ice and Snow and Barren classes accounting for most disagreement.

Disturbed Class Agreement

LC Trends and CCDC Disturbed classes agreed in only a small minority of cases (6.1% producer’s agreement and 5.0% user’s agreement). When all path/rows were summarized together, LC Trends interpreters mapped 21.9% more area as disturbed than did CCDC. Summarized by path/row the differences appear extreme (Figure 4 and Table 13a,b). In the Washington path/row, LC Trends interpreters mapped nine times more area as Disturbed, with LC Trends identifying 3.5% of area as Disturbed and CCDC identifying only 0.39% as Disturbed. In the California path/row, CCDC mapped seven times more area as Disturbed than did LC Trends. For Arizona, CCDC mapped 14.2% of the map as disturbed, but LC Trends did not identify any area as disturbed.

Areas where LC Trends mapped Disturbed and CCDC disagreed were heavily concentrated, with 85% of the cases occurring in just three path/rows (Washington—46/27, Minnesota—27/27, and Arkansas—23/37). Almost all of this disagreement occurred in forest harvest footprints, which in many cases were several years old.

Areas where CCDC mapped Disturbed and LC Trends disagreed were also highly concentrated, with over 85% occurring in just two path/rows (Arizona—36/38 and California—43/34). In Arizona 83% of this was where LC Trends had mapped Grass/Shrub and CCDC mapped Disturbed. In California most of the LC Map disagreement with CCDC Disturbed was classified as Agriculture (67%).

4. Discussion

We undertook this analysis to assess the performance of a continuous change-detection algorithm for mapping thematic land cover across a variety of landscape settings and to evaluate the efficacy of applying data from a national study of land cover, LC Trends, to train the classifier in preparation for operational continuous monitoring of land change. We subjectively selected 14 path/row areas that offered different types of challenges for mapping land cover and made opportunistic use of an existing, high-quality land cover dataset to characterize results. We intended to benefit both from the actual analysis as well as from gaining familiarity with the workflow that will be needed to support eventual operations.

The number of sample blocks (and therefore area of comparison) available for each of the 14 test path/rows varied greatly because the samples were originally selected based on ecoregion strata for the LC Trends project, rather than on Landsat path/rows. For example, the Florida path/row (16/40) had only two sample blocks, accounting for only 1.1% of the total area in this study, but the Washington path/row (46/27) had 36 sample blocks and represented 18.7% of the study area. Our results do not provide a statistical description of either error in the CCDC land cover or of the relation between Land Cover LC Trends and CCDC annual land cover outside of the areas compared. The results will, however, feed back into understanding the suitability of LC Trends data as a source for training data and the types of confusion that may be introduced by the LC Trends classification scheme, minimum mapping unit, level of mapping generalization, and contrasting interpretation approach.

We found good consistency in map results across time periods for all but two (Arizona and California) study areas (Table 3). Rates of agreement between LC Trends and CCDC maps varied geographically (ranging from 75% to 94% in 2000 and 77% to 98% in 1992), but nine of 14 path/rows had rates exceeding 86% agreement (Table 3). At the class level, we observed that Forest, Agriculture, Grass/Shrub, and Water had the highest rates of agreement (all >87% for both producer’s and user’s agreement) between LC Trends and CCDC maps pooled across study areas, but typically showed greater rates of agreement in study areas where the classes occupied appreciable portions of the landscape within the LC Trends blocks.

The most important finding was that CCDC’s automated, efficient, and repeatable approach was able to reproduce results obtained through the LC Trends project’s lengthy manual image interpretation process 86% of the time without any post-classification refinement while using the existing LC Trends dataset to guide the selection of training data. We initially questioned the suitability of the LC Trends data for training CCDC’s classifier, as LC Trends data followed a classification scheme that included some land use characteristics in its class definitions, and the LC Trends methods relied on analyst interpretation and a 60 × 60 m minimum mapping unit (the combination of which resulted in spatial generalization of land cover patches). However, the good overall agreement of CCDC annual land cover with LC Trends maps across the 14 path/rows we studied suggests that the national set of LC Trends data can provide an adequate source of training information to enable CCDC to generate wall-to-wall thematic land cover for the conterminous United States for the 1985 to current Landsat record. We also note that in areas of disagreement, ancillary information indicated that CCDC often made the better choice of class labels.

Conversely, where classes only made up a small portion of the LC Trends blocks in a given path/row we observed several issues. For training, a minimum of 600 pixels for any class was found to provide the best classification result [30]. For several of the 14 path/rows the limited number of LC Trends blocks available did not provide class populations of 600 pixels for one or more classes (for example, see Table 10a for Montana and Arizona). Furthermore, where classes were represented in small fragmented patches, registration and spatial generalization were observed to be a potential problem due to selection of some pixels representing other-than-intended classes within the training data. These small patches also impacted the CCDC/LC Trends comparison process, sometimes measuring disagreement that was the result of misregistration rather than misclassification. The effect of misregistration error on training data selection, which had resulted from reprojection of the LC Trends blocks from Albers to UTM, will be eliminated in operational LCMAP classification when the input Landsat data will be processed to the USGS Albers/NAD83 grid, the native projection of the LC Trends blocks. To address the underrepresentation of small classes in the training data, we have since expanded the collection of training data to LC Trends blocks beyond the area being classified. Preliminary results suggest that this helps in meeting the 600 pixel minimum and better represents landscape variability near the edges of the area being classified. Collection of training pixels from within LC Trends blocks available in a window including and surrounding the area being classified has so far produced superior classification for the Puget Sound, our initial test ground.

Another key finding was the incompatibility of the methods by which LC Trends and CCDC interpreted the Disturbed class. For example, LC Trends land cover was mapped at intervals of 6 to 8 years, and interpreters showed a strong tendency to label pixels as disturbed long after the actual event had occurred to make certain the disturbance was recorded. CCDC only labeled disturbance for the brief intervals in which a time series model could not be fitted following an abrupt change in land cover, and only if this interval overlapped the defined anniversary date of 1 July. This interval often lasted for a period of months, rather than years. CCDC products will be generated annually, and a new formulation being evaluated labels Disturbance in the annual land cover map regardless of the date the change occurred within the year, overcoming the problem posed by selecting a specific anniversary date to survey for disturbance and removing much of the disagreement between CCDC and Trends Disturbance classes. Cumulating change across multiple annual maps output by CCDC then will produce results more comparable with those mapped by LC Trends interpreters across their multiyear mapping intervals.

A second source of incompatibility between the Disturbance classes resulted from CCDC’s sensitivity to change, including changes in land surface condition where the land cover type did not actually change. For example, sequences such as Grass/Shrub to Disturbed to Grass/Shrub were observed in Arizona and California, where the Disturbed interval was apparently caused by multiyear wet or dry periods that created a measurable shift in vegetation response with no removal of vegetation cover or change in vegetation type. Such changes in vegetation condition were not recorded by LC Trends. There were two factors that led to CCDC identifying these changes in vegetation condition as “disturbance”. First, changes in condition caused legitimate breaks in the time series response trajectories, but there was no means to distinguish breaks caused by shifts in condition with breaks caused by changes in cover type. Recent refinements to the CCDC algorithm are incorporating steps to filter changes in land cover condition from changes in type so that the latter can be better isolated for mapping changes in thematic cover. Second, unlike other class types, the Disturbance class was not trained for classification with Random Forest; it was instead defined by breaks in the time series models, as described in Section 2.1.2 (see also Figure 1). Training data now are being developed so that Random Forest can be used to classify disturbance directly.

We found that the confusion between the Grass/Shrub class and the Agriculture class appeared to be due mostly to land use characteristics embedded in the LC Trends class definitions. The Trends Agriculture class definition includes “... cultivated and uncultivated croplands, haylands, [and] pasture ... ,“ which in many cases led to capture by image interpreters of hayland and pasture that were spectrally indistinguishable from more natural grassland, particularly in Kansas (28/33). This land use distinction could be made because analysts employed a variety of contextual clues. CCDC generally mapped these areas of lightly managed hayland and pasture to the Grass/Shrub class. Redefinition of the Agriculture class to exclude natural grassland that is lightly grazed or occasionally hayed might be suggested by this finding.

Another finding related to how successional stages were handled for vegetation stands. The most obvious and widespread case we observed was the difference in how CCDC and LC Trends handled the recovery of forest following clearcut harvest. LC Trends interpreters distinguished early stages of this progression as Grass/Shrub before trees became dominant, then labeled the pixels as Forest once trees regained dominance. In comparison, CCDC fit models to the full length of a time series between the periods of abrupt change, then fed the coefficients from these models to the classifier. The coefficients therefore represented a long-term forest trajectory, rather than the individual stand stages along the trajectory, and the resulting thematic label ended up as Forest. This finding prompted modifying the algorithm to run a separate classification each year to enable the classifier to focus on the evolving stand structure through time. The problem is further being addressed by developing training data indicative of early successional stages of forest recovery. We observed that the grass and shrub cover in these post-disturbance stands have different spectral characteristics from areas of perpetual (or long-term) grass/shrub and therefore require separate, representative training data.

Results from our comparisons corroborated the expected difficulty in classifying woody wetlands or wetlands obscured by tree canopies. Our evaluation of Wetland class agreement was less certain in areas of Forested wetlands. Visual interpretation of forested wetlands was hindered where direct observation of flooding conditions or specific wetland vegetation was obscured by tree cover. We compared areas of class confusion with data from the National Wetlands Inventory to augment our evaluation of Wetland class agreement. Although this informed our perspective, we note that NWI data also were used as an ancillary reference source by LC Trends analysts and are a component of the Wetland Potential Index (WPI) layer that was used as an ancillary input to the CCDC Random Forest classification process. The WPI is a categorical ranking-index map generated based on convergence of evidence from information in the National Land Cover Database 2006 map [31], NWI data [32], and Soil Survey Geographic (SSURGO) hydric soils maps [33]. This complicates any conclusions we might draw from this comparison.

CCDC annual land cover products should be evaluated with an independent dataset developed specifically to determine the accuracy of the thematic outputs, rather than only quantifying the level of agreement with another product. An independent evaluation is planned for the next stage of development towards operational continuous monitoring of land cover.

5. Conclusions

We found 86% agreement between thematic land cover maps generated from two very different approaches applied with Landsat data, one based on manual interpretation of individual time periods spaced at 6- to 8-year intervals (LC Trends) and one based on automated interpretation of mathematical models constructed with dense time series of all available clear observations (CCDC). This agreement did not necessarily reflect the accuracy of the CCDC annual land cover maps, but rather the agreement between results from the two approaches encompassing the footprint of the 186 sample blocks used in this study. We observed consistency in results across time and across study areas of similar landscape types, and found relatively high levels of agreement for land cover classes that were well represented in the training data. Examination of the land cover associated with areas of disagreement suggested that, despite LC Trends classes being somewhat generalized and often incorporating contextual components of land use into class definitions, the annual land cover maps generated by CCDC generally differed from the LC Trends classification in ways that were not problematic. For example, where LC Trends generalized highly fragmented and geometrically complex land cover features, CCDC adhered to the spatial detail represented in the spectral characteristics—often successfully. Whether CCDC was more accurate than the LC Trends data was not made clear by this analysis. Comparison with independent reference data will begin to address this question and is planned for the next stage of evaluation of CCDC annual land cover products. These efforts will help move the USGS towards operational implementation of a continuous monitoring capability.