Comparison of Office and Field Techniques for Validating Landscape Change Classification in Pacific Northwest National Parks

Abstract

We evaluated the feasibility of using aerial photo-based office methods rather than field-collected data to validate Landsat-based change detection products in national parks in Washington State. Landscape change was performed using LandTrendr algorithm. The resulting change patches were labeled in the office using aerial imagery and a random sample of patches was visited in the field by experienced analysts. Comparison of the two labels and associated confidence shows that the magnitude or severity of the change is a strong indicator of whether field assessment is warranted, and that confusion about patches with lower magnitude changes is not always resolved with a field visit. Our work demonstrates that validation of Landsat-derived landscape change patches can be done using office based tools such as aerial imagery, and that such methods provide an adequate validation for most change types, thus reducing the need for expensive field visits.

1. Introduction

Background

As computing power has improved and satellite imagery has become more readily available, broad scale, moderate resolution, multi-temporal mapping, classification, and monitoring of landscape change has become increasingly feasible [1,2]. Moreover, improved algorithms provide the ability to discern among an ever larger group of change processes, including both anthropogenic and natural events. With this increasing power comes a heightened requirement for validation of landscape change detection products [3]. Field work to collect assessment or validation data is expensive and logistically challenging, particularly for large-area projects that include private land or remote and inaccessible terrain, including mountainous wilderness areas [4]. Strategies to address these challenges vary, including limiting the sample size or making use of existing field validation data collected for different projects and applications. However, small sample sizes reduce power, and external data rarely match the temporal and spatial qualities needed to align with the remote sensing products [3].

High-resolution aerial and satellite imagery offer a possible solution. As tools such as Google Earth™ and CollectEarth ease access to and use of historical aerial imagery, it has become possible for broad scale projects to use independently derived, remotely sensed datasets rather than field collected data to validate change detection and labeling results [3,5,6,7]. To create a validation dataset, a skilled analyst views the landscape using appropriately dated imagery via a desktop software package such as Google Earth™. If these methods offer information content comparable to that collected in the field, they could represent an attractive solution for landscape change projects, particularly when costs drive project planning. However, to our knowledge, studies designed to explicitly compare field and office-based validation of landscape change are lacking.

Observations of landscape change from above or from under the canopy each have strengths and weaknesses. Office-based interpretation of aerial imagery offers the opportunity to view the changed area within a broader landscape context that is not always easily viewed in the field. Aerial imagery also offers the viewing perspective similar to that of the satellite imagery, and can importantly provide the opportunity to compare conditions before and after a landscape event has occurred. However, interpretation depends on the availability of high resolution images bracketing the change events, and even with available images, some types of change may be too subtle or obscured to detect. This may be particularly challenging when the target change processes are more complex or variable in effect than on the ground. On the other hand, field visits can provide information that is difficult to see in aerial imagery, such as subtleties in vegetation vigor or the impacts of landscape change processes occurring under the forest canopy. However, field visits can only examine current conditions, and may lack the historical and landscape context to understand what happened in the past.

Resolving this issue is particularly important for natural resources agencies that must manage large land holdings with relatively modest resources. The North Coast and Cascades Inventory and Monitoring Network (NCCN) of the National Park Service (NPS) precisely faced this issue. Tasked with monitoring landscape change inside and outside several large national parks in the U.S.’s Washington State, the NCCN developed a methodology to map, label, and monitor landscape change in the large national parks of the Pacific Northwest [8]. To be cost-effective, this original monitoring protocol utilized office-based aerial photo interpretation for validation, but questions remained about the reliability of the method.

Here, we report on a study designed to evaluate the consequences of using aerial photo-based office methods rather than field-collected data to validate Landsat-based change detection products. Using a restricted random sampling approach, we identified landscape change patches across three large wilderness parks and compared interpretations derived from field versus aerial photo interpretation. Key questions included: Is office validation sufficient to replace field validation entirely, or is some combination of the two types needed? Under what situations does the office change category label differ from the field label? While focused on our own region of interest, we anticipate that our findings may guide other natural resource managers faced with similar tradeoffs between cost and validation robustness.

2. Materials and Methods

2.1. Description of the Study Area

The study area encompasses Mount Rainier (MORA, 46.860709°N; 121.703529°W), North Cascades Complex (NOCA, 48.690548°N; 121.140239°W), and Olympic (OLYM, 47.803139°N; 123.676729°W) National Parks, located in Washington State, USA (Figure 1). Over 90% of each park is designated wilderness, making them an ideal location to study natural disturbances, but also very challenging places to access due to the lack of roads and a minimal trail network. The three parks share a common mix of mountainous terrain, forested valley walls, and lower elevation riverine watersheds.

All three parks feature tall peaks or mountain ranges at their core, creating steep precipitation gradients due to the strong rainshadow effects. Annual precipitation on the West side of the parks exceeds 300 cm, decreasing to 60 cm or less to the East. Snow dominates winter precipitation above 1800 m in all the parks, with 8–10 m accumulating on average each winter. Glaciers can be found in each of the parks, forming the headwaters of regionally significant rivers including the Hoh, Skagit, and Nisqually. The Western lowlands in all parks are dominated by mesic Douglas-fir (Pseudotsuga menziesii) and Western hemlock (Tsuga heterophylla) forests which grade to Pacific silver fir (Abies amabilis) forests at middle elevations. The montane silver fir forests transition into mountain hemlock (Tsuga mertensiana) and subalpine fir (Abies lasiocarpa) woodlands and subalpine meadows as elevation increases. Above the upper limit of treeline, alpine meadows segue to rock and ice at the highest elevations. In the Eastern side of the parks, dry forests and woodlands of pine (Pinus spp), Douglas-fir, subalpine fir, and larch (Larix lyallii) predominate.

2.2. Change Detection

To identify and label landscape change, we utilized methods based on those described in Kennedy et al. [5]. The underlying change detection method was LandTrendr, a temporal segmentation algorithm that attempts to characterize durable change in the spectral trajectories of individual pixels over the entire time period of the Landsat Thematic Mapper (TM) sensors [9]. LandTrendr traces the spectral trajectory of each pixel through the time period of the Landsat imagery and uses statistical line-fitting algorithms to break the trajectory into a series of segments. Significant changes in pixel reflectance were marked as inflection points in the yearly trajectory, and from those inflection points we identified the year, magnitude, and duration of change. For this study, we used the normalized burn ratio (NBR) as the spectral index to identify change [10] and mapped change from 1985 to 2009 at North Cascades Complex (NOCA) and Mount Rainier (MORA) and 1985 to 2010 at Olympic (OLYM) National Parks. Following methods described in our original protocol [8], our workflow initially focused on pixel-level changes whose duration was four years or less and whose magnitude of change was at least 10% of the starting index value. Then, once pixels of change were identified, we grouped pixels of the same disturbance year together into disturbance patches, and removed any patches with fewer than nine pixels (less than 0.8 ha). We used Random Forest classification algorithms [11] to assign a category of change to all LandTrendr-generated patches inside the park boundaries following methods described in Kennedy et al. [5]. We summarized the classification results in a series of NPS Natural Resource Data Series reports [12,13,14]. These patches formed the basis of our comparisons.

2.3. Categories of Landscape Change Monitored

We focused on natural disturbances that significantly removed or altered vegetation. This included fires, landslides, avalanches, tree topples, river course alterations, or forest health declines (

Table 1. List of landscape change types and description.

Landscape Change Type	Definition
Annual Variability	Categorizes landscape changes caused by annual differences in cloud cover, sun angle, phenology, and soil moisture. Annual Variability patches are explicitly modeled so that they can be eliminated from the analysis and results, because they do not capture change of interest to the NPS.
Avalanche	Long, linear change areas which originate in snow receiving zones or valley walls. Typically remove some but not all of the vegetation.
Fire	Often corroborated from outside sources, wildland fires vary in intensity from full canopy removal to partial burns which leave behind a mixture of dead and singed trees.
Mass Movement	Category includes a variety of vegetation-removing changes that expose rock and bare ground: landslides, which are found on valley walls and away from streams; creeps, which are slow downward movements of slope-forming soil or rock; and debris flows that are associated with steep gullies and involve water. Mass Movement is distinguished from the Riparian category because it occurs on slopes greater than 15 degrees.
Progressive Defoliation	Assigned to polygons where the forest cover remains but has undergone slow change in spectral values representing a loss of greenness and wetness. Several patterns of decline in tree health can be seen, reflecting various causal agents including insects, pathogens and drought-stress.
Riparian	Change areas of this type are restricted to the valley floor in areas where either conifer or broadleaf vegetation previously existed and has been converted to active river channel, with water or river banks.
Tree Toppling	Forest areas where trees have been broken off or topped by some action other than Avalanche, generally due to wind but sometimes due to root rot.

, Appendix A). Additionally, we explicitly identified a type of false positive change common in our change detection products. This type of change results when non-durable changes caused by ephemeral phenomena trigger the LandTrendr algorithm to falsely identify a pixel as change, when in fact it was merely subject to unusual phenology, changes in soil moisture, or imperfect masking of shadow or cloud. Consistently found in the subalpine zone of the study area, we labeled this type “Annual Variability.”

2.4. Change Label Validation

2.4.1. Office Validation

Following the Random Forest classification, an NPS Geographic Information Systems (GIS) specialist reviewed all change category labels (primary label) inside the park boundaries in the office by using the time series of United States Geological Survey (USGS) and the National Agricultural Imagery Program (NAIP) orthomosaic imagery available in Google Earth™ (Figure 2). The individual park reports list the images used for office validation [12,13,14].

In addition to aerial photography, the analyst performing the office validation visualized the spectral trajectory of a group of pixels surrounding the center of the patch (Figure 3) and viewed the time series of Landsat image chips centered over the patch (Figure 4). The analyst also referenced US Forest Service Insect and Disease Aerial Surveys [15] as an independent data source of areas with affected vegetation.

Each patch was also assigned a subjective level of “confidence” in the choice of landscape change label ranging from one (least confident) to three (most confident). In general, we assigned a confidence of three when three criteria were all consistent with expectations for that change category: the visual assessment of the patch size and shape, its spectral trajectory characteristics, and its location on the landscape. We chose a confidence of two when one of the evaluation criteria either did not match the expected pattern or could not be assessed (as with subtle spectral change). Additionally, we assigned a confidence of two when a patch appeared to be mixed: pixels of different change types were combined into the same patch because they were spatially adjacent and occurred during the same year. A confidence of one was assigned when at least two of the evaluation criteria did not match the expected pattern or could not be assessed due to lack of high resolution imagery. If a confidence of two or one was chosen, the analyst recorded a second choice for the patch label (alternative label). In the case of a mixed patch, the primary label was given to the change category that occupied more than 50% of the patch.

2.4.2. Field Validation

Random Sample

Following office labeling, a subset of the patches in each park from the years 2006–2010 was randomly chosen for field validation using the Alaska Pak tool in ArcGIS [17]. Most patches were located within a 500 m buffer of accessible trails and roads and within 5 km of a trailhead. Our previous work has demonstrated that this restricted area of inference is representative of the larger study area. Some change types such as Mass Movements, occurred infrequently during our analysis period. For these rare change types, we expanded the sample further than 500 meters from trails and roads in order to have a minimum of five patches. Fire patches were not included in the field validation of randomly selected patches because we could use existing NPS data to identify them reliably [18]. While the map dataset went back to 1985, we only selected patches from the most recent years for two reasons: (1) vegetation recovery since the time of disturbance can make it difficult to assess the original change agent, and (2) our long-term monitoring protocol repeats periodically, allowing us to evaluate landscape changes every 3–5 years. In 2011–2016 NPS ecologists and/or GIS specialists visited 132 randomly selected patches in the field in all three parks (Figure 5), including 25 patches in MORA, 54 in NOCA, and 53 in OLYM. Field crews were only provided information about the location and the year of the disturbance. A Garmin GPS unit was used to navigate to a Landsat pixel near the center of the patch. In cases where distance or barriers such as streams or steep slopes prevented a field visit, but the patch could be accurately located on the landscape from a nearby vantage point, the field team collected data from a distance. Twenty-eight (21.3 %) of patches were assessed from a distance. Photos were taken of each patch. Confidence in the type was recorded using the same scale as the office validation. Patches with a single, clear change category were assigned a confidence of three. Mixed patches, where evidence of more than one change type was present, and patches where the field team was not sure if the observed change on the ground would have been picked up by satellite overhead because of canopy cover, were assigned a confidence of two. Patches with more than two change types present were assigned a confidence of one. An alternative label was recorded for patches with lower confidence levels. In the case of a mixed patch, the primary label was given to change category that occupied more than 50% of the patch.

Incidental Field Observations

As field sampling is expensive and time consuming, during field-work, we opportunistically extended our dataset by sampling any change patch from 2006–2010 that was accessible on the route to the target random sample. Through this process, we added another 134 patches that we named “incidental samples” (Figure 5). While the incidental samples do not meet requirements for design-based sampling, their selection depended entirely on the random arrangement of change patches along access routes and was not affected by human preference. Given the cost in reaching the randomly-selected plots, the addition of these plots provided an opportunity to double our dataset size. In our later analysis, we treat these samples both separately and combined, and allow the reader to judge the degree to which they add useful information.

See Appendix A for examples of field photos and corresponding office validation materials for each landscape change type.

2.5. Data Analysis

We used a series of agreement matrices to assess how frequently the office label differed from the field label for each of the change categories and sample types. We first compared the office and field labels for the random sample using only primary labels from both field and office, i.e. the alternative label was not considered. We then repeated the process for a dataset that included both random and incidental observations to determine if the general patterns noted for the random plots held. A second matrix was generated for randomly-selected plots that considered both primary and alternative labels to determine a label match. In this case, if one of the primary or alternative office labels matched one of the primary or alternative field labels, the combination was considered a match. While the agreement matrix uses the same format as a contingency matrix, one significant difference in this application is that one view is not considered “truth” and the other the “observation”; therefore, we do not apply labels like “user’s accuracy” and “producer’s accuracy.”

Finally, we examined agreement matrices with respect to confidence values of office and field labels to determine how confidence was distributed between the two validation types and among change categories.

3. Results and Discussion

3.1. Random sample

Office- and field-assigned change category labels for randomly selected patches had an overall agreement of 88.5% (

Table 2. Agreement matrix comparing primary labels assigned in office and field for the random sample only.

		Office Label
	Row Labels	Annual Variability	Avalanche	Fire	Mass Movement	Progressive Defoliation	Riparian	Tree Toppling	Grand Total
Field Label	Annual Variability	15				1			16	93.8
	Avalanche	1	15			1			17	88.2
	Fire						1		1	0.0
	Mass Movement		1		14	1			16	87.5
	Progressive Defoliation					34			34	100.0
	Riparian						19		19	100.0
	Tree Toppling		1		1	7		19	28	67.9
	Grand Total	16	17	0	15	44	20	19	131
		93.8	88.2	0.0	93.3	77.3	95.0	100.0
	Overall Agreement	88.5

). Most change categories had a high degree of agreement (> 85%) between office and field labels. The Riparian category had the highest agreement, likely because this category has the most spatially and spectrally discrete signal and a clear landscape context. The Annual Variability, Avalanche, and Mass Movement categories all had levels of agreement above 85%, with Annual Variability showing agreement above 90%. Avalanche and Mass Movement patches were easily recognizable in the field and in the office based on landscape position and the quality of vegetation disturbance associated with them (see Table 1). Annual Variability was unique among the change categories because it denoted a lack of lasting change on the ground. It was most often identified in the field by eliminating all other categories.

Tree Toppling and Progressive Defoliation labels had the lowest agreement and were often confused with each other. Five out of seven patches with a primary field label of Tree Toppling and primary office label of Progressive Defoliation had a secondary field label of Progressive Defoliation. Only one office patch out of seven had Tree Toppling as a second option, but Annual Variability was listed four times. This suggests that the office analyst had difficulty interpreting mild Tree Toppling on aerial photography, but did see subtle change in the spectral signal of the pixel trajectory that signified some type of change in forest canopy.

Detailed evaluation of individual cases of disagreement between the office and field labels for the random sample of patches showed three general patterns.

1. Timing of field visit

We found several instances where the field visit occurred after a later disturbance event had superseded the disturbance event of interest, resulting in the field crew evaluating the wrong event. This was the case for one disagreement in the Riparian category, with Fire occurring after the change was detected by the mapping algorithm, but before the field sampling occurred. The sequence of disturbance could easily be discerned when looking at the time series of aerial photos, but was not evident in the field. A similar scenario took place where an Avalanche path was later covered by debris from a Mass Movement, confusing identification in the field.

2. Labeling of process as opposed to result

This category of disagreement includes patches where there may have been some confusion between identifying the cause and effect of landscape change being labeled. From the office view, the analyst usually only sees the effect of change: reduced canopy vigor (Progressive Defoliation), for example, or fallen trees (Tree Topple). During field sampling, however, one might be able to more readily identify the cause of change by looking at what happened under the canopy: reduced canopy vigor having been caused by a small debris flow that covered the base of trees with mud (Mass Movement); or Tree Toppling having been caused by water inundation, killing and toppling the trees (Riparian). The field calls in these cases recorded the cause of change and not the effect as it was visible to the satellite and the analyst in the office. Patches labeled as Progressive Defoliation in the office were most frequently found to have some other disturbance under the canopy. Both cause and effect were often labeled in the field using confidence of two and an alternate agent.

3. Mixed change

This category of disagreement includes patches that had two or more categories of change evident during the field and/or office labeling. Occurrence of several landscape altering events within one year is not distinguishable when using the LandTrendr algorithms, as these utilize a single image per year for change detection, and patches are delineated by combining adjacent pixels labeled with the same year. Therefore, if two separate weather events in the same year caused different categories of change to be adjacent to each other, the resulting patch would contain the effects of both events. Similarly, a single weather event could cause two or more categories of change to occur next to each other if the landscape context was different. This was seen frequently at OLYM and MORA along river corridors after a 2007 winter windstorm event, when riparian flooding and debris flows were seen alongside windthrow patches that caused tree toppling. In both field and office evaluation, such mixed patches were usually labeled with the confidence of two and an alternate category was recorded. When evaluating such labels for agreement, we often found that even though the same categories of change were present in both types of labeling, their order of priority was switched. This was in part caused by different perspectives of field vs. office analysis, where one category appeared to be more prominent under the canopy vs. above, but we also found it difficult to assess the extent of each category of change on the ground when the entire patch was not visible. Indeed, when given the opportunity, we found it easier to label patches in the field when surveying them from afar, especially for certain categories, particularly Progressive Defoliation and Annual Variability.

To determine how the degree of agreement would change if alternative labels were taken into account, we generated a second matrix where labels were considered to be in agreement if either the primary or alternative office label matched either the primary or alternative field label (

Table 3. Agreement matrix comparing primary and secondary labels assigned in office and field for the random sample only.

		Office Agent
	Row Labels	Annual Variability	Avalanche	Fire	Mass Movement	Progressive Defoliation	Riparian	Tree Toppling	Grand Total
Field Agent	Annual Variability	16							16	100.0
	Avalanche		15						15	100.0
	Fire								0	0.0
	Mass Movement		1		15				16	93.8
	Progressive Defoliation					42			42	100.0
	Riparian						20		20	100.0
	Tree Toppling		1			2		19	22	86.4
	Grand Total	16	17	0	15	44	20	19	131
		100.0	88.2	0.00	100.0	95.5	100.0	100.0
	Overall Agreement	96.9

). While the disagreement resulting from the timing of the field visit was not improved by this approach, we saw a few of the patches in the other two categories switch to being considered as having a matching label, with the overall agreement increasing to 96.9%. This indicated that the same two categories of change were consistently chosen in office and field, but a different order was used.

3.2. Random and Incidental Sample

We repeated our evaluation using both random and incidental samples and only the primary labels for each patch, and found that the patterns remained largely consistent in this larger dataset (

Table 4. Agreement matrix comparing primary labels assigned in office and field for both random and incidental samples.

		Office Labels
	Row Labels	Annual Variability	Avalanche	Fire	Mass Movement	Progressive Defoliation	Riparian	Tree Toppling	Grand Total
Field Labels	Annual Variability	23				1			24	95.8
	Avalanche	1	22		1	1	1		26	84.6
	Fire			13		1	1		15	86.7
	Mass Movement		1		14	1			16	87.5
	Progressive Defoliation		1			81		1	83	97.6
	Riparian						47		47	100.0
	Tree Toppling		1		1	6	2	42	52	80.8
	Grand Total	24	25	13	16	91	51	43	263
		95.8	88.0	100.0	87.5	89.0	92.2	97.7
	Overall Agreement	92.02

). Avalanche, Annual Variability, and Mass Movement categories showed agreements similar to that of the random sample. Fire also had a high degree of agreement, with labeling being more confused in the office in cases of lower intensity fires. The accuracy of Progressive Defoliation and Tree Toppling categories improved, with the majority of newly added incidental patches having agreement between office and field labels in these two categories. The Riparian category remained clearly labeled in the field, but slightly less clear in office, with a few patches labeled Avalanche and Tree Toppling.

3.3. Evaluation of Confidence Rankings

For the random sample using primary labels only, we evaluated the distribution of the confidence in the labeling (

Table 5. Agreement matrix comparing agreement between office and field confidence levels for primary labels assigned in office and field for the random sample only: (a) raw numbers and (b) percent of sample.

		Office Confidence
		1	2	3	Grand Total
Field Confidence	1	0	0	1	1
	2	4	16	21	41
	3	2	16	72	90
	Grand Total	6	32	94	131
(a)
		% Sample Office Confidence
		1	2	3	Grand Total
% Sample Field Confidence	1	0	0	0.76	0.76
	2	3.1	12.2	16.0	31.3
	3	1.5	12.2	55.0	68.7
	Grand Total	4.6	24.4	71.8	100.0
(b)

a,b). For more than 50% of the patches, the field and office analysts shared high confidence. Rarely was the separation of confidence extreme: Only in 4.6% of the patches did the office analyst have lowest confidence and the field analyst highest confidence, and only 1% of the time was the reverse the case. However, there were several areas where confidence scores were close, but not matching: in 12% of the cases, field confidence was lower than in the office, and in 16% of the cases, the reverse was true.

Dissecting these areas of disagreement in confidence paints a more nuanced picture of the relative strengths of the field and the office-based assessments (

Table 6. Comparison of high and medium confidence levels by change category.

Category	% Patches: 2 Office/3 Field	% Patches: 3 Office/2 Field
Annual Variability	4	13
Avalanche	8	27
Fire	33	7
Mass Movement	31	19
Progressive Defoliation	5	18
Riparian	1	0
Tree Toppling	25	10

and Figure 6). Progressive Defoliation, Avalanche, and Annual Variability categories had higher percentage of high confidence labels in the office, where Tree Toppling and Mass Movement patches were labeled with more confidence in the field (Table 6). Field analysts had difficulty labeling Annual Variability in the field, with several patches having a confidence of one, largely because the category can only be labeled by eliminating all other categories first, often leaving uncertainty in the choice selection and possibility of other choices (Figure 6). In contrast, the office analyst can utilize landscape position and highly variable spectral trajectory to assign the label of Annual Variability.

Progressive Defoliation had relatively lower percentage of high confidence labels in both office and field, with a number of plots having the lowest confidence level in the office (Figure 7). Progressive Defoliation is difficult to assess from the office because of the quality and resolution of the one meter NAIP imagery typically used. Orthorectification errors, color balancing, and variable acquisition times all degrade the image quality and make it difficult to visualize the subtle changes in canopy condition detected by LandTrendr for this change type. While assigning labels in the office, altered canopy condition, such as clusters of red needles, are difficult to see in the NAIP. This might encourage the analyst to look for evidence—even subtle or on a fine scale—of any other change spotted in the image, such as broken trees. As with Annual Variability, consulting the spectral trajectory is often helpful, although less so if the change is subtle. In the field, Progressive Defoliation is often hard to see in dense forest where the condition of the canopy is difficult to assess from below and there is no strong evidence of insect or disease.

Mass Movements were harder to identify in the office (lower confidence scores), possibly because smaller Mass Movements under canopies were not visible or because the change resulted in secondary change that was visible. Evidence of Mass Movements, such as scattered rock and debris, is easier to see in the field.

While Tree Toppling had a similar distribution of confidence levels in the office and in the field, the reasons for assigning a confidence of two were quite different. In the office, a minor Tree Toppling, with only a few trees down, was hard to identify on aerial photos, and Progressive Defoliation was often assigned as an alternative label. In the field, the Tree Toppling was often seen together with another change category and the confidence of two was assigned because the patch was “mixed.”

4. Conclusions and Recommendations

As computational and algorithmic advances improve our ability to map natural change processes over broad landscapes, the assessment datasets used to evaluate the maps must similarly improve. The cost of conducting field visits to build reference or validation datasets can limit sample size, making methods that are based on aerial photo interpretation conducted in an office setting attractive. This is particularly the case in the large wilderness parks of Washington state, U.S.A., where access for field visitation is time consuming and costly, but where inventories of natural disturbance processes are important to understand landscape dynamics inside and outside protected areas. In order to optimize our validation effort, we compared the relative strengths of field versus office-based validation of landscape change in Mount Rainier, North Cascades Complex, and Olympic National Parks.

In general, we found that the office and field labels strongly agreed. For five of the seven landscape change types we evaluated (Annual Variability, Avalanche, Fire, Mass Movement, and Riparian), the office and field labels had high degree of agreement. For two remaining categories (Progressive Defoliation and Tree Toppling), office-based evaluation of aerial photos was slightly less effective at identifying change, in larger part due to inadequate image resolution. These findings generally held true both for the sample dataset that included only random samples as well as the expanded dataset that included opportunistically acquired samples. Thus, even though we limited our random samples to areas within realistic access from trails and roads, we argue that the underlying change processes represented in our sample form a reasonable representation of the conditions operating more broadly in these parks.

As we explicitly recognized that all interpretation is subject to some uncertainty, we included methods to capture ambiguity and could use those observations to gain a more nuanced understanding of the conditions under which agreement was weaker. Confidence scoring showed that office interpreters had difficulty confidently identifying Tree Toppling and Progressive Defoliation, while field interpreters had difficulty confidently identifying Annual Variability. This is encouraging as these were indeed the categories where those respective interpreters had greater disagreement, suggesting that actual error is related to uncertainty. This is corroborated by the observations of primary and secondary labels: by aggregating primary and secondary labels, the disagreement among these types diminished.

Case by case evaluation of individual plots of disagreement provided further insight into the causes of disagreement. Some errors were introduced in the field sample due to the delay in the time since disturbance and the field visit, emphasizing the importance of doing the field work soon after the disturbance occurred. Some errors were introduced because of the inevitable mismatch between the ground perspective and the office (aerial photograph) perspective. In general, field observations allowed greater understanding of the complexities of the change processes, as could be expected. Indeed, some disagreement occurred when field observers were able to observe the process itself, rather than simply the manifestation of that process visible at the top of the canopy from an aerial photo. However, field observations underneath a canopy may not be able to understand what is occurring at the top of the canopy that could be detectable by the satellite imagery.

Indeed, this raises an important issue to consider whenever designing a validation study. For certain change processes, field visits will be more able to understand the processes involved. However, the downward-looking view afforded by high resolution imagery provides the same perspective as that of the satellite imagery on which the landscape maps are made, and indeed might be more appropriate as a means of corroborating the detection noted by the sensors. When a subtle mass movement under a canopy causes trees to topple, it may be just as appropriate to document the impact as the potential underlying cause. Moreover, it appears that even the cases of disagreement are relatively minor.

In general, we conclude that the magnitude or severity of change is a better indicator of whether or not a field assessment is warranted. More damaging and abrupt categories of change, especially those that remove canopy cover, are more suitable for labeling in the office. More subtle changes and changes in forests that do not remove canopy, such as Progressive Defoliation or a minor Tree Topple, could potentially be checked in the field. However, our results suggest that a field visit may not entirely resolve uncertainty about these subtle types of change.

Our work demonstrates that validation of Landsat-derived landscape change patches can be done using office based tools such as aerial imagery, and that such methods provide an adequate validation for most change-types, thus reducing the need for expensive field visits.