Search Results (13)

The Total Operating Characteristic (TOC) is an improvement on the quantitative method called the Relative Operating Characteristic (ROC), both of which plot the association between a binary variable and a rank variable. TOC curves reveal the sizes of the four entries in the confusion matrix at each threshold, which make TOC curves more easily interpretable than ROC curves. The TOC has become popular, especially to assess the fit of simulation models to predict land change. However, the literature has shown variation in how authors apply and interpret the TOC, creating some misleading conclusions. Our manuscript lists best practices when applying and interpreting the TOC to help scientists learn from TOC curves. An example illustrates these practices by applying the TOC to measure the ability to predict the gain of crop in Western Bahia, Brazil. The application compares four ways to design the rank variable based on the distance to either pixels or patches of either the presence or change of crop. The results show that the gain of crop during the validation time interval is more strongly associated with the distance to patches rather than pixels of crop. The Discussion Section reveals that if authors show the TOC curves, then readers can interpret the results in ways that the authors might have missed. The Conclusion encourages scientists to follow best practices to learn the wealth of information that the TOC reveals. Full article

This manuscript provides guidance concerning how to use the Total Operating Characteristic (TOC) when (1) analyzing change through time, (2) ranking a categorical independent variable, and (3) constraining the extent for a gaining category. The illustrative variable is the marsh land-cover category in the Plum Island Ecosystems of northeastern Massachusetts, USA. The data are an elevation map and maps showing the land categories of water, marsh, and upland in 1938, 1971, and 2013. There were losses and gains near the edge of the marsh between 1938 and 1972 and between 1972 and 2013. The TOC curves show that marsh gained most intensively at intermediate elevations during the first time interval and then had a weaker association with elevation during the second time interval. Marsh gains more intensively from water than from upland during both time intervals. The TOC curves also demonstrate that the marsh gains occurred where marsh was previously lost, a phenomenon called Alternation. Furthermore, eliminating far distances and extreme elevations from the spatial extent decreased the area under the curve (AUC) for distance and increased the AUC for elevation. We invite scientists to use the TOC because the TOC is easier to interpret and shows more information than the Relative Operative Characteristic. Full article

The Flow matrix is a novel method to describe and extrapolate transitions among categories. The Flow matrix extrapolates a constant transition size per unit of time on a time continuum with a maximum of one incident per observation during the extrapolation. The Flow matrix extrapolates linearly until the persistence of a category shrinks to zero. The Flow matrix has concepts and mathematics that are more straightforward than the Markov matrix. However, many scientists apply the Markov matrix by default because popular software packages offer no alternative to the Markov matrix, despite the conceptual and mathematical challenges that the Markov matrix poses. The Markov matrix extrapolates a constant transition proportion per time interval during whole-number multiples of the duration of the calibration time interval. The Markov extrapolation allows at most one incident per observation during each time interval but allows repeated incidents per observation through sequential time intervals. Many Markov extrapolations approach a steady state asymptotically through time as each category size approaches a constant. We use case studies concerning land change to illustrate the characteristics of the Flow and Markov matrices. The Flow and Markov extrapolations both deviate from the reference data during a validation time interval, implying there is no reason to prefer one matrix to the other in terms of correspondence with the processes that we analyzed. The two matrices differ substantially in terms of their underlying concepts and mathematical behaviors. Scientists should consider the ease of use and interpretation for each matrix when extrapolating transitions among categories. Full article

The profession debates how to encode a categorical variable for input to machine learning algorithms, such as neural networks. A conventional approach is to convert a categorical variable into a collection of binary variables, which causes a burdensome number of correlated variables. TerrSet’s Land Change Modeler proposes encoding a categorical variable onto the continuous closed interval from 0 to 1 based on each category’s Population Evidence Likelihood (PEL) for input to the Multi-Layer Perceptron, which is a type of neural network. We designed examples to test the wisdom of these encodings. The results show that encoding a categorical variable based on each category’s Sample Empirical Probability (SEP) produces results similar to binary encoding and superior to PEL encoding. The Multi-Layer Perceptron’s sigmoidal smoothing function can cause PEL encoding to produce nonsensical results, while SEP encoding produces straightforward results. We reveal the encoding methods by illustrating how a dependent variable gains across an independent variable that has four categories. The results show that PEL can differ substantially from SEP in ways that have important implications for practical extrapolations. If users must encode a categorical variable for input to a neural network, then we recommend SEP encoding, because SEP efficiently produces outputs that make sense. Full article

The Total Operating Characteristic (TOC) measures how the ranks of an index variable distinguish between presence and absence in a binary reference variable. Previous methods to generate the TOC required the reference data to derive from a census or a simple random sample. However, many researchers apply stratified random sampling to collect reference data because stratified random sampling is more efficient than simple random sampling for many applications. Our manuscript derives a new methodology that uses stratified random sampling to generate the TOC. An application to flood mapping illustrates how the TOC compares the abilities of three indices to diagnose water. The TOC shows visually and quantitatively each index’s diagnostic ability relative to baselines. Results show that the Modified Normalized Difference Water Index has the greatest diagnostic ability, while the Normalized Difference Vegetation Index has diagnostic ability greater than the Normalized Difference Water Index at the threshold where the Diagnosed Presence equals the Abundance of water. Some researchers consider only one accuracy metric at only one threshold, whereas the TOC allows visualization of several metrics at all thresholds. The TOC gives more information and clearer interpretation compared to the popular Relative Operating Characteristic. Our software generates the TOC from a census, simple random sample, or stratified random sample. The TOC Curve Generator is free as an executable file at a website that our manuscript gives. Full article

Conventional methods to analyze a transition matrix do not offer in-depth signals concerning land changes. The land change community needs an effective approach to visualize both the size and intensity of land transitions while considering possible map errors. We propose a framework that integrates error analysis, intensity analysis, and difference components, and then uses the framework to analyze land change in Nanchang, the capital city of Jiangxi province, China. We used remotely sensed data for six categories at four time points: 1989, 2000, 2008, and 2016. We had a confusion matrix for only 2016, which estimated that the map of 2016 had a 12% error, while the temporal difference during 2008–2016 was 22% of the spatial extent. Our tools revealed suspected errors at other years by analyzing the patterns of temporal difference. For example, the largest component of temporal difference was exchange, which could indicate map errors. Our framework identified categories that gained during one time interval then lost during the subsequent time interval, which raised the suspicion of map error. This proposed framework facilitated visualization of the size and intensity of land transitions while illustrating possible map errors that the profession routinely ignores. Full article

Several factors influence the performance of land change simulation models. One potentially important factor is land category aggregation, which reduces the number of categories while having the potential to reduce also the size of apparent land change in the data. Our article compares how four methods to aggregate Corine Land Cover categories influence the size of land changes in various spatial extents and consequently influence the performance of 114 Cellular Automata-Markov simulation model runs. We calculated the reference change during the calibration interval, the reference change during the validation interval and the simulation change during the validation interval, along with five metrics of simulation performance, Figure of Merit and its four components: Misses, Hits, Wrong Hits and False Alarms. The Corine Standard Level 1 category aggregation reduced change more than any of the other aggregation methods. The model runs that used the Corine Standard Level 1 aggregation method tended to return lower sizes of changing areas and lower values of Misses, Hits, Wrong Hits and False Alarms, where Hits are correctly simulated changes. The behavior-based aggregation method maintained the most change while using fewer categories compared to the other aggregation methods. We recommend an aggregation method that maintains the size of the reference change during the calibration and validation intervals while reducing the number of categories, so the model uses the largest size of change while using fewer than the original number of categories. Full article

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience. Full article

The Verified Carbon Standard (VCS) recommends the Figure of Merit (FOM) as a possible metric to confirm models that simulate deforestation baselines for Reducing Emissions from Deforestation and forest Degradation (REDD). The FOM ranges from 0% to 100%, where larger FOMs indicate more-accurate simulations. VCS requires that simulation models achieve a FOM greater than or equal to the percentage deforestation during the calibration period. This article analyses FOM’s mathematical properties and illustrates FOM’s empirical behavior by comparing various models that simulate deforestation and the resulting carbon disturbance in Bolivia during 2010–2014. The Total Operating Characteristic frames FOM’s mathematical properties as a function of the quantity and allocation of simulated deforestation. A leaf graph shows how deforestation’s quantity can be more influential than its allocation when simulating carbon disturbance. Results expose how current versions of the VCS methodologies could conceivably permit models that are less accurate than a random allocation of deforestation, while simultaneously prohibit models that are accurate concerning carbon disturbance. Conclusions give specific recommendations to improve the next version of the VCS methodology concerning three concepts: the simulated deforestation quantity, the required minimum FOM, and the simulated carbon disturbance. Full article

This paper presents a method to optimise the calibration of parameters and land use transition rules of a cellular automata (CA) urban growth model using a self-adaptive genetic algorithm (SAGA). Optimal calibration is achieved through an algorithm that minimises the difference between the simulated and observed urban growth. The model was applied to simulate land use change from non-urban to urban in South East Queensland’s Logan City, Australia, from 1991 to 2001. The performance of the calibrated model was evaluated by comparing the empirical land use change maps from the Landsat imagery to the simulated land use change produced by the calibrated model. The simulation accuracies of the model show that the calibrated model generated 86.3% correctness, mostly due to observed persistence being simulated as persistence and some due to observed change being simulated as change. The 13.7% simulation error was due to nearly equal amounts of observed persistence being simulated as change (7.5%) and observed change being simulated as persistence (6.2%). Both the SAGA-CA model and a logistic-based CA model without SAGA optimisation have simulated more change than the amount of observed change over the simulation period; however, the overestimation is slightly more severe for the logistic-CA model. The SAGA-CA model also outperforms the logistic-CA model with fewer quantity and allocation errors and slightly more hits. For Logan City, the most important factors driving urban growth are the spatial proximity to existing urban centres, roads and railway stations. However, the probability of a place being urbanised is lower when people are attracted to work in other regions. Full article

The aim of this study is to improve the understanding of land changes in the Jiulong River watershed, a coastal watershed of Southeast China. We developed a stratified classification methodology for land mapping, which combines linear stretching, an Iterative Self-Organizing Data Analysis (ISODATA) clustering algorithm, and spatial reclassification. The stratified classification for 2002 generated less overall error than an unstratified classification. The stratified classifications were then used to examine temporal differences at 1986, 1996, 2002, 2007 and 2010. Intensity Analysis was applied to analyze land changes at three levels: time interval, category, and transition. Results showed that land use transformation has been accelerating. Woodland’s gains and losses were dormant while the gains and losses of Agriculture, Orchard, Built-up and Bare land were active during all time intervals. Water’s losses were active and stationary. The transitions from Agriculture, Orchard, and Water to Built-up were systematically targeting and stationary, while the transition from Woodland to Built-up was systematically avoiding and stationary. Full article

The Receiver Operating Characteristic (ROC) is widely used for assessing the performance of classification algorithms. In GIScience, ROC has been applied to assess models aimed at predicting events, such as land use/cover change (LUCC), species distribution and disease risk. However, GIS software packages offer few statistical tests and guidance tools for ROC analysis and interpretation. This paper presents a suite of GIS tools designed to facilitate ROC curve analysis for GIS users by applying proper statistical tests and analysis procedures. The tools are freely available as models and submodels of Dinamica EGO freeware. The tools give the ROC curve, the area under the curve (AUC), partial AUC, lower and upper AUCs, the confidence interval of AUC, the density of event in probability bins and tests to evaluate the difference between the AUCs of two models. We present first the procedures and statistical tests implemented in Dinamica EGO, then the application of the tools to assess LUCC and species distribution models. Finally, we interpret and discuss the ROC-related statistics resulting from various case studies. Full article

Intensity Analysis has become popular as a top-down hierarchical accounting framework to analyze differences among categories, such as changes in land categories over time. Some aspects of interpretation are straightforward, while other aspects require deeper thought. This article explains how to interpret Intensity Analysis with respect to four concepts. First, we illustrate how to analyze whether error could account for non-uniform changes. Second, we explore two types of the large dormant category phenomenon. Third, we show how results can be sensitive to the selection of the domain. Fourth, we explain how Intensity Analysis’ symmetric top-down hierarchy influences interpretation with respect to temporal processes, for which changes during a time interval influence the sizes of the categories at the final time, but not at the initial time. We illustrate these concepts by applying Intensity Analysis to changes during one time interval (2000–2004) in a part of Central Kalimantan for the land categories Forest, Bare and Grass. Full article