Next Article in Journal
Development of a 3D Real-Time Atmospheric Monitoring System (3DREAMS) Using Doppler LiDARs and Applications for Long-Term Analysis and Hot-and-Polluted Episodes
Previous Article in Journal
Editorial for the Special Issue “Remote Sensing of the Terrestrial Hydrologic Cycle”

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Graduate School of Geography, Clark University, Worcester, MA 01610, USA
School for the Environment, University of Massachusetts Boston, Boston, MA 02125, USA
Radiant Earth Foundation, San Francisco, CA, 94105, USA
Department of Geography, University of California, Santa Barbara, CA 93013, USA
Bren School of Environmental Science and Management, University of California, Santa Barbara, CA 93013, USA
Azavea, Inc., Philadelphia, PA 19123, USA
Department of Earth and Environment, Boston University, Boston, MA 02215
School for Environment and Sustainability, University of Michigan, Ann Arbor, MI 48109, USA
Faculty of Geo-Information Science & Earth Observation (ITC), University of Twente, 7514 AE Enschede, The Netherlands
Center for Earth Observation and Citizen Science, Ecosystems Services and Management Program, International Institute for Applied Systems Analysis (IIASA), Laxenburg A-2361, Austria
National Security Emerging Technologies, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA
Department of Geography and Geospatial Analysis Center, Miami University, Oxford, OH 45056, USA
Environmental Sciences Initiative, CUNY Advanced Science Research Center, New York, NY 10065, USA
Department of Geography and Environmental Science, Hunter College, New York, NY 10065, USA
Development Seed, Washington, DC 20001, USA
Author to whom correspondence should be addressed.
Remote Sens. 2020, 12(6), 1034;
Received: 8 February 2020 / Revised: 6 March 2020 / Accepted: 18 March 2020 / Published: 23 March 2020
(This article belongs to the Section Environmental Remote Sensing)
Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience. View Full-Text
Keywords: training data; machine learning; map accuracy; error propagation training data; machine learning; map accuracy; error propagation
Show Figures

Graphical abstract

MDPI and ACS Style

Elmes, A.; Alemohammad, H.; Avery, R.; Caylor, K.; Eastman, J.R.; Fishgold, L.; Friedl, M.A.; Jain, M.; Kohli, D.; Laso Bayas, J.C.; Lunga, D.; McCarty, J.L.; Pontius, R.G.; Reinmann, A.B.; Rogan, J.; Song, L.; Stoynova, H.; Ye, S.; Yi, Z.-F.; Estes, L. Accounting for Training Data Error in Machine Learning Applied to Earth Observations. Remote Sens. 2020, 12, 1034.

AMA Style

Elmes A, Alemohammad H, Avery R, Caylor K, Eastman JR, Fishgold L, Friedl MA, Jain M, Kohli D, Laso Bayas JC, Lunga D, McCarty JL, Pontius RG, Reinmann AB, Rogan J, Song L, Stoynova H, Ye S, Yi Z-F, Estes L. Accounting for Training Data Error in Machine Learning Applied to Earth Observations. Remote Sensing. 2020; 12(6):1034.

Chicago/Turabian Style

Elmes, Arthur, Hamed Alemohammad, Ryan Avery, Kelly Caylor, J. Ronald Eastman, Lewis Fishgold, Mark A. Friedl, Meha Jain, Divyani Kohli, Juan Carlos Laso Bayas, Dalton Lunga, Jessica L. McCarty, Robert Gilmore Pontius, Andrew B. Reinmann, John Rogan, Lei Song, Hristiana Stoynova, Su Ye, Zhuang-Fang Yi, and Lyndon Estes. 2020. "Accounting for Training Data Error in Machine Learning Applied to Earth Observations" Remote Sensing 12, no. 6: 1034.

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

Back to TopTop