Context for Reproducibility and Replicability in Geospatial Unmanned Aircraft Systems

Abstract

Multiple scientific disciplines face a so-called crisis of reproducibility and replicability (R&R) in which the validity of methodologies is questioned due to an inability to confirm experimental results. Trust in information technology (IT)-intensive workflows within geographic information science (GIScience), remote sensing, and photogrammetry depends on solutions to R&R challenges affecting multiple computationally driven disciplines. To date, there have only been very limited efforts to overcome R&R-related issues in remote sensing workflows in general, let alone those tied to unmanned aircraft systems (UAS) as a disruptive technology. This review identifies key barriers to, and suggests best practices for, R&R in geospatial UAS workflows as well as broader remote sensing applications. We examine both the relevance of R&R as well as existing support for R&R in remote sensing and photogrammetry assisted UAS workflows. Key barriers include: (1) awareness of time and resource requirements, (2) accessibility of provenance, metadata, and version control, (3) conceptualization of geographic problems, and (4) geographic variability between study areas. R&R in geospatial UAS applications can be facilitated through augmented access to provenance information for authorized stakeholders, and the establishment of R&R as an important aspect of UAS and related research design. Where ethically possible, future work should exemplify best practices for R&R research by publishing access to open data sets and workflows. Future work should also explore new avenues for access to source data, metadata, provenance, and methods to adapt principles of R&R according to geographic variability and stakeholder requirements.

1. Introduction

What would happen if we ceased being able to trust the results of scientific findings? With scientific experiments and results informing policy, catalyzing the development of medicines, new technologies, and more, a lack of trust would lead to urgent problems of vast proportions. Some researchers have found themselves facing this very problem over the past few years, leading to what many have termed a “reproducibility crisis” [1] (p. 1), or essentially, an inability to verify that the results of studies are valid and sound. This issue has spread in part because a lack of reproducibility and replicability (R&R) undermines the credibility of valid science and affects both scientific practitioners, consumers, and other stakeholders [2]. Though not every researcher believes the R&R challenges have reached crisis levels, there is a general consensus that it is certainly a problem needing to be addressed, especially in high technology fields where validation of workflows and computer code are paramount [3,4,5,6].

Recent advances in geographic information systems (GIS), digital cartographic analysis, automated photogrammetric workflows, satellite image processing, and unmanned aircraft systems (UAS) have heavily influenced geography, GIScience, and related disciplines. These now increasingly require heavy computational work and thus demand an increased focus on R&R [7], a need which has not been adequately addressed [8]. This is especially true of remote sensing and photogrammetric workflows combined with UAS, where the juxtaposition of hardware and software technologies can become remarkably resistant to R&R if not carefully documented and presented [9].

While awareness of the issues related to R&R in science appears to have grown in recent years [10], the encouragement of open data practices that are both reproducible and replicable is an ongoing challenge in the scientific community at large, including GIScience, remote sensing, and related fields. There are far-reaching implications that extend from data validation and the creation of robust methodologies, to otherwise disadvantaged groups’ access to critical geospatial workflows and supporting data [11,12]. Though efforts to increase R&R in scientific studies are being undertaken across various aspects of GIScience, this review places a special focus on an area of rapid growth: UAS-based remote sensing and photogrammetric workflows [13].

Given the above issues and context, we address the following questions:

• Why does R&R matter in geography, GIScience, remote sensing, UAS, etc.?
• How does the literature incorporate R&R into GIScience and UAS research?
• What are key barriers to R&R affecting geospatial UAS workflows?
• What are best practices scientists can incorporate into future research to achieve a standard of R&R that expands the value of its impact to more stakeholders?

To capture the current trends in the field, and to provide information about ongoing efforts, we examine the nature of R&R among UAS-based remote sensing and photogrammetric workflows. We discuss in particular the incorporation of open-source software (OSS) into these processes to facilitate greater R&R. This open-data movement, though in its nascent stages, is shaping the idea of technological convergence as it drives rapid progress and expands the user base for emergent technology and discovery [14]. Remote sensing and photogrammetric workflows are then analyzed in the broader context of R&R in the scientific community at large.

The intent of this paper is not to conduct an extensive review of every paper involving drone-based image processing workflows in remote sensing and geography, etc.; there are multiple reviews of that topic already available [15,16,17,18,19]. Instead, we review relevant UAS-based remote sensing publications to assess which approaches to R&R have been most effective. We then identify areas of GIScience that require further development to reach an achievable level of R&R that will enable validation of results and benefit other researchers and end-users of the remote sensing products in question.

Following this introduction, Section 2 provides an overview of the topics of R&R and carefully reviews definitions, especially as they pertain to GIScience. Section 3 includes a discussion of current trends in R&R as seen in remote sensing and photogrammetry studies focused on UAS-based workflows and discusses key barriers to R&R. Section 4 contains a comparative examination of two case studies involving different applications of UAS data and OSS workflows. Finally, Section 5 covers a discussion of overall findings and provides recommendations for augmenting R&R in research when appropriate for stakeholder requirements.

2. Context and Rise of Reproducibility and Replicability

Though issues and ideas related to R&R have been prevalent throughout the history of specific disciplines, there has been a recent growth in interest regarding their wider application throughout the broader scientific community [1,10]. This interest has been largely driven by the trend in scientific applications being increasingly reliant upon highly technical digital processes. Additionally, the diversity in methods for publishing, including conference proceedings, digital archives, and online-only publications requires special scrutiny of results reported [20]. Considering the overall tendency to measure the progress of science upon authoritative publications, it is increasingly important to be able to verify that these numerous publications are of a high quality and based on sound methodologies.

The quality and validity of scientific literature is not only important for the overall ability to mark progress in specific disciplines, but also to establish trust in the results of work which can inform policy and impact members of the public on an individual level. While many researchers provide their own metrics designed to evaluate the validity and accuracy of a study’s results, one of the best ways to test the verity of results is by outside verification from another party. If it is considered an extra step, time constraints would certainly prevent experimental results being reproduced or replicated for every single study published; however, many experts are arguing for a paradigm shift to ensure that R&R could be achieved for each publication if desired, even if not acted upon. Currently, most publications do not contain sufficient materials, descriptions, or metadata to permit reproduction, thus leading some to declare a crisis [1]. Before steps can be carefully recommended to ameliorate this issue, researchers should come to a consensus regarding the main issues surrounding R&R and the meaning and significance of the associated terminology. The following section addresses definitions and terminology both within the scientific community at large, and more specifically within GIScience.

2.1. Convergence in Definitions

Part of the reason that progress regarding the improvement of reproducibility and replicability of research remains a challenge is rooted in general confusion over definitions. Though the words themselves have been in use since the mid-19th century [21], common use in the scientific domain did not occur regularly until relatively recently [22]. Even upon their adoption, the terms have not been attached to one singular meaning, thus increasing confusion. Scholars have noted that different professional and academic groups and organizations tends to attach their own meanings to the terms, or even conflate them completely [11]. Even official organizations such as the Open Science Collaboration use the terms interchangeably [1,23]. This confusion inhibits progress in solving issues related to reproducibility and replicability as collaborative efforts are pending when individuals do not yet agree upon the common issue.

We argue the definitions embraced by the National Academies of Sciences, Engineering and Medicine (NASEM) should be used. In their 2019 report, Reproducibility and Replicability in Science, they define reproducibility as: “Obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis” [24] (p. 1). Replicability, on the other hand, is defined as: “Obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data” [24] (p. 1). Essentially, reproducibility is the ability of someone to rework an experiment or workflow using the original data and the exact methods described in the publication. Replicability, however, is the ability of a group or research to take the methods from one study and use them to analyze a distinct set of data and obtain similar results to those reported in the first study. Interestingly, the NASEM-approved definitions include the provision that the methods used to analyze a distinct set of data can be the same or similar as those used in the original study. Though studies using different definitions for each of these terms are included in this reviewed, we make sole use of the NASEM definitions and offer clarification, if necessary, when citing other works that make use of the terms.

2.2. Scope of the Problem

Perhaps a paramount argument for considering R&R in research is the observation that “the ability to independently verify results is the fundamental, self-correcting mechanism of the scientific method” [25] (p. 135). This sentiment has been echoed by myriad researchers from a variety of disciplines who agree that in order for work to be credible, other researchers need to be able to confirm that the methodology and results of new ideas are reliable [3,26,27,28]. Without this paradigm, scientists might publish nearly any result without a mechanism to carefully identify problems that might affect the nature of the findings. Consequently, there is a need for groups to periodically attempt reproductions of seminal works to ensure that conclusions and generalizations are valid.

How far reaching is this need in the scientific community? A 2016 Nature survey found that among respondents working in the field of medicine, roughly 65% had been unable to reproduce another’s experiment, while over 50% reported having been unable to repeat their own work [1]. This not only undermines the validity of the results and the status of scientific inquiry among the community, but it can actually prove dangerous for experiments and projects that provide the basis for action that affects consumers and other stakeholders [12]. Results from such studies could heavily influence the manufacturing of drugs, vaccines, and other treatments, all while potentially based on flawed conclusions.

Though some may not consider the field of GIScience to have as seemingly dire consequences resulting from a lack of reproducibility as does medicine, its impact on trust in scientific findings as well as its promotion of further discoveries is still critically important. Despite this importance, GIScience appears to be facing a similar problem as other scientific disciplines regarding R&R. Just how big is this problem? Separately from the articles reviewed in this paper, we examined 200 remote sensing and photogrammetry related papers published from 2014 to 2022 in a variety of academic journals to obtain an idea of the status of R&R in the geospatial UAS field. Out of the 200 articles, only 37 articles included any sort of access to their source data and a mere 16 included access to any of the source code or other materials that would enable R&R (Figure 1).

Though these types of publishing practices are the norm for most scientific disciplines, there are many benefits that will result from greater adoption of R&R in methods of research and publication. Increased access to reproducible research provides additional benefits even beyond increased potential for scientific advancement. The gap between discovery and implementation results in a widespread phenomenon where individuals are plagued by issues that have solutions, but those same individuals lack access to the solutions. Open data publications focusing on R&R can help provide the necessary information to bridge that gap that is so often caused by lack of communication and dissemination of research and its relevance [29].

Increasing this access and other benefits to stakeholders of any form of geospatial technology is the primary objective of the technological convergence movement [14]. Within GIScience specifically, convergence is a topic of crucial importance because the patterns present in natural phenomena allow certain methodologies to solve a variety of problems. For instance, equations that can accurately model the spatial distribution of water features in caves can be used to understand the spatial patterns of snow present in aerial imagery [30]. Thus, the benefits of publishing open, reproducible research extend not only to those conducting the research, but additional convergent stakeholders including individuals or communities who can benefit from the application of the methods and use of new knowledge.

It is important to note, however, that not all individuals find R&R greatly important to science. Some of the participants in the Nature survey noted that they simply did not feel the need for R&R in their work [1]. Similarly, Guttinger [31] suggested that replicability was only important within specific sciences and should not play a widespread role across all scientific disciplines. Sui [27] reinforced this counterpoint by suggesting that it was important to consider the progress made without the practice of R&R. Certainly, such progress could be more meaningful if verified through the reproduction of experiments and workflows. Thus, even if a discipline-specific situation does not warrant the term “crisis”, there are still potential benefits from incorporating R&R where appropriate into future research design [32].

3. Literature on Reproducibility and Replicability in GIScience

Recently, trends in remote sensing and photogrammetric workflows are reflecting a shift from traditional satellite and aerial-based image acquisition to imagery gathered from UAS. UAS imagery is currently being used for a variety of applications including: land mapping [33], precision agriculture [34], forestry [35], security and reconnaissance [36], utility inspections [37], emissions monitoring and compliance [38], disaster recovery [39], coastal process monitoring [40], wildlife biology [41], crop phenology [42], 3D building reconstruction [43], population estimation [44], environmental hazard assessments [45], and more.

The variety in application areas of remotely sensed data necessitates the use of a variety of platform categories (airborne, satellite, or UAS); each of which contains dozens of different specific platforms and sensors. Combined with the myriad software options for postprocessing data collected, the potential for creating unique workflows to address specific applications or problems is enormous. Thus, there is a need for systematic reproducibility in order to build consensus that workflows are sound and that results from varied configurations can be validated.

As previously discussed, there are myriad uses for UAS in remote sensing and photogrammetric workflows. Changchun [46], Colomina [47], Singhal [48], Yao [15], and Zahari [49] provide excellent overviews of these uses and different remote sensing and photogrammetry applications for which UAS acquired imagery has been, and is currently being, used. The majority of these applications require somewhat complex computational workflows in order to align and process imagery, extract features or classify areas, calculate indices or other helpful metrics, and overlay these data with relevant source information to aid whichever application to which it is being applied. The relative complexity of these workflows stems from the basis of the UAS itself. Unlike traditional satellite or airborne platforms, UAS comprises a complex system of instruments, tools, software, expertise, and personnel all needed for the system to perform its intended function [50]. Each time a UAS is used for a scientific application, the researchers must successfully integrate “hardware, software, sensors, actuators, and communication components” all within the mechanical system of the aircraft itself [51] (p. 2).

This use and integration of so many distinct parts poses a unique challenge for R&R among UAS applications. Documenting each of a variety of systems can be a challenge, and small changes to any portion of the system can wreak havoc for those trying to reproduce or replicate results. The complexity of objects and systems that must be considered when attempting to conduct replicable and reproducible research using UAS is significant (Figure 2). Each facet shown plays a key role in the correct execution of a UAS project or workflow, and an error in the implementation of any part can limit R&R. Thus, there is a need to adequately document the complex nature of a UAS workflow to facilitate successful reproduction or replication. Additionally, a careful review of the literature is required to best understand the most useful methods for increasing R&R, and the barriers that prevent it from being adopted, not only in the scientific community as a whole, but also for GIScience and the specific complexities of UAS.

In examining the R&R of UAS-based computational workflows, we focus on a subset of the literature that contains elements of R&R within the published research methodology. Several of the authors of reviewed articles state that R&R are designated goals of their work, while others exemplify an open data concept within their experimental design. No particular application area of remote sensing or photogrammetry is highlighted; instead, all papers we identified containing UAS-based remote sensing or photogrammetry workflows were included if they either mention open data concepts or specifically outline steps to increase R&R in their research. This provides insights into current R&R trends in these fields, as well as showcases useful techniques that can serve as recommendations for future researchers.

3.1. UAS Remote Sensing and Photogrammetry Workflows

In the fields of remote sensing and photogrammetry, one common use for UAS imagery is the 3D reconstruction of buildings or topographical features. Clapuyt et al. [52] explored methods for assessing R&R of such workflows based on Structure-from-Motion (SfM) algorithms. They collected imagery over the same test site multiple times and compared the results of the workflow to identify potential sources of variation or error. They found that their workflow generated an acceptably low amount of error between the end products and thus deem their method to support R&R. Following this analysis, they gathered imagery using a different focal length on the camera and found that this imagery resulted in higher reconstruction variations due to its coarser resolution. They conclude that the degree to which R&R can be achieved is determined by input data quality and recommended using high quality data when attempting to reproduce or replicate a workflow.

Mlambo et al. [53] conducted a similar study to investigate the ability of SfM photogrammetric algorithms to accurately model tree canopies using UAS imagery as the input data. Unlike Clapuyt et al. [52], they did not attempt to replicate their own workflow. However, they compared a method using only OSS to one using proprietary software to identify the likelihood that open-source, low-cost methods could replace more expensive workflows in developing countries. Thus, this work was designed with reproducibility in mind, though the authors did not automate their workflow or publish it for others to access and use which would make reproducibility of their methods difficult at this point in time. Their results were supported by the work of Lisein et al. [54], who were also able to successfully use OSS to create a photogrammetric workflow for forest canopy height modeling. A study by Wallace et al. [55] also focused on utilizing OSS for forestry applications. They utilized OSS out of a desire to create a low-cost solution for point cloud creation from a UAS-LiDAR system which could be applied to forest inventory among other applications. In a comparable study, researchers used workflows composed of entirely OSS to monitor invasive species using a low cost UAS [56]. Though the exact applications for each workflow certainly differed, both studies demonstrated that the cost of monitoring products could be significantly lowered by making use of UAS and OSS.

In another study, Goncalves et al. [57] attempted to use only OSS solutions to photogrammetrically reconstruct topographical features including foredunes in Portugal. The researchers did not compare OSS with proprietary software, but simply attempted to create a useful workflow for this particular application of UAS data-based reconstruction. Though the authors did not specifically mention R&R, they did use open practices to construct their workflow and included detailed descriptions of it in their publication which certainly would facilitate R&R other groups seeking to reconstruct topography in coastal areas.

Jaud et al. [58] conducted similar work that compared high resolution Digital Surface Model (DSM) creation from two workflows, one using OSS and the other using a proprietary solution. Like the previous studies, their workflows were based on imagery gathered using a UAS, but they specifically flew in sub-optimal conditions to identify variations that might be caused by poor global positioning system (GPS) reception or tricky weather and survey conditions. The results supported Goncalves et al.’s [57] assertion that OSS workflows can generate high quality digital surface models (DSMs). The researchers addressed an important point related to open data and OSS models: they acknowledged the higher technical knowledge threshold needed to utilize OSS compared with the simpler, proprietary solution. However, they recommended OSS because it did not lose accuracy or precision and provided the ability for researchers to adjust more variables and steps in the workflow.

The increased parameter control that accompanies open-source workflows has been found to be beneficial. For example, studies focused on georeferencing point clouds [59], geolocating orthomosaic time series [60], and extending the remote access range of a camera on a UAS [61] all found that OSS better suited their needs and allowed them to further customize tools and aspects of each workflow. Ahmadabadian et al. [62] even found that OSS outperformed proprietary software for many photogrammetry applications, while Galland et al. [63] demonstrated its benefit to the modeling of surface deformations. Continued use of OSS benefits groups seeking R&R as they will not struggle with the increased result variability that can often arise from the hidden nature of algorithms used in workflows consisting entirely of off-the-shelf, proprietary software products [64]. Rocchini et al. [65] advocated for educational initiatives to teach researchers and students about the benefits of open data and OSS for applications of remote sensing.

While many researchers have increased R&R capacity associated with their works by utilizing OSS workflows, other groups have focused instead on R&R of their own workflows in order to validate their methods and demonstrate the benefits of UAS-based remote sensing workflows. Ludwig et al. [60] tested their time series geolocation efforts by comparing orthomosaics generated from the same image source and using the same processing parameters, but collected at different times. Because they found the orthomosaics to differentiate only within the error bounds they set, they report that their R&R efforts were successful and that their workflow can be used to generate accurate orthomosaics from UAS imagery.

Benassi et al. [66] performed block orientation with different software packages on several different sets of imagery gathered using the same UAS. Their results demonstrated that OSS could be used in the right circumstances to handle robust photogrammetric workflows. Perhaps their most important result, however, is that they were able to repeat each of their tested workflows and also identify variation thresholds that would be acceptable for the results from each type of workflow. This type of result analysis is important to include in published research to ensure both that the methods themselves are sound, but to also ensure that other researchers can understand what amounts of variation have been observed if they choose to replicate this research using their own data and want to evaluate the accuracy and precision of their work.

An analogous study was conducted by Teodoro and Araujo [67]. They wanted to explore the ability of OSS to run quality object-based image analysis (OBIA) on UAS data that could be used for landcover mapping. Like Benassi et al. [66], they replicated their own workflows with imagery gathered from the same location but from a different time period. They also found the OSS solution to be suitable for their needs and encouraged its use among other researchers in hopes that it would catalyze further innovation of algorithms specifically suited to UAS landcover classification.

Another study tried to replicate their own surface model workflow using UAS imagery collected over the same area, but at varying heights to simulate real-life alterations to source data that may occur when conducting fieldwork [68]. The authors found that these alterations did not cause end product variations to exceed an acceptable threshold for landcover mapping, but they found widely varied results in elevation change mapping. This is an example of the importance of gauging workflow R&R as the researchers’ own tests highlighted an area where additional work may need to be done to ensure their methodology is robust and will not be invalidated simply due to natural variations present in imagery collected over multiple UAS flights.

There is a body of research conducted by groups who have found R&R in their own research, but who have taken different steps to address it. The authors of the following studies all published results and findings in a way that promoted R&R in UAS remote sensing workflows by giving other researchers access to necessary data and metadata to reproduce workflows. Meng et al. [69] created a system for real-time ground object detection using UAS. While though they did not mention R&R as a specific goal of their research, they built a reusable graphical user interface (GUI) that could incorporate different UAS imagery as input, thus enabling others to replicate the workflow using their own data from different locations. Baca et al. [70] furthered the standard of open data access in the publication process. They published not only their workflows, but also simulated environments that could be used to test the rigor of novel approaches in applied UAS remote sensing. They argued that new systems do need verification through independent R&R but acknowledged that materials and information were often lacking to do this with UAS remote sensing workflows. The steps they recommend are exemplified in their work as a way to rectify this issue.

Knoth et al. [71] took a similar approach to ensuring that their results could be independently verified via reproduction. Their work demonstrates novel approaches to OBIA using OSS. Echoing Jaud et al. [58], they reiterate the technical skill level required to implement their workflow which is quite intricate and difficult due to the OSS used. However, they sought to rectify, rather than just acknowledge this issue, publishing a containerized version of their workflow in a publicly available GitHub repository. Similarly, Baca et al. [70] published the entirety of their OSS-designed system, and above-mentioned simulation environment, in a GitHub repository. Importantly, they note that the repository is not only publicly available, but also “well-documented” and “actively maintained” (p. 1). Publication trends like these eliminate the need for other researchers to understand the exact nuances of working with a variety of OSS packages and instead exemplify a standard for current and future work that enables R&R.

3.2. Key Barriers to Reproducibility and Replicability Affecting Geospatial UAS

If awareness regarding R&R is growing, what barriers have prevented it from being widely embraced? What barriers, or perhaps merely a lack of incentives, prevent scientists who may even see the need to conduct replicable and reproducible research, from conducting it themselves? While individuals’ motivations vary, we argue there are some specific problems that could potentially increase reproducible research if ameliorated. This section begins with a discussion of barriers that face the broader scientific community and which may require systemic change to overcome. It concludes with a narrow, focused discussion of additional barriers specifically present in geography and by implication fields such as GIScience and remote sensing that make use of geographic data.

Though the list of potential factors impacting R&R research is long, two that appear to be quite prominently cited are time and finance [1,10,25,28]. Essentially, due to the additional work needed to adequately document a workflow for reproduction and replication, many researchers are dissuaded from engaging in that work when it is not a required element of publication [8]. Given the existing pressure to publish research quickly, it is hardly surprising that spending the time required to make research open and reproducible is often viewed as too high a cost [8,72]. This can be further influenced if the work is being funded by grants. In general, researchers may not want to spend time on optional work that may delay set deadlines from being met.

Beyond a lack of financial incentives, there are some natural disadvantages within academia that accompany the publication of open and reproducible research. Konkol et al. [8] note that the very culture of academic research is founded upon the idea of publishing original findings and tying one’s reputation to the quantity and quality of unique research produced. Typically, reproduction of work is not looked upon favorably as a standalone publication. Conversely, original experiments are lauded and approved for publication in leading journals [73]. Similarly, Singleton [12] noted that many scientists feared open publication methods would prevent them from being able to capitalize on their work if others were able to reproduce their work and potentially take credit for its impact. Some researchers even feared that open publishing methods would enable others to steal their work; in turn ruining their reputations as scientists [8]. As long as publication is tied to funding, employment opportunities, and tenure, there may be small likelihood of a paradigm shift that will result in increased focus on open data access and the publication of fully reproducible and replicable research [3].

There are additional potential barriers related to current publishing standards. Many researchers are unable to produce reproducible research under the current context of peer-reviewed, journal articles as the main source of publication due to the confines of the medium. This is especially true of those conducting computationally based research who are limited by the scope of what they can communicate in a mainly text-based report [28]. Likewise, results of a 2019 study showed that many researchers felt they did not currently have access to the tools necessary to support presenting their work in an open and reproducible way [8]. Though proposals have been made for different publication methods that would better facilitate the communication of computational workflows by including access to appropriate metadata and provenance information for stakeholders and end users, none have yet widely permeated the traditional scientific publishing arena [74]. A final issue related to publication involves the significant subset of scientific research that is conducted on behalf of national security or institutions that handle confidential and sensitive data. The nature of this type of data may be an inhibiting factor in and of itself as it often cannot be published in a manner consistent with open data practices or with the detail needed to ensure R&R [12].

These broad issues are only compounded by specific factors that often prohibit the development of R&R in the field of geography. Geography is a unique discipline due to its involvement and treatment of place. Because different phenomena are studied in various locations around the world, it can be difficult to understand how, and if, that research may be replicated in another environment [75]. Kedron [25] notes that this problem is unique to the discipline of geography, and describes the difficulty of needing to foster research that can be adapted to places with many different characteristics. Geographers have long been aware of this discipline-specific issue. In 1968, Davies [26] noted that spatial variation was the entire reason for the study of geography as a whole, but that it also poses the largest underlying problem to the field. It is difficult to make generalizations and laws about phenomena when the phenomena themselves can vary so widely by spatial location [27,76]. To place this issue in proper context, consider a problem posed in a discussion of results of the 2016 cross-discipline study on R&R. Baker [1] noted that biologists and chemists cited variability of reagents as a factor that prohibited replication of experimental results. If natural variations in materials of the same chemical structure can preclude successful replication of a method, what havoc might arise from trying to replicate a workflow among locations with varied landscapes, environments, climates, and cultures?

The unique nature of the geography and related disciplines working with geographic data) poses yet an additional challenge to widespread R&R. Scientists with a wide variety of backgrounds may be found in the field. Researchers may study human interactions, code complicated software packages, or measure and observe natural processes; all with different educational backgrounds and possessing a variety of functional skillsets. Researchers can range from social scientists with a focus on the humanities to highly technical data analysts with a background in machine learning. Thus, it is easy to understand how two researchers working with geographic data may not possess the background knowledge or skillset to reproduce or replicate another’s work, despite both being practitioners of geographical methods and studies [77]. Similarly, it can be hard to conduct metanalyses or other studies that help determine the state of reproducible research in such a field due to the sheer complexity and diversity of subjects studied [23].

Furthermore, even if this geographic influence is accompanied by the required skillset needed to reproduce or replicate another’s work, it can hinder conceptualization of certain phenomena [25]. Like with many disciplines, the conceptual frameworks of many geographic phenomena are not set in stone and may be open to interpretation. Location often plays a critical role in how a phenomenon is understood as catalysts and environmental stimuli often vary by region. Consequently, failure to clearly communicate the conceptual framework underlying a workflow could make it nearly impossible for another scientist to accurately reproduce or replicate that work.

The fundamental differences in geographical locations provides a unique challenge to the idea of replicability [25]. Phenomena behave differently in different geographic locales, and thus assumptions about one location do not necessarily extend to another. There is a need to identify mechanisms that take natural geographic location variability into account and can thus be incorporated into future research in order for it to be replicated by individuals in different environments, or even reproduced by people with a different native research configuration. These individuals need the requisite information to make the modifications necessary to successfully reproduce work. In this context, the nuanced definition of replicability provided by NASEM is particularly relevant [24]. By this definition, replicability can be achieved even when different, though similar, methods are used to measure the same variable or attempt to evaluate the end result of a study. Varied methods are likely to be needed when groups try to replicate research in different areas of the world as altered results can sufficiently rectify the inherent differences caused by location and can be used to achieve comparable results in a replication attempt.

Konkol et al. [8] noted that the field of geography is behind in addressing and working to solve the R&R issue. This lag in progress is likely due to a combination of the various factors affecting both science as a whole and issues specific to geography (

Table 1. Summary of barriers to reproducibility and replicability (R&R) applicable to both many scientific disciplines as well as to geography and GIScience, with example scenarios.

Category	Barrier	Applied Example
	Terminology [11]	A researcher publishes work and claims it is “reproducible” but does not provide access to source code or original data.
	Time [1,25,28]	A research plan specifies five different workflow trials, but as the deadline for submission draws near, the researcher only finds time to run the workflow once before analyzing the results.
Applicable to many scientific disciplines	Finance [10,28]	A researcher wants a graduate student to create a script automating their workflow that can be published alongside their upcoming journal article. However, the project is quickly running out of grant funding. They would need to petition for additional funds to pay the hourly wages needed for the graduate student to complete the script, so instead they decide to publish the results without it.
	Publication pressure [8,72]	A researcher believes that to be considered for tenure next year, he should publish five manuscripts over the next year. This only leaves enough time to run through each project workflow once, and to minimize writing time by outlining only the basic steps of each experiment.
	Article format [28]	A researcher constructs a complex image processing workflow that requires four different open-source software (OSS) packages. He does his best to describe each step in the methods section of his paper, but another researcher finds the workflow impossible to follow based only on the description. She really needs the actual script to correctly replicate the work on her own image data.
	Issue of place [26,27,76]	A researcher publishes a useful image processing workflow and includes a link to the script used to conduct the original experiment. Another researcher downloads the script and runs it on his own imagery from a different area of the world. He finds that the hard-coded script variables do not accurately account for the landscape features in his imagery and his results vary significantly from the original research.
Geography and GIScience barriers	Conceptualization of phenomena [25,75]	A researcher publishes a workflow that analyzes imagery for potential environmental hazards. A researcher from another country replicates the workflow, using her own data to try to assess the risk in her own country. The agency funding the work is unhappy because the analysis failed to identify a specific type of environmental hazard common to their country, because it was not common to the country in which the original workflow was created and excluded from the workflow.
	Different educational backgrounds [77]	A researcher with a background in computer science publishes a remote sensing workflow with what he considers to be a very detailed write up of necessary steps to complete the workflow. Another geoscientist wants to replicate the workflow for her own remote sensing project. She tries to follow the write up in the article, but finds the instructions too high level for anyone without a computer science background to follow.

). Nevertheless, there are unique benefits that result from adopting reproducible and replicable research methods within the field. The very issue of varied locales and different environments can in turn benefit an unprecedented number of people. Advances in replicable and reproducible geographic research can extend to validate ethnographic and cultural research and learning, improve accuracy and confidence of physical landscape process modelling and hazard mitigation, all while simultaneously allowing for the expansion of spatially informed development and analysis across the world.

4. Case Studies

In order to extend understanding of how to conduct research that supports R&R in remote sensing and GIScience, we review two case studies. Both studies involve remote sensing workflows utilizing UAS data processed with OSS. We analyze the result of each study, highlighting successes that can be adopted by other researchers and acknowledging problems that still require viable solutions before they can be widely adopted.

4.1. Case Study: Open Source Application for UAS Photogrammetry

Along with the growing use of UAS in remote sensing to perform low-cost image acquisition, there has been a corresponding trend to adopt open-source image processing software in order to extend the cost feasibility of such platforms and products. In 2017, a team from the University of Porto conducted a study in which they created an open-source application that could perform photogrammetric operations on UAS imagery using the open-source platform MicMac [78]. The object of the study was to create a completely open-source application that could be used to generate point clouds and DSMs. The authors note that other research has already proven MicMac to be a robust software when compared to other proprietary solutions, thus the main focus of their study was not to compare the accuracy of the results of open-source and proprietary processing, but to assess the ability of MicMac to be utilized in a larger open-source GIS application that could successfully perform a series of photogrammetric tasks.

The authors built their application as a plug-in to be used with Quantum GIS (QGIS), a robust open-source GIS platform. They utilized the specific structure developed by QGIS to guide their application building process. The first step was to incorporate MicMac into the application. MicMac requires two separate executable files to run, but the authors created a batch script that would allow these programs to be installed automatically, thus reducing user-burden and minimizing chance for errors that could occur as a result of an improperly installed working environment. The entire application was developed as a widget so that users could interact with a GUI, rather than the traditional command line otherwise required to use MicMac.

Depending on the selection of georeferencing preferences, the user may be guided through a series of steps in a GUI to validate the position of each ground control point (GCP). Then, through the use of many of the available commands in MicMac, the authors enabled their application to process the imagery correctly and automatically output an orthophotograph, DSM, and shaded relief of each area shown in the input imagery.

The authors then tested their workflow on data collected from two locations in Portugal: Aguda Beach and Coimbra (Figure 3). In the first location, the researchers utilized a fixed-wing UAS, made by senseFly, called the singlet. They mounted a canon IXUS 220HS camera to the aircraft and utilized it to take 35 photos covering the entire study area. In the second location, they used an entirely open source UAS built by a local company called AIRBORNE PROJECTS. It was a multirotor UAS which provided contrasting experience to data collection with the fixed-wing craft. This UAS was equipped with a Sony Alpha 5000 camera which allowed for excellent image quality in a sensor light enough for use on a UAS. Both series of images were then run through the same workflow using the developed application. Then, each resultant orthoimage was analyzed for accuracy. Both were found to have high accuracy and a good coincidence of features within the imagery to the reference datasets.

Throughout this study, many steps were taken to ensure this work could meet a high standard of open, reproducible, and replicable research and overcome the main barriers that prevent R&R in research (Table 1). First, the study was conducted using entirely OSS. This removes cost and hardware parameters that would prevent some groups from being able to replicate this study if they chose to do so. Additionally, the authors chose to create a GUI application to guide users through this process. This is an essential, time-saving step for ensuring reproducibility of research and expanding access of the workflow by reducing the user burden and enabling individuals with less programming and software configuration experience to be able to successfully utilize this workflow. Due to the significant number of software installations that needed to take place for this workflow to be executable, it would have been unlikely that the results would have been able to be reproduced accurately, as any slight differentiation in installation choices could prevent the workflow from operating in the same way it did for the original researchers. Scripting the installation process largely decreases chance of issues with software installations and configurations.

Another key element of this research that promoted R&R was the inclusion of a workflow diagram within the publication itself. This was accompanied by a very detailed textual log of all the commands used from each software and was written in the order they were executed in the workflow. This alone may have enabled someone to follow the workflow even without the availability of the GUI to guide the entire process as it presented a more user-friendly description of the work than the solely textual descriptions often presented due to current trends in publication formats.

While other authors have reproduced their own work during a study to assess workflows and identify issues, this study included a replication of the workflow. The use of a different location, different UAS platform, and a different sensor established that the workflow is robust to changes in study location, an issue that can often prevent research from being replicated. This step can offer encouragement to other parties seeking to apply this workflow to their own study areas, using their own materials, by establishing that they would be likely to have success in using this workflow as well.

Finally, the authors published their entire application for free use. This may be one of the most fundamentally important steps required to achieve a standard of open data and encourage R&R. It is unknown if the original data gathered were published as well, which might hinder the reproducibility of the study, but the replicability of the study is certainly encouraged. However, at the time of writing, the author attempted to access the application using the provided web address and found that the link was no longer valid. This not-uncommon phenomenon represents a hindrance to the reproducibility of current research. Data storage problems prove problematic and there is no current single solution for storing applications and data in a location that cannot potentially be rendered inaccessible over the passage of time. However, it may have been useful to publish the application in a popular Git-based repository another platform less likely to suffer hosting and location issues than a personal website that may be altered or removed entirely from the internet.

Overall, this case study showcased an exemplary effort towards producing reproducible and replicable science. Improvements upon this effort could include providing access to the original study data to facilitate reproduction studies, finding a more permanent method for hosting the application, and perhaps utilizing some form of version-controlled application container as it is likely that the application would need updates to continue functioning well due to software version changes that have occurred since publication.

4.2. Case Study: Open Source Landcover Mapping Applications for UAS

UAS imagery is increasingly being used to create land classification maps. Recently, an interdisciplinary team sought to examine the ability of open-source machine learning workflows to create quality landcover maps from UAS imagery [80]. The main objective of the study was to identify a workflow that would yield the highest accuracy of a landcover map product. The authors remarked that this goal arose from years of automating landcover mapping from satellite imagery and sometimes achieving less than desirable results. Thus, they wanted to identify an optimal method that would use UAS imagery as the input and would focus on an open-source solution. It is unclear what the exact reasoning was for utilizing OSS, but the authors do mention that future work needs to be done to lower the barriers to use of OSS capable of producing high quality outputs such as the methods described in this article.

The authors chose to test four different machine learning workflows on their imagery. The four types of workflows included: ilastic, segmentation, fully connected neural networks (FCNs), and convolutional neural networks (CNNs). The Ilastik software is based on a random forest algorithm and facilitates the rapid creation of a classified image of high quality. It was reported to be fairly easy to implement and the software website contains many useful resources for researchers new to the software to be able to use. The segmentation workflow came from the Orfeo Toolbox library, another open-source repository of useful machine learning tools. The researchers supplemented this set of tools with their own script they wrote using R to calculate summary statistics for each segment and to label each image segment with its appropriate landcover class. For the neural network workflows, the authors used an open-source package called the Neural Network Image Classifier. It is an OSS package designed to classify landcover using various neural network algorithms. The authors used one workflow that applied a neural network algorithm to vectors (FCN) and another that applied the algorithm to image chips (CNN).

To assess the level of accuracy of each workflow, three different images were captured using a stock RGB camera mounted on a DJI Phantom 3 Pro quadcopter. The images were taken over the same location, but varied in altitude: 10 m, 45 m, and 90 m. Utilizing QGIS, polygons were drawn around certain landcover types and then pixels were extracted from each type to serve as training data for each of the different machine learning workflows. This was done three times—once for the image at each altitude. Then, the machine learning workflows were each run on the three different images, resulting in a total of twelve workflows being run in the study (Figure 4).

After running each workflow, the team assessed the accuracy using a set of pixels labeled in the initial step that was not used to train the algorithms. The authors found the ilastic workflow to be the most accurate. In their discussion, the authors noted that overall, automated classification of imagery gathered by low flying UASs is still difficult, especially in rural areas where the boundaries between classes are not so distinct as they may be in more urban settings. Additionally, they acknowledged that their results only reflected their specific project and they recommended that other studies still utilize all four workflows because their imagery may be processed more accurately than one of the three workflows that did not perform as well in this study. They concluded by referencing the problem of needing to tune hyperparameters and test network layouts, which can require the testing of thousands of models, in order to improve the selection and evaluation of training data that will be used in neural network classifiers. Thus, they recommended that collaborative workflows be developed and promoted the development of an open system that would allow different classification workflows to be systematically compared and evaluated.

Though this study did not specifically reference the ideas of reproducibility or replicability (a barrier to R&R that can be mitigated by spreading awareness of the terms and their importance to GIScience), its methods allow us to see how these ideas can best be practiced in a machine learning-based, UAS imagery workflow. Like Duarte et al. [78], this study used entirely OSS to complete their workflow and took several other steps that contributed to overcoming the R&R barriers mentioned in Table 1. As discussed previously, using OSS greatly improves access to the technology used in this article and eliminates cost as a barrier to other individuals seeking to reproduce or replicate this work. Beyond cost-related issues, using OSS is also important as it allows for full control of parameters that are input into each algorithm and does not contain tools with hidden processes that can induce variability into workflows without the ability to understand where the variation is coming from.

While no workflow diagrams were included in this article, the authors took an additional step towards making their data open and reproducible by publishing their scripts, and an accompanying user guide, in a GitHub repository. The scripts are not fully automated like those written by Duarte et al. [78], but the provision of a user guide would increase the likelihood of correct reuse of these scripts by individuals regardless of educational background. Additionally, storing these scripts in a version-controlled repository is incredibly beneficial as it would allow the authors to update the scripts to ensure they continue to work despite new R software versions being released. This increases the longevity of the applicability of this research.

The authors also published accompanying supplemental material with their print article. The supplemental material includes any scripts used in the workflow as well as original images, training data, and the variables used for each workflow. Including this information along with their print publication is crucial for ensuring future reproducibility of their work. (It should be noted that at the time of writing, the link to the webpage hosting the supplemental materials is still in working order.)

While the Duarte study published excellent workflows, it was unclear if their original data were published as well. This would enable replicability, but perhaps not reproducibility. This study, however, has achieved both by ensuring access to scripts as well as original data and useful metadata. However, some of the steps in these machine learning workflows were implemented in QGIS and not using scripts. This could potentially inhibit replicability if users are unsure of how to complete those same types of steps for their own data. Examples like these demonstrate the importance of fully scripted workflows to increase replicability by users with less software experience. However, since replicability can be achieved by use of varied workflows that assess the same idea, as long as an accurate description is included in the article to demonstrate what necessary steps were taken in each software component, it is certainly feasible that other users could substitute a software package with which they are more familiar to complete a similar workflow and achieve replicability of a study. It is still recommended to automate as much of software-based workflows as possible however, as this leaves less room for errors in workflow use and replication.

The final aspect of Horning et al.’s [80] work that is exemplary in their promotion of reproducible and replicable practices is that they published their work in an open access journal, Remote Sensing in Ecology and Conservation. Open access journals promote the concept of open data and remove the barrier of institutional affiliation or cost from the ability of scientists and researchers to access projects and workflows demonstrating advances at the forefront of scientific discovery. This study is a good representation of the type of publication which needs to be adopted by researchers on a wide scale to better support R&R. The method of publishing the scripts and workflows still requires further improvement. Because software versions change at different times, workflows that use multiple software packages, which will comprise nearly every open-source workflow, can break quite easily if the scripts are not regularly updated and maintained. Unless the original researchers plan to reuse their work in the future, there is little incentive for them to ensure their published scripts and workflows are in working order as this takes considerable time to maintain. A containerized solution may be a better solution for this type of publication as it enables other individuals to use an instance of each software contained in the workflow and is less prone to versioning-induced errors.

The research conducted by Duarte et al. [78] and Horning et al. [80] showcases various steps that can be taken to increase R&R in technical GIScience research. Both projects adopted some of the recommendations for conducting R&R research as noted in current R&R literature including: using OSS, automating workflows and technical analyses, including source data alongside findings, granting access to workflows, and publishing study results in an open and accessible format (

Table 2. Comparative view of reproducible and replicable methods used in case studies.

R&R Recommendations	Case Study: Duarte et al. [78]	Cast Study: Horning et al. [80]
Use of OSS throughout workflow	Yes	Yes
Automated workflow or script	Yes	No
GUI interface for automated workflow	Yes	No
Publication of source data for open access	No	Yes
Publication of workflow for open access	Yes	Yes
Working link to data or workflow repository	No	Yes
Sufficient accompanying workflow metadata	Yes	Yes
Reproduced or replicated workflow before publication	Yes	Yes
Publication of article in open access journal	No	Yes

). These steps showcased important efforts to overcoming many of the barriers to R&R such as time, cost and finances, article formatting limits, differing educational backgrounds of researchers, and the problems introduced by differing study areas, or the issue of place. While no one paper or project may perfectly implement all ideal R&R practices, or overcome every barrier to R&R, an analysis of existing methods in papers such as these can help researchers find simple ways to incorporate R&R into future work and identify gaps in their current research reporting practices.

5. Key Recommendations

Though R&R are beginning to receive further attention in the discipline of geography, only a few researchers conducting UAS-based remote sensing appear to be focused on applying these principles and incorporating these practices into their research. Though such research in an important ideal for many scientific disciplines, GIScience specifically is an area in which researchers need to make a concentrated effort towards achieving these ideals. Work in the GISciences typically includes computationally heavy workflows which are subject to the versioning of software or workflow parameters.

Previous sections of this literature review have discussed several examples of remote sensing and photogrammetric workflows created by researchers who made a particular effort to create replicable and reproducible research. However, even among these early adopters, there is no consensus regarding steps to be taken and standards needing to be met for work to be classified as reproducible or replicable. The next section reviews existing recommendations for reproducibility in the computational geosciences, as well as the author’s conclusions and recommendations for the broader scientific community based on the work discussed in this review.

5.1. Communicating the Importance Replicability and Reproducibility

Until recognition of the importance of these concepts is widespread, it is unlikely that any sort of lasting change regarding the nature of research will occur. Thus, a widespread effort to communicate the importance of R&R research needs to be undertaken. This can be done most efficiently by beginning to educate undergraduate and graduate students about the importance of R&R [81]. Smaller educational efforts can encourage widespread change if students are taught to conduct reproducible research and then apply reproducible methods throughout their careers, potentially also influencing lab mates and classmates to do the same [82]. These efforts can benefit from a consensus regarding terminology using NASEM’s definitions of R&R [24]. Further education should delve into the ideas of R&R beyond a surface level. Indeed, one study found that many participants in a survey responded that they published reproducible research, but then stated that they rarely published useable links to source code or data [8]. This disparity exemplifies an unfortunate trend—even among those who understand the need for R&R, that education regarding execution of these ideals is lacking. Courses examining the ideas of R&R, and then demonstrating these ideals in practice, could be incorporated into higher education curriculum to facilitate the spread of R&R in practice throughout academia.

Attempting to reconcile all issues related to R&R at once does not seem possible. Frery et al. [83] and Wilson et al. [74] have suggested mechanisms for labeling and rewarding scientists for the work they do that will contribute to their data being replicable or reproducible. Different levels are reached as more and more steps are taken in line with the recommendations reviewed above. Such reward systems could be implemented in education efforts and then rolled out to the broader research community. This type of system would be an excellent mechanism for establishing discipline-wide standards and informing researchers about the steps they can continue to take to improve their research and make it more supportive of R&R.

5.2. Increasing Access to Provenance and Metadata

Beyond an understanding of the need for R&R, there are functional barriers that often prevent UAS and other research from achieving a high standard of R&R. One such barrier is access, or lack thereof, to sufficient metadata and provenance information. Specifically, there is a need for researchers to explicitly specify instrument types in workflows to enable other teams to follow exact methods when attempting reproductions [52]. Additionally, workflows need to make use of quality instruments with supporting metadata to facilitate reproducible workflow creation [84]. Clapuyt et al. [52] argue that inferior instruments might have more variation in measurements which could lead towards reproduction efforts having variation that falls outside acceptable limits leading to failure of reproduction. Similarly, Bollen et al. [85] focused on the need for researchers to include precision estimates and standard errors when publishing their research. This would enable future researchers to understand the variability they might find when reproducing an experiment and to know which variations are significant, and which are not. Overall, it appears that researchers would need to increase the amount of metadata they record for their projects to ensure that throughout the process, other researchers can locate the necessary information to successfully reproduce data and understand the type of variation in results that might arise from workflow replication.

Similar to the recommendation to increase computational metadata, is the recommendation to improve access to the data itself. The goal of furthering access to necessary components of a study for reproducibility can best be achieved by adoption of an open data model [75]. When replicating an experiment, the researcher will be collecting his or her own data, but for reproduction, access to the original data is necessary. Bollen et al. [85] suggested that all data from each stage of a project need to be publicly accessible and stored online. Konkol et al. [8] supported this idea and argued for a standard of ORR (open reproducible research). This standard would include making every component used in a research project publicly available online, so that all interested parties could view and access the data, thereby allowing workflows to be easily reproduced [3,86].

Reverting to the aforementioned suggestion, Gil et al. [28] argued that not only should data and workflows be available, but they need to contain relevant metadata that allows a researcher to understand the full process and be able to fairly easily reproduce the experiment. This would include metadata necessary to support workflow format conversion if an individual is interested in replicating a study and adapting the workflow to fit their own system and preferences [87]. Anselin et al. [88] discussed a method utilizing OSS to track metadata and provenance to ensure that detailed records are kept of not only each dataset, but each action performed on a dataset throughout a workflow—a critical need for reproduction and replication [11]. PROV-DM is one such open-source standard that can be used to capture the necessary information to render a workflow replicable and reproducible (Figure 3 and Figure 4). It is a conceptual model that relates entities to their method and time of creation as well as derived entities further along in a process or workflow [79]. Utilizing this model, or a similar one, is an effective way to communicate both the major points and nuances of detailed, computational workflows to others seeking to reproduce or replicate data.

As Tullis and Kar [11] argue, provenance as a form of contextual metadata is a key to R&R, though its free exchange may be limited by privacy assurance, intellectual property, export control, and other stakeholder interests. The fact that R&R has competing interests does not mean that multi-stakeholder solutions cannot be found. Instead, they argue that provenance services can be developed to support the competing interests. In this sense, R&R can be conceptualized as just one of many applications of provenance. They reference Code Ocean which has demonstrated R&R services that also addresses privacy using access control of repositories where a relatively complete provenance record is curated.

5.3. Adapting Publishing Practices

It would be remiss to advocate for open data publication and storage without acknowledging several issues that would arise from the achievement of the ideal of freely accessible data stored online. Primarily, there are questions regarding overall publication practices and the added issues of related storage and data maintenance. For instance, as beneficial as it would be to have a large, publicly accessible repository of data and results from previous studies, how would that vast amount of data be stored? How would it be properly organized and documented? Would there be a governing body in charge of organization, database maintenance, and storage? If so, who would compose the governing body? These are all questions that still need to be answered before any substantial movement towards fully open research could occur. However, there are some smaller steps that researchers can take in the meantime to still progress towards an R&R ideal for all research until answers to the larger questions can be found.

One recommended step is to increase the amount of research published in open access journals. This prevents financial means from acting as a barrier for researchers who wish to access workflows for reproduction or replication. Additionally, many open access journals either encourage [89], or require [90] authors to make source code and data available in public repositories before the accompanying article can be published. This is part of their initiative to increase R&R in the sciences and it helps researchers by offering outside motivations to create reproducible research while also helping encourage other journals to implement requirements of source data publication alongside article publication. Colom et al. [91] reviewed a journal that publishes not only literary descriptions of work, but also source code, a series of test examples, and online environments where other researchers may test code. Efforts like these exemplify the changes to publication methods that can promote R&R among research in all disciplines.

Another recommendation regarding article publication is the expansion of publication formats, or even a complete restructuring of how published scientific literature is formatted. The currently accepted format of a text-heavy journal article often does not provide enough useful description of a computational workflow or the results and accompanying statistics and data. Thus, many have argued that it would be better to publish digital artifacts rather than a written, research article [77]. These could include data, workflows, and possibly even the entire computing environment which would be much more useful for individuals trying to reproduce work than attempts to derive the full complexity of a workflow and its variables from a text-based description [13].

Nüst et al. [92] suggested the use of executable research compendium (ERC) which would package all of the necessary components of a full report and serve as a new medium for publication. In a less comprehensive, but perhaps more immediately useful approach, integrated text and code platforms such as Jupyter Notebooks can solve this problem of balancing text and code, and offer a new format for publications [10,23,28,77]. Additionally, publication of containerized software environments could prevent version changes and different software environments from precluding successful reproduction of research [71,93]. This type of publication would also speed the time needed to process through data and workflows accompanying a written article in attempts to implement methods described in said article.

Some researchers balk at the idea of R&R publication practices due to well-founded fears of losing claim to their work or having their methodologies appropriated without requisite recognition. This is a valid concern, however, promoting open and accessible research does not mean that researchers must forfeit the opportunity to receive appropriate attribution for their work. Stodden [94] proposes a framework that creates a standard for reproducible research by removing the restrictions of copyright but enabling the attribution aspects of open software type licenses. It is the belief of the author that a similar type of framework needs to be widely enacted before proprietary licensing of work will cease to preclude the publication of reproducible research on a wide scale.

Finally, it has been recognized that some form of change needs to occur to facilitate a shift in data publication standards. Because there is arguably little to no incentive for scientists to take the extra time and effort to ensure that their research is replicable and reproducible, outside of intrinsic motivations, it is recommended that institutions place less emphasis on the number of publications as a measure of a researcher’s stature, accept more formats for the presentation of findings, and encourage open forms of workflow and protection and licensing that does not limit access to data and methodologies. Until such emphases change, it will be difficult to widely shift towards open, replicable, and reproducible research.

5.4. Addressing the Issue of Geographic Variability

As mentioned in Section 3, the geographic component of GIScience present an additional complexity to the issue of R&R. Because geographic phenomena vary by location, it is possible that methods shared by one researcher will not work for a person tackling a similar problem in a different region. Thus, it is crucial for researchers to give access to both source data, but also a detailed account of their workflow and code used to conduct analyses. This allows other researchers to replicate work with their own source data and see where slight modification might need to be made to the workflow to adjust to their specific locale. For example, the authors of a recent paper using deep learning to identify weeds from UAS imagery made sure to publish their workflow in a GitHub repository while also including their source images used to train the algorithm as well as their weights file for the algorithm [95]. This allows scientists desiring to reproduce the work for validation purposes to simply use the weights file and run the algorithm as the authors did. The inclusion of the original images, however, also allows for scientists desiring to replicate the work using their own images to understand how the model was trained so they can adapt it accordingly to their own images which may look different depending on the geographic area of interest being studied.

Careful presentation of both metadata and provenance information, as mentioned previously, will not only aid in overall R&R efforts, but will prove paramount to enabling these concepts to be applied in the GISciences. Additionally, some changes may need to occur in the conceptualization of R&R to account for geographic variability. While a consensus regarding the R&R terminology is certainly beneficial to the concepts being able to be more readily adopted, differences in the nature of some scientific disciplines may prevent one pan-disciplinary term definition from serving the true needs of each field of study [96]. Thus, in the field of GIScience, it is possible that in order to have achieved a replication of work, the definition of “consistent results” may need to be broadened to account for the natural variability of locations and phenomena [24] (p. 1). Similar adjustments may need to be considered as future research seeks to achieve a more ideal R&R standard.

6. Conclusions

The ideas of R&R have gained increased value in the eyes of scientists across a variety of disciplines in the past several years. This is especially true of scientists who focus on computational results and data analysis. Within the field of geography, this makes the creation of R&R research especially important to individuals who utilize remote sensing and photogrammetric workflows in their research. Currently, there is no discipline-wide standard for R&R research. In order to bridge this gap, we have conducted a review of past implementations of R&R in geoscience, reviewed current trends in UAS-based remote sensing and photogrammetry workflows, and proposed recommendations for future research. The information has yielded insights into methods that can be used by current researchers, including increasing the quantity of metadata and other descriptors of workflow processes, publishing source data and code alongside journal articles, and publishing results in open access journals. The review also highlights areas where further study is needed: management of online data repositories, facilitation of movement towards a rewards system not based on publication of proprietary methods, and increased education regarding R&R. It is hoped that addressing these areas in further study will provide solutions that will lead to an increase in reproducible and replicable publications in the GIScience, thus validating groundbreaking methods and expanding access to these scientific methods to all convergent stakeholders who may benefit from their application and use.