Next Article in Journal
Semi-Supervised Convolutional Long Short-Term Memory Neural Networks for Time Series Land Cover Classification
Previous Article in Journal
Combining Remote Sensing and Meteorological Data for Improved Rice Plant Potassium Content Estimation
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Comparing Interpretation of High-Resolution Aerial Imagery by Humans and Artificial Intelligence to Detect an Invasive Tree Species

1
Department of Molecular Biosciences and Bioengineering, University of Hawai’i at Manoa, Honolulu, HI 96822, USA
2
Department of Geography and Environmental Science, University of Hawai’i at Hilo, Hilo, HI 96720, USA
3
Center for Aquatic and Invasive Plants Aquatic and Invasive Plants, Department of Agronomy, University of Florida, Gainesville, FL 32653, USA
4
Security in Silicon Laboratory, Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32653, USA
5
Department of Computer Science, University of Hawai’i at Hilo, Hilo, HI 96720, USA
6
Spatial Data Analysis and Visualization Lab, University of Hawai’i at Hilo, Hilo, HI 96720, USA
*
Author to whom correspondence should be addressed.
Remote Sens. 2021, 13(17), 3503; https://doi.org/10.3390/rs13173503
Submission received: 20 July 2021 / Revised: 19 August 2021 / Accepted: 31 August 2021 / Published: 3 September 2021
(This article belongs to the Section AI Remote Sensing)

Abstract

:
Timely, accurate maps of invasive plant species are critical for making appropriate management decisions to eliminate emerging target populations or contain infestations. High-resolution aerial imagery is routinely used to map, monitor, and detect invasive plant populations. While conventional image interpretation involving human analysts is straightforward, it can require high demands for time and resources to produce useful intelligence. We compared the performance of human analysts with a custom Retinanet-based deep convolutional neural network (DNN) for detecting individual miconia (Miconia calvescens DC) plants, using high-resolution unmanned aerial system (UAS) imagery collected over lowland tropical forests in Hawai’i. Human analysts (n = 38) examined imagery at three linear scrolling speeds (100, 200 and 300 px/s), achieving miconia detection recalls of 74 ± 3%, 60 ± 3%, and 50 ± 3%, respectively. The DNN achieved 83 ± 3% recall and completed the image analysis in 1% of the time of the fastest scrolling speed tested. Human analysts could discriminate large miconia leaf clusters better than isolated individual leaves, while the DNN detection efficacy was independent of leaf cluster size. Optically, the contrast in the red and green color channels and all three (i.e., red, green, and blue) signal to clutter ratios (SCR) were significant factors for human detection, while only the red channel contrast, and the red and green SCRs were significant factors for the DNN. A linear cost analysis estimated the operational use of a DNN to be more cost effective than human photo interpretation when the cumulative search area exceeds a minimum area. For invasive species like miconia, which can stochastically spread propagules across thousands of ha, the DNN provides a more efficient option for detecting incipient, immature miconia across large expanses of forested canopy. Increasing operational capacity for large-scale surveillance with a DNN-based image analysis workflow can provide more rapid comprehension of invasive plant abundance and distribution in forested watersheds and may become strategically vital to containing these invasions.

Graphical Abstract

1. Introduction

Invasive species are one of the main threats to native ecosystems worldwide, altering plant community structure and function, i.e., reducing biodiversity and compromising ecosystem services [1,2,3,4]. Invasive species detection and control programs typically consume a significant portion of natural resource management budgets, and provide fertile ground for technological innovations to reduce costs by increasing efficiency in protecting large landscapes [5,6]. A compelling example of this management challenge can be found in the state of Hawai’i, where conservation land managers are confronted by a multitude of invasive species threats to critical habitats and native ecosystems. Research and development of emerging technologies has become an institutional component of invasive species management strategies in Hawai’i, as a measure to gain advantages on large, often expensive, problems [7]. Strategically, early detection and rapid response (EDRR) offers the last opportunity to consider an aggressive small-scale eradication program [8]. Beyond that, naturalized invasive species populations often become established beyond feasible eradication and are consequently relegated to containment strategies attempting to confine populations to their occupied areas [9]. Regardless of the management strategy, a majority of resources to combat invasive species are dedicated to reconnaissance and surveillance [10,11]. There have been many technological advancements to this effort, starting with the advent of civilian GPS, geographical information systems (GIS), and remote sensing, leading to better spatial and temporal tracking of dynamic species invasions [12,13,14,15].
Miconia (Miconia calvescens DC) is a high-priority invasive plant target for the Hawai’i Invasive Species Council with over US$1M invested annually in search and control efforts [16,17,18,19,20]. It is a mid-story canopy tree native to South and Central America and originally introduced as an ornamental specimen to the Big Island of Hawai’i in 1961. It is presently invading more than 100,000 ha of forested watersheds across the Hawaiian archipelago. Technological innovations in herbicide application, data collection, and search strategy have enhanced control and containment efforts, but miconia continues to spread [21,22,23,24,25,26].
Miconia is an autogomous species that is prone to passive long-distance dispersal by frugivores [27] and capable of establishing isolated founder populations stochastically spread out over large areas [28,29]. Moreover, seed from several miconia species are reported with extended physiological dormancy that results in latent germination and persistent recruitment from a deposited seedbank over several decades [30,31], a trait also specifically observed in miconia [32,33]. Miconia can germinate in low light conditions and can remain visually hidden under a multi-tiered forest canopy for several years [24,31]. These life-history traits make miconia a highly problematic and aggressive invasive species, despite arduous search efforts and long-term intervention schedules [32].
Surveillance programs from manned aerial platforms (e.g., helicopter) have been described as random search efforts, which predictably translate to imperfect detection [23,34,35,36]. The randomness affecting miconia detection can be inferred from several factors. Importantly, an on-board observer’s experience, acuity and stamina are observed factors in miconia detection [23]. Visual discrimination of individual species in a diverse, heavily vegetated, wet forest is difficult, and may also be explained by color contrast and signal to clutter ratio [37,38]. Miconia was imported to Hawaii and other locations as a striking ornamental; its leaves are large, elliptic to obovate (e.g., up to 80 cm in length) with three acrodomous veins, and dark green dorsal and reddish-purple ventral surfaces [39,40]. These prominent features also assist with its detectability. Random (imperfect) search efforts often follow an exponential function where the probability of detection is dependent on the cumulative amount of search effort applied uniformly to an area [36,41]. Thus, the only practical option for increasing the probability of detection is to compensate with repeated search efforts, usually at a great expense dictated by the terms of helicopter service contracts well over US$1000 h−1.
The availability of high-resolution aerial imagery derived from unmanned aerial systems (UAS) has dramatically increased over the past decade due to reduced costs, reduced regulatory barriers, and technological advancements in flight endurance, GPS-precision flight planning, image sensors, post-processing algorithms and cloud-based computing [42,43,44,45,46,47,48]. Mapping applications with UAS have become economical and routine, resulting in a growing demand for high spatial and temporal resolution data from a wide range of industries and services [49,50,51,52]. Adoption of this technology for invasive species surveillance is still being cultivated through proof-of-concept demonstrations and protocol development [53,54,55,56]. Many industrial applications may pertain to small-scale site inspections, while natural area conservation, including invasive species management, is more likely to be obfuscated by the enormity and remoteness of the location. While resolution and comprehension are desirable features of remotely sensed data, operational and post-process workflow efficiencies inherently dictate usability and adoption by practitioners.
Increased use of UAS can create a backlog of large aerial image data sets. Artificial intelligence and deep neural network (DNN) algorithms provide a means of automating image analysis for object detection, following investments in image collection, annotation, and model training [57,58]. Early adopters of neural networks with UAS imagery have focused on agricultural systems [59,60,61,62], but a growing number of studies are now exploring ways to detect and map invasive plant species in more complex forest environments [63,64,65].
Detection and mapping these invasive plants fall within the domain of object detection. Convolutional neural networks have numerous variations but will consist of convolutional and pooling layers grouped into modules with a final layer outputting the class label [66,67]. Convolutional neural network based object detection models may be separated into two categories: a two-stage approach and one-stage approach [68,69]. In the two-stage approach, such as Fast R-CNN [70], Faster R-CNN [71], Mask R-CNN [72], and Feature pyramid network [73], object detection is separated into an initial region proposal phase, during which regions where the object may exist are identified, and the detection phase, where candidate regions are classified into different classes. One-stage detectors, such as YOLO [74], SSD [75], and RetinaNet [76], use anchors, which are sets of pre-defined bounding boxes of varying scales and rations, for initial region proposals and the detector classifies these pre-defined regions. One stage detectors are typically faster but will have reduced accuracy compared to two stage detectors [69].
Here, we present a convolutional deep neural network (DNN) based on a one-stage RetinaNet model [77] specific to detection of miconia in wet, heavily vegetated tropical forests and compare its performance to experienced human image analysts. RetinaNet was selected was selected due to its improved performance over other networks for tree crown detection [78,79].
While advances in automated remote sensing classification techniques are rapidly evolving, human interpretation of high-resolution imagery continues to play an important role in forestry and conservation [80,81,82,83], including Human-in-the-loop applications [84]. Trained human analysts can readily detect cryptic understory species such as miconia in high-resolution imagery, but distraction and fatigue become factors of concern when processing large numbers of images [85,86]. The motivation of this study was to advance the adoption of UAS technologies in invasive plant species surveillance with a need to understand the efficacies and efficiencies of human- and DNN based image analyses. Here we report on a study that compares the performance of human analysts against a custom DNN for image scanning and detection of miconia in high-resolution imagery derived from UAS. We measured human performance under a controlled experimental setting using three linear image scrolling speeds and compared those detection recalls against a customized miconia detection DNN algorithm. We further examined the importance of nine different optical features relating to miconia canopy geometry, size, and visual characteristics on detection recall. We also compared simple linear cost models based on the workflows in field mapping and image analyses performed manually by human analysts versus a semi-automatic computational approach using the DNN algorithm.

2. Materials and Methods

2.1. Image Collection

We collected aerial images over miconia-infested areas on the island of Hawai’i with a small multirotor UAS (Inspire 2, SZ DJI Technology Co., Ltd., Shenzhen, China) equipped with an RGB camera with a 4/3 sensor (Zenmuse X5S with 15 mm MFT lens, SZ DJI Technology Co., Ltd.). Flight surveys were conducted at an altitude of 50 m above ground level with a groundspeed of 5 m s−1 on parallel flight paths with the camera oriented in the nadir position and automatic settings for focus and white balance (Figure 1, Table 1). These surveys captured images (5280 × 3956 pixels) with a ground sampling distance of approximately 1.1 cm px−1. No geometric correction, radiance correction, or reprojection was performed on the aerial images. Three different locations known to have miconia (sites A–C; Figure 1) were surveyed on the windward side of Hawai’i Island. The study areas ranged from 1.2–3.2 ha with the resident vegetation canopy comprised primarily of other invasive tree species including Albizia (Falcataria moluccana), Common and Strawberry Guava (Psidium guajava and cattleianum), Strangling Banyan (Ficus sp.), Trumpet Tree (Cecropia obtusifolia), Octopus Tree (Schefflera actinophylla), Bingabing (Macaranga mappa), and Princess Flower (Tibouchina heteromalla). Invasive liana species were also present, including Stink Vine (Paederia foetida) and Passion Flower (Passiflora sp.). The understory vegetation was dominated by two native fern species, Uluhe (Dicranopteris linearis) and Hāpu’u (Cibotium glaucum).
We collected a total of 649 images were captured from these flights. We selected six individual images for interpretation, representing a range of miconia abundances from sparse to densely infested. In each image, we careflully outlined all contiguous miconia leaf canopy and manually digitized them into vector polygon features using QGIS (v. 3.4.15) by two separate analysts spending at least 30 min per image for quality assurance (Figure 2). The six images from all three sites contained a total of 150 feature polygons. As a final step, we created a 53-pixel buffer, calculated based on 25% of the average characteristic dimension, around each feature polygon to accommodate human errors with hand-eye coordination when marking the feature polygons and thereby reducing false-positive scoring, especially for the smaller features. The characteristic dimension is equal to the square root of the enclosed area. A single feature polygon may not necessarily designate an individual tree. Instead, several polygons could have been derived from a single plant, i.e., background tree canopy were to overlap with large sprawling miconia canopy. Alternatively, some polygons might correspond to multiple individual plants in close proximity.

2.2. Human Analyst Trials

We wanted to formally compare the performance of trained human analysts, operating at three different scrolling speeds, against a custom DNN for the detection of miconia. To do this, we needed to generate baseline data for human detection performance. Institutional Review Board approval for the human experiments was obtained from the University of Hawai’i Human Studies Program (IRB protocol no. 2017-00863). We recruited forty test participants from a pool of local volunteers with professional experience in identifying miconia but unfamiliar with the specific regions of interest in this study. Participants first answered a questionnaire to provide relevant information on their experience with identifying miconia and other background information that might impact their ability to visually detect miconia. We screened participants for visual acuity, using a LogMAR chart [87], and color-blindness, using the Ishihara test [88]. If a participant demonstrated visual acuity below normal as defined by the visual acuity measurement standard [89] or a color blindness deficiency we removed them from the pool. In total, 38 participants were ultimately included to participate in the human analyst trials.
We developed a custom Python script to display image sections within a 500 × 500 px field of view, continuously scrolling through each test image at one of three fixed linear speeds (100, 200, or 300 px/s). The fixed scrolling speed ensured a uniform viewing time for the image to reduce fixation in search effort that may come from a static view of each section. The viewing monitors were 24-inch diagonals with 1920 px × 1200 px resolution (P2419H, Dell Inc., Round Rock, TX, USA) and were calibrated to ensure consistent color displays (Datacolor Spyder5, Lawrenceville, NJ, USA). Participants were seated approximately 50 cm from the screen with a 30° viewing angle for a field of view encompassing the entire screen. The optical mouse was located on the right or left side based on subject preference. Participants were instructed to mark each suspected miconia plant distinguished as a contiguous leaf canopy cluster. The marking procedure was performed by placing a mouse cursor over the suspected feature and clicking to secure a reference point on the image section. They were further instructed that a mark anywhere within the contiguous leaf canopy cluster would be recorded as a successful detection, while multiple clicks within the same contiguous area did not affect total detection counts. Participants were presented with a total of three images randomly selected from the pool of six, as described above. Each image corresponded to one of the three speeds assigned randomly. Points created by each participant were saved separately as a comma separated variable (CSV) file accompanying the image section. The experiment was administered for each participant within a 10-min period, to eliminate fatigue as a factor. Points were classified as true positives when contained inside the buffered polygons. Points outside of the buffered polygons were classified as false positives. Buffered polygons with no points occurring inside the polygon were classified as false negatives.

2.3. Deep Convolutional Neural Network Searches

Convolutional neural networks have been successfully used to identify invasive plants in UAS imagery [67,90]. However, to our knowledge no one has developed a DNN for the detection of miconia in nadir aerial imagery or done a rigorous comparison of DNN performance with trained human analyst trials. The miconia detection algorithms developed for this study were based on RetinaNet [77] with a ResNet-101 [91] backbone pre-trained on ImageNet designed for fast and accurate detection of densely packed objects within images [92]. The model was pre-trained to be capable of general image understanding by using a transfer learning technique with the large ImageNet database (i.e., 14 million images) consisting of ground-level photos of common objects. The final DNNs were obtained by freezing the first 80 layers of the ResNet backbone before performing specialized transfer learning with a custom miconia dataset.
We used cross fold validation to train 10 models on 3636 miconia annotations spread across 86 training images taken from all three sites. These training images were cropped 1000 × 1000 subsections of the original images. Two skilled human analysts sequentially annotated and verified each image section with bounding boxes. A 10-fold cross validation was performed with the 86 training images by splitting into ten roughly even folds, with six folds having nine images and four having eight. Each model was trained using a unique nine-fold combination, with the remaining folds used as a validation set. Hyperparameters used during training are provided in Table 2. This was then followed by standard ImageNet preprocessing as described by [77]. To select the final model for each fold, we chose the model with the lowest validation loss across all epochs. The six images used in the human analyst trials were withheld from our DNN training and validation sets and processed by each of the ten developed DNNs. This methodology is depicted in Figure 3. The output of each DNN consisted of a set of generated bounding boxes for detected targets. Vector points were created from the centroids of each bounding box and assessed for recall against the miconia feature polygons described above. Algorithm development, calibration, and validation procedures were performed with a computer workstation with a Titan RTX graphical processing unit (NVIDIA, Corp., Santa Barbara, CA, USA) and an i9-9900K CPU (Intel, Santa Clara, CA, USA).

2.4. Recall and Effect of Optical and Search Properties

The recall of image interpretation in this study was measured as the aggregate probability of that each feature polygons will be correctly identified as a true positive by human participants or DNN models [93]. This decision was based on the combination of data types involved in the study (point data from the human analysts and the DNN and polygons for the miconia reference data) and our treatment of detection counts, where multiple marks within the same contiguous area did not affect the totals to allow a fair comparison between the results of the DNN and human interpretation. Due to the definition of a true positive and false negative being defined based on detections within a polygon, false positives, i.e., detections outside of the defined polygons, lacked an equivalent definition to aggregate these points, preventing the calculation of precision. Therefore, the false positive rate, number of false positive detections divided by the total number of detections, was used in lieu of precision [94].
We examined the effects of nine factors relating to optical and search properties on the detection recall for the human analysts and the DNN algorithms. Geometric factors (n = 2) included the relative size of the miconia leaf clusters and distance to nearest miconia neighbors. Optical factors (n = 6) included miconia contrast to surrounding background and signal to clutter ratio (SCR) for each of the three color channels. Image scrolling speeds during human analyst trials (n = 1) were also examined.
The relative size of the plant was quantified as the number of pixels contained in the delimiting miconia polygon (NM). The distance to the nearest miconia, D, was determined using a nearest neighbor analysis that measured the shortest distance between two polygons within an image using the NNJoin plugin (QGIS 3.4.15).
Optical characteristics were analyzed in square regions centered on each miconia feature polygon. The sides of these squares were twice as long as the characteristic dimension of the bounded polygon, which in turn was calculated as the square root of the total number of contained pixels within the polygon (Figure 2). Within each square region, pixels belonging to miconia were denoted with a subscript M and pixels belonging to the background (i.e., not belonging to the miconia pixels under consideration) are denoted with a subscript B.
Contrast values between miconia and local background (CM) were calculated for each color channel as the sum of the squares of the difference between the digital value of each miconia pixel for that channel, yS for s M , and the corresponding mean digital value of the local background pixels, μB [95]:
C M = s P y s μ B 2 ¯ N M
where μB is the mean of the digital values for the background pixels ( s B ):
μ B = 1 N B s B y s
The SCR for each color channel in the image was calculated as the ratio of the difference between the mean digital values of miconia and background pixels and the variance of the local background pixels for the corresponding color [96]:
S C R = μ M μ B σ B 2
where µM is
μ M = 1 N M s M y s
and σB2 is
σ B 2 = 1 N B s B y s μ B 2
Contrast and SCR calculated for each channel were designated by a subscript of the corresponding channel as red (R), green (G), or blue (B), respectively.
Statistical analysis on the effects of nine factors (CP and SCR for each color band, size NM and nearest neighbor distance D, and image scrolling speed S) on detection was conducted with a multivariate analysis of variance (MANOVA) for each factor using R software [97]. Eta-squared (η2) was used to measure the effect size for each MANOVA factor. All dependent variables were determined to be normally distributed based on Shapiro-Wilk’s method (p > 0.05). Mean separation was performed with Tukey’s Honest Significant Difference test for interpretation by humans at the various scrolling speeds and the DNN.

Cost Model

Cost models were constructed for workflows using human analysts or a DNN to perform invasive species detection from aerial imagery with the following base equation:
C T = C V F S + C V I A + C F X
where CT is the total cost, (CV)FS is the variable cost to conduct a UAS flight survey, (CV)S is the variable cost to analyze the resulting imagery, and CFX is the fixed cost. The fixed cost for conducting UAS flight operations largely pertains to the upfront investment and continued maintenance of aircraft and sensors estimated here to be $6000 USD which applies to both human and DNN interpretation. The DNN interpretation has an added fixed cost for the human labor associated with generating image annotations (approximately 240 h at $25 h−1) investment and maintenance of computer workstations capable of performing automated image analyses for a total of $15,000 USD. The variable cost to conduct a UAS flight was calculated as
C V F S = A S I W G S D 1 O S v C L
where AS is the survey area, IW was the image width, GSD is the ground sampling distance, OS is the proportional side overlap, v is the flight speed, and CL is the cost of labor set at $25 USD per hour. The variable cost to conduct image analyses with humans (HA) was calculated as
C V H A = A S F O V H S G S D 2 C L
where FOVH is the height of the field of view equal to 500 pixels in these trials and S is the linear scrolling speed. The variable cost to conduct a DNN analysis was calculated as
C V S = A S I H I W t I G S D 2 C L
where IH is the image height and tI is the time to process an image which was calculated based on the average time to process six images. While the DNN process is automated, here, we assume human involvement with routine tasks and oversight being performed until completion and include the cost of labor as well (CL). Additional parameters are described in Table 3.

3. Results

3.1. Recall

Human recalls were 75 ± 3%, 60 ± 3%, and 50 ± 3% at scrolling speeds of 100 px s−1, 200 px s−1 and 300 px s−1, respectively (Figure 4). The DNN had an overall recall of 83 ± 3% and analyzed each test image in less than 5 s, over 32 times faster than the fastest scrolling speed. False positive rates were 7.4 ± 1.8%, 12.0 ± 3.0%, and 18.6 ± 2.4% for the three human analyst scrolling speeds of 100 px s−1, 200 px s−1 and 300 px s−1, and 9.9 ± 0.2% for the DNN.

3.2. Effect of Optical and Search Properties on Recall

Seven of the nine factors significantly affected miconia detection by human analysts (p < 0.05; Table 4). The two factors with the greatest effects (based on magnitudes of η2 as indicated by asterisks) [98] were scrolling speed and relative size. These factors exhibited a non-linear association curve and an inverse relationship, respectively (Figure 5). Red and green contrasts (CM) and all three SCRs also had significant effects on human detection recall. Only three factors were significant for efficacy of DNN detection: CM,R, SCRR and SCRG (p < 0.05, magnitudes of η2; Table 5), which were also significant for the human detections.

3.3. Cost Comparisons between Human and DNN Image Analyses

A simple exponential model was fit (R2 = 0.995) to the human analyst efficacy results
recall = 100 e 0.00242 S
to estimate the scrolling speed needed to match the DNN recall of 83.3%. This scrolling speed was determined to be 78 px s−1 and is equivalent to a search effort of 0.23 s m−2 with FOV of 500 px and GSD of 1.1 cm2 px−1. The existing DNN was determined to be more cost effective than a human search conducted at a scrolling speed of 78 px/s once the cumulative area searched exceeds 617.3 ha (Figure 6). If recall is sacrificed and search speeds of 100, 200, and 300 px/s are used, the cumulative area searched must exceed 793.6, 1606.6, and 2439.9 ha, respectively, for the DNN to be more cost effective.

4. Discussion

We tested the ability of human analysts and a custom DNN to detect the invasive miconia plant in visible-wavelength UAS imagery collected over complex canopy forest and found that the DNN outperformed the human analysts. While similar results have been reported in other image classification studies [91,99], we are not aware of any prior studies involving rigorous time-controlled human trials for detecting invasive species in high-resolution UAS imagery. The significance of optical contrast and SCR as factors for human recall agrees with previous studies [100], and recent work has further associated poor accuracies with low SCR for both humans and DNNs [101,102]. The lowland tropical forest canopy environment imaged in this study is a complex community of diverse species and functional groups, creating fully vegetated and highly cluttered backgrounds for detecting even a highly conspicuous plant such as miconia. The DNN’s relatively high recall may be diminished in imagery collected in regions with different species composition from the training dataset. Therefore, additional training data for the miconia detection DNN, particularly including image training sets with low red contrast and red and green SCRs, would likely improve the robustness of the DNN. Additionally, alternative DNN architectures, such as EfficientDet [103,104], YOLOv5 [105], and Mask R-CNN [72,106,107], should be considered to improve accuracy and inference speed.
Developing a DNN with perfect accuracy in miconia detection is improbable and may not be necessary, based on the biology and life history traits of the species. Strategically, management interventions must outpace invasion by eliminating miconia before it reaches maturity within 3–4 years of germination [32]. Seed bank germination is asynchronous and can survive for multiple decades [32,33], and miconia can remain cryptic in the understory for some time before becoming visible from above. Thus, even with the capability of 100% target detection, there is an inherent commitment to repeated surveillance of an area to ensure extinction. However, UAS surveillance platforms integrated with an automated detection workflow could greatly enhance our ability to detect miconia and other species, as well as our understanding of biological invasions, with a constant stream of data providing posterior updates improving predictability in forecasting management outcomes [108].
Comprehensive field intelligence on invasive species abundance and distribution is anything but routine. In reality, management programs struggle with budgetary decisions on how to proportionally allocate resources between detection and intervention. For miconia programs in Hawai’i, efforts are often combined by combining surveillance and intervention operations by treating targets as they are found. However, this remains operationally insufficient to meeting the demands of gathering intelligence across large, remote landscapes with repeated measures, even with the most cost-effective options [32]. The parallel advancements of UAS mapping capability and artificial intelligence are surpassing human capacity in surveillance and may transcend invasive species management towards a better comprehension of the invasion problem with rapid deliverables and more precise and effective interventions. We recognize that different species may require different amounts of training data to produce similar results. Recognizing those limitations, we believe this study establishes that DNN interpretation of aerial imagery provides a more effective path for invasive detection at the landscape level than manual image interpretation, freeing up valuable human resources for management interventions and other activities.

5. Conclusions

UAS imagery can provide valuable intelligence for natural resource managers, but the current bottleneck of time and human resources required to exhaustively search through these images reduces the scalability of this approach. Automated classification of miconia with a deep neural network exhibited a higher degree of recall than any of our tested human search speeds and did not exhibit the biases toward large plants seen in human searches. This makes deep neural networks particularly appropriate for detection of incipient miconia populations which tend to consist of small, sporadically located plants. In FY20, the Hawai’i Invasive Species Council funded projects to search 792,368 acres across the Hawai’i an islands. Implementation of a deep neural network for invasive plant detection can result in cost savings due to the substantially faster processing time compared to human searches of UAS imagery. Further improvements to the deep neural network, through advancements in deep neural network architectures or other approaches and incorporating additional training data, to improve accuracy of detection and applications to other invasive species will further advance the utility of UAS in natural resource management.

Author Contributions

Conceptualization, R.R.III; methodology, R.R.III, M.P. and T.M.; software, R.R.III, M.P. and P.P.; validation, R.R.III and R.L.P.; formal analysis, R.R.III; investigation, R.R.III, D.J., J.L. and R.L.P.; resources, R.R.III, D.J., J.L. and R.L.P.; data curation, R.R.III and R.L.P.; writing—original draft preparation, R.R.III, R.L.P., J.L. and D.J; writing—review and editing, M.P., P.P. and T.M.; visualization, R.R.III; supervision, R.L.P., D.J., J.L. and T.M.; project administration, R.L.P. and J.L.; funding acquisition, J.L. and R.L.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded in part by the Hawai’i Invasive Species Council awards FY 18 and 19.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of the University of Hawai’i at Manoa at (protocol no. 2017-00863 and approved on 10 January 2018).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Individual data are not publicly available to protect the privacy and confidentiality of human participants.

Acknowledgments

The authors would like to thank Joannie Dobbs for assistance in developing protocols for human studies and Charlie Tommy and Jason Henson for annotating miconia in collected imagery. The authors would like to thank the Panaewa Recreational Complex, University of Hawai’i at Hilo College of Agriculture, Forestry and Natural Resource Management, and Celia Bardwell-Jones for providing sites to collect the miconia imagery used in this study.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Pearson, D.E.; Ortega, Y.K.; Eren, Ö.; Hierro, J.L. Community Assembly Theory as a Framework for Biological Invasions. Trends Ecol. Evol. 2018, 33, 313–325. [Google Scholar] [CrossRef] [PubMed]
  2. Pejchar, L.; Mooney, H.A. Invasive Species, Ecosystem Services and Human Well-Being. Trends Ecol. Evol. 2009, 24, 497–504. [Google Scholar] [CrossRef] [PubMed]
  3. Vitousek, P.M.; D’Antonio, C.M.; Loope, L.L.; Remanek, M.; Westbrooks, R.G. Introduced Species: A Significant Component of Human-Caused Global Change. N. Z. J. Ecol. 1997, 21, 1–16. [Google Scholar]
  4. Weidlich, E.W.A.; Flórido, F.G.; Sorrini, T.B.; Brancalion, P.H.S. Controlling Invasive Plant Species in Ecological Restoration: A Global Review. J. Appl. Ecol. 2020, 57, 1806–1817. [Google Scholar] [CrossRef]
  5. Kirk, N.; Kannemeyer, R.; Greenaway, A.; MacDonald, E.; Stronge, D. Understanding Attitudes on New Technologies to Manage Invasive Species. Pac. Conserv. Biol. 2020, 26, 35–44. [Google Scholar] [CrossRef] [Green Version]
  6. Martinez, B.; Reaser, J.K.; Dehgan, A.; Zamft, B.; Baisch, D.; McCormick, C.; Giordano, A.J.; Aicher, R.; Selbe, S. Technology Innovation: Advancing Capacities for the Early Detection of and Rapid Response to Invasive Species. Biol. Invasions 2020, 22, 75–100. [Google Scholar] [CrossRef] [Green Version]
  7. Pejchar, L.; Lepczyk, C.A.; Fantle-Lepczyk, J.E.; Hess, S.C.; Johnson, M.T.; Leopold, C.R.; Marchetti, M.; McClure, K.M.; Shiels, A.B. Hawaii as a Microcosm: Advancing the Science and Practice of Managing Introduced and Invasive Species. BioScience 2020, 70, 184–193. [Google Scholar] [CrossRef]
  8. Reaser, J.K.; Burgiel, S.W.; Kirkey, J.; Brantley, K.A.; Veatch, S.D.; Burgos-Rodríguez, J. The Early Detection of and Rapid Response (EDRR) to Invasive Species: A Conceptual Framework and Federal Capacities Assessment. Biol. Invasions 2020, 22, 1–19. [Google Scholar] [CrossRef] [Green Version]
  9. Hobbs, R.J.; Humphries, S.E. An Integrated Approach to the Ecology and Management of Plant Invasions. Conserv. Biol. 1995, 9, 761–770. [Google Scholar] [CrossRef] [Green Version]
  10. Cacho, O.J.; Spring, D.; Pheloung, P.; Hester, S. Evaluating the Feasibility of Eradicating an Invasion. Biol. Invasions 2006, 8, 903–917. [Google Scholar] [CrossRef]
  11. Cacho, O.J.; Spring, D.; Hester, S.; Mac Nally, R. Allocating Surveillance Effort in the Management of Invasive Species: A Spatially-Explicit Model. Environ. Model. Softw. 2010, 25, 444–454. [Google Scholar] [CrossRef]
  12. Asner, G.P.; Jones, M.O.; Martin, R.E.; Knapp, D.E.; Hughes, R.F. Remote Sensing of Native and Invasive Species in Hawaiian Forests. Remote Sens. Environ. 2008, 112, 1912–1926. [Google Scholar] [CrossRef]
  13. Holcombe, T.; Stohlgren, T.J.; Jarnevich, C. Invasive Species Management and Research Using GIS. In Managing Vertebrate Invasive Species; USDA/APHIS Wildlife Services, National Wildlife Research Center: Fort Collins, CO, USA, 2007. [Google Scholar]
  14. Joshi, C.M.; de Leeuw, J.; van Duren, I.C. Remote Sensing and GIS Applications for Mapping and Spatial Modelling of Invasive Species. In Proceedings of the ISPRS 2004: Geo-Imagery Bridging Continents Congress, Istanbul, Turkey, 12–23 July 2004; pp. 669–677. [Google Scholar]
  15. Pettorelli, N.; Laurance, W.F.; O’Brien, T.G.; Wegmann, M.; Nagendra, H.; Turner, W. Satellite Remote Sensing for Applied Ecologists: Opportunities and Challenges. J. Appl. Ecol. 2014, 51, 839–848. [Google Scholar] [CrossRef]
  16. Duffy, D.C.; Martin, C. Cooperative Natural Resource and Invasive Species Management in Hawai’i. In Proceedings of the International Conference on Island Invasives 2017, Dundee, UK, 10–14 July 2017; IUCN: Gland, Switzerland, 2019; p. 497. [Google Scholar]
  17. Loope, L.; Kraus, F. Preventing establishment and spread of invasive species: Current status and needs. In Conservation of Hawaiian Forest Birds: Implications for Island Birds; Yale University Press: New Haven, CT, USA, 2009. [Google Scholar]
  18. Kaiser, B.A.; Burnett, K.M.; Roumasset, J.A. Control of Invasive Species: Lessons from Miconia in Hawaii; Agricultural and Applied Economics Association: Milwaukee, WI, USA, 2006. [Google Scholar]
  19. Chimera, C.G.; Medeiros, A.C.; Loope, L.L.; Hobdy, R.H. Status of Management and Control Efforts for the Invasive Alien Tree Miconia Calvescens DC. (Melastomataceae) in Hana, East Maui; Pacific Cooperative Studies Unit, Department of Botany, University of Hawaii at Manoa: Honolulu, HI, USA, 2000. [Google Scholar]
  20. Medeiros, A.C.; Loope, L.L.; Conant, P.; McElvaney, S. Status, Ecology and Management of the Invasive Plant Miconia Calvescens DC. (Melastomataceae) in the Hawaiian Islands. Bish. Mus. Occas. Pap. 1997, 48, 23–26. [Google Scholar]
  21. Jorgensen, N.; Leary, J.; Renz, M.; Mahnken, B. Characterizing the Suitable Habitat of Miconia Calvescens in the East Maui Watershed. Manag. Biol. Invasions 2020, 12, 313. [Google Scholar] [CrossRef]
  22. Leary, J.; Gooding, J.; Chapman, J.; Radford, A.; Mahnken, B.V.; Cox, L.J. Calibration of an Herbicide Ballistic Technology (HBT) Helicopter Platform Targeting Miconia Calvescens in Hawaii. Invasive Plant Sci. Manag. 2013, 6, 292–303. [Google Scholar] [CrossRef]
  23. Leary, J.; Mahnken, B.V.; Cox, L.J.; Radford, A.; Yanagida, J.; Penniman, T.; Duffy, D.C.; Gooding, J. Reducing Nascent Miconia (Miconia calvescens) Patches with an Accelerated Intervention Strategy Utilizing Herbicide Ballistic Technology. Invasive Plant Sci. Manag. 2014, 7, 164–175. [Google Scholar] [CrossRef]
  24. Perroy, R.L.; Sullivan, T.; Stephenson, N. Assessing the Impacts of Canopy Openness and Flight Parameters on Detecting a Sub-Canopy Tropical Invasive Plant Using a Small Unmanned Aerial System. ISPRS J. Photogramm. Remote Sens. 2017, 125, 174–183. [Google Scholar] [CrossRef]
  25. Rodriguez, R.; Jenkins, D.M.; Leary, J.J.K. Design and Validation of a GPS Logger System for Recording Aerially Deployed Herbicide Ballistic Technology Operations. IEEE Sens. J. 2015, 15, 2078–2086. [Google Scholar] [CrossRef]
  26. Rodriguez, R.; Leary, J.J.K.; Jenkins, D.M.; Mahnken, B.V. Herbicide Ballistic Technology: Spatial Tracking Analysis of Operations Characterizing Performance of Target Treatment. Trans. ASABE 2016, 59, 803–809. [Google Scholar] [CrossRef]
  27. Spotswood, E.N.; Meyer, J.-Y.; Bartolome, J.W. Preference for an Invasive Fruit Trumps Fruit Abundance in Selection by an Introduced Bird in the Society Islands, French Polynesia. Biol. Invasions 2013, 15, 2147–2156. [Google Scholar] [CrossRef]
  28. Moody, M.E.; Mack, R.N. Controlling the Spread of Plant Invasions: The Importance of Nascent Foci. J. Appl. Ecol. 1988, 25, 1009–1021. [Google Scholar] [CrossRef]
  29. Shigesada, N.; Kawasaki, K.; Takeda, Y. Modeling Stratified Diffusion in Biological Invasions. Am. Nat. 1995, 146, 229–251. [Google Scholar] [CrossRef]
  30. Martínez-Ghersa, M.A.; Ghersa, C.M. The Relationship of Propagule Pressure to Invasion Potential in Plants. Euphytica 2006, 148, 87–96. [Google Scholar] [CrossRef]
  31. Pearson, T.R.H.; Burslem, D.F.R.P.; Goeriz, R.E.; Dalling, J.W. Interactions of Gap Size and Herbivory on Establishment, Growth and Survival of Three Species of Neotropical Pioneer Trees. J. Ecol. 2003, 91, 785–796. [Google Scholar] [CrossRef]
  32. Leary, J.; Mahnken, B.; Wada, C.; Burnett, K. Interpreting Life-History Traits of Miconia (Miconia Calvescens) through Management over Space and Time in the East Maui Watershed, Hawaii (USA). Invasive Plant Sci. Manag. 2018, 11, 191–200. [Google Scholar] [CrossRef]
  33. Meyer, J.Y.; Malet, J.P. Study and Management of the Alien Invasive Tree Miconia Calvescens DC. (Melastomataceae) in the Islands of Raiatea and Tahaa (Society Islands, French Polynesia) 1992–1996; Cooperative National Park Resources Studies Unit, University of Hawaii at Manoa, Department of Botany: Honolulu, HI, USA, 1997. [Google Scholar]
  34. Cacho, O.J.; Hester, S.; Spring, D. Applying Search Theory to Determine the Feasibility of Eradicating an Invasive Population in Natural Environments. Aust. J. Agric. Resour. Econ. 2007, 51, 425–443. [Google Scholar] [CrossRef] [Green Version]
  35. Frost, J.R.; Stone, L.D. Review of Search Theory: Advances and Applications to Search and Rescue Decision Support; National Technical Information Service: Springfield, VA, USA, 2001. [Google Scholar]
  36. Koopman, B.O. The Theory of Search. II. Target Detection. Oper. Res. 1956, 4, 503–531. [Google Scholar] [CrossRef]
  37. Verghese, P. Visual Search and Attention: A Signal Detection Theory Approach. Neuron 2001, 31, 523–535. [Google Scholar] [CrossRef] [Green Version]
  38. Verghese, P.; McKee, S.P. Visual Search in Clutter. Vis. Res. 2004, 44, 1217–1225. [Google Scholar] [CrossRef] [Green Version]
  39. Michelangeli, F.A.; Almeda, F.; Goldenberg, R.; Judd, W.S.; Bécquer, E.R.; Tulig, T.M. A Complete Web-Based Monograph of the Tribe Miconieae (Melastomataceae); New York Botanical Garden: Bronx, NY, USA, 2009. [Google Scholar]
  40. Weber, E. Invasive Plant Species of the World: A Reference Guide to Environmental Weeds; CAB International: Wallingford, UK, 2003; ISBN 978-0-85199-695-0. [Google Scholar]
  41. Koopman, B.O. Search and Screening: General Principles with Historical Applications; Pergamon Press: New York, NY, USA, 1980. [Google Scholar]
  42. Bassi, E. From Here to 2023: Civil Drones Operations and the Setting of New Legal Rules for the European Single Sky. J. Intell. Robot. Syst. 2020, 100, 493–503. [Google Scholar] [CrossRef]
  43. Pagallo, U.; Bassi, E. The Governance of Unmanned Aircraft Systems (UAS): Aviation Law, Human Rights, and the Free Movement of Data in the EU. Minds Mach. 2020, 30, 439–455. [Google Scholar] [CrossRef]
  44. Srivastava, S.; Gupta, S.; Dikshit, O.; Nair, S. A Review of UAV Regulations and Policies in India. In Proceedings of the UASG 2019, Roorkee, India, 6–7 April 2019; Jain, K., Khoshelham, K., Zhu, X., Tiwari, A., Eds.; Springer: Cham, Switzerland, 2020; pp. 315–325. [Google Scholar]
  45. Cummings, A.R.; McKee, A.; Kulkarni, K.; Markandey, N. The Rise of UAVs. Photogramm. Eng. Remote Sens. 2017, 83, 317–325. [Google Scholar] [CrossRef]
  46. Scott, B.I. The Law of Unmanned Aircraft Systems: An Introduction to the Current and Future Regulation under National, Regional and International Law; Kluwer Law International B.V.: Philadelphia, PA, USA, 2016; ISBN 978-90-411-6132-1. [Google Scholar]
  47. Rango, A.; Laliberte, A.S. Impact of Flight Regulations on Effective Use of Unmanned Aircraft Systems for Natural Resources Applications. J. Appl. Remote Sens. 2010, 4, 043539. [Google Scholar] [CrossRef]
  48. Stöcker, C.; Bennett, R.; Nex, F.; Gerke, M.; Zevenbergen, J. Review of the Current State of UAV Regulations. Remote Sens. 2017, 9, 459. [Google Scholar] [CrossRef] [Green Version]
  49. De Michele, C.; Avanzi, F.; Passoni, D.; Barzaghi, R.; Pinto, L.; Dosso, P.; Ghezzi, A.; Gianatti, R.; Della Vedova, G. Using a Fixed-Wing UAS to Map Snow Depth Distribution: An Evaluation at Peak Accumulation. Cryosphere 2016, 10, 511–522. [Google Scholar] [CrossRef] [Green Version]
  50. Jeziorska, J. UAS for Wetland Mapping and Hydrological Modeling. Remote Sens. 2019, 11, 1997. [Google Scholar] [CrossRef] [Green Version]
  51. Kuželka, K.; Surový, P. Mapping Forest Structure Using UAS inside Flight Capabilities. Sensors 2018, 18, 2245. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  52. Papakonstantinou, A.; Topouzelis, K.; Doukari, M. UAS Close Range Remote Sensing for Mapping Coastal Environments. In Proceedings of the Fifth International Conference on Remote Sensing and Geoinformation of the Environment (RSCy2017), Paphos, Cyprus, 20–23 March 2017; International Society for Optics and Photonics: Bellingham, WA, USA, 2017; Volume 10444, p. 1044418. [Google Scholar]
  53. Baron, J.; Hill, D.J.; Elmiligi, H. Combining Image Processing and Machine Learning to Identify Invasive Plants in High-Resolution Images. Int. J. Remote Sens. 2018, 39, 5099–5118. [Google Scholar] [CrossRef]
  54. Jiménez López, J.; Mulero-Pázmány, M. Drones for Conservation in Protected Areas: Present and Future. Drones 2019, 3, 10. [Google Scholar] [CrossRef] [Green Version]
  55. Lehmann, J.R.K.; Prinz, T.; Ziller, S.R.; Thiele, J.; Heringer, G.; Meira-Neto, J.A.A.; Buttschardt, T.K. Open-Source Processing and Analysis of Aerial Imagery Acquired with a Low-Cost Unmanned Aerial System to Support Invasive Plant Management. Front. Environ. Sci. 2017, 5, 44. [Google Scholar] [CrossRef] [Green Version]
  56. Müllerová, J. UAS for Nature ConservationMonitoring Invasive Species. In Applications of Small Unmanned Aircraft Systems: Best Practices and Case Studies; CRC Press: Boca Raton, FL, USA, 2019; ISBN 978-0-429-52085-3. [Google Scholar]
  57. Zhao, Z.-Q.; Zheng, P.; Xu, S.-T.; Wu, X. Object Detection with Deep Learning: A Review. IEEE Trans. Neural Netw. Learn. Syst. 2019, 30, 3212–3232. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  58. Ajmal, H.; Rehman, S.; Farooq, U.; Ain, Q.U.; Riaz, F.; Hassan, A. Convolutional Neural Network Based Image Segmentation: A Review. In Proceedings of the Pattern Recognition and Tracking XXIX, Orlando, FL, USA, 18–19 April 2018; International Society for Optics and Photonics: Bellingham, WA, USA, 2018; Volume 10649, p. 106490N. [Google Scholar]
  59. Kaya, A.; Keceli, A.S.; Catal, C.; Yalic, H.Y.; Temucin, H.; Tekinerdogan, B. Analysis of Transfer Learning for Deep Neural Network Based Plant Classification Models. Comput. Electron. Agric. 2019, 158, 20–29. [Google Scholar] [CrossRef]
  60. Chandra, A.L.; Desai, S.V.; Guo, W.; Balasubramanian, V.N. Computer Vision with Deep Learning for Plant Phenotyping in Agriculture: A Survey. arXiv 2020, arXiv:2006.11391. [Google Scholar] [CrossRef]
  61. Chiu, M.T.; Xu, X.; Wei, Y.; Huang, Z.; Schwing, A.; Brunner, R.; Khachatrian, H.; Karapetyan, H.; Dozier, I.; Rose, G.; et al. Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis. arXiv 2020, arXiv:2001.01306. [Google Scholar]
  62. Espejo-Garcia, B.; Mylonas, N.; Athanasakos, L.; Fountas, S. Improving Weeds Identification with a Repository of Agricultural Pre-Trained Deep Neural Networks. Comput. Electron. Agric. 2020, 175, 105593. [Google Scholar] [CrossRef]
  63. Pearse, G.D.; Tan, A.Y.S.; Watt, M.S.; Franz, M.O.; Dash, J.P. Detecting and Mapping Tree Seedlings in UAV Imagery Using Convolutional Neural Networks and Field-Verified Data. ISPRS J. Photogramm. Remote Sens. 2020, 168, 156–169. [Google Scholar] [CrossRef]
  64. Fricker, G.A.; Ventura, J.D.; Wolf, J.A.; North, M.P.; Davis, F.W.; Franklin, J. A Convolutional Neural Network Classifier Identifies Tree Species in Mixed-Conifer Forest from Hyperspectral Imagery. Remote Sens. 2019, 11, 2326. [Google Scholar] [CrossRef] [Green Version]
  65. Kattenborn, T.; Eichel, J.; Fassnacht, F.E. Convolutional Neural Networks Enable Efficient, Accurate and Fine-Grained Segmentation of Plant Species and Communities from High-Resolution UAV Imagery. Sci. Rep. 2019, 9, 17656. [Google Scholar] [CrossRef] [PubMed]
  66. Rawat, W.; Wang, Z. Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review. Neural Comput. 2017, 29, 2352–2449. [Google Scholar] [CrossRef] [PubMed]
  67. Kattenborn, T.; Leitloff, J.; Schiefer, F.; Hinz, S. Review on Convolutional Neural Networks (CNN) in Vegetation Remote Sensing. ISPRS J. Photogramm. Remote Sens. 2021, 173, 24–49. [Google Scholar] [CrossRef]
  68. Sultana, F.; Sufian, A.; Dutta, P. A Review of Object Detection Models Based on Convolutional Neural Network. In Intelligent Computing: Image Processing Based Applications; Mandal, J.K., Banerjee, S., Eds.; Advances in Intelligent Systems and Computing; Springer: Singapore, 2020; pp. 1–16. ISBN 9789811542886. [Google Scholar]
  69. Liu, Y.; Sun, P.; Wergeles, N.; Shang, Y. A Survey and Performance Evaluation of Deep Learning Methods for Small Object Detection. Expert Syst. Appl. 2021, 172, 114602. [Google Scholar] [CrossRef]
  70. Girshick, R. Fast R-CNN. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 11–18 December 2015; pp. 1440–1448. [Google Scholar]
  71. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1137–1149. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  72. He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 42, 386–397. [Google Scholar] [CrossRef]
  73. Lin, T.-Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; Belongie, S. Feature Pyramid Networks for Object Detection. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 936–944. [Google Scholar]
  74. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 779–788. [Google Scholar]
  75. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. SSD: Single Shot MultiBox Detector. In Proceedings of the Computer Vision—ECCV 2016, Amsterdam, The Netherlands, 11–14 October 2016; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer: Cham, Switzerland, 2016; pp. 21–37. [Google Scholar]
  76. Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollar, P. Focal Loss for Dense Object Detection. In Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2980–2988. [Google Scholar]
  77. Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollar, P. Focal Loss for Dense Object Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 318–327. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  78. Culman, M.; Delalieux, S.; Tricht, K.V. Palm Tree Inventory From Aerial Images Using Retinanet. In Proceedings of the 2020 Mediterranean and Middle-East Geoscience and Remote Sensing Symposium (M2GARSS), Tunis, Tunisia, 9–11 March 2020; pp. 314–317. [Google Scholar]
  79. dos Santos, A.A.; Marcato Junior, J.; Araújo, M.S.; Di Martini, D.R.; Tetila, E.C.; Siqueira, H.L.; Aoki, C.; Eltner, A.; Matsubara, E.T.; Pistori, H.; et al. Assessment of CNN-Based Methods for Individual Tree Detection on Images Captured by RGB Cameras Attached to UAVs. Sensors 2019, 19, 3595. [Google Scholar] [CrossRef] [Green Version]
  80. Lister, A.; Lister, T.; Weber, T. Semi-Automated Sample-Based Forest Degradation Monitoring with Photointerpretation of High-Resolution Imagery. Forests 2019, 10, 896. [Google Scholar] [CrossRef] [Green Version]
  81. Schepaschenko, D.; See, L.; Lesiv, M.; Bastin, J.-F.; Mollicone, D.; Tsendbazar, N.-E.; Bastin, L.; McCallum, I.; Laso Bayas, J.C.; Baklanov, A.; et al. Recent Advances in Forest Observation with Visual Interpretation of Very High-Resolution Imagery. Surv. Geophys. 2019, 40, 839–862. [Google Scholar] [CrossRef] [Green Version]
  82. Tompalski, P.; White, J.C.; Coops, N.C.; Wulder, M.A.; Leboeuf, A.; Sinclair, I.; Butson, C.R.; Lemonde, M.-O. Quantifying the Precision of Forest Stand Height and Canopy Cover Estimates Derived from Air Photo Interpretation. For. Int. J. For. Res. 2021. [Google Scholar] [CrossRef]
  83. White, A.R. Human Expertise in the Interpretation of Remote Sensing Data: A Cognitive Task Analysis of Forest Disturbance Attribution. Int. J. Appl. Earth Obs. Geoinf. 2019, 74, 37–44. [Google Scholar] [CrossRef]
  84. García Rodríguez, C.; Vitrià, J.; Mora, O. Uncertainty-Based Human-in-the-Loop Deep Learning for Land Cover Segmentation. Remote Sens. 2020, 12, 3836. [Google Scholar] [CrossRef]
  85. Colwell, R.N. Manual of Photographic Interpretation; American Society of Photogrammetry: Washington, DC, USA, 1960; Volume 10. [Google Scholar]
  86. Colwell, R.N. Four Decades of Progress in Photographic Interpretation since the Founding of Commission VII (IP). Int. Arch. Photogramm. Remote Sens. 1993, 29, 683. [Google Scholar]
  87. Bailey, I.L.; Lovie, J.E. New Design Principles for Visual Acuity Letter Charts. Optom. Vis. Sci. 1976, 53, 740–745. [Google Scholar] [CrossRef]
  88. Ishihara, S. Tests for Color Blindness. Am. J. Ophthalmol. 1918, 1, 376. [Google Scholar] [CrossRef]
  89. International Council of Opthamology. Visual Acuity Measurement Standard. Ital. J. Ophthamol. 1988, II, 15. [Google Scholar]
  90. Qian, W.; Huang, Y.; Liu, Q.; Fan, W.; Sun, Z.; Dong, H.; Wan, F.; Qiao, X. UAV and a Deep Convolutional Neural Network for Monitoring Invasive Alien Plants in the Wild. Comput. Electron. Agric. 2020, 174, 105519. [Google Scholar] [CrossRef]
  91. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar] [CrossRef] [Green Version]
  92. Deng, J.; Dong, W.; Socher, R.; Li, L.; Li, K.; Li, F.-F. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
  93. Olson, D.L.; Delen, D. Advanced Data Mining Techniques; Springer: New York City, NY, USA, 2008; ISBN 978-3-540-76917-0. [Google Scholar]
  94. Davis, J.; Goadrich, M. The Relationship between Precision-Recall and ROC Curves. In Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA, 25–29 June 2006; Association for Computing Machinery: New York, NY, USA, 2006; pp. 233–240. [Google Scholar]
  95. Chen, C.L.P.; Li, H.; Wei, Y.; Xia, T.; Tang, Y.Y. A Local Contrast Method for Small Infrared Target Detection. IEEE Trans. Geosci. Remote Sens. 2014, 52, 574–581. [Google Scholar] [CrossRef]
  96. Kim, S.; Lee, J. Scale Invariant Small Target Detection by Optimizing Signal-to-Clutter Ratio in Heterogeneous Background for Infrared Search and Track. Pattern Recognit. 2012, 45, 393–406. [Google Scholar] [CrossRef]
  97. R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2019. [Google Scholar]
  98. Cohen, J. Eta-Squared and Partial Eta-Squared in Fixed Factor Anova Designs. Educ. Psychol. Meas. 1973, 33, 107–112. [Google Scholar] [CrossRef]
  99. Guirado, E.; Alcaraz-Segura, D.; Cabello, J.; Puertas-Ruíz, S.; Herrera, F.; Tabik, S. Tree Cover Estimation in Global Drylands from Space Using Deep Learning. Remote Sens. 2020, 12, 343. [Google Scholar] [CrossRef] [Green Version]
  100. Tidhar, G.; Reiter, G.; Avital, Z.; Hadar, Y.; Rotman, S.R.; George, V.; Kowalczyk, M.L. Modeling Human Search and Target Acquisition Performance: IV. Detection Probability in the Cluttered Environment. OE 1994, 33, 801–808. [Google Scholar] [CrossRef]
  101. Dodge, S.; Karam, L. A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions. In Proceedings of the 2017 26th International Conference on Computer Communication and Networks (ICCCN), Vancouver, BC, Canada, 31 July–3 August 2017; pp. 1–7. [Google Scholar]
  102. Geirhos, R.; Janssen, D.H.J.; Schütt, H.H.; Rauber, J.; Bethge, M.; Wichmann, F.A. Comparing Deep Neural Networks against Humans: Object Recognition When the Signal Gets Weaker. arXiv 2018, arXiv:1706.06969. [Google Scholar]
  103. Tan, M.; Pang, R.; Le, Q.V. EfficientDet: Scalable and Efficient Object Detection. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 14–19 June 2020; pp. 10778–10787. [Google Scholar]
  104. Ammar, A.; Koubaa, A.; Benjdira, B. Deep-Learning-Based Automated Palm Tree Counting and Geolocation in Large Farms from Aerial Geotagged Images. Agronomy 2021, 11, 1458. [Google Scholar] [CrossRef]
  105. Jocher, G.; Stoken, A.; Borovec, J.; Chaurasia, A.; Changyu, L.; Laughing, V.A.; Hogan, A.; Hajek, J.; Diaconu, L.; Kwon, Y.; et al. Ultralytics/Yolov5: V5.0—YOLOv5-P6 1280 Models, AWS, Supervise.Ly and YouTube Integrations; Zenodo: Geneve, Switzerland, 2021. [Google Scholar]
  106. Iqbal, M.S.; Ali, H.; Tran, S.N.; Iqbal, T. Coconut Trees Detection and Segmentation in Aerial Imagery Using Mask Region-Based Convolution Neural Network. IET Comput. Vis. 2021, 15, 428–439. [Google Scholar] [CrossRef]
  107. Hao, Z.; Lin, L.; Post, C.J.; Mikhailova, E.A.; Li, M.; Chen, Y.; Yu, K.; Liu, J. Automated Tree-Crown and Height Detection in a Young Forest Plantation Using Mask Region-Based Convolutional Neural Network (Mask R-CNN). ISPRS J. Photogramm. Remote Sens. 2021, 178, 112–123. [Google Scholar] [CrossRef]
  108. Cook, A.; Marion, G.; Butler, A.; Gibson, G. Bayesian Inference for the Spatio-Temporal Invasion of Alien Species. Bull. Math. Biol. 2007, 69, 2005–2025. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Locations of imagery collection.
Figure 1. Locations of imagery collection.
Remotesensing 13 03503 g001
Figure 2. Example of human search showing true positive (green) and false positive (red) marked points in image. Yellow polygon delimits contiguous area of visible miconia (treated as a “plant”). Green buffer corresponds to area of positive detection based on average characteristic length of polygons. Blue squares are buffers with sides equal to twice the characteristic length (square root of the enclosed area) of the enclosed polygon. Pixels within the yellow boundary correspond to the plant subset, P, while pixels within the blue boundary but outside of the yellow boundary correspond to background subset, B.
Figure 2. Example of human search showing true positive (green) and false positive (red) marked points in image. Yellow polygon delimits contiguous area of visible miconia (treated as a “plant”). Green buffer corresponds to area of positive detection based on average characteristic length of polygons. Blue squares are buffers with sides equal to twice the characteristic length (square root of the enclosed area) of the enclosed polygon. Pixels within the yellow boundary correspond to the plant subset, P, while pixels within the blue boundary but outside of the yellow boundary correspond to background subset, B.
Remotesensing 13 03503 g002
Figure 3. Flow chart of experimental methodology. Following image acquisition images used for training and validation of the DNN were annotated with bounding boxes of individual miconia leaves while the test set used for final evaluation was annotated with polygons delimiting contiguous areas of miconia.
Figure 3. Flow chart of experimental methodology. Following image acquisition images used for training and validation of the DNN were annotated with bounding boxes of individual miconia leaves while the test set used for final evaluation was annotated with polygons delimiting contiguous areas of miconia.
Remotesensing 13 03503 g003
Figure 4. Frequency distribution of recall (portion of total miconia polygons in analyzed imagery identified within individual recall % bins of human subjects or DNN models) of each search type with mean recall values (vertical dashed lines). An increase in the search speed of human searches results in a smoothing of the frequency distribution as recall diminishes. The deep neural network has a bimodal distribution, either identifying the plant with high recall or failing to identify it completely. Inset: Results of Tukey’s Honest Significant Difference test.
Figure 4. Frequency distribution of recall (portion of total miconia polygons in analyzed imagery identified within individual recall % bins of human subjects or DNN models) of each search type with mean recall values (vertical dashed lines). An increase in the search speed of human searches results in a smoothing of the frequency distribution as recall diminishes. The deep neural network has a bimodal distribution, either identifying the plant with high recall or failing to identify it completely. Inset: Results of Tukey’s Honest Significant Difference test.
Remotesensing 13 03503 g004
Figure 5. Non-linear regression of relationship between detection recall by human participants and relative size of miconia plant and search speed.
Figure 5. Non-linear regression of relationship between detection recall by human participants and relative size of miconia plant and search speed.
Remotesensing 13 03503 g005
Figure 6. Linear cost analysis of implementing deep neural network (DNN) and human searches at varied search speeds for miconia.
Figure 6. Linear cost analysis of implementing deep neural network (DNN) and human searches at varied search speeds for miconia.
Remotesensing 13 03503 g006
Table 1. Conditions during image acquisition at sites A–C.
Table 1. Conditions during image acquisition at sites A–C.
Locations
ABC
Coordinates (DMS)19°49′27.9″N
155°7′58.2″W
19°39′32.7″N
155°4′13.5″W
19°39′9.6″N
155°3′8.3″W
Date (YYYY/MM/DD)2018/04/172017/12/272018/03/23
Time (HH:MM HST)2:55 p.m.10:34 a.m.10:54 a.m.
Sky conditionOvercastSunnyOvercast
Mapped Area (ha)3.161.171.48
Overlap (Front/Side)80/8085/8560/60
Table 2. Hyperparameters for RetinaNet training used in this study.
Table 2. Hyperparameters for RetinaNet training used in this study.
ItemValue
Optimization MethodAdam
Initial Learning Rate0.0001
Learning Rate ScheduleValidation loss does not decline for 3 epochs, the learning rate was cut in half
Batch Size3
Training Epochs600 epochs at 500 steps per epoch
Table 3. Values for linear cost analysis of human and deep neural network searches of UAS imagery.
Table 3. Values for linear cost analysis of human and deep neural network searches of UAS imagery.
VariableValueVariableValue
CF (UAS)$6000IH5280 px
CF (DNN)$18,000IW3956 px
CL$25/hOS80%
FOVH500 pxt5 s
GSD1.1 cm/pxv5 m/s
Table 4. ANOVA table of factors (NM, size; S, speed of search, CP,R, contrast in red channel; CP,G; contrast in green channel; CP,B, contrast in blue channel; SCRR, signal to clutter ration in red channel; SCRG, signal to clutter ratio in green channel; SCRB, signal to clutter ratio in blue channel; D, distance to nearest miconia plant) affecting human detection recall for miconia. * indicates significant factors with largest values of η2.
Table 4. ANOVA table of factors (NM, size; S, speed of search, CP,R, contrast in red channel; CP,G; contrast in green channel; CP,B, contrast in blue channel; SCRR, signal to clutter ration in red channel; SCRG, signal to clutter ratio in green channel; SCRB, signal to clutter ratio in blue channel; D, distance to nearest miconia plant) affecting human detection recall for miconia. * indicates significant factors with largest values of η2.
PredictorSum of SquaresDegrees of FreedomMean SquareFpη2
NM48,220148,22045.0847<0.001 *0.072
S45,692222,84621.3605<0.001 *0.068
CP,R8450184507.90050.0050.013
CP,G12,877112,87712.0395<0.0010.019
CP,B161160.01450.9040.000
SCRR34,414134,41432.1767<0.0010.051
SCRG29,784129,78427.8471<0.0010.044
SCRB24,689124,68923.0839<0.0010.037
D44414440.41520.5200.001
Table 5. ANOVA table of factors (NM, size; S, speed of search, CP,R, contrast in red channel; CP,G; contrast in green channel; CP,B, contrast in blue channel; SCRR, signal to clutter ration in red channel; SCRG, signal to clutter ratio in green channel; SCRB, signal to clutter ratio in blue channel; D, distance to nearest miconia plant) affecting DNN detection recall for miconia. * indicates significant factors with largest values of η2.
Table 5. ANOVA table of factors (NM, size; S, speed of search, CP,R, contrast in red channel; CP,G; contrast in green channel; CP,B, contrast in blue channel; SCRR, signal to clutter ration in red channel; SCRG, signal to clutter ratio in green channel; SCRB, signal to clutter ratio in blue channel; D, distance to nearest miconia plant) affecting DNN detection recall for miconia. * indicates significant factors with largest values of η2.
PredictorSum of SquaresDegrees of FreedomMean SquareFpη2
NM2153121532.05770.1530.012
CP,R10,267110,2679.81050.002 *0.057
CP,G761760.07280.7880.000
CP,B2531125312.41860.1220.014
SCRR4716147164.50650.036 *0.026
SCRG9146191468.73960.004 *0.051
SCRB1083110831.03500.3110.006
D2241122412.14140.1460.013
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Rodriguez, R., III; Perroy, R.L.; Leary, J.; Jenkins, D.; Panoff, M.; Mandel, T.; Perez, P. Comparing Interpretation of High-Resolution Aerial Imagery by Humans and Artificial Intelligence to Detect an Invasive Tree Species. Remote Sens. 2021, 13, 3503. https://doi.org/10.3390/rs13173503

AMA Style

Rodriguez R III, Perroy RL, Leary J, Jenkins D, Panoff M, Mandel T, Perez P. Comparing Interpretation of High-Resolution Aerial Imagery by Humans and Artificial Intelligence to Detect an Invasive Tree Species. Remote Sensing. 2021; 13(17):3503. https://doi.org/10.3390/rs13173503

Chicago/Turabian Style

Rodriguez, Roberto, III, Ryan L. Perroy, James Leary, Daniel Jenkins, Max Panoff, Travis Mandel, and Patricia Perez. 2021. "Comparing Interpretation of High-Resolution Aerial Imagery by Humans and Artificial Intelligence to Detect an Invasive Tree Species" Remote Sensing 13, no. 17: 3503. https://doi.org/10.3390/rs13173503

APA Style

Rodriguez, R., III, Perroy, R. L., Leary, J., Jenkins, D., Panoff, M., Mandel, T., & Perez, P. (2021). Comparing Interpretation of High-Resolution Aerial Imagery by Humans and Artificial Intelligence to Detect an Invasive Tree Species. Remote Sensing, 13(17), 3503. https://doi.org/10.3390/rs13173503

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop