Accessing the Accuracy of Citizen Science Data Based on iNaturalist Data
Round 1
Reviewer 1 Report
This is a nice study that assesses the accuracy of citizen science data obtained from the free app iNaturalist. With the recent increase in citizen science projects, studies such as this are vital to validate findings, and the authors have done a good job of identifying potential issues with the use of iNaturalist data. However, I have two main concerns that prevent me from recommending the current version of the manuscript for publication. These are set out below and explained in more detail (along with other minor comments) in the attached annotated word file.
My first concern is about the explanations given in the methods. There appears to be information missing without which it is difficult to assess exactly what the authors did. For example, it is unclear exactly what data was harvested from iNat and how exactly it was treated. Another example concerns the verification of species identification. It seems that the sole criterion for assuming misidentification was record location – surely a quick morphological assessment would be more robust.
My second concern may be a separate issue or maybe a result of the lack of information mentioned above. I am not sure if the results are just an artifact of 1) the very different sample sizes, and 2) the possibility of single individuals providing a disproportionate number of records. For example, there are only 35 opportunities for citizen scientists to misidentify Ds, but over 6 times as many opportunities for them to misidentify Pc, and this may be further confounded if multiple records for single users are included. It might be an idea to analyze the proportions of accurate records, or a randomly selected subsample, rather than (or in addition to) the absolute values.
Overall, this can be a very worthwhile paper and I wish the authors success in revising it.
With best regards,
Kit Magellan
Comments for author File: 
Comments.pdf
Author Response
Respond to reviewer 1
The reviewer’s comments greatly helped in enhancing the quality of this paper. We hope that this paper gets published in the renowned Diversity journal to identify problems in civic science as well as improving it.
Reviewer comment:
This is a nice study that assesses the accuracy of citizen science data obtained from the free app iNaturalist. With the recent increase in citizen science projects, studies such as this are vital to validate findings, and the authors have done a good job of identifying potential issues with the use of iNaturalist data. However, I have two main concerns that prevent me from recommending the current version of the manuscript for publication. These are set out below and explained in more detail (along with other minor comments) in the attached annotated word file.
Point 1. My first concern is about the explanations given in the methods. There appears to be information missing without which it is difficult to assess exactly what the authors did. For example, it is unclear exactly what data was harvested from iNat and how exactly it was treated. Another example concerns the verification of species identification. It seems that the sole criterion for assuming misidentification was record location – surely a quick morphological assessment would be more robust.
- As the reviewer mentioned that morphology is the most reliable and fast sorting key. In this study we focused on the accuracy of the reported location on the two species. For analysis, only photos were used to check data accuracy, excluding unidentified data.
- Line 76: We added the sentence with what kind of data we used -> We only used data that included photos and can be accurately identified. In addition, we excluded from the analysis if multiple records were recorded at one site.
Point 2. My second concern may be a separate issue or maybe a result of the lack of information mentioned above. I am not sure if the results are just an artifact of 1) the very different sample sizes, and 2) the possibility of single individuals providing a disproportionate number of records. For example, there are only 35 opportunities for citizen scientists to misidentify Ds, but over 6 times as many opportunities for them to misidentify Pc, and this may be further confounded if multiple records for single users are included. It might be an idea to analyze the proportions of accurate records, or a randomly selected subsample, rather than (or in addition to) the absolute values.
- The two species were selected in this study because of their habitat-specific characteristics. In Korea, chsenicus (Class 2) and D. suweonensis (Class 1) are critically endangered species, so they are rarely found. Moreover, since only accurate and certain data were subject for verification, the amount of data for D. suweonensis was much smaller than the actual data uploaded. Just like the reviewer said, there is a limit to accurate comparison because of the difference in the number of samples between the two species. Nevertheless, it is important to understand the problems that may arise in citizen science through the results of this study and to make efforts for improvement in the future.
Author Response File: Author Response.pdf
Reviewer 2 Report
This is a very interesting and good study concerning the use of citizen science data, especially iNaturalist, one of the popular app in citizen science.
There are several comments:
Line 74-75: iNaturalist has 3 categories of observations: Needs ID, Casual and Research Grade. I presumed that the authors did not use the needs ID data but either use the Casual or Research Grade data. Could the authors confirm which observation category of quality is used?
Line 144: iNaturalist mostly relies on pictures for confirmation ID of species although people can also upload calls. How many from the data of D. suweonsis also include with recorded call? How can you assume that people used the mating calls to ID the frog.
Line 168-169: in this you said "..we propose a method". How? Could you elaborate more? Do you want the habitat where the picture of species taken also will be uploaded in iNaturalist?
Line 176: Not really sure with what you term as "controversial". What I summarized is that we need to test data from citizen science to analyze its reliability. If it is unreliable then we need to delete those data and only used data that has been checked thoroughly. Could you elaborate more what you mean with "controversial"?
Comments for author File: Comments.pdf
Author Response
Respond to reviewer 2
The reviewer’s comments greatly helped in enhancing the quality of this paper. We hope that this paper gets published in the renowned Diversity journal to identify problems in civic science as well as improving it.
Reviewer comments:
This is a very interesting and good study concerning the use of citizen science data, especially iNaturalist, one of the popular app in citizen science.
There are several comments:
Point 1. Line 74-75: iNaturalist has 3 categories of observations: Needs ID, Casual and Research Grade. I presumed that the authors did not use the needs ID data but either use the Casual or Research Grade data. Could the authors confirm which observation category of quality is used?
- Yes, it has three categories. In this research, we used all data available, mostly because some data classified as “Needs ID” had matched species and distribution. On the other hand, a lot of data from “Research Grade” were not accurate. Thus, regardless of the category, all information about discovery location and species photos were used in the study after direct examination by author.
Point 2. Line 144: iNaturalist mostly relies on pictures for confirmation ID of species although people can also upload calls. How many from the data of D. suweonsis also include with recorded call? How can you assume that people used the mating calls to ID the frog.
- suweonensis have high morphological similarites to their close relatives, D. japonicus, making them difficult for identification just by their shape. Therefore, experts also check the location and species based on the mating call, a definite classification key. In other words, the investigation of D. suweonensis is generally conducted in the order of 1) hearing call, 2) find species and 3) record location.
Point 3. Line 168-169: in this you said "..we propose a method". How? Could you elaborate more? Do you want the habitat where the picture of species taken also will be uploaded in iNaturalist?
- Yes, in this study, we found how the credibility of data was affected by changes in sites according to passage in time. For example, the location of suweonensis was identified as rice fields, while it is residential areas these days and therefore does not match. Of course, all researchers could check the uploading period and the time in Google Map of all data. However, if photos (including environment) of Google map at the time of uploading the data can be saved as a single data, researchers will be able to check the changes in the environment over time and thus address one problem that could debilitate the credibility.
- Line 169: we revised the sentence
Point 4. Line 176: Not really sure with what you term as "controversial". What I summarized is that we need to test data from citizen science to analyze its reliability. If it is unreliable then we need to delete those data and only used data that has been checked thoroughly. Could you elaborate more what you mean with "controversial"?
âž”We agree with reviewer’s opinion. In this paper, we argue for the necessity in improving the credibility of the data collected from citizen science. Accordingly, we revised it to mean that there is not enough credibility from “controversial.”
âž” Line 176: still controversial -> not enough using for science
Author Response File: Author Response.pdf
