Experimental Evaluation and Consistency Comparison of UAV Multispectral Minisensors

Round 1

Reviewer 1 Report

The manuscript “Experimental Evaluation and Consistency Comparison of UAV Mini Multi-spectral Sensors” presents an interesting comparison of UAV sensors, a topic that is very timely and actively growing because of low cost, high resolution and thus feasible promise to answer research questions that are currently answered by much more expensive technical applications. Thus, the manuscript has potential to be of interest to a wide audience.

However, I have two major concerns:

1. How large was the study area? If one estimates the area in Figure 1, it could be 240 m × 100 m, which is hardly enough for a comprehensive comparison study if it was the only study area. Were there several study areas or is this study based on only one study area? Was the data for both sensors gathered the same day? In which time window? Did the study area(/s) include agricultural land? Which crop species were included? When assessing the suitability of UAV sensors in determining vegetation indices, such as the NDVI, GNDVI and OSAVI in this study, it would be important to include agricultural land, preferably with different nutrient treatments or crop species.
2. The discussion of the results seems rather weak with a total of five references of which the first four are situated in the first chapter of the discussion that appears to be text that should be in the introduction since it does not refer to the results of this study. I am aware that this study area is new and actively growing, which might mean that now there are articles available that were not published a few months ago, but the newest results published should be cited and the results compared to the results of the current study. At least these seem relevant, found with a quick search within the journal of interest:

Fawcett D., Panigada C., Tagliabue G., Boschetti M., Celesti M., Evdokimov A., Biriukova K., Colombo R., Miglietta F., Rascher U., Anderson K. 2020. Multi-Scale Evaluation of Drone-Based Multispectral Surface Reflectance and Vegetation Indices in Operational Conditions. Remote Sens., 12, 514; doi:10.3390/rs12030514

Poncet, A.M., Knappenberger, T., Brodbeck, C., Fogle, M.Jr., Shaw, J.N., Ortiz, B.V. 2019. Multispectral UAS Data Accuracy for Different Radiometric Calibration Methods. Remote Sens. 2019, 11, 1917; doi:10.3390/rs11161917

Minor comments:

-The text would benefit of language checking. For example, the presentation of the results of the current study should be written in past tense, not in present tense.

-The abbreviations should be written in full in the main manuscript also, not only in abstract.

-Lines 134-149: No need to repeat in text what can be found in Table 1.

-Materials and Methods: Advertisement type of text for the sensors should be avoided

-In Figure 3, how was the spectral response calculated? Is it average reflectance for the whole study area?

-Why were these three indices selected when there would be so many others available in the literature?

-Lines 389-395: The first chapter of the discussion could be transferred to the introduction

-Lines 431-433: The pixel size description seems to be text that should be in the materials and methods, not in the discussion in the current form

Author Response

Dear Reviewer:

Thank you very much for your careful review and constructive suggestions with regard to our manuscript " Experimental Evaluation and Consistency Comparison of UAV Mini Multi-spectral Sensors "(ID: remotesensing-863540).

Those comments are all valuable and very helpful for revising and improving our paper, as well as the importance guiding significance to our researches. We have studied comments carefully and have made correction which we hope meet with approval. Our main corrections in the manuscript and the responds to the reviewer's comments are as follows and a word file " remotesensing-863540-revision " (main corrections are marked in red color). The main corrections in the paper and the responds to the reviewer's comments are as following:

 

Main corrections:

1. We optimized the description about methods, results and discussion.
2. We have added the experiment about Leaf Chlorophyll Index compare.
3. We added the limitations of this experiment in discussion.

Author Response File: Author Response.docx

Reviewer 2 Report

Dear Colleagues,

The topic of your paper is in my opinion interesting for RS community. In nowadays, there is still discussion about the needs of radiometric/reflectance calibration of UAV-based multispectral images. However, the presented study is in many parts insufficiently described (mainly in methodology) and therefore in this time without major changes does not reach the level of Remote Sensing journal.

GENERAL EVALUATION

The abstract and introduction chapters are generally well written. There are some statements, which should be cleared or rewrite (see detail comments below). The methodology part of the manuscript is sometimes confusing for me and should be improved. A lot of important information is missing here. Fundamental is accurate information about using the camera; were used Parrot SEQUOIA or Parrot SEQUOIA+? In the case of Parrot SEQUOIA, using calibration targets is necessary for further pre-processing in Pix4D software, when the reflectance is calculated. However, this information missing in the manuscript (both targets and software). Ordinary, the manuscript represents a typical case study, which is based on a relatively small piece of UAV data and a couple of ground points. Therefore, the robust evaluation of UAV sensors isn’t, in my opinion, relevant and is very site-sensitive.

DETAIL COMMENTS

ABSTRACT

LINE 19: NOTE – In my opinion is surface reflectance important in analysis, where the vegetation indices are calculated. All available methods for converting DN to surface reflectance could be tried; using DN values as input is, in my opinion, the last option in the process of vegetation indices calculation.

LINE20: I disagree with the statement: “Some UAV mini multi-spectral sensors use reflectance to calculate the VI…” Is the reflectance or VI really calculate by sensors? I think that both are rather calculated in pre/post-processing phases in specialized software by users.            

Personally, I think that this statement: “…others directly use the digital number (DN) to avoid errors in the process of converting DN to reflectance” is not appropriate and must be rewritten.

LINE 21/109: “But the consistency and accuracy of spectral data and VIs obtained by these sensors have not been investigated in detail before.” – I would be careful with this statement. Are you really convinced that no study focused on this topic exist?

INTRODUCTION

LINE 46: Once again (first time was mentioned in Abstract), I would explain the acronym UAV.

LINE 48 – 51: There are a lot of commas in one sentence. The sentence is hard to read.

LINE 53: Once again (first time was mentioned in Abstract), I would explain the acronym VI.

LINE 60: Once again (first time was mentioned in Abstract), I would explain the acronym DN.

LINE 106 – 107: In my opinion, the current research (previous paragraph) is still quite new.

MATERIAL AND METHODS

LINE 123: Information about altitude, topography and size of the study area is missing here.

LINE 128: Fill the name of small hexacopter UAV.

LINE 128: Parrot SEQUOIA or Parrot SEQUOIA+ is used in the study?

LINE 136 – 138: Redundant with Table 1. I would leave it only in the text or in the table.

LINE 144 – 146: Redundant with Table 1. I would leave it only in the text or in the table.

LINE 134 – 159: Remove the hyperlinks and replace by citations with concrete links on equipment in the list of references.        

LINE 154: InertiaL UNIT?

LINE 164: Date of acquisition differs from Table 2.

LINE163 – 172: The description of using UAV data is confusing. Rewrite more clearly.

LINE163 – 172: Were calibration targets using in the data acquisition process? If yes, must be stated in the manuscript.

LINE 169: Why P4M obtained only one image set, against Sequoia? The information must be in the text of the manuscript, not only in the header of Table 2. Why didn't you repeat the experiment again?

LINE 193 – 195: Two very similar sentences.

LINE 195: How many ground points measured by ASD were selected? Were selected randomly or stratified randomly with regards on different surfaces? New overview map represent points distribution would be valuable here.  

LINE 210: Were ground control points (GCP) using in the data processing? If no, how the horizontal shift of the individual data sets was handled?

FIGURE 5 and 6: The resolution of images is poor. The legend of both images is unreadable. In legend, there is also missing information about yellow/black squares (also some squares missing in the individual images).

254: Represent selected ROIs the different surfaces?

270: I don’t think that 8 ASD ground points are enough for the complex analysis focused on sensor consistency and accuracy.

271: Why did you use selected vegetation indices? There is plenty of VI, why didn’t you, for example, select VI calculated also from Red Edge band, when the correlation of Red Edge was higher than NIR band in your study?

336 a 338: Typos DNVI àNDVI.

383: Why didn’t you compare the individual UAVs bands with ASD values, same as you compare the VI? It would be valuable information about their reflectance accuracy. Could you add this comparison to the manuscript?

457: “we came to a robust conclusion“. Are your conviction, that your results are valid and applicable everywhere, not only for study areas and for sensors similar to your study?

Despite the above shortcomings, I suggest a major revision.

With regards!

Author Response

Author Response File: Author Response.docx

Reviewer 3 Report

Dear authors,

The presented manuscript aims to evaluate two mini multispectral sensors' performance. To achieve this objective, the authors analyze the consistency of spectral values and vegetation indices, and the NDVI accuracy between two mini multispectral sensors. In my opinion, introduction provides enough information about previous studies and the objective of this research. Methodology is well organized, providing enough information to understand the experiment. However, from this reviewer point of view, results section provides just many correlations between the two sensors' measurements. Authors repeat many times the same finding at a different spatial resolution or vegetation index. In addition, sixteen scatter plots are provided to highlight that data from both sensors are equivalent. To compare two sensors measurements does not contribute to scientific knowledge. If that is the case, we should compare every available sensor in the market. At this point, this reviewer would reject this manuscript for publication. However, the discussion section is potentially interesting. In my opinion, that should be the essence of this research, to understand why two sensors are equivalent or not in its measurements. Authors can contribute more if they focus this manuscript in the ideas written from line 389 to 395. My suggestion is to focus this research on the reasons that create a measurement difference between two comparable sensors, and not just to show that two sensors are equivalent on its measurements. Therefore, major improvements should be done on this manuscript before publication.

Minor comments:

In keywords, unmanned aerial vehicle (UAV) and multi-spectral sensor are words written in the title, therefore, they should not be included in keywords.

Figure 2 please explain that sequoia is the solid line and P4M is the dashed line in the captions not in the figure.

Line 154 including=includes

Please don't use adjectives such as good, better, strong, relatively weak, etc. When referring to either correlations or the experiment results.

Line 306 red band is again=red band

Lines 336 and 338, what DNVI is?

Line 441 Sequoiaand= Sequoia and

Lines 463:472, these lines sound more like a suggestion or opinion about the sensors. This is not a conclusion of the experiment, please remove.

 

 

 

         

 

Author Response

Dear Reviewer:

Thank you very much for your careful review and constructive suggestions with regard to our manuscript " Experimental Evaluation and Consistency Comparison of UAV Mini Multi-spectral Sensors "(ID: remotesensing-863540).

Those comments are all valuable and very helpful for revising and improving our paper, as well as the importance guiding significance to our researches. We have studied comments carefully and have made correction which we hope meet with approval. Our main corrections in the manuscript and the responds to the reviewer's comments are as follows and a word file " remotesensing-863540-revision " (main corrections are marked in red color). The main corrections in the paper and the responds to the reviewer's comments are as following:

 

Main corrections:

1. We optimized the description about methods, results and discussion.
2. We have added the experiment about Leaf Chlorophyll Index compare.
3. We added the limitations of this experiment in discussion.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Especially the abstract and discussion would benefit from English correction and thorough checking of the use of past tense whenever the results of this study are presented.

Author Response

Dear Reviewer: Thank you very much for your careful review and constructive suggestions with regard to our manuscript " Experimental Evaluation and Consistency Comparison of UAV Mini Multi-spectral Sensors "(ID: remotesensing-863540). Those comments are all valuable and very helpful for revising and improving our paper, as well as the importance guiding significance to our researches. We have studied comments carefully and have made correction which we hope meet with approval. Our main corrections in the manuscript and the responds to the reviewer's comments are as follows and a word file " remotesensing-863540-revision-2 " (main corrections are marked in blue color).

Author Response File: Author Response.docx

Reviewer 2 Report

Dear Colleagues,

Thank you for the very careful revision of your manuscript, incorporating not only my comments. The study (mainly methodology) is much clearer for readers now. However, I have still couple minor revisions/comments, which must be solved (revisions are numbered according to your replies).

9. Information about altitude, topography and size of the study area is missing here.

Reviewer's Reply

It's a pity that the size of your study area is only 0.03 km². In my opinion, 3 hectares is not enough for complex validation. If you would acquire data at even more different sites, the study would be much more valuable, not only small case study.

19. Why P4M obtained only one image set, against Sequoia? The information must be in the text of the manuscript, not only in the header of Table 2. Why didn't you repeat the experiment again?

Author's Reply:

Thanks. To maintain generally similar condition between each surveyed set of images, these three flights all started within a 60-minute period (11:27 to 12:22). Although P4M obtained only one image set, we can resample it to obtain the other image set. We have explained it in the second paragraph of Section2.3.1.

“In order to maintain generally similar condition between each surveyed set of images, the P4M sensor did not collected images with the resolution of 10cm“.

Reviewer's Reply

Your answers are contradictory. In the first version of manuscript you wrote, that the four UAV missions were performed but the images were not acquired in case of P4M with 10 cm. In this new version, you write about the needs of similar conditions and therefore you performed only three flights. Which version is true?

25. Represent selected ROIs the different surfaces?

Author's Reply:

Thanks. These ROIs were selected with different surfaces like grassland and stones. We have explained it in the second paragraph of Section2

Reviewer's Reply

From a methodological point of view is not, in my opinion, appropriate validate the values of vegetation indices outside the vegetation (for example on stones etc.). Were these surfaces used for validation?

28. Why didn’t you compare the individual UAVs bands with ASD values, same as you compare the VI? It would be valuable information about their reflectance accuracy. Could you add this comparison to the manuscript?

Author's Reply:

Thanks. We didn’t compare the spectral value of UAV’s individual bands with the ASD value because we can’t obtain the reflectance directly from P4M images. We have explained it in the Section2.4.3

Reviewer's Reply

From my point of view is the comparison of spectral values of UAVs with ASD the most demonstrable indicator of data consistency. Vegetation indices are based on mathematical formulas, which can potentially balance sensors differences. Therefore, provable results can be caused by coincidence in your experiment.

Why you do not compare the surface reflectance values and DN values with ASD in the same way as you compare the reflectance and DN values (dived by 10^5)?

Thank you for your responses.

With regards!

Author Response

Dear Reviewer:

Thank you very much for your careful review and constructive suggestions with regard to our manuscript " Experimental Evaluation and Consistency Comparison of UAV Mini Multi-spectral Sensors "(ID: remotesensing-863540).

Those comments are all valuable and very helpful for revising and improving our paper, as well as the importance guiding significance to our researches. We have studied comments carefully and have made correction which we hope meet with approval. Our main corrections in the manuscript and the responds to the reviewer's comments are as follows and a word file " remotesensing-863540-revision-2 " (main corrections are marked in blue color).

With regards!

Author Response File: Author Response.docx

Reviewer 3 Report

Dear Authors,

Thank you for accepting and include my suggestions on the presented version of the manuscript. I have checked this new version and it improved respect to the previous one. My only minor comment is to correct line 474 toanalyze= to analyze.    

Author Response

Dear Reviewer:

Thank you very much for your careful review and constructive suggestions with regard to our manuscript " Experimental Evaluation and Consistency Comparison of UAV Mini Multi-spectral Sensors "(ID: remotesensing-863540).

Those comments are all valuable and very helpful for revising and improving our paper, as well as the importance guiding significance to our researches. We have studied comments carefully and have made correction which we hope meet with approval. Our main corrections in the manuscript and the responds to the reviewer's comments are as follows and a word file " remotesensing-863540-revision-2 " (main corrections are marked in blue color).

With regards!

Author Response File: Author Response.docx