Random Forest Classification of Multitemporal Landsat 8 Spectral Data and Phenology Metrics for Land Cover Mapping in the Sonoran and Mojave Deserts

Round 1

Reviewer 1 Report

The authors have implemented a random forest classification method for the Landsat8 dataset. The manuscript looks okay but a lot of flaws are there in the manuscript that could mislead the readers. They computed a couple of vegetation indices that were fed for the classification for the random forest method which is not new and innovative work. Therefore, the manuscript is not suitable for publication. The specific comments are summarized below.

·         The authors used the terminology “augmented” in the abstract and conclusion only. This is not explained either in the method or results sections. The authors have included this terminology without doing any work in the manuscript which greatly misleads the work.

·         The presentation of the objective is not correct, the authors claimed in objective 1 that they developed an ML algorithm for the classification that was never done, they simply fit the RF model

·         The terminologies used in objective 2 are repeated and not convincing.

·         Is there any topographical or climatic or ecological differences in BCR reason to harmonize into the transboundary region? No, means they are working in a homogenous topography and climatic condition means nothing done for objective 3.

·         The quality of the figures specifically fig,2 is very poor, the caption of fig 2 is not understandable.

·         Citation is not proper, lines 181, 202 among others

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

General Comments

The manuscript describes the approach implemented to elaborate the first 30 m land cover map of the Bird Conservation Region (BCR 33) in the Sonoran and Mojave Deserts across the international boundary US-Mexico. The investigated topic is certainly of interest to RS readers as well as to the broad ecological community.

The manuscript is well structured and written. The sections are clear and correctly balanced.

However, clarifications and additional information are required to improve it, particularly for the method section. An example is the selection of the 2-band EVI index (exclusion of blue band) in an environment with heavy aerosol presence. The Overall Accuracy metric, unsuitable for unbalanced datasets as already highlighted by the authors, has to be substituted with a more appropriate metric. The authors have to clarify how the reference land cover map is corrected; in the present form, it seems that the map has been adjusted with the data to be validated. Some definitions need to be clarified and harmonized (e.g., SOS and EOS phenology metrics). Point-by-point comments are listed below.

Moreover, a formal issue has to be evaluated with the editorial team about the previous publication of the thesis including the proposed study (https://repository.arizona.edu/handle/10150/666145 ).

Specific comments

Introduction

Line 50: Please verify the full name list of editors for reference number 5.

Lines 60 and 62: Please complete the information of the report of Ref. 8 (…Final report for Science Support Partnership Project for US Geological Survey and US Fish and Wildlife Service. Cooperative Agreement, 2017, G15AC00133.) as well as the recommended Citation of Ref. 9 (Flesch, A. D. 2018. Cactus Ferruginous Pygmy-Owl monitoring and habitat assessment on Pima County Conservation Lands. Report to Pima County Office of Sustainability and Conservation, University of Arizona, School of Natural Resources and the Environment. Contract No. CT-SUS-17-211).

Data and methods

Line 152: “ephemeral riparian woodland”, do you mean plants with short life cycles? or a narrow and stretched land cover (ref. woodland)? As in botany ephemeral indicates any short-lived plant, if you are referring to this, please add a few species as examples. Conversely, if you are referring to the second option, please select a different term (e.g., elongated).

Line 174: “Recently disturbed or modified land cover classes were also removed”. How were the land covers to be removed identified?

Line 176: Even if derived from GAP classification, please explain “developed land cover classes”. Generally, land cover maps report Urban / Built-up for land covered by buildings and other man-made structures/infrastructure. Therefore, explaining what is included in these classes is helpful for readers.

Figure 2: In the image, the pattern of the study area seems to correspond only to the Mexican portion. Please identify the whole study area with a bold line (without patterns to not overlay the GAP map) or alternatively indicate in the legend that the patterned area corresponds to the Mexican study area).

Moreover, how does the algorithm discriminate between Natural and Introduced Riparian and Wetland vegetation?

Line 210: Why is the last release (2021) of the US Cropland Data Layer used in the study? The analyzed Landsat series covers the period 2013-2020, whereas the CDL is based on 2017-2021 data (also previous releases are available). Please add comments in the text.

Line 225: What is the rationale for selecting the 2-band EVI index (EVI2)? The blue band is available in OLI data and already used by the authors. The reported reference (https://doi.org/10.1016/j.rse.2008.06.006 ) suggests EVI2 as an acceptable and accurate substitute of EVI when atmospheric influences are insignificant for sensors without a blue band. In the study area, arid conditions, sparse vegetation, and the presence of the desert facilitate the incorporation of natural particles into the atmosphere as well as the surrounding cities, active mining, and smelting operations favor wind to diffuse anthropogenic aerosol (https://doi.org/10.1016/S0016-7169(13)71480-6, https://doi.org/10.1175/JTECH-D-21-0114.1, https://doi.org/10.1029/2021JD035830)

Moreover, the adoption of the MODIS coefficients for the OLI bands (having different spectral ranges compared to the MODIS ones) has to be supported.

Lines 226-228: Please explicit the reason for complementarity between NDVI and EVI2.

Line 246: Do “the NDVI annual profiles” correspond to the long-term monthly average ? Please explicit in this sub-section.

Lines 257-262: More details are needed to explain how the parameters (minimum and maximum ranges of NDVI values, errors or noise in the time series, and changes over time, as well as the minimum length of a season) are tuned. Do you apply some thresholds? How are these threshold values evaluated?

Moreover how “changes over time” are defined?

Table 1: The phenology metrics SOS and EOS are defined in the table as the day identified by a consistent upward (downward) trend in NDVI. In lines 249-256, the use of the modified threshold (0.35) of the half-max approach is previously described. Please revise the definitions in table 1 accordingly.

Moreover, are the GUR and GDR (positive and negative rate of change in NDVI) evaluated on the 3-month intervals as for the other statistics (line 240)?

Lines 304-305: “sample sites were visually validated”, by checking which data the sample was validated? Please provide details. Did you use orthophotos or UAV images? It is quite difficult to correctly identify all the investigated classes by visual inspection of 30 m OLI images.

Lines 306-308: Please clarify how the GAP classification (REFERENCE MAP) has been CORRECTED. It seems that the reference map has been adjusted with the data to validate. A detailed explanation is mandatory. How did you identify mislabeled sites in the GAP classification? And how did you correct such mislabeling?

Line 339: Please add the total number of pixels used for validation.

Lines 344-348: As correctly highlighted by the authors, standard accuracy metrics provide misleading information in the presence of unbalanced datasets. Thus, while the use of Recall and Precision allows the authors to interpret better the F1-score, the analysis of the Overall Accuracy is meaningless in the context of the proposed study. Therefore, such a metric can be eliminated.

A practical alternative metric for thematic map accuracy is the Balanced Accuracy (https://doi.org/10.1109/ICPR.2010.764) that account for the unbalance of class distributions representing the mean of the individual class accuracies. It is widely adopted in different disciplines for ML classifications (https://doi.org/10.1007/s10994-020-05913-4). For details and applications on Balanced Accuracy in land cover mapping see https://doi.org/10.1016/j.deveng.2018.100039, https://doi.org/10.3390/rs14205127, https://doi.org/10.3390/ijgi9040277

Results

Table 4: As the F-Score level is already highlighted by bold and italic, please list the classes as in the confusion matrix (Figure 6) to make it easier for readers to match land cover names to class numbers.

Figure 6: Please add a comment in the caption that the values reported in the confusion matrix represent RF model accuracy. Generally, error matrices report the number of pixels.

Figures 7 and 8: I understand that there is a large number of classes, but it is really difficult to distinguish them on the map. Please select more distinguishable colors or use patterns to increase the readability of the maps. Alternatively, you can add the Geotif or vector map of the final classification as Supplementary material.

Figures 9 and 10: Please select more distinguishable colors or use different symbols (e.g., points, triangles, sqares) for land cover phenologies.

Discussion

Lines 450-451: In Figure 6 is reported the model accuracy by land cover class (lines 375-376). There is no number of pixels per class; thus, how can the readers identify the dominant classes? it can be useful to add the population size percentage (as in Figure 4) under the class code of the confusion matrix.

Lines 457-459: such a comment seems to contradict what has just been said in the previous sentences (lines 450-451). If the sampling choice compromises the performance of the majority classes, how can these classes be the most accurate? Please better explain the comment.

Lines 494-495: “EVI2 has a smaller dynamic range but better separation capabilities as it responds more to the NIR reflectance” in contrast to NDVI. Please clarify such a concept. The NIR band is utilized in both the indices in the same configuration (position and coefficient). Conversely, the coefficient of red band and the soil adjustment factor largely modify the EVI2 dynamic range.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

The Article text looks clear, and well-structured. Methodology is described, and the description appears appropriate to be used as a reference when designing study of ML application to remote sensing data analysis.

There are no any fundamental criticism from my side, besides the proposition of some cosmetic corrections:

- It is better to point additionally/duplicate the References number for the article in the row 69, In the phrase “The methods used by Elkind et al. …”.

- The scale bar is needed in side maps (in every map image) in the figures 2, 3, 7 and in the maps in figures 5, 8, 11.

- The legend colors in the figures 2, 3, 5, 8 (and, consequently in the figures 9, 10) have to be revised, as now there are almost no differences are observed for some pairs of land cover classes (for instance, Chihuahuan Stabilized Coppice Dune and Sand Flat Scrub class color looks the same to Undiffirentiated Barren Land class). As an idea to discuss, I may propose to use come basic colors (green, gray, brown) for high-level land cover types (shrubland, woodland, desert, etc.) and differentiate low-level classes by color tone and brightness. Unfortunately, the currently used legend tells nothing when you see it for the first time.

- The Web link in row 548 have to be formatted as active Web link, similar to the links in rows 180, 202, and etc.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

The authors present an interesting study producing a cross-border land cover (LC) map of BCR 33 at 30m resolution using the Random Forest (RF) machine learning (ML) method. They incorporate innovative sampling methods for creating training data for the ML model, however, some LC classes could not be predicted due to the absence of training samples. This is discussed. Generally, the paper is well-written and provides a solid methodology to build on.

How do the 31 classes used for training, map to the 24 classes for validation as the legend is different? Please provide some explanation to how this was achieved. 

For the summarized phenology, what were the climatic trends over the study period? Could the phenology be affected by annual rainfall or land cover change? How does the rainfall variability affect the mean phenology? Please link this to the 0.35 threshold mentioned in L254.

L376/397 The confusion matrix in Figure 6 should be labelled as a table IMO

L403-4 comparisons of the area's true colour image, INEGI data, and classification results obtained from the RF model. Do these somehow correspond with Figure 7 and if not, where are these areas located?

L423-433 – colours in Figures 9 and 10 are so similar – very difficult to tell apart – possibly use different symbols?

L444 – use “filtered out” instead of “tuned out”

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

What is the difference between objectives 1 and 2? As they seem to have the same meaning, could you combine them into a single objective?

Author Response

Please see the attachment.

Author Response File: Author Response.docx