Detecting Land Abandonment in Łódź Voivodeship Using Convolutional Neural Networks

Round 1

Reviewer 1 Report

The paper is addressing an interesting research problem but in general, I find it challenging to follow the manuscript because of poor presentation. Here are some major comments:

The English of this manuscript needs to be strengthened by a NATIVE speaker. The wording and sentences read quite weird to me. In addition, there are typos and grammatical errors that are quite noticeable. The structure is equally a major drawback of this manuscript. Each section consists of numerous small paragraphs, which is not good. I suggest merging some of them to convey messages in brief. I expect some important papers to be cited regarding the use of deep learning (convolutional neural networks in particular) for land cover/use classification, but this is given neither in Introduction nor in Background. Why did you want to test deep learning for detecting land abandonment? Can traditional classification methods do? You should justify this somewhere, probably in the section Introduction. CNN is useful for pattern recognition and image classification but not all the readers of Land are familiar with such machine learning techniques. The authors should provide a brief introduction to it and explaining some key terms involved. It is important to provide a technical flowchart for the classification process as it would help better illustrate how abandoned land is identified. The confusion matrix provided in Table 1 seems a little confusing. What is the prediction data? An accuracy of 0.78 for the trained model is not high. Figures in this manuscript require extensive revision as well. For example, Figure 1 gives little useful information on the study. More places labels are mandatory. Country/province boundaries? Where are the regional/national capital cities if applicable? It is bounding_boxes in the legend, not boudning_boxes. Good figure captions in research articles should be self-contained, so readers are not forced to read the text above them. 

Author Response

Dear Sir/Madame,

thank you for providing us with the opportunity to submit a revised draft of our manuscript. We appreciate the time and effort that you have dedicated to the review process. We have strived to incorporate changes reflecting the bulk of the suggestions provided by your review.

Here is a point-by-point response to your comments and concerns:

Manuscript structure:

As suggested, we have merged a number of groups of related smaller paragraphs into larger ones. Furthermore, we have decided to adjust the ordering of the paragraphs describing the model’s prediction results.

Literature review (line 87):

The manuscript has been expanded to explicitly reference additional significant research results regarding the use of CNNs in land cover classification.

Justification of utilizing CNNs to detect land abandonment (lines 76 - 87):

Thank you for pointing out a possible clarity problem with the explanation of our motivation when choosing CNNs over traditional methods. In our current research, we focus on approaching computer vision problem with CNNs, as is industry standard; however, this is no excuse for not providing a clear justification for doing so. We have amended the relevant section with a short list of benefits that CNNs provide, including their ability to intrinsically represent complex features. 

Prediction data (lines 214 - 217):

As stated in the manuscript, the prediction data set was collected from Sentinel Hub satellite images acquired in August 2018. Image identifiers were collected within a CSV file, and attached to the manuscript as supplementary data. Thank you for drawing our attention to the fact that this might not be completely apparent. We have included an additional note describing the data collection process within the portion of the “Materials and Methods” chapter outlining the prediction phase.

Confusion matrix (Table 1):

We agree that the placement of the confusion matrix lacked context. We added a relevant reference to the text, and highlighted that the matrix is a tool for augmenting the understanding of the overall performance of the model. Furthemore, the data source, as well as precision, recall, and F-score, derived from the confusion matrix, have been made explicit.

Model performance (lines 209 - 212):

Regarding accuracy in general: we are aware that a value of 0.78 is, per se, not a production-grade result, and that there exists potential for improvement. However, we have focused on AUC as a measure that better evaluates binary classification problems. Furthermore, we have, in fact, obtained additional observations. Firstly, we have determined that providing the model with data related to fruit orchards, which were always improperly classified as abandoned land, eliminated some of the model’s deficiencies, and increased the accuracy to 0.83. Secondly, we experimented with transfer learning, and noted a marked, positive impact on the result. In the end, we decided against modifying our stated experimental setup “on the fly”, in order to maintain consistency of the research process.

Flow diagram (Figure 2):

A flow diagram demonstrating the model’s training process has been included into the “Materials and Methods” chapter. We have settled on a more generalized version of the diagram, in order to simultaneously assist the accompanying text in describing the process, and to provide a “bird's eye” level of complexity; both due to the need to maintain legibility, and because a “lower-level” version can be deduced from the supplied code repository.

 

In addition to the modifications outlined above, the paper underwent a professional language and style correction process. Finally, figures and their descriptions were also revised and/or improved. Once again, thank you for the in-depth review.

On behalf of the research team,

Maciej Adamiak 

Reviewer 2 Report

Dear authors,

the paper is well designed and the topic is interesting.

1. A fine english revision is needed;

2. Figure 8: improve Figure description. It is  not clear.

3. Figure 3 & Figure 4. Improve the representation. 

Author Response

Dear Sir/Madame,

we appreciate the time and effort that you have dedicated for providing your feedback on our manuscript. We have been able to incorporate changes to reflect all of the suggestions provided by your review.

Here is a point-by-point response to your comments and concerns:

Factors (previously figure 8, currently figure 9):

The description of the figure has been clarified. Thank you for pointing out the ambiguity of the previous caption. We have decided to rephrase it to emphasize that the figure shows occurrences of each factor within identified cold- and hotspots.

Samples (previously figures 3 and 4, currently figures 4 and 5):

As suggested the representation has been revised. The labels have been changed to highlight that each row represents a different data point of a single grid element. Additionally, we have chosen more representative data samples to better emphasize the differences between classification grades. Furthermore, the resolution of the images has been increased.

In addition to the above comments, as suggested, the paper underwent a professional language and style correction process.

We look forward to hearing from you in due time regarding our submission and to respond to any further questions and/or comments you may have.

On behalf of the research team,

Maciej Adamiak

Reviewer 3 Report

The paper describes an interesting application of Convolutional Neural Networks to the detection of land abandonment for a region in Poland.
The approach is sound and the post classification assessment of the results, using Getis-Ord Gi* spatial correlation analysis, is very detailed.

L37 "of 2010 , fallow" -> "of 2010, fallow"

L93 PCA -> Principal Component Analysis (PCA)

L94 "DCLE/eCognition": explain acronyms at their first use.

L104 "Generally, the preference seems to be to take both visual and infrared bands into account.": this probably is related to data availability.

L196 describe the elements of the formula.

L216 Just a comment: QGIS provides a plugin for this, HotSpot, and a branch on the git repository (https://github.com/danioxoli/HotSpotAnalysis_Plugin) has been recently updated to use the current PySAL 2.0 library. It is an experimental plugin, though.

L250 It would be interesting to quantify the influence of landscape configuration using landscape metrics.

Author Response

Dear Sir/Madame,

we appreciate the time and effort that you have dedicated for providing your feedback on our manuscript. We have been able to incorporate changes to reflect all of the suggestions provided by your review.

Here is a point-by-point response to your comments and concerns:

PCA (previously line 93, currently line 104): 

The Principal Component Analysis (PCA) abbreviation has been expanded.

DCLE/eCognition (previously line 94, currently line 105):

After discussion we have decided to change the term used to Deep Convolutional Neural Network and Transfer Learning (DTCLE). “DCLE/eCognition” does not appear in the referenced text and therefore cannot be used. Thank you for pointing this out.

Choice of spectral bands (previously line 104, currently 115): 

Data availability could indeed be one of the main reasons for choosing RGB + NiR over a different set of bands. It is worth noting that, in contemporary writing, not only data access is important, but also ensuring interoperability between different data sources. In addition, one of the lessons that we have learned during this research is that the time needed to train the model and its accuracy are equally important. In the early stage of the CNN’s training we have utilized more than four Sentinel bands. After multiple rounds of hyperparameter tuning with different combinations of bands, we have reached a state in which the model performance did not improve when providing additional frequency information to the base  RGB + NiR set. At the same time resource consumption was rising proportionally to the number of image channels, including the training time per epoch increasing dramatically. We assume that other authors also reached this conclusion via similar observations.

In the text, we have additionally elaborated on the possible reasons of limiting the number of bands used during similar studies.

Formula (previously line 196, currently line 207):

The relevant elements of the formula have been generalized and explained.

QGIS plugin (previously line 216, currently line 229):

We have decided to interoperably use QGIS and ArcGIS only because of individual preferences of the research team’s members. The suggested plugin has been examined and indeed offers the needed feature - thank you for bringing it to our attention, since it may prove useful in later research.

Landscape metrics (previously line 250, currently 259):

Thank you for the suggestion. Without a doubt quantifying the influence of landscape configuration would increase the value of the acquired results and subsequent analysis. We are certain we will take it into consideration when planning a larger study.

In addition to the above comments, all punctuation, spelling and grammatical errors pointed out by the reviewers have been corrected. We look forward to hearing from you in due time regarding our submission, and to respond to any further questions and comments you may provide.

On behalf of the research team,

Maciej Adamiak 

Round 2

Reviewer 1 Report

My comments have been well addressed and I am fine with the revised version of the manuscript except for the following small issues:

(1) Line 87 Reference 41.

(2) Figure 7 should be placed after the text.

(3) The black frames of Figures 7 and 11 should be removed.

Author Response

Dear Sir/Madame,

thank you, once again, for providing us with the opportunity to submit a revised draft of our manuscript.

Regarding the issues pointed out in the latest review:

 

Line 87:

The relevant reference (now indexed as 40) has been appended to the given citation sequence.

Figure 7:

The figure has been repositioned accordingly.

Frames of figures:

Figures 7 and 11 have had their outer frames removed. In addition, the corresponding change has also been applied to Figure 10, as it is identical, representation-wise, to Figure 11.

 

On behalf of the research team,

Maciej Adamiak