A Metric for Evaluating the Geometric Quality of Land Cover Maps Generated with Contextual Features from High-Dimensional Satellite Image Time Series without Dense Reference Data
Round 1
Reviewer 1 Report
The paper presents very clearly a comparison between several spatial supports that can be used to apply image analysis methods on high-resolution time-series image data from remote sensors.
The problem is worth to be analyzed since the new satellite missions are demanding for new methods to better assess and produce land cover maps and understand their longitudinal variations.
Three spatial supports that are commonly used are considered and defined. The definitions and the state of the art are intermixed with discussions that are of interest but that makes difficult to follow the main research line of the paper.
The production of these spatial supports is evaluated with respect to very well know features with respect to standard statical performance indicators and localization precision metrics, derived through the help a particular kind of corner detection.
Then the goodness of the spatial supports is evaluated on a dataset consisting of 33 dates, obtaining, as a result, a deeper understanding of the interdependency between the choice of spatial supports and descriptive features for guiding segmentation and labelling.
It is clear that the authors have worked hard to produce the paper and they provide their insight into the problem, which is of a good level.
However, the research contribution of the paper is still somewhat unclear at this stage.
It was asked to the authors to make precise claims about the novelty of the paper but that still seems to be absent.
A clear point by point response to the comments raised by the reviewers at the first round should be given
Changes made in the manuscript should be clearly commented.
I appreciated that reference:
D. Derksen, J. Inglada and J. Michel, "Spatially Precise Contextual Features Based on Superpixel Neighborhoods for Land Cover Mapping with High-Resolution Satellite Image Time Series," IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, 2018, pp. 200-203.
was now included but the provided explanation of differences is still not sufficient.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Well written introduction with rich information is perhaps too long. The author may wish to edit and remove some of the sections without disturbing the writing flow. Some minor editing works are needed. As an example, line 106 says; “…geometric quality of a classification map,…”, which is not clear by the direct meaning. Is it a classified map? Figure 1 needs a title and if possible, an insert map. The corner detection map in figure 2 is not clear, make the figure larger. Need to enlarge the figure 6 too to show “Pixel” clearly. The overall writing has done very well and each section has explained well. Please mark the “area chosen for the detailed experiments” on figure 12 and add a map frame to figure 13. Some of the figure captions (figure 16) and the caption for Table 4 are too long. Make them shorter and add the needed explanation into the text.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
The authors carried out a major revision and now present a framework for measuring the geometric precision of land cover maps generated using contextual features. I think the paper is suitable for publication on Remote Sensing, but I suggest some minor adjustments.
- The legend texts of Figure 4 (a and b) are too close, please fix it.
- It is important to standardize some maps elements in the Figures (i.e. north arrow and scale bar thickness). The scale bars in Figure 12 and 13 are thicker than those found in Figures 1 and 1 1. Also, use the same unit (km or meter) in all the scale bars.
- I suggest the authors to include coordinate grids in Figure 11, 12 and 13.
- Conclusion section is still too long. This section should be shorter and some parts could be moved to a discussion section or even to the results section.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.
Round 1
Reviewer 1 Report
Text needs some revisions. Use coma as thousand separator. Standardize some expressions (dataset or data set, benchmark or bench-mark, etc).
L23. Avoid listing keywords already used in the title.
Figure 3. The authors present the methods following this sequence: Sliding Window, Mean Shift and Superpixel. In order to make the reading easier, please keep this sequence when calling them in the Figures and also in the discussion.
L.203. Authors must provide more information about the Sentinel-2 time series. What period the images refer to? What did they deal with time series dataset to run the classification tests?
L327. Although this paper uses a 17-class land cover mapping published by another paper, the authors should provide more information about the map.
Figures 6 and 7: Improve the legend text, it’s not clear what “Window” or “Pixel” mean without reading the sections.
L417. Conclusion section is too long. First paragraph doesn’t show any conclusion.
Reviewer 2 Report
A useful field of study with high-resolution sentinel data. Abstract has written well.
35: There were costly high-resolution data in the market and Sentinel became a free-off charge resource.
36: “inaccessible to coarse spatial resolution sensors” check English. Only one or two places will be mentioned in this review, but the authors must check the whole report.
55: “same shape”, is this same spectral value? For one pixel, what is the meaning of shape? The shape is formed by a number of pixels.
86: Is there a need for explaining the flow of the study in the introduction section? Better to add a “Research objective” section.
96: Defining the spatial support section should enrich with appropriate diagrams. Authors have added a large number of citations, but no figure.
215: Figure 01 text is now clearly separated for (a) and (b). Also, it is worth to include pixel level zoom-in images to show the separation.
215: Figures should say where the location of the image is. Figure 3, well presented. Can add some points of highlights to the figure caption.
280: There is no indication of the image location and image scale (number of pixels) in the figure caption. Must address these minor but important aspects before the consideration for publication. Authors have mentioned some information in figure 9. However, it is too late to indicate after presenting other figures.
332: Red dotted line is too thick.
335: Figure 10 shows the urban or built-up areas in purple color? No legend is given to support this image. Also, there are no purple color pixels for the urban features in figure 3.4.and 11 image tile?
397: Result analysis is good, but it is better to present the tabled data in an illustration and highlight the findings.
Reviewer 3 Report
English should be review. The sentences are too long and carry to much ideas at once. There are also repeat in the text e.g. l.331 and l.341.
The graphs could use more clarity.
The "features" could be explained with more details to allow reproducibility of the experiments
Reviewer 4 Report
The paper presents very clearly a comparison between several spatial supports that can be used to apply image analysis methods on high-resolution time-series image data from remote sensors.
The problem is worth to be analyzed since the new satellite missions are demanding for new methods to better assess and produce land cover maps and understand their longitudinal variations.
Three spatial supports that are commonly used are considered and defined. The definitions and the state of the art are intermixed with discussions that are of interest but that makes difficult to follow the main research line of the paper.
The production of these spatial supports is evaluated with respect to very well know features with respect to standard statical performance indicators and localization precision metrics, derived through the help a particular kind of corner detection.
Then the goodness of the spatial supports is evaluated on a dataset consisting of 33 dates, obtaining, as a result, a deeper understanding of the interdependency between the choice of spatial supports and descriptive features for guiding segmentation and labelling.
It is clear that the authors have worked hard to produce the paper and they provide their insight into the problem, which is of a good level.
However, the research contribution of the paper is somewhat unclear at this stage.
The last line of the introduction fails in describing precisely the contributions of the paper. Indeed line 66-78 are just another description of the content of the paper, similarly to lines 86-91. The author should try to make more precise claims about their findings.
Additionally, there appears to be no novelty from a methodological point of view, since the described methods are all well know to the community. Section 4 with the evaluation metrics perhaps can be improved with a more vast analysis of metrics and emphasizing the new metric based on corner detection and feature matching. But at the moment Section 4 opens with a long discussion about the loss of localization capabilities e.g. in using major, which is true, but that can be postponed to the discussion.
The used dataset is just from one region and can be extended to prove the general validity of the findings.
In general, I suggest the authors rework the paper trying to transforming it into a well-focused survey of spatial supports and of their impact in localization precision by extending: - the number and quality of considered features, - the evaluation metrics, -the datasets.
Add also a proper "discussion" section besides the conclusion one.
I notice also one omission in the references. Indeed the paper:
D. Derksen, J. Inglada and J. Michel, "Spatially Precise Contextual Features Based on Superpixel Neighborhoods for Land Cover Mapping with High Resolution Satellite Image Time Series," IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, 2018, pp. 200-203.
doi: 10.1109/IGARSS.2018.8518961
appears to have some overlap with this one.
