Thematic Classification Accuracy Assessment with Inherently Uncertain Boundaries: An Argument for Center-Weighted Accuracy Assessment Metrics

Round 1

Reviewer 1 Report

This paper proposes an improved accuracy assessment for the cases of inherently uncertain boundaries. This method is developed based on center-weighting map segment area, and offers an alternative assessment metric for land cover classification.

The introduction and background are written with details, the methodology sounds correct. Six examples show that the quantitative values derived from traditional method and the proposed center-weighted method are quite different. The reviewer thinks that this topic is a rare but important in remote sensing community. However, there are a few issues that are encouraged to improve. The comments are as follows:

 

Major comments:

 

1. What is “thematic” classification? Is there any definition for it?

 

2. The Introduction and Background are suggested to be simplified and re-organized. Although the authors wrote lots of contents in those parts, the readers may be hard to quickly get the idea about this work. Thus, The authors are encouraged to remove some redundant/duplicate contents. On the other hand, Figure 1 could not tell what are the inherently uncertain boundaries. The readers could not understand why we need a new quality assessment metric. Thus, the authors should make more annotation on it.
3. As an engineering paper, the authors should use mathematical equations/symbols to describe the proposed method in Section 3.
   
   
4. Example 6 simulates a multi-class classification assessment. The authors adopt SAL data as reference (ground truth), and NWI data as predictions. So, this experiment is not a typical classification task. Are there any precedents or reference to follow?

 

 

Minor comments:

1. Line 149: F1 score(Equation 7) à F1 score(Equation 8)
2. Line 150: specificity(Equation 8) à specificity(Equation 7)
3. Line 414: specificity should be 0.966 à 0.996 (Table 6)
4. Line 406: Example 4.5à Example 5
5. The Legend format of Figure 7 and Figure 8 is inappropriate. There are no black and black dash lines in the figure.
6. In Figure 3, what dosed “(h) or (i)” means?

Author Response

Please see the attached file. 

Author Response File: Author Response.docx

Reviewer 2 Report

The submitted manuscript deals with a prominent challenge in remote sensing, namely the accuracy assessment of spatial objects that do not contain sharp or easily detectable boundaries, e.g. mixed pixels. The authors propose a center-weighing map segment approach which further allows to define the impact of distance from map segment edges based on a distance weighting exponent and a saturation threshold distance, after which the weighting ceases to grow.

The proposed approach is novel and interesting, and the submitted manuscript deserves publication. I listed a couple of questions and remarks below which require revisions in the introduction, background and discussion sections. 

 

General comments for manuscript improvement:

1. Wetlands are used as an example, but what other applications can benefit from the newly proposed approach? I was in particular thinking about the used term of "mixed pixels" which is a common phenomenon in the global south caused by landscape mosaics such as mixed farming systems, crop rotations or agroforestry, degraded forests, (small-scale) deforestation transition zones. Could the authors elaborate more on that issue in the discussion.

2. Does the size of AOI influence the weighing metrices, e.g what will happen once the approach is applied to a single global land cover map or the whole USA? Would it eventually make sense to do that, and if not why? Given the discussed drawbacks of the newly approach, do the authors propose its use for certain AOIs such as Wetlands to be a general necessity or as add-on to the conventional approaches?

3. The authors highlight in the introduction of the added-value in using the distance weighting method for GEBIOA segmented polygons. But I understood from the reading that only example 6 would refer to this case. Did I misunderstood? If not please make the connection of the distance weighting method to the GEBIOA more clear throughout the manuscript.

4. The application of the distance weighting method requires a reference data set that includes the boundaries of the featured mapped. This would refer to a watershed, or administrative unit, or a country? It not is really clear for the reader what "the boundaries of the features mapped refer to". Please add more explanations.

 

Typos:

Line 321: "Predication" should be "Prediction"?

Line 406: "Example 4.5" should be "Example 5"?

 

Author Response

Please see the attachment. 

Author Response File: Author Response.docx

Reviewer 3 Report

The paper covers assessment metrics; it argues and shows that the proposed strategy is better. There is very little to show that the authors have suggested classification methods or any distance to quantify the goodness of any classification method. The scope of the work is limited.

Author Response

Please see the attachment

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors has made appropriate modifications on the revised version. The reviewer has no other comments.

Reviewer 2 Report

Dear Authors,

Thanks for your detailed reponse which satisfies all previously stated concerns.