Positional Accuracy Assessment of Digital Elevation Models and 3D Vector Datasets Using Check-Surfaces

Round 1

Reviewer 1 Report (Previous Reviewer 2)

The subject and the authors' proposition are scientifically relevant. The questions were answered and the text of the article was improved.

The most worrisome questioning in the first review was answered. However, some questions can be improved in the text, as presented below.

 

 

::::::::::::::::::::::::::

R: 1. Introduction

>> Page 1 - lines 21 and 23: Uses ISO 19.113. This standard has been replaced by ISO 19157, which contains 6 quality elements.

A: I have replaced this reference to ISO 19157. However, the document I have consulted describes 5 data quality elements:

• completeness – presence and absence of features, their attributes and relationships

• logical consistency – degree of adherence to logical rules of data structure, attribution and relationships (data structure can be conceptual, logical or physical)

• spatial accuracy – accuracy of the position of features in relation to Earth

• temporal accuracy – accuracy of the temporal attributes and temporal relationships of features

• thematic accuracy – accuracy of quantitative attributes and the correctness of non-quantitative attributes and of the classifications of features and their relationships

 

Reviewer's reply: The author is mistaken. In section 7.3 of ISO 19157:2013, Figure 4 with six (6) elements is presented. The author has forgotten the element "usability", exposed in section 7.3.7, page 10 of ISO 19157.

 

::::::::::::::::::::::::::

R: >> Page 2 – line 51: Name the institutions.

A: See page 36 of [11]. For example ISPRS: (1+7.5 tan α) at scale 1:50,000.

 

Reviewer's reply: Quote in the text the ISPRS institution

 

::::::::::::::::::::::::::

R: >>Page 3 – line 113: Reference authors who mention the term 2.5D.

A: I cannot find any reference about the first use and definition of this term. This aspect is commonly assumed by the scientific community. As an example, it is described by the ArcGIS software website: https://desktop.arcgis.com/en/arcmap/latest/extensions/3d-analyst/what-is-a-functional-surface-.htm  

 

Reviewer's reply: If it is "commonly referred to", one should cite recent bibliographies of the term 2.5D.

 

::::::::::::::::::::::::::

R: 3. Application

>> Page 7 – lines 305-311: The reference data does not meet the level of 3x more accurate than the evaluated data. What's the justification? Is this threshold appropriate for large scale mapping?

A: The application developed in this study is focused on testing the proposed methodology. With this goal in mind, I selected a reference that meets the requirement of being more accurate than the dataset to be assessed (in this case is 2x). The level of being 3x more accurate is a recommendation for assess spatial databases. In this study, I am not trying to assess the BCA10 database. As you indicated, this assessment should involve, among other aspects, the development of a sampling procedure and the use of reference data that meet this level.  

 

Reviewer's reply: The author should cite in the text the explanation mentioned above for not meeting the 3x most accurate level

 

::::::::::::::::::::::::::

R: >> Page 9 – line 334: The evaluation carried out in the article encompasses the vertical and 3D component, so when using the ASPRS standard it should be 25 points, instead of 20.

A: Sorry, but I cannot find this number in the ASPRS standard documentation. In [8] Table 7 indicates 20 points. Can you clarify your comment? Thank you in advance.

 

Reviewer's reply: In table C.1. of the ASPRS standard, for areas smaller than 500 km² the total number of checkpoints for evaluating Horizontal and Vertical accuracy is 25. Of these, 20 checkpoints are for areas without vegetation (NVA) and 5 for areas with vegetation (VVA)

https://www.asprs.org/wp-content/uploads/2015/01/ASPRS_Positional_Accuracy_Standards_Edition1_Version100_November2014.pdf

 

::::::::::::::::::::::::::

 

Author Response

R: 1. Introduction

>> Page 1 - lines 21 and 23: Uses ISO 19.113. This standard has been replaced by ISO 19157, which contains 6 quality elements.

A: I have replaced this reference to ISO 19157. However, the document I have consulted describes 5 data quality elements:

• completeness – presence and absence of features, their attributes and relationships
• logical consistency – degree of adherence to logical rules of data structure, attribution and relationships (data structure can be conceptual, logical or physical)
• spatial accuracy – accuracy of the position of features in relation to Earth
• temporal accuracy – accuracy of the temporal attributes and temporal relationships of features
• thematic accuracy – accuracy of quantitative attributes and the correctness of non-quantitative attributes and of the classifications of features and their relationships

Reviewer's reply: The author is mistaken. In section 7.3 of ISO 19157:2013, Figure 4 with six (6) elements is presented. The author has forgotten the element "usability", exposed in section 7.3.7, page 10 of ISO 19157.

Author: Changed. Thank you.

::::::::::::::::::::::::::

R: >> Page 2 – line 51: Name the institutions.

A: See page 36 of [11]. For example ISPRS: (1+7.5 tan α) at scale 1:50,000.

Reviewer's reply: Quote in the text the ISPRS institution

Author: Done. Thank you.

::::::::::::::::::::::::::

R: >>Page 3 – line 113: Reference authors who mention the term 2.5D.

A: I cannot find any reference about the first use and definition of this term. This aspect is commonly assumed by the scientific community. As an example, it is described by the ArcGIS software website: https://desktop.arcgis.com/en/arcmap/latest/extensions/3d-analyst/what-is-a-functional-surface-.htm 

Reviewer's reply: If it is "commonly referred to", one should cite recent bibliographies of the term 2.5D.

Author: Done. Thank you.

 

 

 

::::::::::::::::::::::::::

R: 3. Application

>> Page 7 – lines 305-311: The reference data does not meet the level of 3x more accurate than the evaluated data. What's the justification? Is this threshold appropriate for large scale mapping?

A: The application developed in this study is focused on testing the proposed methodology. With this goal in mind, I selected a reference that meets the requirement of being more accurate than the dataset to be assessed (in this case is 2x). The level of being 3x more accurate is a recommendation for assess spatial databases. In this study, I am not trying to assess the BCA10 database. As you indicated, this assessment should involve, among other aspects, the development of a sampling procedure and the use of reference data that meet this level. 

Reviewer's reply: The author should cite in the text the explanation mentioned above for not meeting the 3x most accurate level

Author: Done. I have included these aspects in text.

::::::::::::::::::::::::::

R: >> Page 9 – line 334: The evaluation carried out in the article encompasses the vertical and 3D component, so when using the ASPRS standard it should be 25 points, instead of 20.

A: Sorry, but I cannot find this number in the ASPRS standard documentation. In [8] Table 7 indicates 20 points. Can you clarify your comment? Thank you in advance.

Reviewer's reply: In table C.1. of the ASPRS standard, for areas smaller than 500 km² the total number of checkpoints for evaluating Horizontal and Vertical accuracy is 25. Of these, 20 checkpoints are for areas without vegetation (NVA) and 5 for areas with vegetation (VVA)

https://www.asprs.org/wp-content/uploads/2015/01/ASPRS_Positional_Accuracy_Standards_Edition1_Version100_November2014.pdf

Author: Done. Thank you.

::::::::::::::::::::::::::

Reviewer 2 Report (Previous Reviewer 3)

The paper changes proposed by the authors are adequate.

I have only one suggestion:

LINE 497 CHANGE On the other hand, BY Moreover, 

Author Response

The paper changes proposed by the authors are adequate.

I have only one suggestion:

LINE 497 CHANGE On the other hand, BY Moreover,

Author: Done. Thank you.

 

 

Reviewer 3 Report (Previous Reviewer 4)

The comments we have provided from the previous round remains to be unaddressed in the manuscript, nor there are responses provided apart from the revised manuscript with changes reflecting revisions suggested by other reviewers perhaps.

For the author's reference, the comments from the previous review are provided below.

First, the introduction section must be separated with a section for related studies. The introduction section insufficiently discussed the importance and significance of the study, as well as the knowledge gap/clarity regarding the subject. The related studies section is sufficient in content, but separating it from the introduction will improve the paper’s readability for the reader’s understanding.

The paper’s title indicates that this paper focuses on accuracy assessment of DEMs and TINs. However, the paper mainly illustrates its methods applicable for DEMs. The author mentions that this method can be adapted for TIN models, it is not illustrated how, through an illustration of how this adaptation could work, or an illustrative example. Should the authors not put this in the paper, maybe it is better to focus the manuscript only on DEMs.

The selection procedure and the matching procedure are still not sufficiently discussed in the methodology. An illustrative figure may be added to explain this procedure.

In Chapter 4 (Results), the values for each parameter at each buffer distance (say, in Figure 4) is tabulated as a single value for the entire study area. How this single value is obtained for the numerous buildings is still unexplained if it is an average value, a total value, etc. Variability (ie standard deviation) for each value (if it is the mean for all polygons) must be illustrated in the figures too.

For Figure 6, how are urban and rural zones selected? Is this selected by checking a land-use map of the study area? It is not reliable to assess whether an area is urban or rural bases on terrain undulation and building density only. Please also label the graphs properly (type of zone) instead of the zone numbers.

Figure 7 also does not illustrate well what it aims to, nor does it provide additional evidence regarding the objective. It is better to illustrate this figure in 3D to visualize the errors in height/differences in the buffers.

 

In the conclusions section, the paper mentions that the paper demonstrated the feasibility, consistently (sic), and validity of its positional accuracy assessment. However, this paper mainly demonstrates its feasibility, but its reliability and validity is not proved in its methodologies and experimental results. Consistency (reliability) may be demonstrated by using different experimental (test) datasets in the experiment, and validity may be proved by using other ground truth datasets, neither of which are present in the paper. I believe all three is necessary to prove that the proposed method can be a useful method for accuracy assessment and that this paper has sufficient scientific rigor that can be published in IJGI.

The English writing level of this paper is in line with the publication level, further checking for minor errors is necessary.

Author Response

The comments we have provided from the previous round remains to be unaddressed in the manuscript, nor there are responses provided apart from the revised manuscript with changes reflecting revisions suggested by other reviewers perhaps.

For the author's reference, the comments from the previous review are provided below.

Reviewer: First, the introduction section must be separated with a section for related studies. The introduction section insufficiently discussed the importance and significance of the study, as well as the knowledge gap/clarity regarding the subject. The related studies section is sufficient in content, but separating it from the introduction will improve the paper’s readability for the reader’s understanding.

Author: Thank you for your suggestion, but we tried to follow those sections suggested by the IJGI journal (Introduction, Materials and Methods, Results, Discussion, and Conclusions). However, we have included some subsections in the introduction to facilitate the readability of the manuscript. In this sense, the related studies are included in those sections focused on the positional accuracy of vector databases and DEMs.

Reviewer: The paper’s title indicates that this paper focuses on accuracy assessment of DEMs and TINs. However, the paper mainly illustrates its methods applicable for DEMs. The author mentions that this method can be adapted for TIN models, it is not illustrated how, through an illustration of how this adaptation could work, or an illustrative example. Should the authors not put this in the paper, maybe it is better to focus the manuscript only on DEMs.

Author: We agree with you that the application of the methodology performed in this study used a DEM grid and not a DEM based on a TIN model. However, we found no problem in applying it to TIN models using vertex positions instead of cell positions (now included in text). To confirm this, we propose the application of the method to a DEM based on a TIN model or directly to point clouds in a future work.

Reviewer: The selection procedure and the matching procedure are still not sufficiently discussed in the methodology. An illustrative figure may be added to explain this procedure.

Author: Obviously, these aspects are important in cases of performing a quality assessment of a specific dataset. However, the aim of the application developed in is study was to test the methodology. Therefore, the selection and matching procedure was carried out visually considering a large amount of data. This is included in the text. Thank you.

Reviewer: In Chapter 4 (Results), the values for each parameter at each buffer distance (say, in Figure 4) is tabulated as a single value for the entire study area. How this single value is obtained for the numerous buildings is still unexplained if it is an average value, a total value, etc. Variability (ie standard deviation) for each value (if it is the mean for all polygons) must be illustrated in the figures too.

Author: The percentage of inclusion is obtained considering the number of cell positions that are included within each buffer (considering each case: 2DInc, 2DBd, HBd and 3DBd) in relation to the total number of cells. This is part of the procedure and it was described in the methodology section. Thank you.

Reviewer: For Figure 6, how are urban and rural zones selected? Is this selected by checking a land-use map of the study area? It is not reliable to assess whether an area is urban or rural bases on terrain undulation and building density only. Please also label the graphs properly (type of zone) instead of the zone numbers.

Author: This aspect is explained in lines 404-409. We take into account the average area of the polygons, considering that lower values of average area are related to rural zones with isolated buildings, while the highest values are related to urban zones with large blocks of buildings.

Reviewer: Figure 7 also does not illustrate well what it aims to, nor does it provide additional evidence regarding the objective. It is better to illustrate this figure in 3D to visualize the errors in height/differences in the buffers.

Author: Thank you for your suggestion, but in this case is difficult to provide this figure in 3D. We think that the graph is clear because the color of the cells indicates the type of inclusion obtained, which allows to detect problems in the 3D geometry of polygons (this is the purpose of this figure).

Reviewer: In the conclusions section, the paper mentions that the paper demonstrated the feasibility, consistently (sic), and validity of its positional accuracy assessment. However, this paper mainly demonstrates its feasibility, but its reliability and validity is not proved in its methodologies and experimental results. Consistency (reliability) may be demonstrated by using different experimental (test) datasets in the experiment, and validity may be proved by using other ground truth datasets, neither of which are present in the paper. I believe all three is necessary to prove that the proposed method can be a useful method for accuracy assessment and that this paper has sufficient scientific rigor that can be published in IJGI.

Author: The methodology has been tested with a large amount of data. We think that it demonstrated its feasibility (indicated in the text) in the assessing of the positional accuracy of these datasets. The other of aspects you indicate could be checked in future work using other datasets. Thank you.

Reviewer: Comments on the Quality of English Language

The English writing level of this paper is in line with the publication level, further checking for minor errors is necessary.

Author: The manuscript has been reviewed by a native English speaker. Thank you.

Reviewer 4 Report (New Reviewer)

The Manuscript is very interesting as it has attempted to develop the new methods of assessing the accuracy of DEMs.

In Methodology section in Line 227-234, the authors have applied such as automatic matching methodologies… Instead of “recommend”, authors need to argue the reasoning of applying this methods…avoid the words like “recommend”.

In Line 357…the authors talks about developing of “new software in Java”. It is suggested the rational for developing that software and also its does not make sense of either it is “New” or “old”…So, it might be interesting to brief about the software and its usability in the methods

In Figure 2, the authors came up about the 3D, so while developing DEM will authors consider other feature? Will this be applicable in all geographical urban setting? Do, there be any influence by the characteristic of urban areas including building types, terrain etc. Although the authors have made discussion in the result section, it is suggested to highlight what physical characteristic need to be consider while applying this methodology.

Which sampling methods applied for selecting the buildings? Why not selected other features.

Suggested to go for moderate editing and suggested to avoid I and We.

Author Response

Reviewer: The Manuscript is very interesting as it has attempted to develop the new methods of assessing the accuracy of DEMs.

In Methodology section in Line 227-234, the authors have applied such as automatic matching methodologies… Instead of “recommend”, authors need to argue the reasoning of applying this methods…avoid the words like “recommend”.

Author: We have used visual matching due to, among other reasons, the amount of data used in the application. In cases of larger datasets, we recommend the use of automatic matching algorithms to improve processing efficiency. Thank you.

Reviewer: In Line 357…the authors talks about developing of “new software in Java”. It is suggested the rational for developing that software and also its does not make sense of either it is “New” or “old”…So, it might be interesting to brief about the software and its usability in the methods

Author: Done. I have removed the term “new”.

Reviewer: In Figure 2, the authors came up about the 3D, so while developing DEM will authors consider other feature? Will this be applicable in all geographical urban setting? Do, there be any influence by the characteristic of urban areas including building types, terrain etc. Although the authors have made discussion in the result section, it is suggested to highlight what physical characteristic need to be consider while applying this methodology.

Author: As you indicate, the results have shown that building geometry is very important and the methodology proposed in this study is sensitive to this aspect. In fact, the lack of information in the case of the vector dataset used in this study has shown worse results mainly in complex buildings. This aspect has been discussed extensively throughout the manuscript. Thank you.

Reviewer: Which sampling methods applied for selecting the buildings? Why not selected other features.

Author: We did not used any sampling method because this study was not focus on performing a quality assessment of a specific dataset. The application was only focused on testing the methodology. Therefore, the selection of the buildings was visual considering a large amount of areas with different features. The use of buildings instead of other elements was conditioned by the availability of datasets. Thank you.

Reviewer: Comments on the Quality of English Language

Suggested to go for moderate editing and suggested to avoid I and We.

Author: The manuscript has been reviewed by a native English speaker. Thank you.

Round 2

Reviewer 3 Report (Previous Reviewer 4)

The reviewer agrees with the responses to the comments and believes the current version of the paper may be accepted to IJGI.

Final checks to minor spelling and grammar errors prior to publication.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The paper entitled "Positional Accuracy assessment of Digital elevation models..." presents an interesting and valuable proposal.

 

Structure:

The structure is adequate.

 

Contents and length:

The length of the introduction section is excessive. For example, lines 54 to 62 add little to the document. The length given to the review of the methods of using uncertainty bands for linear elements is also excessive. You can go more to the point. However, there is a lack of aspects more related to statistical hypotheses about the data, and also about sample sizes, a critical aspect for the evaluations.

The methodology section is a bit confusing. Figure 1 requires a legend. The acronyms used are not mnemonic, for example, 2DInc, 2DBd, 3DInc and 3DBd would make more sense.

Within the methodology, sub-methodologies appear, such as the triangulation process. This should be out of the paper. A standardized process must be applied with a reliable and recognized algorithm.

The description of the data sets requires clarification whether the indicated accuracy is horizontal or vertical. In addition, in relation to the comments made in the conclusions, the production process should be better known, here it should be indicated whether or not these characteristics may affect the results. The results would show the evidence of it.

The example of application supposes the realization of a sampling whose design is not justified in a solvent way. Then in the conclusions it is indicated that in the future progress will be made to determine sample sizes. In this case, I suggest the author not present in the paper something similar to a sample (which he does), and simply analyze what happens in a large and representative area. That is to say, that he clearly shows an experiment with real data but that he does not confuse it with carrying out a sample. Table 1 is confusing to me, the information of the products and the information of the sample must be separated.

In the results the interpretation of the important figure 5 requires knowing what BH means, but this is not indicated until the end of the section, several pages later. In the results section there are contents that correspond to analyzes that have not been proposed in the objectives. This is a lack of consistency. For example, if it is known that the height of buildings presents definition problems (lack of internal structure, lin.509), this analysis is perfectly justified.

The conclusion section is very long. Some of its content is a summary, this is not a conclusion. Another part of its content is more typical of the discussion section. A true conclusion must be made about the contribution.

 

Some minor, but important aspects:

Lin.21. The ISO 19113 standard is mentioned, but this standard is no longer in force. Currently, ISO 19157:2013 is in force and a revised version will be available soon.

Lin.44. The ASPRS standard (2015) is also mentioned, but there is a new version that is under review. It would be nice to consider it.

Lin. 54. In relation to the description of vector geometries, it is more appropriate to mention ISO 19107 or 19137.

Lin.63. The use of the word “errors” is not appropriate. Error is not the same as uncertainty. Here he talks about uncertainty. In the case of comparing two DEMs, one must speak of a discrepancy. Review the document.

Lin.188. Please replace “resolution” with “spatial resolution”.

Lin.276. From what I understand, they are not 3D segments, they are 3d triangular facets.

Lin.333. The accounts you present are for one area (map sheet), but what about the entire region of Andalusia?

Lin.385-389. There is a change in font size.

Lin.393. Replace “100” with “1000”.

Several places: The author indicates that the reference data should be "a more accurate source". This is not the only requirement; the reference data must also be independent. In a good paper from your peers (Ariza-López F.J., García-Balboa, J.L., Rodríguez-Avi, J., Robledo J., 2021. Guide for the Positional Accuracy Assessment of Geospatial Data. Pan American Institute of Geography and History, Occasional Publication # 563) they stated three requirements for the reference.

 

In conclusion. I think it's a new and interesting idea, but the document requires many changes and some corrections. Therefore I suggest a major review.

Author Response

Reviewer #1:

Author: Thank you for your suggestions and comments. I have rewritten the paper following your suggestions.

R: The paper entitled "Positional Accuracy assessment of Digital elevation models..." presents an interesting and valuable proposal.

Structure:

The structure is adequate.

Contents and length:

The length of the introduction section is excessive. For example, lines 54 to 62 add little to the document. The length given to the review of the methods of using uncertainty bands for linear elements is also excessive. You can go more to the point. However, there is a lack of aspects more related to statistical hypotheses about the data, and also about sample sizes, a critical aspect for the evaluations.

A: I have reduced the length of the introduction section. Lines 54 to 62 have been removed. I have changed this section following your suggestions.

R: The methodology section is a bit confusing. Figure 1 requires a legend. The acronyms used are not mnemonic, for example, 2DInc, 2DBd, 3DInc and 3DBd would make more sense.

A: I have included a legend in Figure 1 and improved this section following your suggestions. The acronyms follow a simple rule that I think is clear:

• 2DInc -> 2D Inclusion
• 2DBd -> 2D Buffer of distance “d”
• HBd -> Height Buffer of distance “d”
• 3DBd -> 3D Buffer of distance “d”

When the distance “d” is specified (case of Figure 7), this letter is changed to the value used (e.g. “(3m)”).

R: Within the methodology, sub-methodologies appear, such as the triangulation process. This should be out of the paper. A standardized process must be applied with a reliable and recognized algorithm.

A: I have included a legend in Figure 1 and improved this section. In the case of the triangulation process, I have changed the procedure suggesting the use of Delaunay algorithm.

R: The description of the data sets requires clarification whether the indicated accuracy is horizontal or vertical. In addition, in relation to the comments made in the conclusions, the production process should be better known, here it should be indicated whether or not these characteristics may affect the results. The results would show the evidence of it.

A: I have clarified this aspect. The discussion and conclusion sections include results related to production process, such as the lack of internal structure of the polygons, which shows great influence on the results of urban zones.

R: The example of application supposes the realization of a sampling whose design is not justified in a solvent way. Then in the conclusions it is indicated that in the future progress will be made to determine sample sizes. In this case, I suggest the author not present in the paper something similar to a sample (which he does), and simply analyze what happens in a large and representative area. That is to say, that he clearly shows an experiment with real data but that he does not confuse it with carrying out a sample. Table 1 is confusing to me, the information of the products and the information of the sample must be separated.

A: I agree with you. I have changed this part trying to avoid presenting data as a sample. Instead, data selection is related to analyzing what is happening in a large area. I have changed Table 1 to clarify. I think it is not worth splitting it into two tables.

R: In the results the interpretation of the important figure 5 requires knowing what BH means, but this is not indicated until the end of the section, several pages later. In the results section there are contents that correspond to analyzes that have not been proposed in the objectives. This is a lack of consistency. For example, if it is known that the height of buildings presents definition problems (lack of internal structure, lin.509), this analysis is perfectly justified.

A: I have corrected this part because the term “BH” was a mistake. The correct one is “HBd”. I have added the lack of internal structure in the objective section.

R: The conclusion section is very long. Some of its content is a summary, this is not a conclusion. Another part of its content is more typical of the discussion section. A true conclusion must be made about the contribution.

A: I have improved this section following your suggestion.

R: Some minor, but important aspects:

Lin.21. The ISO 19113 standard is mentioned, but this standard is no longer in force. Currently, ISO 19157:2013 is in force and a revised version will be available soon.

A: Done.

R: Lin.44. The ASPRS standard (2015) is also mentioned, but there is a new version that is under review. It would be nice to consider it.

A: Thank you. I will consider it when it is published.

R: Lin. 54. In relation to the description of vector geometries, it is more appropriate to mention ISO 19107 or 19137.

A: This part has been removed following your suggestion.

R: Lin.63. The use of the word “errors” is not appropriate. Error is not the same as uncertainty. Here he talks about uncertainty. In the case of comparing two DEMs, one must speak of a discrepancy. Review the document.

A: Done.

R: Lin.188. Please replace “resolution” with “spatial resolution”.

A: Done.

R: Lin.276. From what I understand, they are not 3D segments, they are 3d triangular facets.

A: I have corrected it by removing “3D”.

R: Lin.333. The accounts you present are for one area (map sheet), but what about the entire region of Andalusia?

A: This study does not pretend to assess the whole region of Andalusia or the BCA10 database. I selected 34 sheets that cover a large area to test the method. Therefore, the mention of this region was included only to show the large amount of data used and to describe its origin.

R: Lin.385-389. There is a change in font size.

A: Done.

R: Lin.393. Replace “100” with “1000”.

A: Done.

R: Several places: The author indicates that the reference data should be "a more accurate source". This is not the only requirement; the reference data must also be independent. In a good paper from your peers (Ariza-López F.J., García-Balboa, J.L., Rodríguez-Avi, J., Robledo J., 2021. Guide for the Positional Accuracy Assessment of Geospatial Data. Pan American Institute of Geography and History, Occasional Publication # 563) they stated three requirements for the reference.

A: Done.

R: In conclusion. I think it's a new and interesting idea, but the document requires many changes and some corrections. Therefore I suggest a major review.

A: Thank you for your suggestions and comments. I think the manuscript has improved after this revision.

Reviewer 2 Report

General comments

--------------------------------------------------

The author proposes to evaluate the positional quality of 3D geospatial data using check surfaces, instead of the traditional approaches of check points and lines. The article is relevant and brings a scientific gain when analyzing the positional accuracy from this perspective. Specifically, the methodology is presented in the evaluation of 3D vector databases and Digital Surface Models (DSM).

In general the text is well structured. The problem, hypothesis and research objectives are well defined.

Some parts of the text in section 2 and 3 need to be clearer for the methodology to be reproducible.

I notice an inconsistency in the results of 2D, Z and 3D accuracy, in the proposed methodology, which need to be justified and discussed in the text.

The results presented in section 5 have errors.

The more detailed questions are presented below in the specific comments.

 

Specific Comments

--------------------------------------------------

Title: Positional accuracy assessment of Digital Elevation Models and 3D vector databases using check-surfaces

>> Title is very specific in dealing only with 3D vector databases and DEM. Could the methodology be applicable to any type of 3D geospatial data? If so, I suggest changing the title.

>> The abstract and the article addresses DEM evaluation, but the model used in the methodology is DSM. I suggest revising the article to make these terms compatible.

>> The abstract cites 4 results, but the correct would be 3 (2D, Z and 3D)

 

1. Introduction

>> Page 1 - lines 21 and 23: Uses ISO 19.113. This standard has been replaced by ISO 19157, which contains 6 quality elements.

>> It uses the terms “true values” and “true position”. In Geospatial Data, the true position is not known. The movement of the Earth and its tectonic plates is dynamic, and positioning instruments having uncertainties in the survey process. A better term would be “reference” rather than “true”.

>> Page 1 – lines36 and 37: “As an example, the ratio between accuracies of reference and the product to be assessed should be at least three [3]”. I agree with this value for medium and large scale cartographic data. What about for large scale map (1:1,000, 1:500)? Is this value still feasible?

>> Page 2 – line 51: Name the institutions.

>> Page 2 – Lines 65 and 66: “Positional uncertainty can be considered as an area (2D) or 65 volume (3D) surrounding each entity where the most probable position is located”. It would be interesting to have a figure to illustrate.

>>Page 3 – line 113: Reference authors who mention the term 2.5D.

>> Page 3 – lines 127 e 129: These articles cover this subject ( https://pubs.er.usgs.gov/publication/70005960)

 

2. Methodology

>> Page 4 – lines 184-188: The author must reference the text. There are articles that address this (https://doi.org/10.1590/s1982-21702020000200007 )

>> Page 5. Figure 1: Instead of using 3 boxes (DEM, DTM and DSM elements) you could use only 1 (DSM). The DSM involves the entire terrain surface plus vegetation and buildings. The article uses only the elevation of the buildings. When doing a subtraction of the DEM with the DTM, only vegetation and buildings would remain. Wouldn't it have to be another nomenclature, instead of the DSM?

It does not address, or is unclear, the sampling process in the flowchart in Figure 1.

The Maching box should be formed from the 3D vector and the DSM, generating the selection of features and cells. The text has to be clearer.

>> Page 5. - lines 214: “the same Coordinate Reference System (CRS)”. Explain in the text the altimetric reference. It is important to mention that the use of different (almost) geoid models, for the conversion of altimetric references, can bring uncertainties in the process of evaluating the positional accuracy. This is very important and should be addressed in the text of the article.

>> Page 6 – lines259-273: quotes 3D triangulation based on 3D polygons. However, in item 3 of the article a DSM is used as a reference. How is triangulation performed on a DSM grid? Why not use Delaunay's algorithm for triangulation?

>> Page 6 - Figure 2.d: This part left me confused. In case the 3D polygon is a reference, why do the triangulation if the data is already 3D and has the surface created? How would it be if the DSM was the reference? Would you triangulate based on the centroid of the cells?

>> Figure 2.e and 2.f: the illustration could be improved by including specific features related to altimetry.

 

3. Application

>> Page 7 – lines 305-311: The reference data does not meet the level of 3x more accurate than the evaluated data. What's the justification? Is this threshold appropriate for large scale mapping?

>>Page 8 – lines 316 and 317: The author should give more details about the origin of the altimetric reference. What is the original reference of each data (test and reference)? Was there an altitude conversion? Which geoid models were used? How can this conversion propagate uncertainties in the evaluated data?

>> Figure 3: 3.a and 3.b must have legends to inform the symbologies. Figure 3.a must be presented as a map, containing legend, coordinates grid and cartographic information. Figure 3.b is confusing without the legend.

>> Page 9 – line 322: What is the total area (km²) of the data region (Andalucía)? Cite in text.

>> Page 9 – line 323 and 324: Review area values. 7.3km x 4.7km = 34.31 km². 34 sheets by 34.31 = 1166.54 km². In Table 1 these calculations are not equal.

>> Page 9 – line 325: What aspects? Cite in the text.

>> Page 9: The sampling process needs to be made clearer in the text. How did you arrive at the number of 34 cards used?

>> Page 9 – line 330: Zone is each sheet selected in the sampling process?

>> Page 9 – line 334: The evaluation carried out in the article encompasses the vertical and 3D component, so when using the ASPRS standard it should be 25 points, instead of 20.

>> Page 9 - lines 337-339: In section 2 you recommended to use automatic matching methodologies. Why not use this automatic methodology? Was the matching performed visually or manually?

>> Page 9 – line 344: The numbers do not match. In the text it says 1.8 and in Table 1 the value corresponds to 1.79.

>> Table 1: Review the values in the “Average (by zone)” column. Use 1 decimal place.

>> Page 9 – line 349: What is the name of the software? Is this software proprietary or freeware?

 

4. Results

>> Positional accuracy results of the tested product (analyzing Figure 4.a):

·         2D: 2.25m (95%)

·         Z: 8m (80%) ** Does not cross at 95% limit.

·         3D: 6m (95%)

These values should be highlighted in the text of section 4, 5 and 6.

These values are compatible with the 1:10,000 scale of the evaluated product. What is the final quality of the product?

Considering that 3D² = 2D² + Z², shouldn't the accuracies of the developed methodology follow this pattern?

If we calculate the 3D accuracy above, using the 2D and Z variables, we would arrive at a value of 8.3m for the 3D accuracy. But the methodology presented the value is 6m.

How does the author justify these results and the consistency of the 3D accuracy values? It is an important part of the article that should be analyzed and discussed in greater depth.

 

>> Page 11 – line 387: Talks about several aspects but mention only two: area and slope.

>> Figure 5: The title of each graph has abbreviations different from those used in the text (2Dinc, 2DBd, HBd, 3DBd).

>> Page 12 – line 398, 412, 414: correct the acronym for HBd.

>> Figure 6 and 7: cite the figure in the text before presenting it.

>> Page 12 – line 419: Did you do any statistical test to say that the zones are representative of the region?

>> Page 13 – lines 429-436: The author cannot conclude about the entire product based on these 4 zones. The analysis has to be for the dataset as a whole.

5. Discussion

>> Page 14 - lines 455-464: The author sets the vertical accuracy result equal to the 3D accuracy result. As per Figure 4a and comments made by this reviewer in the previous section, the results are not the same. Furthermore, it has an inconsistency with the 3D accuracy value. The author needs to give explanations in the text.

>> Page 14 - lines 466-474:: The author cannot conclude about the entire product based on the 4 zones analysed. The text has to be explicit that the results achieved are for the data used in the research. Other data sets may show different results.

 

6. Conclusions

>> The proposed methodology is indicated only for large-scale geospatial data, since the use of building roofs is the checking surface. For medium and low scale data could this methodology be applied? Address this analysis in the text.

Author Response

Reviewer #2:

Author: Thank you for your suggestions and comments. I have rewritten the paper following your suggestions.

R: The author proposes to evaluate the positional quality of 3D geospatial data using check surfaces, instead of the traditional approaches of check points and lines. The article is relevant and brings a scientific gain when analyzing the positional accuracy from this perspective. Specifically, the methodology is presented in the evaluation of 3D vector databases and Digital Surface Models (DSM).

In general the text is well structured. The problem, hypothesis and research objectives are well defined.

Some parts of the text in section 2 and 3 need to be clearer for the methodology to be reproducible.

I notice an inconsistency in the results of 2D, Z and 3D accuracy, in the proposed methodology, which need to be justified and discussed in the text.

The results presented in section 5 have errors.

The more detailed questions are presented below in the specific comments.

Specific Comments

--------------------------------------------------

Title: Positional accuracy assessment of Digital Elevation Models and 3D vector databases using check-surfaces

>> Title is very specific in dealing only with 3D vector databases and DEM. Could the methodology be applicable to any type of 3D geospatial data? If so, I suggest changing the title.

A: I have tested this methodology to these datasets. Future work (see conclusion section) includes the possibility of applying to other types of data.

R: >> The abstract and the article addresses DEM evaluation, but the model used in the methodology is DSM. I suggest revising the article to make these terms compatible.

A: As you said, the dataset used to test the methodology was a DSM. I selected it because I wanted to test the method using buildings. However, the methodology does not limit the type of element to be used. In this sense, we could use any element included in any DEM. The only premise is that the elements must be clearly defined in both XY and height.

R: >> The abstract cites 4 results, but the correct would be 3 (2D, Z and 3D)

A: I agree with you. We obtained four partial results but actually, we obtained 3 accuracy values (2D, vertical and 3D). I have changed this aspect to clarify.

R: 1. Introduction

>> Page 1 - lines 21 and 23: Uses ISO 19.113. This standard has been replaced by ISO 19157, which contains 6 quality elements.

A: I have replaced this reference to ISO 19157. However, the document I have consulted describes 5 data quality elements:

• completeness – presence and absence of features, their attributes and relationships
• logical consistency – degree of adherence to logical rules of data structure, attribution and relationships (data structure can be conceptual, logical or physical)
• spatial accuracy – accuracy of the position of features in relation to Earth
• temporal accuracy – accuracy of the temporal attributes and temporal relationships of features
• thematic accuracy – accuracy of quantitative attributes and the correctness of non-quantitative attributes and of the classifications of features and their relationships

R: >> It uses the terms “true values” and “true position”. In Geospatial Data, the true position is not known. The movement of the Earth and its tectonic plates is dynamic, and positioning instruments having uncertainties in the survey process. A better term would be “reference” rather than “true”.

A: I have included the term “reference” in this sentence.

R: >> Page 1 – lines36 and 37: “As an example, the ratio between accuracies of reference and the product to be assessed should be at least three [3]”. I agree with this value for medium and large scale cartographic data. What about for large scale map (1:1,000, 1:500)? Is this value still feasible?

A: Positional accuracy assessment methods generally indicate that accuracy should be at least three-times higher but I cannot find any specific reference related to large-scale maps.

R: >> Page 2 – line 51: Name the institutions.

A: See page 36 of [11]. For example ISPRS: (1+7.5 tan α) at scale 1:50,000.

R: >> Page 2 – Lines 65 and 66: “Positional uncertainty can be considered as an area (2D) or 65 volume (3D) surrounding each entity where the most probable position is located”. It would be interesting to have a figure to illustrate.

A: Thank you for your suggestion but I must reduce the introduction section following the comments of other reviewers.

R: >>Page 3 – line 113: Reference authors who mention the term 2.5D.

A: I cannot find any reference about the first use and definition of this term. This aspect is commonly assumed by the scientific community. As an example, it is described by the ArcGIS software website: https://desktop.arcgis.com/en/arcmap/latest/extensions/3d-analyst/what-is-a-functional-surface-.htm  

R: >> Page 3 – lines 127 e 129: These articles cover this subject ( https://pubs.er.usgs.gov/publication/70005960)

A: Thank you but this subject was already referenced [39].

R: 2. Methodology

>> Page 4 – lines 184-188: The author must reference the text. There are articles that address this (https://doi.org/10.1590/s1982-21702020000200007 )

A: I have included this reference [43].

R: >> Page 5. Figure 1: Instead of using 3 boxes (DEM, DTM and DSM elements) you could use only 1 (DSM). The DSM involves the entire terrain surface plus vegetation and buildings. The article uses only the elevation of the buildings. When doing a subtraction of the DEM with the DTM, only vegetation and buildings would remain. Wouldn't it have to be another nomenclature, instead of the DSM?

It does not address, or is unclear, the sampling process in the flowchart in Figure 1.

The Maching box should be formed from the 3D vector and the DSM, generating the selection of features and cells. The text has to be clearer.

A: I have explained this aspect previously. The methodology covers a general procedure that includes any DEM regardless of the application carried out in this study. I have changed the inputs pointing to the “maching” box taking into account your suggestion. Thank you.

R: >> Page 5. - lines 214: “the same Coordinate Reference System (CRS)”. Explain in the text the altimetric reference. It is important to mention that the use of different (almost) geoid models, for the conversion of altimetric references, can bring uncertainties in the process of evaluating the positional accuracy. This is very important and should be addressed in the text of the article.

A: I agree with you. I have included that all data should be referred to the same height reference to highlight this important aspect.

R: >> Page 6 – lines259-273: quotes 3D triangulation based on 3D polygons. However, in item 3 of the article a DSM is used as a reference. How is triangulation performed on a DSM grid? Why not use Delaunay's algorithm for triangulation?

A: Triangulation is always performed to polygons, regardless of the dataset used as reference (see Figure 1).

I agree with you. I suggest the use of Delaunay triangulation algorithm.

R: >> Page 6 - Figure 2.d: This part left me confused. In case the 3D polygon is a reference, why do the triangulation if the data is already 3D and has the surface created? How would it be if the DSM was the reference? Would you triangulate based on the centroid of the cells?

A: This issue is commented previously. Triangulation is always performed on polygons to create check-surfaces.

R: >> Figure 2.e and 2.f: the illustration could be improved by including specific features related to altimetry.

A: I do not understand your comment. Please, clarify it.

R: 3. Application

>> Page 7 – lines 305-311: The reference data does not meet the level of 3x more accurate than the evaluated data. What's the justification? Is this threshold appropriate for large scale mapping?

A: The application developed in this study is focused on testing the proposed methodology. With this goal in mind, I selected a reference that meets the requirement of being more accurate than the dataset to be assessed (in this case is 2x). The level of being 3x more accurate is a recommendation for assess spatial databases. In this study, I am not trying to assess the BCA10 database. As you indicated, this assessment should involve, among other aspects, the development of a sampling procedure and the use of reference data that meet this level.  

R: >>Page 8 – lines 316 and 317: The author should give more details about the origin of the altimetric reference. What is the original reference of each data (test and reference)? Was there an altitude conversion? Which geoid models were used? How can this conversion propagate uncertainties in the evaluated data?

A: Both sources are referred to the official system used in Spain.

R: >> Figure 3: 3.a and 3.b must have legends to inform the symbologies. Figure 3.a must be presented as a map, containing legend, coordinates grid and cartographic information. Figure 3.b is confusing without the legend.

A: Done.

R: >> Page 9 – line 322: What is the total area (km²) of the data region (Andalucía)? Cite in text.

A: Done.

R: >> Page 9 – line 323 and 324: Review area values. 7.3km x 4.7km = 34.31 km². 34 sheets by 34.31 = 1166.54 km². In Table 1 these calculations are not equal.

A: Done.

R: >> Page 9 – line 325: What aspects? Cite in the text.

A: They are indicated in the next sentence.

R: >> Page 9: The sampling process needs to be made clearer in the text. How did you arrive at the number of 34 cards used?

A: The goal of this study is not related to perform an assessment of the BCA10 database. As I mentioned previously, the application carried out only tries to test the proposed methodology. In this sense, I selected 34 sheets in order to analyze what happens in a large area considering specific cases. I have changed all references related to the sampling process in order to avoid confusion.

R: >> Page 9 – line 330: Zone is each sheet selected in the sampling process?

A: Yes, see line 323.

R: >> Page 9 – line 334: The evaluation carried out in the article encompasses the vertical and 3D component, so when using the ASPRS standard it should be 25 points, instead of 20.

A: Sorry, but I cannot find this number in the ASPRS standard documentation. In [8] Table 7 indicates 20 points. Can you clarify your comment? Thank you in advance.

R: >> Page 9 - lines 337-339: In section 2 you recommended to use automatic matching methodologies. Why not use this automatic methodology? Was the matching performed visually or manually?

A: In my case, the development of an automatic methodology was not possible when I developed this application. Thus, I performed a manual matching between both datasets supported by a previous automatic selection of cells located close to polygons (using a buffer of 10 meters).

R: >> Page 9 – line 344: The numbers do not match. In the text it says 1.8 and in Table 1 the value corresponds to 1.79.

A: I used one decimal place both in the text and in Table 1. Therefore, 1793764 square meters is approximately 1.8 square kilometers.

R: >> Table 1: Review the values in the “Average (by zone)” column. Use 1 decimal place.

A: Done.

R: >> Page 9 – line 349: What is the name of the software? Is this software proprietary or freeware?

A: It is a specific software developed for this purpose.

R: 4. Results

>> Positional accuracy results of the tested product (analyzing Figure 4.a):

• 2D: 2.25m (95%)
• Z: 8m (80%) ** Does not cross at 95% limit.
• 3D: 6m (95%)

These values should be highlighted in the text of section 4, 5 and 6.

These values are compatible with the 1:10,000 scale of the evaluated product. What is the final quality of the product?

Considering that 3D² = 2D² + Z², shouldn't the accuracies of the developed methodology follow this pattern?

If we calculate the 3D accuracy above, using the 2D and Z variables, we would arrive at a value of 8.3m for the 3D accuracy. But the methodology presented the value is 6m.

How does the author justify these results and the consistency of the 3D accuracy values? It is an important part of the article that should be analyzed and discussed in greater depth.

A: The result of the height accuracy is not 8 meters but 5.6 meters. Those percentage values of BHd shown in Figure 4a are related to the total number of cells. However, we must take into account that those cells located planimetrically outside the polygon are not considered when calculating BHd. Therefore, the percentages should be related to the number of cells included in the polygons (2DInc). This is shown in Figure 4b and clarify in text. I have included a final sentence to summarize the accuracy results: 2.25, 5.6 and 6 meters. Therefore, If we consider 2.25 and 5.6 meters (2D and height accuracies), we would arrive at a value of 6.0 meters for the 3D accuracy (following 3D² = 2D² + Z²). This is the same value that I have obtained.   

R: >> Page 11 – line 387: Talks about several aspects but mention only two: area and slope.

A: Ok, I have changed to “two aspects”.

R: >> Figure 5: The title of each graph has abbreviations different from those used in the text (2Dinc, 2DBd, HBd, 3DBd).

A: Done.

R: >> Page 12 – line 398, 412, 414: correct the acronym for HBd.

A: Done.

R: >> Figure 6 and 7: cite the figure in the text before presenting it.

A: Done.

R: >> Page 12 – line 419: Did you do any statistical test to say that the zones are representative of the region?

A: I have selected them considering their statistics about number of polygons and average slope.

R: >> Page 13 – lines 429-436: The author cannot conclude about the entire product based on these 4 zones. The analysis has to be for the dataset as a whole.

A: This conclusion only focuses on the selected sheets, but confirms the results shown previously for the whole dataset (shown in Figure 5).

R: 5. Discussion

>> Page 14 - lines 455-464: The author sets the vertical accuracy result equal to the 3D accuracy result. As per Figure 4a and comments made by this reviewer in the previous section, the results are not the same. Furthermore, it has an inconsistency with the 3D accuracy value. The author needs to give explanations in the text.

A: Sorry, This was a mistake. I have changed these values considering the results commented previously. Thank you.

R: >> Page 14 - lines 466-474:: The author cannot conclude about the entire product based on the 4 zones analysed. The text has to be explicit that the results achieved are for the data used in the research. Other data sets may show different results.

A: This comment is related to the results shown in Figure 5 for the whole dataset (34 sheets). The analysis of specific cases (4 zones) was developed to confirm the hypotheses raised previously.

R: 6. Conclusions

>> The proposed methodology is indicated only for large-scale geospatial data, since the use of building roofs is the checking surface. For medium and low scale data could this methodology be applied? Address this analysis in the text.

A: We can apply this methodology to any dataset where the elements involved in the assessment have approximately the same level of detail as the reference.

Thank you for your suggestions and comments. I think the manuscript has improved after this revision.

Reviewer 3 Report

In this paper the author deals with an interesting and important subject: the assessment of positional accuracy of Digital Elevation Models (DEMs) and 3D vector features, using methods that compare data (absolute or relative positions) with respect to other independent datasets that are considered as true values. Indeed, positional accuracy has been studied and assessed by many authors and standards have been proposed. These methods are based on data comparison with a set of checkpoints extracted from the dataset to be assessed and the homologous elements obtained from an independent and more accurate data source. When it comes to 3D data, most studies and standards distinguish horizontal and vertical accuracies which are assessed independently. The author suggests that these horizontal and vertical accuracies should also be considered jointly. The assessment of the positional accuracy of DEMs has been traditionally undertaken using checkpoints. However, points are usually not well defined in DEMs. Recent approaches have been proposed to facilitate the identification and positioning of points to be used in positional assessment; as for example, plane intersections to assess LiDAR data. In this context, the author suggests to take advantage of using surfaces to assess the positional accuracy of a DEM rather than checkpoints: he proposes a new method to assess DEMs using surfaces from a 3D vector source and vice versa. The proposed approach considers horizontal and vertical accuracies independently and jointly and is based on 3D uncertainty models. The method is presented in Section 2, each step being carefully described and justified; including selection of features and identification of polygons, selection of cells (in the case of a DEM based on a grid model), identification of buffers (external 2D and 3D as well as height buffers), computation of metrics (percentage of inclusion: 2DInc, 2DBd, HBd, 3dBd). These metrics consider the inclusion of cell positions within polygons (2DInc) or buffers (2DBd, HBd and 3DBd). In Section 3, the author presents the application of the proposed method to two data sources obtained from Spanish institutions. He provides detailed information about these sources and indicates which pre-processing steps need to be carried out. Section 4 presents the results and discusses the interest of using each of the above-mentioned metrics. Other contextual issues are also considered such as the analysis of data from urban and rural areas, as well as the terrain characteristics (average slope values) and the influence of buildings with different footprints. All in all, a detailed analysis of the application of the proposed method!

In Section 5 the author discusses these results and concludes that this study demonstrates the feasibility of the proposed approach and the interest of using it to assess both the horizontal and vertical accuracies independently and jointly. This approach generates a wide variety of results in the form of distribution functions describing the 2D accuracy, the vertical accuracy and the 3D accuracy of data. Some limitations are also discussed, but they are mainly related to the quality of the selected reference data and do not impair the proposed method. Section 6 concludes the paper, mentioning some future work that would show how lower sample sizes are used for the data accuracy comparisons with the proposed method. Finally, the author evokes a large number of application domains for his method.

Conclusion:

The paper is well written and contains all the necessary information to understand the proposed method and its application. The problem is well defined in relation to a good literature review. I appreciated the clear presentation and justification of the method, as well as the presentation and discussion of the results.

One issue might be raised about the paper: it is the length of the introduction. Indeed, in the current version, the reader has to read until page 4 to understand what are the objectives of this work and the content of the paper. In fact, Section 1 merges the introduction and the literature review that presents the scientific and technical backgrounds to understand the objectives of the study. The author might consider reorganizng the current Section 1 by writing a shorter and more synthetic introduction followed by a section containing the technical background material and literature review.

DETAILED COMMENTS:

PAGE 2 CHANGE this band model is modified BY this band model has been modified

PAGE 4 CHANGE. In the other direction, BY However,

PAGE 4 CHANGE can suppose an improvement in identification BY can improve the identification

PAGE 5 CHANGE Firstly, there an initial BY Firstly, there is an initial

PAGE 5 CHANGE the use of an DTM obtained BY the use of a DTM obtained

PAGE 6 CHANGE As result, a distribution function BY As a result, a distribution function

PAGE 6 CHANGE this methodology supposes the obtaining of four BY this methodology supposes that we can obtain four

PAGE 6 CHANGE be considered because they allow us BY be considered because this allows us

PAGE 9 CHANGE using Serval raster editing tools BY using several raster editing tools

PAGE 11 CHANGE this division is to analysis cases BY this division is to analyse cases

PAGE 12 CHANGE these polygons shows lower percentages BY these polygons show lower percentages

PAGE 14 CHANGE The obtaining of distribution functions BY Obtaining distribution functions

PAGE 16 REFERENCE 15 CHANGE Bulletin de I’ Academic Polonaise des Sciences BY Bulletin de I’Académie Polonaise des Sciences

Author Response

Reviewer #3:

Author: Thank you for your suggestions and comments. I have rewritten the paper following your suggestions.

R: In this paper the author deals with an interesting and important subject: the assessment of positional accuracy of Digital Elevation Models (DEMs) and 3D vector features, using methods that compare data (absolute or relative positions) with respect to other independent datasets that are considered as true values. Indeed, positional accuracy has been studied and assessed by many authors and standards have been proposed. These methods are based on data comparison with a set of checkpoints extracted from the dataset to be assessed and the homologous elements obtained from an independent and more accurate data source. When it comes to 3D data, most studies and standards distinguish horizontal and vertical accuracies which are assessed independently. The author suggests that these horizontal and vertical accuracies should also be considered jointly. The assessment of the positional accuracy of DEMs has been traditionally undertaken using checkpoints. However, points are usually not well defined in DEMs. Recent approaches have been proposed to facilitate the identification and positioning of points to be used in positional assessment; as for example, plane intersections to assess LiDAR data. In this context, the author suggests to take advantage of using surfaces to assess the positional accuracy of a DEM rather than checkpoints: he proposes a new method to assess DEMs using surfaces from a 3D vector source and vice versa. The proposed approach considers horizontal and vertical accuracies independently and jointly and is based on 3D uncertainty models. The method is presented in Section 2, each step being carefully described and justified; including selection of features and identification of polygons, selection of cells (in the case of a DEM based on a grid model), identification of buffers (external 2D and 3D as well as height buffers), computation of metrics (percentage of inclusion: 2DInc, 2DBd, HBd, 3dBd). These metrics consider the inclusion of cell positions within polygons (2DInc) or buffers (2DBd, HBd and 3DBd). In Section 3, the author presents the application of the proposed method to two data sources obtained from Spanish institutions. He provides detailed information about these sources and indicates which pre-processing steps need to be carried out. Section 4 presents the results and discusses the interest of using each of the above-mentioned metrics. Other contextual issues are also considered such as the analysis of data from urban and rural areas, as well as the terrain characteristics (average slope values) and the influence of buildings with different footprints. All in all, a detailed analysis of the application of the proposed method!

In Section 5 the author discusses these results and concludes that this study demonstrates the feasibility of the proposed approach and the interest of using it to assess both the horizontal and vertical accuracies independently and jointly. This approach generates a wide variety of results in the form of distribution functions describing the 2D accuracy, the vertical accuracy and the 3D accuracy of data. Some limitations are also discussed, but they are mainly related to the quality of the selected reference data and do not impair the proposed method. Section 6 concludes the paper, mentioning some future work that would show how lower sample sizes are used for the data accuracy comparisons with the proposed method. Finally, the author evokes a large number of application domains for his method.

Conclusion:

The paper is well written and contains all the necessary information to understand the proposed method and its application. The problem is well defined in relation to a good literature review. I appreciated the clear presentation and justification of the method, as well as the presentation and discussion of the results.

One issue might be raised about the paper: it is the length of the introduction. Indeed, in the current version, the reader has to read until page 4 to understand what are the objectives of this work and the content of the paper. In fact, Section 1 merges the introduction and the literature review that presents the scientific and technical backgrounds to understand the objectives of the study. The author might consider reorganizng the current Section 1 by writing a shorter and more synthetic introduction followed by a section containing the technical background material and literature review.

A: I have reduced the length of the introduction following your suggestion.

R: DETAILED COMMENTS:

PAGE 2 CHANGE this band model is modified BY this band model has been modified

A: Done.

R: PAGE 4 CHANGE. In the other direction, BY However,

A: Done.

R: PAGE 4 CHANGE can suppose an improvement in identification BY can improve the identification

A: Done.

R: PAGE 5 CHANGE Firstly, there an initial BY Firstly, there is an initial

A: Done.

R: PAGE 5 CHANGE the use of an DTM obtained BY the use of a DTM obtained

A: Done.

R: PAGE 6 CHANGE As result, a distribution function BY As a result, a distribution function

A: Done.

R: PAGE 6 CHANGE this methodology supposes the obtaining of four BY this methodology supposes that we can obtain four

A: Done.

R: PAGE 6 CHANGE be considered because they allow us BY be considered because this allows us

A: Done.

R: PAGE 9 CHANGE using Serval raster editing tools BY using several raster editing tools

A: Done.

R: PAGE 11 CHANGE this division is to analysis cases BY this division is to analyse cases

A: Done.

R: PAGE 12 CHANGE these polygons shows lower percentages BY these polygons show lower percentages

A: Done.

R: PAGE 14 CHANGE The obtaining of distribution functions BY Obtaining distribution functions

A: Done.

R: PAGE 16 REFERENCE 15 CHANGE Bulletin de I’ Academic Polonaise des Sciences BY Bulletin de I’Académie Polonaise des Sciences

A: Done.

Thank you for your suggestions and comments. I think the manuscript has improved after this revision.

Reviewer 4 Report

The paper proposes a method to assess the positional accuracy of DEMs and 3D vector databases using what it calls “check surfaces”. While the topic of the paper is significant and interesting, the authors must consider revising the manuscript significantly before it can be published in IJGI.

First, the abstract seems insufficient to describe the goal of the paper. Briefly and concisely, it must attract readers' attention while sufficiently discussing the paper's significance, objectives, and results. Similarly, the introduction section lacks sufficient discussion in the motivation and background of the study. After a single paragraph, it jumps straight into discussing existing methods for measuring positional accuracy. However, this discussion may be better placed in a separate section, along with the extensive discussion on related studies on the topic- which this manuscript currently lacks. Additionally, the paper does not sufficiently discuss the significance or the reason for pursuing its objectives, which are not also clearly discussed.

The title mentions the use of “check-surfaces” as the main method of accuracy assessment in this study. This term must be formally defined and discussed first to establish its significance in the paper’s methodology.

This paper also seems to use 3D vector data(dataset), and 3D vector database. The study does not seem to discuss accuracy assessment of data in the context of it being contained or maintained in a database, so the authors must re-examine this.

The paper claims to propose a method for accuracy assessment for both DEMs and 3D vector (data, or database? See previous comment), through a raster-vector comparison but the methodology and the experiment presented demonstrates only checking accuracy of a 3D vector versus a raster (DEM) taken as the reference value, and not vice versa. Also, if the DEM is based on a TIN (as mentioned in line 206) wouldn’t this be a vector-to-vector comparison and not a raster-vector comparison?

The selection procedure and the matching procedure are not sufficiently discussed in the methodology. The concept of uncertainty band must also be introduced and well-established in previous sections before it springs up in the methodology.

In the 3D triangulation process, selection of the third side presents an ambiguity in this process. The generation of the third side of the triangle depends on which second side is selected relative to the seed polygon segment. Even if the restriction of the third side being contained inside the polygon is present, there will still be two possibilities of triangulations. The authors are encouraged to look into Delaunay Triangulation for this concern.

In Chapter 4 (Results), the values for each parameter at each buffer distance (say, in Figure 4) is tabulated as a single value for the entire study area. How this single value is obtained for the numerous buildings is unclear. If this is an average value, the variability for each data point must be illustrated in the figure to account for possible outliers.

The categorization of the data in Figure 5 does not have a clear basis. For example, if Table 1 illustrates an average slope of 10%, wouldn’t the categorization of the slope values skew the distribution of each category? The categories are vague and possibly skewed.

Overall the process lacks statistical analysis in terms of 1) whether the absolute values in Figure 4 calculated for the parameters are significant, or the differences between the calculated parameters for each category made in Figure 5 significant. The paper claims too (in line 394) that rural areas have lower values of average areas for building features and urban areas have large blocks of buildings. What is the basis of this claim? Intuitively, more urbanized and densely populated areas have smaller building footprints because of the scarcity of bare land.

Moreover, the paper presents hypotheses (such as in Line 404: “As a hypothesis, we consider that this issue is related to the presence of more complexity in building structures in urban landscapes.”), which have no relation whatsoever to the objective of the paper and cannot be verified by the method this paper presents. The claim made in Line 415 of slope having a “great influence” has no statistical basis within the manuscript.

While the manner of sampling presented in Figure 6 of contrasting urban-rural and flat-undulating terrain is interesting, the categorization between urban and rural is unfounded, and the analysis in Figure 6 does not provide additional insight relative to the paper’s objectives. Figure 7 also does not illustrate well what it aims to, nor does it provide additional evidence regarding the objective.

The paper’s main point in the discussion is the feasibility (whether the process can be carried out) of its presented parameters as a method to assess the accuracy when comparing vector and raster data. However, in accuracy assessment, it is rather more important is its reliability (whether an assessment tool can consistently provide results) and its validity (whether it measures what it is supposed to measure). The paper does not provide these in its current form.

 

The paper lacks organization and structure. While there are minimal grammar errors, punctuation errors, and spelling errors, the structure of the manuscript is difficult to read. In majority of the sections, the author goes back and forth between one topic within one paragraph. The inclement use of acronyms in the text and in the figures makes it also difficult to understand the discussion. For example, some acronyms (2DInc, 2DBD, etc.) are introduced by the author and not yet widely used in the scientific community, so the reader might find themselves revisiting their meaning in previous pages to understand the meaning of statements further in the paper. Overall, the authors must consider re-organizing the paper for it to be understood easier by readers.

Author Response

Reviewer #4:

Author: Thank you for your suggestions and comments. I have rewritten the paper following your suggestions.

R: The paper proposes a method to assess the positional accuracy of DEMs and 3D vector databases using what it calls “check surfaces”. While the topic of the paper is significant and interesting, the authors must consider revising the manuscript significantly before it can be published in IJGI.

First, the abstract seems insufficient to describe the goal of the paper. Briefly and concisely, it must attract readers' attention while sufficiently discussing the paper's significance, objectives, and results. Similarly, the introduction section lacks sufficient discussion in the motivation and background of the study. After a single paragraph, it jumps straight into discussing existing methods for measuring positional accuracy. However, this discussion may be better placed in a separate section, along with the extensive discussion on related studies on the topic- which this manuscript currently lacks. Additionally, the paper does not sufficiently discuss the significance or the reason for pursuing its objectives, which are not also clearly discussed.

A: I have improved the manuscript following your suggestions. For example, I have changed the abstract including the main goal and other aspects about the results.

R: The title mentions the use of “check-surfaces” as the main method of accuracy assessment in this study. This term must be formally defined and discussed first to establish its significance in the paper’s methodology.

A: I have included an explanation of this term.

R: This paper also seems to use 3D vector data(dataset), and 3D vector database. The study does not seem to discuss accuracy assessment of data in the context of it being contained or maintained in a database, so the authors must re-examine this.

A: I agree with you. I consider a database as a collection of datasets. In this sense, I have changed the term “database” to “dataset” both in the title and in the rest of the manuscript.

R: The paper claims to propose a method for accuracy assessment for both DEMs and 3D vector (data, or database? See previous comment), through a raster-vector comparison but the methodology and the experiment presented demonstrates only checking accuracy of a 3D vector versus a raster (DEM) taken as the reference value, and not vice versa. Also, if the DEM is based on a TIN (as mentioned in line 206) wouldn’t this be a vector-to-vector comparison and not a raster-vector comparison?

A: In this study I have analyzed the case of gridded DEMs. In the case of TIN, the adaptation of the methodology should be easy by using vertexes instead of cells. However, I have not included it in this study because of the large differences related to the matching procedure between polygons and TIN vertexes. In this sense, I have proposed the adaptation of this approach to include TIN models for future work.

R: The selection procedure and the matching procedure are not sufficiently discussed in the methodology. The concept of uncertainty band must also be introduced and well-established in previous sections before it springs up in the methodology.

A: I have improved this part of the manuscript. Thank you.

R: In the 3D triangulation process, selection of the third side presents an ambiguity in this process. The generation of the third side of the triangle depends on which second side is selected relative to the seed polygon segment. Even if the restriction of the third side being contained inside the polygon is present, there will still be two possibilities of triangulations. The authors are encouraged to look into Delaunay Triangulation for this concern.

A: I have changed the part related to the 3D triangulation process. Following your suggestion, I propose to use the Delaunay triangulation algorithm to obtain the check-surfaces. Thank you.

R: In Chapter 4 (Results), the values for each parameter at each buffer distance (say, in Figure 4) is tabulated as a single value for the entire study area. How this single value is obtained for the numerous buildings is unclear. If this is an average value, the variability for each data point must be illustrated in the figure to account for possible outliers.

A: In this case, I have adapted the methodology described by [30]. Considering this approach, I have checked the results and eliminated the outliers.

R: The categorization of the data in Figure 5 does not have a clear basis. For example, if Table 1 illustrates an average slope of 10%, wouldn’t the categorization of the slope values skew the distribution of each category? The categories are vague and possibly skewed.

A: The categorization of slopes is related to the commonly used classification of the terrain (flat, slightly undulated, undulated-mountainous) without considering the average slope.

R: Overall the process lacks statistical analysis in terms of 1) whether the absolute values in Figure 4 calculated for the parameters are significant, or the differences between the calculated parameters for each category made in Figure 5 significant. The paper claims too (in line 394) that rural areas have lower values of average areas for building features and urban areas have large blocks of buildings. What is the basis of this claim? Intuitively, more urbanized and densely populated areas have smaller building footprints because of the scarcity of bare land.

A: The paper describes the final results in order not to complicate its reading. The statistical analysis of the intermediate results followed the method described by [30].

In the case of rural areas, buildings are mainly composed of family houses, which are characterized by small footprints. In contrast, buildings in urban areas are composed of large blocks and, as consequence, show larger footprints. 

R: Moreover, the paper presents hypotheses (such as in Line 404: “As a hypothesis, we consider that this issue is related to the presence of more complexity in building structures in urban landscapes.”), which have no relation whatsoever to the objective of the paper and cannot be verified by the method this paper presents. The claim made in Line 415 of slope having a “great influence” has no statistical basis within the manuscript.

A: I have included the analysis of the level of detail of (LoD) of the internal structure representing the element in the objectives section. I have changed the claim made in line 415 following your suggestion.

R: While the manner of sampling presented in Figure 6 of contrasting urban-rural and flat-undulating terrain is interesting, the categorization between urban and rural is unfounded, and the analysis in Figure 6 does not provide additional insight relative to the paper’s objectives. Figure 7 also does not illustrate well what it aims to, nor does it provide additional evidence regarding the objective.

A: I have improved this part by including statistical data to explain the categorization carried out to classify into urban and rural areas.

R: The paper’s main point in the discussion is the feasibility (whether the process can be carried out) of its presented parameters as a method to assess the accuracy when comparing vector and raster data. However, in accuracy assessment, it is rather more important is its reliability (whether an assessment tool can consistently provide results) and its validity (whether it measures what it is supposed to measure). The paper does not provide these in its current form.

A: I have included these aspects in the conclusions section.

R: The paper lacks organization and structure. While there are minimal grammar errors, punctuation errors, and spelling errors, the structure of the manuscript is difficult to read. In majority of the sections, the author goes back and forth between one topic within one paragraph. The inclement use of acronyms in the text and in the figures makes it also difficult to understand the discussion. For example, some acronyms (2DInc, 2DBD, etc.) are introduced by the author and not yet widely used in the scientific community, so the reader might find themselves revisiting their meaning in previous pages to understand the meaning of statements further in the paper. Overall, the authors must consider re-organizing the paper for it to be understood easier by readers.

A: I have improved the manuscript following your suggestions. The method only introduces four acronyms (2DINC, 2DBd, HBd and 3DBd), which follow a simple rule that I think is clear:

• 2DInc -> 2D Inclusion
• 2DBd -> 2D Buffer of distance “d”
• HBd -> Height Buffer of distance “d”
• 3DBd -> 3D Buffer of distance “d”

When the distance “d” is specified (case of Figure 7), this letter is changed by the value used (e.g. “(3m)”).

Thank you for your suggestions and comments. I think the manuscript has improved after this revision.

Round 2

Reviewer 1 Report

The paper entitled "Positional Accuracy assessment of Digital elevation models..." presents an interesting and valuable proposal.

 

The document has been improved with the revision. Fixed bugs and clarified confusing aspects. All of this now allows a better reading of the document and a better understanding of the proposal being made and its significance. However, the document still has weaknesses that need to be corrected.

 

The structure is adequate.

 

Contents and development

 

Section 1.1. It focuses on the positional accuracy of vector databases. This section is a review of techniques that are basically 2D. Well, it is important to review the buffering techniques since they are going to be applied in the proposal; however, a review is missing that presents cases related to 3D positional accuracy.

 

Section 1.2. Following your publications I have found a communication from you in the following congress: https://www.age-geografia.es/site/wp-content/uploads/2022/11/Actas-XIX-Congreso-de-TIG_Zaragoza-2022-1.pdf. I understand that this work is not exactly the same, but the idea of the control surfaces is there. This paper should be indicated as a reference in this section.

 

Section 1.3. You added "we also must consider the level of detail of the internal structure...". In your study, for your dataset, you do not have adequate information (LoD) and therefore you encounter certain problems derived from the dataset used. In the conclusions you rightly indicate that LoD is important. I consider that, in the objectives, you cannot talk about LoD since they do not really appear in the development of your methodology. But there is something missing in your objectives and in your methodology: You carry out (in section 4), the analysis of the influence of the density of elements and the slope. This is an interesting analysis. The problem is that this analysis has not been indicated in the objectives, nor in the methodological proposal. That is, they appear unexpectedly in section 4. This is the main flaw in your work and must be corrected.

 

Section 2.

Line 267. Be careful and qualify the sentence. A larger sample made up of correlated data... should take this into account: the autocorrelation.

Line 279: More accurate but also “independent”.

 

Please, include in this section the presentation of the method of analysis related to density and slope. I believe that figure 1 does not have to be modified, you can distinguish between a general methodology (Figure 1) and a method for these two specific analyzes (add another figure if you consider it).

 

I understand that the entire calculation process described between lines 329 and 357 is based on considering the center of the cells (point-in-polygon algorithm line 330). Please clarify this. I understand that for the development of the method it is suitable. But if your work is based on the center of the cell, you could propose in your conclusions a line of improvement based on using the entire cell. In this case we are talking about voxels (3D cells).

 

Section 5

In the previous review I told you:” The conclusion section is very long. Some of its content is a summary, this is not a conclusion. Another part of its content is more typical of the discussion section. A true conclusion must be made about the contribution” But, the conclusions section has changed very little. You have just removed some content. Please redo the discussion and conclusions. There are many aspects that can be indicated in the discussion: method, data, results, implementation, the interest of each of the proposed parameters (2DbD, etc.), comparison with other proposals, etc.

 

In conclusion, the document requires relevant changes and therefore I suggest a major review.

 

 

Reviewer 4 Report

The authors responded to the previous comments and improved the manuscript. But there are still several fatal problems, and I don't think the current version of the paper will be published in IJGI. I can see the positive efforts of the author, but most problems pointed out in the first round of review is still unresolved.