Applying a Robust Empirical Method for Comparing Repeated LiDAR Data with Different Point Density

Round 1

Reviewer 1 Report

This manuscript used a method to compare LiDAR data with different point density.

The biggest concern is the novelty should be highlighted. For example, it is said ‘applying a method’ in the title, while the conclusion part indicates ‘our approach is novel’. It is unclear whether the author used an existing method or proposed a new method. In addition, the author compared several factors that impact on vertical errors of DEM and chose the best method. This makes the manuscript more like an experimental report rather than research paper. It is fine to use the current method, however, I suggest the author first highlights the proposed new framework/flowchart, and then clarifies that the specific methods included in the framework/flowchart. This would bring the novelty of this manuscript to a higher level.

Another concern also relates to the previous one is whether the best method is universal. If it is, then the author must clearly explain theoretically the advantages of choosing the method over others, rather than only by comparison. If not, the proposed framework/flowchart should be highlighted and similar methods can be replaced in different environments should be stated.

The upper image in Figure 1 should use a regional map which can show distributions of 6 sites.

Line 238-239, grammar error.

Line 299, what does it mean by ‘site (site)’?

Line 327-328, confusing expression. How can RMSE overstated the vertical error and P50 underestimated it?

Line 662, ‘Although …’ is not a complete sentence.

Author Response

REFEREE 1

Comments and Suggestions for Authors

This manuscript used a method to compare LiDAR data with different point density.

The biggest concern is the novelty should be highlighted. For example, it is said ‘applying a method’ in the title, while the conclusion part indicates ‘our approach is novel’. It is unclear whether the author used an existing method or proposed a new method.

Thank you for your advice. I have included some sentences in the introduction as well as in both the discussion and conclusion sections indicating that the proposed method (DEMs of difference) is a robust and standard approach to estimate vertical errors in DEMs and, it is novel in this manuscript because it was applied to correct raw elevation data using the entire interpolated elevation differences as a “local pseudo-geoide”, instead of only a collection of checkpoints, as it was applied in previous works. In addition, I highlighted the importance of using “the best DEM of difference” approach to correct more accurately the target elevations in each environment; mainly because it allowed choosing the best methods (i.e., classification filter, interpolation method and resolution) for capturing the micro-topography of each site. Using all interpolated points from the best DEM of difference, random or methodological errors that would be very difficult to correct by using only checkpoints were significantly reduced.

INTRODUCTION:

“This method has been robust to estimate vertical errors but only in few times, it has been used to correct raw elevation values [12,13]. In the latter cases, elevation differences were corrected by means regression analysis using several checkpoints but those corrected DEMs continued being affected by the random nature of elevation errors. Using DEMs of difference as local pseudo-geoide (i.e., all interpolated points with elevation deviations) allow adjusting less accurate elevation data (e.g. low-density LiDAR) by means of continuous elevation surfaces instead of only checkpoints. This procedure reduces random and other methodological elevation errors increasing the comparability between DEMs and derived vegetation products.”

“The aim of this study was to apply an improved empirical method (i.e., DEMs of difference) to make comparable multitemporal LiDAR datasets collected from different sensors with different point density. The improvement consisted of using continuous surfaces of elevation differences, that worked as a local and dense "pseudo-geoid", instead of a collection of checkpoints, for adjusting elevations of low-density LiDAR to a high-density LiDAR benchmark. Moreover, we propose to use “the best DEM of difference” (with the lowest vertical error) obtained from comparing the vertical errors raised from different methodologies (i.e., classification filters, interpolation methods and spatial resolutions) to correct more accurately the elevation of low-density LiDAR in each environment. This approach will allow capturing the micro-topography of each site and will reduce random or methodological errors that would be very difficult to correct by using only checkpoints.”

DISCUSSION:

“Here, the raw elevation of low-density LiDAR was adjusted to the elevation of high-density LiDAR using “the best DEM of difference” as local pseudo-geoide (i.e., all interpolated points). This approach improved the vertical accuracy of the target DEMs, minimized the random nature of vertical errors and decoupled vegetation changes from ground elevation errors.”

CONCLUSIONS:

“Our approach was novel mainly because continuous surfaces of elevation differences, instead of a collection of checkpoints, were used to correct the raw elevation of low-density LiDAR. Moreover, using the “best DEM of difference” approach, based on comparing the vertical errors raised from different methodologies (i.e., classification filters, interpolation methods and spatial resolutions), the elevation of each site was more accurately corrected reducing random or methodological errors that would be very difficult to correct by using only checkpoints.”

In addition, the author compared several factors that impact on vertical errors of DEM and chose the best method. This makes the manuscript more like an experimental report rather than research paper.

I am sorry for differing with this statement. More than half of the references cited in this manuscript came from papers that focused only on assessing the factors (systematic, methodological or site-specific) that affect vertical errors in DEMs. All of them have been published as research papers.

It is fine to use the current method, however, I suggest the author first highlights the proposed new framework/flowchart, and then clarifies that the specific methods included in the framework/flowchart. This would bring the novelty of this manuscript to a higher level.

Thank you for your suggestion. I highlighted the novelty of this work, indicating that the aim was to correct the target elevations using an improved standard procedure (the best DEM of difference). Please, see my previous responses and the sections in which I highlighted these issues.

Another concern also relates to the previous one is whether the best method is universal. If it is, then the author must clearly explain theoretically the advantages of choosing the method over others, rather than only by comparison. If not, the proposed framework/flowchart should be highlighted and similar methods can be replaced in different environments should be stated.

Thank you for your suggestion. In the paper, I have indicated that “the best DEM of difference” approach was very accurate because it was adapted to the site-specific conditions of each environment and, reduced considerably, the effects of elevation errors on vegetation height changes. Moreover, I highlighted that the classification filters based on TIN densification algorithm using default parameters or low spikes (TIN-DEF or TIN-SF), the kriging interpolation method and 5 m of pixel size worked adequately for all sites analysed. However, those best methods were not universal; mainly because none of the classification filters worked perfectly in each environment, and all of them were susceptible to both omission and commission errors. Accordingly, I recommended the possibility of combining different classification filters in the same environment to exploit the strengths of each other. We have highlighted those issues in conclusions section:

CONCLUSIONS:

“Overall, it was observed that the classification filters based on TIN densification algorithm using default parameters or low spikes (TIN-DEF or TIN-SF), the kriging interpolation method and 5 m of pixel size corrected adequately elevation deviations in all sites. After correction, vertical errors dropped drastically reaching low values (from 0.04 m to 0.08 m) using the best DEM of difference in each site. Nevertheless, none of the classification filters worked perfectly in each site, and all of them are susceptible to both omission and commission errors. Accordingly, it is recommended the possibility of combining different classification filters in the same environment to exploit the strengths of each other.”

The upper image in Figure 1 should use a regional map which can show distributions of 6 sites

Thank you for your suggestion. I have used a regional map to assess the distribution of the sites.

Line 238-239, grammar error.

Thank you. Corrected

Line 299, what does it mean by ‘site (site)’?

Thank you for noticing that mistake. It was corrected.

Line 327-328, confusing expression. How can RMSE overstated the vertical error and P50 underestimated it?

Line 662, ‘Although …’ is not a complete sentence.

Thank you for noticing that mistake. It was corrected.

Author Response File: Author Response.docx

Reviewer 2 Report

Dear Authors

    His article: "Applying a robust empirical method for comparing repeated LiDAR data with different point density" very properly explores an important theme: the extraction of forest metrics by combining LiDAR data from two different periods and the errors inherent to this processing. The sources of errors were explored in depth at all stages since the acquisition of the Lidar data, local characteristics (vegetation relief), and associated with filtering, interpolation, and spatial resolution. Setting the distance to the nearest geoid point was very smart to improve the results.
    After a thorough reading of your article and the supplementary file, I would like to make some suggestions:
     1) Would it be possible to present the area values ​​(ha) for each flight?
     2) Line 268: “Finally, we built the Canopy Height Models (CHMs) at 2 and 5 m pixel size”. Better explore the choice of spatial resolution of DEM and CHM relating it to the accuracies obtained and also characteristics of the vegetation.
      3) Line 287: It would be interesting to present the NMAD formula in equation format, the main text part.
      4) Was the semivariogram analysis performed for kriging interpolation? What lag is adopted? Was spatial dependence observed?
       5) What is the kriging method used? What theoretical model was used in the interpolation?
       6) What were the hardware resources used to process the data?

      I conclude my review by congratulating you for all the effort to process the data and for the excellent article presented.

Sincerely,

Comments for author File: Comments.pdf

Author Response

REFEREE 2

Dear Authors

    His article: "Applying a robust empirical method for comparing repeated LiDAR data with different point density" very properly explores an important theme: the extraction of forest metrics by combining LiDAR data from two different periods and the errors inherent to this processing. The sources of errors were explored in depth at all stages since the acquisition of the Lidar data, local characteristics (vegetation relief), and associated with filtering, interpolation, and spatial resolution. Setting the distance to the nearest geoid point was very smart to improve the results.

After a thorough reading of your article and the supplementary file, I would like to make some suggestions:

     1) Would it be possible to present the area values ​​(ha) for each flight?

Of course. I have included this data in Table 1. Nevertheless, that values are given in supplementary material (Table S1).

     2) Line 268: “Finally, we built the Canopy Height Models (CHMs) at 2 and 5 m pixel size”. Better explore the choice of spatial resolution of DEM and CHM relating it to the accuracies obtained and also characteristics of the vegetation.

I have rewritten that sentence in the following way: “Finally, we built the Canopy Height Models (CHMs) selecting the better spatial resolution from the DEM of difference with the lowest vertical errors…”

      3) Line 287: It would be interesting to present the NMAD formula in equation format, the main text part.

Done.

          4) Was the semivariogram analysis performed for kriging interpolation? What lag is adopted? Was spatial dependence observed?

In the kriging interpolation the semivariogram was calculated using the function “vgm” from the gstat package in R software. We used the default parameters, but changed the default kappa smoothness (from 10 to 40), because it resulted in better DEMs

kriging(model = gstat::vgm(0.59, "Sph", 874), k = 40L)

Spherical variogram with sill (0.59), range (874); in case of anisotropy: major range; and Kappa smoothness parameter for the Matern class of variogram models (40). No nugget was included. Default lag was 10. I have not assessed the spatial dependence.

       5) What is the kriging method used? What theoretical model was used in the interpolation?

Spatial interpolation was based on universal kriging using the krige function from gstat (R software). This method combines the KNN approach with the kriging approach. For each point of interest, it kriges the terrain using the k-nearest neighbour ground points. This method is more difficult to manipulate but it is also the most advanced method for interpolating spatial data (LidR package).

 

Pebesma, E.J., 2004. Multivariable geostatistics in S: the gstat package. Computers \& Geosciences, 30: 683-691.

       6) What were the hardware resources used to process the data?

I have used the package lidR and its dependences from gstat package in R software to derive the DEMs

      I conclude my review by congratulating you for all the effort to process the data and for the excellent article presented.

Thank you very much for your positive and fruitful comments,

 

Author Response File: Author Response.docx

Reviewer 3 Report

Comment

This study focused on applying a robust empirical method for comparing repeated LiDAR data with different point density. I agree that this study follows the Special Issue: Laser Scanning of Forest Dynamics. This manuscript is interesting, while some sections might be improved. The special comment was provided as follows.

Special comment

1. Title and Abstract are suitable for this paper. However, audiences might not clearly understand what is robust empirical method? Therefore, I suggested that authors should have more explanations this term in Introduction.
2. The presentation of introduction chapter is suitable. From this chapter, audiences could understand the motivation, background information, and purposes of this study.
3. I suggested that Figure 2 should be improved. If possible, the step 3 to step 6 could put below step 2, not at the right hand.
4. Title of Figure 5 showed “Letters indicated significant differences within groups according to the post-hoc Kruskal-Wallis Dunn test”. It should note significant level, such as α=0.05 or other levels
5. I suggested that Supplementary material did refer in text, such as, Lines, 222, 327, 338, 409, 701. Please check whole text.
6. If Figures 8 and 9 are important result, I suggest moving to Results chapter.
7. Figure 8(b) might consider using “Difference between 2014 and 2019”.
8. Conclusions should briefly reflect the result of this study. However, Conclusions of this study is too long. It might let audiences confuse. Therefore, I suggest that it should be simplified.

Comments for author File: Comments.pdf

Author Response

REFEREE 3

 Comment

This study focused on applying a robust empirical method for comparing repeated LiDAR data with different point density. I agree that this study follows the Special Issue: Laser Scanning of Forest Dynamics. This manuscript is interesting, while some sections might be improved. The special comment was provided as follows.

 

Special comment

1. Title and Abstract are suitable for this paper. However, audiences might not clearly understand what is robust empirical method? Therefore, I suggested that authors should have more explanations this term in Introduction.

Thank you for your comments. I have included some explanations about this term in the introduction and in other sections of the manuscript (see below). I have indicated that proposed method (DEMs of difference) is robust because it has been successfully applied to estimate vertical errors in DEMs and, it is “novel” or “improved” because it was applied to correct raw elevation data using the entire interpolated elevation differences as a “local pseudo-geoide” instead of only a collection of checkpoints, as it was applied in the previous works. Correcting elevation errors by using all interpolated points from the DEMs of difference, the elevation accuracy increased reducing random or methodological errors that would be very difficult to correct by using only checkpoints.

INTRODUCTION:

“This method has been robust to estimate vertical errors but only in few times, it has been used to correct raw elevation values [12,13]. In the latter cases, elevation differences were corrected by means regression analysis using several checkpoints but those corrected DEMs continued being affected by the random nature of elevation errors. Using DEMs of difference as local pseudo-geoide (i.e., all interpolated points with elevation deviations) allow adjusting less accurate elevation data (e.g. low-density LiDAR) by means of continuous elevation surfaces instead of only checkpoints. This procedure reduces random and other methodological elevation errors increasing the comparability between DEMs and derived vegetation products. “

“The aim of this study was to apply an improved empirical method (i.e., DEMs of difference) to make comparable multitemporal LiDAR datasets collected from different sensors with different point density. The improvement consisted of using continuous surfaces of elevation differences, that worked as a local and dense "pseudo-geoid", instead of a collection of checkpoints, for adjusting elevations of low-density LiDAR to a high-density LiDAR benchmark. Moreover, we propose to use “the best DEM of difference” (with the lowest vertical error) obtained from comparing the vertical errors raised from different methodologies (i.e., classification filters, interpolation methods and spatial resolutions) to correct more accurately the elevation of low-density LiDAR in each environment. This approach will allow capturing the micro-topography of each site and will reduce random or methodological errors that would be very difficult to correct by using only checkpoints.”

DISCUSSION:

“Here, the raw elevation of low-density LiDAR was adjusted to the elevation of high-density LiDAR using “the best DEM of difference” as local pseudo-geoide (i.e., all interpolated points). This approach improved the vertical accuracy of the target DEMs, minimized the random nature of vertical errors and decoupled vegetation changes from ground elevation errors.”

CONCLUSIONS:

“Our approach was novel mainly because continuous surfaces of elevation differences, instead of a collection of checkpoints, were used to correct the raw elevation of low-density LiDAR. Moreover, using the “best DEM of difference” approach, based on comparing the vertical errors raised from different methodologies (i.e., classification filters, interpolation methods and spatial resolutions), the elevation of each site was more accurately corrected reducing random or methodological errors that would be very difficult to correct by using only checkpoints.”

 2. The presentation of introduction chapter is suitable. From this chapter, audiences could understand the motivation, background information, and purposes of this study.

Thank you very much for your positive comments.

3. I suggested that Figure 2 should be improved. If possible, the step 3 to step 6 could put below step 2, not at the right hand.

 

The figure has been changed according to your suggestion.

4. Title of Figure 5 showed “Letters indicated significant differences within groups according to the post-hoc Kruskal-Wallis Dunn test”. It should note significant level, such as α=0.05 or other levels

Thank you for advice. The significance level was included in the captions of Tables 5 and 6 indicating that the significance level was p < 0.05. In addition, we included the significance levels in the captions of the supplementary tables where the post-hoc Kruskal-Wallis Dunn test was carried out.

5. I suggested that Supplementary material did refer in text, such as, Lines, 222, 327, 338, 409, 701. Please check whole text.

 The supplementary material is cited in the main text following the nomenclature of the journal. Supplementary tables are referred as “Table S1…Sn” and supplementary figures as “Figure S1…Sn”. Please, check the standard format of supplementary data at the end of the paper:

Supplementary Materials: The following supporting information can be downloaded at: www.mdpi.com/xxx/s1. Table S1: Technical characteristics of LiDAR datasets; Tables S2-S4: Mean and standard deviation of uncorrected elevation errors (i.e., DEMs of differences) measured by the percentile 50^th, NMDA and RMSE, respectively; Tables S5-S7: Mean and standard deviation of corrected elevation errors (i.e., DEMs of differences) measured by the percentile 50th, NMDA and RMSE, respectively; Tables S8-S13: GAMs (with linear terms) for each site using as response variable the uncorrected vertical errors derived from the CSF, TIN-DEF, PMF, TIN-SF, TIN-SW2 and TIN-WILD classification filters and as explanatory ones the site conditions; Table S14: Estimated coefficients of the vertical errors of DEMs in GAMs for explaining the corrected vegetation height changes (2014-2019); and Table S15: The percentage of deviance in the corrected vegetation height changes (2014-2019) explained by the vertical errors of DEMs. Figure S1: Clouds of points from high-density LiDAR data before (left panel) and after (right panel) correction of the boresight misalignment; Figure S2: Normal Q-Q graph for the distribution of the uncorrected elevation differences; Figure S3: Hillshades of DEMs derived from high-density LiDAR data using different classification filters; Figure S4: Histograms of the elevation differences between low- and high-density DEMs before and after correction; Figure S5: Cluster K-means of the elevation differences between low- and high-density LiDAR derived DEMs before and after correction; Figures S6-S7: Scatterplots of the relationship between the vegetation height changes (2014-2019) occurred in site 4 and the vertical errors of the DEM built using the TIN-SF and the TIN-SW2 classification filters, respectively.

 

6. If Figures 8 and 9 are important result, I suggest moving to Results chapter.

Thank you for your suggestion. However, I think that it is better that those figures are kept in discussion because I have not carried out a deep analysis of the vegetation growth processes that justify their inclusion in the results section. Those figures are a preclude of my future work and serve to justify the need of a further analysis.

 

7. Figure 8(b) might consider using “Difference between 2014 and 2019”.

 

Thank you for noticing of this mistake. It has been corrected.

8. Conclusions should briefly reflect the result of this study. However, Conclusions of this study is too long. It might let audiences confuse. Therefore, I suggest that it should be simplified.

 Thank you for your suggestion. I have shortened and simplify the conclusions being more precise in the messages.

 

“Standardization of ground elevation is obligatory for comparison of LiDAR derived forest metrics at different dates. After standardization, it can be reasonably assumed that the relative accuracy between CHMs is like the DEM accuracy. In this paper, it has been assessed that although high-density LiDAR data does not provide a perfect elevation model, it was considerably more accurate compared to low-density LiDAR; and therefore, it was a good benchmark. Our approach was novel mainly because continuous surfaces of elevation differences, instead of a collection of checkpoints, were used to correct the raw elevation of low-density LiDAR. Moreover, using the “best DEM of difference” approach, based on comparing the vertical errors raised from different methodologies (i.e., classification filters, interpolation methods and spatial resolutions), the elevation of each site was more accurately corrected reducing random or methodological errors that would be very difficult to correct by using only checkpoints.

Overall, it was observed that the classification filters based on TIN densification al-gorithm using default parameters or low spikes (TIN-DEF or TIN-SF), the kriging interpolation method and 5 m of pixel size corrected adequately elevation deviations in all sites. After correction, vertical errors dropped drastically reaching low values (from 0.04 m to 0.08 m) using the best DEM of difference in each site. Nevertheless, none of the classification filters worked perfectly in each site, and all of them are susceptible to both omission and commission errors. Accordingly, it is recommended the possibility of combining different classification filters in the same environment to exploit the strengths of each other. On the other hand, the vertical errors observed between low- and high-density LiDAR datasets were partially explained by site factors (maximum deviance explained was 57% ± 0.07). The slope and the distance to the nearest geoide point were the most important explanatory variables. Overall, the higher terrain slope and the distance to the nearest geoide, the higher the vertical errors. The vegetation height played a minor role, but vertical errors increased significantly with vegetation height. Finally, we assessed that changes in vegetation height were decoupled from elevation vertical errors in all sites.

The findings of the study are important to understand the sources of DEMs error by analysing the role of methodological and site-specific factors; and also to correct vertical errors of low-density LiDAR derived DEMs by using “the best DEM of difference” method. This approach reduced considerably the effects of elevation errors on vegetation height changes. The study recommends that, before comparing LiDAR datasets with different point density, “the best DEM of difference” should be considered for correcting the target elevation dataset; mainly because each environment has its own physical properties, and this method allows adapting the best DEM to each site. Finally, the estimation of vegetation growth from repeated LiDAR with different points density must be improved requiring further studies.”

 

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors have well addressed all the concerns.

Reviewer 3 Report

Comment

I reviewed the revised manuscript and Author response to report. Some of my suggestions have been improved. Although some still not improved thoroughly, author has stated in Author response to report. Nevertheless, I agreed author’s viewpoint. I only have slight suggestion in this version. Line 328, in equation (1), please use “ × ” to replace “ * ”。