Critical Points Extraction from Building Façades by Analyzing Gradient Structure Tensor

Round 1

Reviewer 1 Report

This paper presents an approach to extract building facade contours from 3D point clouds. The authors introduce various image and point cloud based approaches and explain how most of them have deficiencies. The proposed approach detects control points using the concept of confidence at every point; then classifies these points into corner, edge, boundary or constant points based on the computed confidence and difference in gradients using a dual-threshold criterion. Then the point cloud is simplified using three different simplification algorithms. The approach is backed by experimental results on various publicly available datasets and is also compared to two existing approaches.

The paper is structured well, although some parts seem verbose. For example the gradient and difference in gradients are explained at great length while the authors could have referred to these techniques from a textbook or an earlier paper. In place of this discussion, I would have preferred a bit more context on the WLOP algorithm.

Currently, the paper is composed of different components with mathematical equations describing each component. Instead, the authors could have chosen a more unified formulation with a single key equation describing the entire process with individual terms corresponding to each of the presented components.

Also, a related comment on unified mathematical formulation: To make the  current approach’s mathematical formulation more unified and robust, why didn’t the authors choose to apply SLAM like techniques ? For example one can apply a SLAM based technique to extract contours from detected and simplified control points similar in spirit to Choi et al [17] ?

I felt that the authors can do a better job at explaining what is novel in this paper. The confidence computation is heavily inspired from [33], the gradient tensor generation is inspired from Harris corner and edge detector, and finally the simplification algorithms are also not novel. So, it would be nice to sharply highlight the novelty of this approach in the paper text

The paper claims that one of the goals is to remove dependence on thresholds and also claim that the algorithm is insensitive to thresholds. The following are most of the thresholds that I picked up from this paper. The authors should explain the sensitivity of these thresholds through experiments.

• Neighborhood sphere radius
• Thresholds for Eigen values to classify points as Corner, edge, boundary and constant (lines 235 — 247)
• Gaussian kernel/weight function parameter (standard deviation)
• Two thresholds in the dual threshold criterion
• Parameters of the simplification algorithm

The paper contains a lot of grammatical errors and typos which should be addressed to improve the quality of the paper. A few suggestions are as below:

Line 31: can be written well

Line 52: did not understand the sentence - “3D reconstruction method representing by …”

Line 54: “The proposed method is …” - I am confused as to whether the authors are referring to Mineo et al. [12] or the current paper. For this review I assumed that the authors are referring to the current paper.

Line 73: This sentence needs to be better written

Line 90: “Except for the above methods” -> “Apart from the above methods” ?

Line 267: “an ingenious combination” -> “an elegant combination” ?

Lines 303 & 304: “trivial work by simultaneously tuning” -> “trivial work to simultaneously tune”

Line 304: “It needs tail and error …” - This sentence may be unfinished

Line 485: “to very” -> “to vary”

In summary, the approach presented in the paper makes sense and the experimental results back most of the claims but then the authors should address the following items:

• Making the text more readable by improving the English text
• Adding a discussion with experimental results on the thresholds’ sensitivities
• Discussion on novelty
• Reducing verbosity and adding discussion on WLOP (optional, unto the authors)

Author Response

Response to Reviewer 1’s Comments for Manuscript (remotesensing-1290665)

Critical Point Extraction from Building Facades by Analyzing Gradient Structure Tensor

Dong Chen, Jing Li, Shaoning Di, Jiju Peethambaran*, Guiqiu Xiang, Lincheng Wan, and Xianghong Li

Dear Anonymous Reviewer,

We would like to thank you for giving us an opportunity to revise and submit our manuscript entitled “Critical Point Extraction from Building Facades by Analyzing Gradient Structure Tensor”. We sincerely appreciate your careful reviewing and valuable comments which helped us improving our paper substantially. Following the reviewers and your comments, we believe that we have been successful in significantly improving the presentation and the technical content of our paper. In the revised manuscript (RM), we have addressed, point by point, all the requests and comments from the reviewers to eliminate the linguistic, technical, and structural deficiencies of our manuscript. For convenience, the comments of the editor and the reviewers are repeated below in light blue color. The modifications made in the revised manuscript are highlighted using the yellow background. The revised manuscript now completely complies with the guidelines of Remote Sensing. We appreciate your time and look forward to getting a response at your earliest convenience.

Question 1: This paper presents an approach to extract building facade contours from 3D point clouds. The authors introduce various image and point cloud-based approaches and explain how most of them have deficiencies. The proposed approach detects control points using the concept of confidence at every point; then classifies these points into corner, edge, boundary or constant points based on the computed confidence and difference in gradients using a dual-threshold criterion. Then the point cloud is simplified using three different simplification algorithms. The approach is backed by experimental results on various publicly available datasets and is also compared to two existing approaches. The paper is structured well, although some parts seem verbose. For example, the gradient and difference in gradients are explained at great length while the authors could have referred to these techniques from a textbook or an earlier paper. In place of this discussion, I would have preferred a bit more context on the WLOP algorithm.

Response 1:  Thank you again for reviewing our manuscript and providing us your valuable comments and suggestions. We have carefully considered each one of your comments and modified the manuscript accordingly. In the RM, we knock off some irrelavent parts as reviewer suggested to make the manuscript more concise. Furthermore we add more detailed descriptions regarding WLOP algorithm. Kindy refer to the relevant content in the subsection of refinement in the updated version.

Question 2: Currently, the paper is composed of different components with mathematical equations describing each component. Instead, the authors could have chosen a more unified formulation with a single key equation describing the entire process with individual terms corresponding to each of the presented components. Also, a related comment on unified mathematical formulation: To make the current approach’s mathematical formulation more unified and robust, why didn’t the authors choose to apply SLAM like techniques? For example one can apply a SLAM based technique to extract contours from detected and simplified control points similar in spirit to Choi et al [17]?

Response 2:  Thank you for your constructive question that inspires us a lot. In reference 17, the author presents a 3D edge detection algorithm from organized RGB-D point clouds. The created contour points are used in two applications, namely pair-wise registration, and RGB-D SLAM system. The author indirectly proves the effectiveness of the created edge features by confirming the superiorities of these two applications through comparisons with the state of the art. This paper emphasizes on how to use the extracted contours and prove the applicability of contour points in the real applications. The contouring method (edge detection method) in reference 17 uses a variant of Canny edge detection considering the particularity of the RGB-D point clouds, which are derived from RGB-D sensor and organized as images (row and columns of pixels).

Inspired by reference 17, in future work, we plan to use our contouring results in MLP autoencoder in deep Learning framework to get more accurate local feature of point clouds. Actually, most of encoder in autoencoder module uses uniformly downsampling strategy, thereby weakening the salient features in subsequent sampling layers. The contour points can be viewed as an intelligent downsampling step that favours points that retain more of the important structure of the scene, thereby improving the accuracy of a series of tasks, such as detection, semantic segmentation, instance segmentation and part segmentation.  By integrating our contours into the practical applications would be very effective way to verify the effectiveness of prosed contouring method. Thank you for your inspiration. Very good question.

Question 3: I felt that the authors can do a better job at explaining what is novel in this paper. The confidence computation is heavily inspired from [33], the gradient tensor generation is inspired from Harris corner and edge detector, and finally the simplification algorithms are also not novel. So, it would be nice to sharply highlight the novelty of this approach in the paper text.

Response 3: Thank you for your constructive comments. In the updated version, we explicitly state the novel contributions of the paper. Kindly see the second to last paragraph in Section 1(Introduction) for more details.

Question 4: The paper claims that one of the goals is to remove dependence on thresholds and also claim that the algorithm is insensitive to thresholds. The following are most of the thresholds that I picked up from this paper. The authors should explain the sensitivity of these thresholds through experiments.

Neighbourhood sphere radius

Thresholds for Eigen values to classify points as Corner, edge, boundary and constant (lines 235 - 247)

Gaussian kernel/weight function parameter (standard deviation)

Two thresholds in the dual threshold criterion

Parameters of the simplification algorithm

Response 4: Thank you for your valuable comments regarding all threshold setting. In the updated version, we add a new sub section (3.2 Parameter Analysing) from which we analyse all the relevant thresholds and explain how select their appropriate values. I suggest the reviewer refer to this part for detailed explanations.  

Question 5: The paper contains a lot of grammatical errors and typos which should be addressed to improve the quality of the paper. A few suggestions are as below:

Line 31: can be written well

Line 52: did not understand the sentence - “3D reconstruction method representing by …”

Line 54: “The proposed method is …” - I am confused as to whether the authors are referring to Mineo et al. [12] or the current paper. For this review I assumed that the authors are referring to the current paper.

Line 73: This sentence needs to be better written

Line 90: “Except for the above methods” -> “Apart from the above methods” ?

Line 267: “an ingenious combination” -> “an elegant combination” ?

Lines 303 & 304: “trivial work by simultaneously tuning” -> “trivial work to simultaneously tune”

Line 304: “It needs tail and error …” - This sentence may be unfinished

Line 485: “to very” -> “to vary”

Response 5: We are sorry for these careless mistakes. We appreciate the time taken to revise our language mistakes. In the revised manuscript, we incorporate all the corrections mentioned by the reviewer. Apart from a few mistakes that you have pointed out, we carefully proofread the manuscript several times to to avoid any such typographical and grammatical errors in the revised version.

Question 6: In summary, the approach presented in the paper makes sense and the experimental results back most of the claims but then the authors should address the following items:

Making the text more readable by improving the English text

Adding a discussion with experimental results on the thresholds’ sensitivities

Discussion on novelty

Reducing verbosity and adding discussion on WLOP (optional, unto the authors)

Response 6: Thank your again for your positive feedback. We totally agree with your comments on the above problems. In the revised version, we believe that we successfully solve these questions. We hope that our revisions can meet your requirements for publication. Many thanks.

In closing, we would like to again thank you for your valuable suggestions and comments which have helped us to significantly improve the technical content and presentation of our paper. We look forward to receiving your decision regarding the publishability of our revised paper.

Dong Chen

College of Civil Engineering, Nanjing Forestry University, No.159 Longpan Rd., Xuanwu Dist., Nanjing 210037, P.R. China

E-mails: [email protected]

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors presented an accurate and performant method for the extraction of critical points on building facades. 

The introduction and related work are presented clearly and systematically.

The methodology is described with an appropriate level of detail, clearly describing key methods. 

The results are given with a lot of detail, describing all relevant aspects of the critical points extraction method: accuracy, compactness, and robustness. 

The figures and tables are very well prepared and help a lot in understanding the text.

Author Response

Response to Reviewer 2’s Comments for Manuscript (remotesensing-1290665)

Critical Point Extraction from Building Facades by Analyzing Gradient Structure Tensor

Dong Chen, Jing Li, Shaoning Di, Jiju Peethambaran*, Guiqiu Xiang, Lincheng Wan, and Xianghong Li

Dear Anonymous Reviewer,

We would like to thank you for giving us an opportunity to revise and submit our manuscript entitled “Critical Point Extraction from Building Facades by Analyzing Gradient Structure Tensor”. We sincerely appreciate your careful reviewing and valuable comments which helped us improving our paper substantially. Following the reviewers and your comments, we believe that we have been successful in significantly improving the presentation and the technical content of our paper. In the revised manuscript (RM), we have addressed, point by point, all the requests and comments from the reviewers to eliminate the linguistic, technical, and structural deficiencies of our manuscript. For convenience, the comments of the editor and the reviewers are repeated below in light blue color. The modifications made in the revised manuscript are highlighted using the yellow background. The revised manuscript now completely complies with the guidelines of Remote Sensing. We appreciate your time and look forward to getting a response at your earliest convenience.

Overall Comments:

The authors presented an accurate and performant method for the extraction of critical points on building facades. 

The introduction and related work are presented clearly and systematically.

The methodology is described with an appropriate level of detail, clearly describing key methods. 

The results are given with a lot of detail, describing all relevant aspects of the critical points extraction method: accuracy, compactness, and robustness. 

The figures and tables are very well prepared and help a lot in understanding the text.

Response : Thank you for reviewing our manuscript and giving us positive feedback. We have revised the manuscript and carefully addressed the concerns of other reviewers. As a result of the revision, we believe that the manuscript has been significantly improved. We greatly appreciate your time and effort for revisions. Many thanks.

In closing, we would like to again thank you for your valuable suggestions and comments which have helped us to significantly improve the technical content and presentation of our paper. We look forward to receiving your decision regarding the publishability of our revised paper.

Dong Chen

College of Civil Engineering, Nanjing Forestry University, No.159 Longpan Rd., Xuanwu Dist., Nanjing 210037, P.R. China

E-mails: [email protected]

Author Response File: Author Response.pdf

Reviewer 3 Report

The work is very valuable from a scientific point of view and is well written. Some minor revisions are necessary before publication.

Introduction: Maybe it would be interesting to discuss the possible error in the measurements of the building contour definition from satellite data.

Methodology: Please better discuss if any information on the structural typology/materials can be detected with your methodology?

Figure 4. There is a perfect alignment of the points on the sides of the building. Is this result present in all the types of buildings? Or is this function of the type/precision of the input data?

Conclusions: How these data can be combined with the deformational satellite data in order to provide remote sensing monitoring?

Author Response

Response to Reviewer 3’s Comments for Manuscript (remotesensing-1290665)

Critical Point Extraction from Building Facades by Analyzing Gradient Structure Tensor

Dong Chen, Jing Li, Shaoning Di, Jiju Peethambaran*, Guiqiu Xiang, Lincheng Wan, and Xianghong Li

Dear Anonymous Reviewer,

We would like to thank you for giving us an opportunity to revise and submit our manuscript entitled “Critical Point Extraction from Building Facades by Analyzing Gradient Structure Tensor”. We sincerely appreciate your careful reviewing and valuable comments which helped us improving our paper substantially. Following the reviewers and your comments, we believe that we have been successful in significantly improving the presentation and the technical content of our paper. In the revised manuscript (RM), we have addressed, point by point, all the requests and comments from the reviewers to eliminate the linguistic, technical, and structural deficiencies of our manuscript. For convenience, the comments of the editor and the reviewers are repeated below in light blue color. The modifications made in the revised manuscript are highlighted using the yellow background. The revised manuscript now completely complies with the guidelines of Remote Sensing. We appreciate your time and look forward to getting a response at your earliest convenience.

Overall Comments:

The work is very valuable from a scientific point of view and is well written. Some minor revisions are necessary before publication.

Response:  Thank you for reviewing our manuscript and giving us positive feedback. We believe that your insightful comments helped us to enhance the clarity and readability of the manuscript. We have carefully considered each one of your comments and modified the manuscript accordingly. Please find below the detailed responses to your comments. 

Question 1: Introduction: Maybe it would be interesting to discuss the possible error in the measurements of the building contour definition from satellite data.

Response 1:   Thank you for your valuable suggestions. It needs to be added this discuss in the section of Introduction because the created building contours from aerial and satellite imagery are only 2D building footprints rather than 2.5D building highmaps or 3D building envelops. The building footprints are totally different from 3D building envelop contours derived from LiDAR scans and photogrammetric point clouds. It is an interesting perspective for building contouring from aerial or satellite images. Kindly see the updated version in Section 1 for more details.

Question 2: Methodology: Please better discuss if any information on the structural typology/materials can be detected with your methodology?

Response 2:  Thank you for your constructive comments. In the updated version, we add the relevant discussions, and we mention that our work cannot maintain topological relationships of the building contour points, thereby causing a difficult work for successive building reconstruction using a contour-based representation. However, this problem is expected to be solved by segmenting critical points into multiple linear and nonlinear clusters. Using these clusters, the building models can be organized by a series of boundary lines. In fact, we are working on that through the semantic labeling of critical points using the graph convolutional network and topological optimization technique.

Question 3: Figure 4. There is a perfect alignment of the points on the sides of the building. Is this result present in all the types of buildings? Or is this function of the type/precision of the input data?

Response 3:  Thank you for your wonderful observation regarding the good alignment of two types of point clouds in Figure 4. The proposed algorithm can adapt to any type of buildings and get promising facades contours. This is demonstrated in Figure 8, from which the enlarged rectangles denoted by purple color have a very complex geometric shape with a high degree of nonlinear structure. Despite this, we obtain a reasonable skeleton for this complex shape.

The obtained building contours are slightly influenced by the precision of different point clouds because the proposed gradient structure tensor is directly derived from the local geometric feature of individual point. This implies that if the precision of the point clouds is relatively low, the derived gradient structure tensor is not robust, as evident in Figure 12. 

Question 4: Conclusions: How these data can be combined with the deformational satellite data in order to provide remote sensing monitoring?

Response 4: Thank you for your constructive question. Actually, most of satellite imageries are 2D raster data, but LiDAR scans and/or photogrammetric point clouds used in this paper are 3D point clouds, which has the capability to describe the details of vertical structures of the objects. For example, it can capture both extrusions and intrusions of building façades. Due to the penetration of the LiDAR scans in the deciduous forest, the vertical structure of the branches could be more clearly extracted. Based on a lidar-derived height distribution and the proportion of laser return from the branches, the users even can discriminate species of individual tree. Take the dynamic monitoring building for example, the satellite image only answers the question of whether the buildings exits or not (instance-level scale), however, the LiDAR scan can provide the building changes in more detailed scale (part/component scale). For example, LiDAR scan can answer the question of whether the façade window or rooftop chimney are newly built or just renovated.

In summary, Fusion of deformational satellite imagery with LiDAR scans is an interesting direction. Although some works, take advantage of the complementary nature of aerial imagery and LiDAR scans by integrating them, the issue of how to optimally integrate data from the multiple data sources with dissimilar characteristics is still an open research problem. In addition, the remote sensing monitoring using multiple data sources demands more complex algorithmic solutions requiring increased computations and higher acquisition costs.

In closing, we would like to again thank you for your valuable suggestions and comments which have helped us to significantly improve the technical content and presentation of our paper. We look forward to receiving your decision regarding the publishability of our revised paper.

Sincerely,

Dong Chen

College of Civil Engineering, Nanjing Forestry University, No.159 Longpan Rd., Xuanwu Dist., Nanjing 210037, P.R. China

E-mails: [email protected]

Author Response File: Author Response.pdf