# From the Semantic Point Cloud to Heritage-Building Information Modeling: A Semiautomatic Approach Exploiting Machine Learning

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

#### Aim of the Paper

- (i)
- A semantic segmentation via machine learning:

- (ii)
- A scan-to-BIM reconstruction:

- -
- Processing of a large amount of relevant data.
- -
- Coordinated management of research results.
- -
- Standardized syntax and representation, also in view of future data updates.

## 2. State of the Art

#### 2.1. 3D Data Acquisition Methods

#### 2.2. From Unstructured Point Clouds to H-BIM Models

- In the geometry recognition phase, a set of simple geometric primitives that closely correspond to the underlying shape can be mapped over a 3D point cloud, by making use of fitting algorithms [38,39]. Available software for point cloud processing, e.g., CloudCompare, an open source 3D visualization and computation software from Daniel Girardeau (http://www.cloudcompare.org/), are equipped with tools for primitive fitting, but they do not produce parametric objects that can be integrated directly into BIM platforms. Macher et al. [19] partially filled this gap by introducing a conversion step, i.e., by enabling the reconstruction of walls and slabs of the indoors of existing buildings as BIM-readable objects starting from point clouds. They exploit the open-source software FreeCAD in order to convert building entities from imported data into the Industry Foundation Classes (IFC) format, an open file format specification compatible with BIM management software. Shah et al. [40] proposed a framework for the fitting of primitives with computer-aided design (CAD) models, applicable to the assembling of mechanical parts in industrial production. Region-growing and boundary detection algorithms [41] can also be applied to identify primitive geometries with plane development.
- Beyond manual tracing and practices of the identification of primitives, more complex architectural components can be reconstructed by procedural modeling techniques to create libraries of heritage buildings’ elements. López et al. [42], based on the amount and type of details required to represent each architectural element, distinguished between regular shapes and irregular, organic surfaces. The first ones are modelled by using pre-packaged BIM families, while the second ones are constructed from scratch and inserted in a library of historic buildings that can be used in future works.

- Distinction of data into homogeneous sections with similar attributes (segmentation);
- Assignment of a class label to each segmented region to insert a semantic meaning (classification) [48].

#### 2.3. Automating Semantic Segmentation via Machine Learning

## 3. Materials

## 4. Methods

#### 4.1. Semantic Segmentation via Machine Learning

- (i)
- Neighborhood selection and feature extraction;
- (ii)
- Feature selection;
- (iii)
- Manual annotation on a reduced portion of the dataset (training set), to identify classes of elements;
- (iv)
- Automated propagation of the class labels to the whole dataset via a RF classifier, and accuracy evaluation;
- (v)
- Annotated 3D point cloud.

- Its combination of learning models increases classification accuracy, averaging noisy and unbiased models.
- It requires a smaller amount of annotated data for learning, compared to other ML algorithms. Furthermore, as specified in Section 2.3, the training dataset is not large enough to train a neural network.
- It presents estimates for features’ importance: the less significant variables can be removed so that the model is trained on a subset of features, greatly reducing the time for learning and increasing predictive accuracy.

#### 4.1.1. Feature Extraction and Selection

#### 4.1.2. Random Forest Classifier and Evaluation of the Trained Model

#### 4.2. Scan-to-BIM

- (i)
- Extraction of single classes of architectural components from the annotated point cloud;
- (ii)
- Creation of libraries of ideal parametric shapes, broken down by each class;
- (iii)
- Reconstruction of parametric components and export to IFC (BIM-readable and interchangeable) format.

- In the most trivial cases, typological elements can be attributed to simple geometric primitives, e.g., cylinders to describe columns or pipes, flat objects to describe walls, roof pitches and floors. The classic approach to the reconstruction of these simple elements is primitive model fitting.
- In the general case, the construction of the parametric element takes places through the modeling of its ideal geometry, derived from constructive and proportional rules as defined in architectural treatises. The representative element of a class is modeled in Autodesk Revit as a parametric adaptive component.

## 5. Results

#### 5.1. Semantic Segmentation via Machine Learning

#### 5.1.1. Neighborhood Selection and Feature Extraction

#### 5.1.2. Feature Selection and Manual Annotation of the Training Set

#### 5.1.3. Random Forest Classifier and Validation Results

- Geometric features;
- Geometric features + Z;
- Geometric features + Z + RGB values;
- Geometric features + Z + Intensity;
- Geometric features + Z + RGB values + Intensity, in order to evaluate the most successful one in making predictions.

#### 5.2. Scan-to-BIM

#### 5.2.1. Extraction of Single Classes of Architectural Components

- By appropriately editing the visibility settings in the various views, it was possible to isolate—or, conversely, hide—some classes of elements rather than others, boosting the reconstruction process.
- By moving across different zones of the point cloud, the class to which they belong was displayed directly.

#### 5.2.2. Libraries of Ideal Parametric Shapes

#### 5.2.3. H-BIM Model and Export to IFC Format

## 6. Discussion

- To access, to amend and to systematically update knowledge-related information, and therefore to enrich the reconstructed H-BIM model. In fact, following the logic of digital information models, semantic annotations associated with both parametric and reality-based representations can be further improved, such as in the example of Figure 20, by means of historical documentation, management reports, and so on.
- To create disparity maps, highlighting the deviations between the ideal model and the real model and their evolution over the time. In that way, differences between individual elements belonging to the same class can be suitably detected, revealing, for example, degradation phenomena and losses of material.

## 7. Conclusions

- The application of ML approaches for the classification of 3D heritage data as a preliminary step towards a more automated construction of H-BIM models;
- A more effective management of 3D data, with insertion of semantic and meaningful tags in both reality-based and parametric representation models;
- Acceleration of the semantic annotation process, as once the training set is annotated and the RF classifier is trained, the application of the trained model to non-annotated parts takes on average between 15 and 20 min;
- Automation in the transition from point-based to parametric representation;
- Time-reduction of the scan-to-BIM process, through the import of an annotated yet semantically segmented point cloud;
- The creation of a semantic bridge between reality-based and parametric models.
- As a future work, some aspects will be further investigated and developed:
- Streamlining of the procedure with the creation of a single development environment that does not require the transition between different software, and that can be tested by non-skilled operators even in the semantic segmentation phase;
- Investigations on a multi-level semantic segmentation, in such a way that components can be hierarchically classified depending on the desired scales of representation and levels of detail;
- Surveys on the interrelation and dependency between the selected features and the labelled dataset type to speed up selection of feature subsets;
- Extension of the work to other case studies to demonstrate the applicability of the proposed approach to other heritage objects, for example, belonging to a different time or architectural style. This also implies exploring the relationship between the most relevant features and the proportions of some elements of the dataset;
- Studies on the connection between represented classes and domain ontologies, to make 3D reality-based and parametric data even more accessible, traceable and reusable by users and applications in the cultural heritage field;
- Enrichment of existing benchmarks of heritage annotated data with insertion of this study to allow a more suitable application of DL algorithms, as already envisaged in [23]. This would boost the development of automatic classification solutions at a larger scale.

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## List of Abbreviations

BIM | Building Information Modeling |

CAD | Computer Aided Design |

DL | Deep Learning |

FN | False Negatives |

FP | False Positives |

H-BIM | Heritage-Building Information Modeling |

IFC | Industry Foundation Classes |

LiDAR | Light Detection and Ranging |

ML | Machine Learning |

RANSAC | Random Sample and Consensus |

RF | Random Forest |

RGB | Red, Green and Blue |

SLAM | Simultaneous Localization and Mapping |

SVM | Support Vector Machine |

TN | True Negatives |

TP | True Positives |

3D | Three-dimensional |

## References

- Bacci, G.; Bertolini, F.; Bevilacqua, M.G.; Caroti, G.; Martínez-Espejo Zaragoza, I.; Martino, M.; Piemonte, A. HBIM Methodologies for the Architectural Restoration. The Case of the Ex-Church of San Quirico All’Olivo in Lucca, Tuscany. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-2/W11, 121–126. [Google Scholar] [CrossRef][Green Version] - Roussel, R.; Bagnéris, M.; De Luca, L.; Bomblet, P. A Digital Diagnosis for the “autumn” Statue (Marseille, France): Photogrammetry, Digital Cartography and Construction of a Thesaurus. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-2/W15, 1039–1046. [Google Scholar] [CrossRef][Green Version] - Chiabrando, F.; Lo Turco, M.; Rinaudo, F. Modeling the Decay in a HBIM Starting from 3D Point Clouds. A Followed Approach for Cultural Heritage Knowledge. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, XLII-2/W5, 605–612. [Google Scholar] [CrossRef][Green Version] - Lo Turco, M.; Mattone, M.; Rinaudo, F. Metric Survey and BIM Technologies to Record Decay Conditions. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, XLII-5/W1, 261–268. [Google Scholar] [CrossRef][Green Version] - Robleda-Prieto, G.; Caroti, G.; Martínez-Espejo Zaragoza, I.; Piemonte, A. Computational Vision in UV-Mapping of Textured Meshes Coming from Photogrammetric Recovery: Unwrapping Frescoed Vaults. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2016**, XLI-B5, 391–398. [Google Scholar] [CrossRef] - Oreni, D.; Brumana, R.; Della Torre, S.; Banfi, F. Survey, HBIM and Conservation Plan of a Monumental Building Damaged by Earthquake. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, XLII-5/W1, 337–342. [Google Scholar] [CrossRef][Green Version] - Stefani, C.; Luca, L.D.; Veron, P.; Florenzano, M. Modeling Buildings Historical Evolutions. In Proceedings of the Focus K3D conference on Semantic 3D Media and Content, Méditerranée, France, 11–12 February 2010. [Google Scholar]
- Apollonio, F.I. The Production of 3D Digital Archives and the Methodologies for Digitally Supporting Research in Architectural and Urban Cultural Heritage. In Digital Research and Education in Architectural Heritage; Münster, S., Friedrichs, K., Niebling, F., Seidel-Grzesińska, A., Eds.; Communications in Computer and Information Science; Springer International Publishing: Cham, Switzerland, 2018; Volume 817, pp. 139–158. ISBN 978-3-319-76991-2. [Google Scholar]
- Apollonio, F.I.; Basilissi, V.; Callieri, M.; Dellepiane, M.; Gaiani, M.; Ponchio, F.; Rizzo, F.; Rubino, A.R.; Scopigno, R.; Sobra’, G. A 3D-Centered Information System for the Documentation of a Complex Restoration Intervention. J. Cult. Herit.
**2018**, 29, 89–99. [Google Scholar] [CrossRef] - Croce, V.; Caroti, G.; De Luca, L.; Piemonte, A.; Véron, P. Semantic Annotations on Heritage Models: 2D/3D Approaches and Future Research Challenges. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2020**, XLIII-B2-2020, 829–836. [Google Scholar] [CrossRef] - Bevilacqua, M.G.; Caroti, G.; Piemonte, A.; Ruschi, P.; Tenchini, L. 3D Survey Techniques for the Architectural Restoration: The Case of St. Agata in Pisa. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, XLII-5/W1, 441–447. [Google Scholar] [CrossRef][Green Version] - Manuel, A.; Véron, P.; Luca, L.D. 2D/3D Semantic Annotation of Spatialized Images for the Documentation and Analysis of Cultural Heritage. Eurographics Workshop Graph. Cult. Herit.
**2016**, 4. [Google Scholar] [CrossRef] - Andrews, P.; Zaihrayeu, I.; Pane, J. A Classification of Semantic Annotation Systems. Semant. Web
**2012**, 3, 223–248. [Google Scholar] [CrossRef][Green Version] - Bruno, N.; Roncella, R. HBIM for Conservation: A New Proposal for Information Modeling. Remote Sens.
**2019**, 11, 1751. [Google Scholar] [CrossRef][Green Version] - Andriasyan, M.; Moyano, J.; Nieto-Julián, J.E.; Antón, D. From Point Cloud Data to Building Information Modelling: An Automatic Parametric Workflow for Heritage. Remote Sens.
**2020**, 12, 1094. [Google Scholar] [CrossRef][Green Version] - Croce, V.; Caroti, G.; Piemonte, A.; Bevilacqua, M.G. Geomatics for Cultural Heritage conservation: Integrated survey and 3D modeling. In Proceedings of the IMEKO TC4 International Conference on Metrology for Archaeology and Cultural Heritage, MetroArchaeo, Florence, Italy, 4–6 December 2019; pp. 271–276. [Google Scholar]
- Tang, P.; Huber, D.; Akinci, B.; Lipman, R.; Lytle, A. Automatic Reconstruction of As-Built Building Information Models from Laser-Scanned Point Clouds: A Review of Related Techniques. Autom. Constr.
**2010**, 19, 829–843. [Google Scholar] [CrossRef] - De Luca, L. Methods, Formalisms and Tools for the Semantic-Based Surveying and Representation of Architectural Heritage. Appl. Geomat.
**2011**, 6, 115–139. [Google Scholar] [CrossRef] - Macher, H.; Landes, T.; Grussenmeyer, P. From Point Clouds to Building Information Models: 3D Semi-Automatic Reconstruction of Indoors of Existing Buildings. Appl. Sci.
**2017**, 7, 1030. [Google Scholar] [CrossRef][Green Version] - Volk, R.; Stengel, J.; Schultmann, F. Building Information Modeling (BIM) for Existing Buildings—Literature Review and Future Needs. Autom. Constr.
**2014**, 38, 109–127. [Google Scholar] [CrossRef][Green Version] - Grilli, E.; Remondino, F. Classification of 3D Digital Heritage. Remote Sens.
**2019**, 11, 847. [Google Scholar] [CrossRef][Green Version] - Grilli, E.; Özdemir, E.; Remondino, F. Application of Machine and Deep Learning Strategies for the Classification of Heritage Point Clouds. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-4/W18, 447–454. [Google Scholar] [CrossRef][Green Version] - Matrone, F.; Lingua, A.; Pierdicca, R.; Malinverni, E.S.; Paolanti, M.; Grilli, E.; Remondino, F.; Murtiyoso, A.; Landes, T. A Benchmark for Large-Scale Heritage Point Cloud Semantic Segmentation. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2020**, XLIII-B2-2020, 1419–1426. [Google Scholar] [CrossRef] - Pierdicca, R.; Paolanti, M.; Matrone, F.; Martini, M.; Morbidoni, C.; Malinverni, E.S.; Frontoni, E.; Lingua, A.M. Point Cloud Semantic Segmentation Using a Deep Learning Framework for Cultural Heritage. Remote Sens.
**2020**, 12, 1005. [Google Scholar] [CrossRef][Green Version] - Breiman, L. Ramdom Forests. Mach. Learn.
**2001**, 45, 5–32. [Google Scholar] [CrossRef][Green Version] - Remondino, F.; Georgopoulos, A.; Gonzalez-Aguilera, D.; Agrafiotis, P. Latest Developments in Reality-Based 3D Surveying and Modelling, 1st ed.; MDPI AG Publisher: Basel, Switzerland, 2018; ISBN 978-3-03842-685-1. [Google Scholar]
- Bevilacqua, M.G.; Caroti, G.; Piemonte, A.; Terranova, A.A. Digital Technology and Mechatronic Systems for the Architectural 3D Metric Survey. In Mechatronics for Cultural Heritage and Civil Engineering; Ottaviano, E., Pelliccio, A., Gattulli, V., Eds.; Intelligent Systems, Control and Automation: Science and Engineering; Springer International Publishing: Cham, Switzerland, 2018; Volume 92, pp. 161–180. ISBN 978-3-319-68645-5. [Google Scholar]
- Weinmann, M. Reconstruction and Analysis of 3D Scenes; Springer International Publishing: Cham, Switzerland, 2016; ISBN 978-3-319-29244-1. [Google Scholar]
- Remondino, F. Heritage Recording and 3D Modeling with Photogrammetry and 3D Scanning. Remote Sens.
**2011**, 3, 1104–1138. [Google Scholar] [CrossRef][Green Version] - Pamart, A.; Morlet, F.; de Luca, L. A Fully Automated Incremental Photogrammetric Processing Dedicated for Collaborative Remote-Computing Workflow. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-2/W9, 565–571. [Google Scholar] [CrossRef][Green Version] - Hichri, N.; Stefani, C.; De Luca, L.; Veron, P.; Hamon, G. From Point Cloud to BIM: A Survey of Existing Approaches. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2013**, XL-5/W2, 343–348. [Google Scholar] [CrossRef][Green Version] - Bruno, S.; Musicco, A.; Fatiguso, F.; Dell’Osso, G.R. The Role of 4D Historic Building Information Modelling and Management in the Analysis of Constructive Evolution and Decay Condition within the Refurbishment Process. Int. J. Archit. Herit.
**2019**, 1, 1–17. [Google Scholar] [CrossRef] - Dore, C.; Murphy, M. Current State of the Art Historic Building Information Modeling. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, XLII–2, 185–192. [Google Scholar] [CrossRef][Green Version] - Murphy, M.; McGovern, E.; Pavia, S. Historic Building Information Modelling—Adding Intelligence to Laser and Image Based Surveys of European Classical Architecture. ISPRS J. Photogramm. Remote Sens.
**2013**, 76, 89–102. [Google Scholar] [CrossRef] - Murphy, M.; McGovern, E.; Pavia, S. Historic building information modelling (HBIM). Struct. Surv.
**2009**, 27, 311–327. [Google Scholar] [CrossRef][Green Version] - López, F.; Lerones, P.; Llamas, J.; Gómez-García-Bermejo, J.; Zalama, E. A Review of Heritage Building Information Modeling (H-BIM). MTI
**2018**, 2, 21. [Google Scholar] [CrossRef][Green Version] - Pocobelli, D.P.; Boehm, J.; Bryan, P.; Still, J.; Grau-Bové, J. Building Information Modeling for Monitoring and Simulation Data in Heritage Buildings. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2018**, XLII–2, 909–916. [Google Scholar] [CrossRef][Green Version] - Kaiser, A.; Ybanez Zepeda, J.A.; Boubekeur, T. A Survey of Simple Geometric Primitives Detection Methods for Captured 3D Data. Comput. Graph. Forum
**2019**, 38, 167–196. [Google Scholar] [CrossRef] - Tulsiani, S.; Su, H.; Guibas, L.J.; Efros, A.A.; Malik, J. Learning Shape Abstractions by Assembling Volumetric Primitives. arXiv
**2018**, arXiv:1612.00404. [Google Scholar] - Shah, G.A.; Polette, A.; Pernot, J.-P.; Giannini, F.; Monti, M. Simulated Annealing-Based Fitting of CAD Models to Point Clouds of Mechanical Parts’ Assemblies. Eng. Comput.
**2020**, in press. [Google Scholar] [CrossRef] - Wang, C.; Cho, Y.K.; Kim, C. Automatic BIM Component Extraction from Point Clouds of Existing Buildings for Sustainability Applications. Autom. Constr.
**2015**, 56, 13. [Google Scholar] [CrossRef] - López, F.J.; Lerones, P.M.; Llamas, J.; Gómez-García-Bermejo, J.; Zalama, E. A Framework for Using Point Cloud Data of Heritage Buildings toward Geometry Modeling in a BIM Context: A Case Study on Santa Maria La Real De Mave Church. Int. J. Archit. Herit.
**2017**, 3816, 1–22. [Google Scholar] [CrossRef] - Capone, M.; Lanzara, E. Scan-to-BIM vs. 3D Ideal Modela HBIM: Parametric Tools to Study Domes Geometry. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-2/W9, 219–226. [Google Scholar] [CrossRef][Green Version] - Baik, A. From Point Cloud to Jeddah Heritage BIM Nasif Historical House—Case Study. Digit. Appl. Archaeol. Cult. Herit.
**2017**, 4, 1–18. [Google Scholar] [CrossRef] - Rodríguez-Moreno, C.; Reinoso-Gordo, J.F.; Rivas-López, E.; Gómez-Blanco, A.; Ariza-López, F.J.; Ariza-López, I. From Point Cloud to BIM: An Integrated Workflow for Documentation, Research and Modelling of Architectural Heritage. Surv. Rev.
**2018**, 50, 212–231. [Google Scholar] [CrossRef] - Pepe, M.; Costantino, D.; Restuccia Garofalo, A. An Efficient Pipeline to Obtain 3D Model for HBIM and Structural Analysis Purposes from 3D Point Clouds. Appl. Sci.
**2020**, 10, 1235. [Google Scholar] [CrossRef][Green Version] - Barazzetti, L.; Banfi, F.; Brumana, R.; Gusmeroli, G.; Previtali, M.; Schiantarelli, G. Cloud-to-BIM-to-FEM: Structural Simulation with Accurate Historic BIM from Laser Scans. Simul. Model. Pract. Theory
**2015**, 57, 71–87. [Google Scholar] [CrossRef] - Grilli, E.; Menna, F.; Remondino, F. A Review of Point Clouds Segmentation and Classification Algorithms. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2017**, XLII-2/W3, 339–344. [Google Scholar] [CrossRef][Green Version] - Fiorucci, M.; Khoroshiltseva, M.; Pontil, M.; Traviglia, A.; Del Bue, A.; James, S. Machine Learning for Cultural Heritage: A Survey. Pattern Recognit. Lett.
**2020**, 133, 102–108. [Google Scholar] [CrossRef] - Grilli, E.; Farella, E.M.; Torresani, A.; Remondino, F. Geometric Features Analysis for the Classification of Cultural Heritage Point Clouds. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-2/W15, 541–548. [Google Scholar] [CrossRef][Green Version] - Özdemir, E.; Remondino, F.; Golkar, A. Aerial Point Cloud Classification with Deep Learning and Machine Learning Algorithms. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-4/W18, 843–849. [Google Scholar] [CrossRef][Green Version] - Weinmann, M.; Jutzi, B.; Hinz, S.; Mallet, C. Semantic Point Cloud Interpretation Based on Optimal Neighborhoods, Relevant Features and Efficient Classifiers. ISPRS J. Photogramm. Remote Sens.
**2015**, 105, 286–304. [Google Scholar] [CrossRef] - Bassier, M.; Klein, R.; Van Genechten, B.; Vergauwen, M. IFCwall Reconstruction from Unstructured Point Clouds. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
**2018**, IV–2, 33–39. [Google Scholar] [CrossRef][Green Version] - Murtiyoso, A.; Grussenmeyer, P. Point Cloud Segmentation and Semantic Annotation Aided by GIS Data for Heritage Complexes. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-2/W9, 523–528. [Google Scholar] [CrossRef][Green Version] - Qi, C.R.; Su, H.; Mo, K.; Guibas, L.J. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. arXiv
**2017**, arXiv:1612.00593. [Google Scholar] - Qi, C.R.; Yi, L.; Su, H.; Guibas, L.J. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. arXiv
**2019**, arXiv:1706.02413. [Google Scholar] - Malinverni, E.S.; Pierdicca, R.; Paolanti, M.; Martini, M.; Morbidoni, C.; Matrone, F.; Lingua, A. Deep Learning for Semantic Segmentation of 3D Point Cloud. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, XLII-2/W15, 735–742. [Google Scholar] [CrossRef][Green Version] - Matrone, F.; Grilli, E.; Martini, M.; Paolanti, M.; Pierdicca, R.; Remondino, F. Comparing Machine and Deep Learning Methods for Large 3D Heritage Semantic Segmentation. IJGI
**2020**, 9, 535. [Google Scholar] [CrossRef] - Scamozzi, V. L’idea Dell’architettura Universale; Per Girolamo Albrizzi: Venezia, Italy, 1615. [Google Scholar]
- Remondino, F.; Rizzi, A. Reality-Based 3D Documentation of Natural and Cultural Heritage Sites—Techniques, Problems, and Examples. Appl. Geomat.
**2010**, 2, 85–100. [Google Scholar] [CrossRef][Green Version] - Tan, K.; Cheng, X. Distance Effect Correction on TLS Intensity Data Using Naturally Homogeneous Targets. IEEE Geosci. Remote Sens. Lett.
**2020**, 17, 499–503. [Google Scholar] [CrossRef] - Bolkas, D. Terrestrial Laser Scanner Intensity Correction for the Incidence Angle Effect on Surfaces with Different Colours and Sheens. Int. J. Remote Sens.
**2019**, 40, 7169–7189. [Google Scholar] [CrossRef] - Xu, T.; Xu, L.; Yang, B.; Li, X.; Yao, J. Terrestrial Laser Scanning Intensity Correction by Piecewise Fitting and Overlap-Driven Adjustment. Remote Sens.
**2017**, 9, 1090. [Google Scholar] [CrossRef][Green Version] - Kashani, A.; Olsen, M.; Parrish, C.; Wilson, N. A Review of LIDAR Radiometric Processing: From Ad Hoc Intensity Correction to Rigorous Radiometric Calibration. Sensors
**2015**, 15, 28099–28128. [Google Scholar] [CrossRef][Green Version] - Hackel, T.; Wegner, J.D.; Savinov, N.; Ladicky, L.; Schindler, K.; Pollefeys, M. Large-Scale Supervised Learning For 3D Point Cloud Labeling: Semantic3d.Net. Photogramm. Eng. Remote Sens.
**2018**, 84, 297–308. [Google Scholar] [CrossRef] - Hackel, T.; Wegner, J.D.; Schindler, K. Fast Semantic Segmentation of 3D Point Clouds with Strongly Varying Density. 8. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2016**, 3, 177–184. [Google Scholar] - Ni, H.; Lin, X.; Zhang, J. Classification of ALS Point Cloud with Improved Point Cloud Segmentation and Random Forests. Remote Sens.
**2017**, 9, 288. [Google Scholar] [CrossRef][Green Version] - Rodriguez-Gonzalvez, P.; Rodriguez-Martin, M. Weld Bead Detection Based on 3D Geometric Features and Machine Learning Approaches. IEEE Access
**2019**, 7, 14714–14727. [Google Scholar] [CrossRef] - Hackel, T.; Wegner, J.D.; Schindler, K. Contour Detection in Unstructured 3D Point Clouds. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 1610–1618. [Google Scholar]
- Weinmann, M.; Jutzi, B.; Mallet, C. Feature Relevance Assessment for the Semantic Interpretation of 3D Point Cloud Data. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
**2013**, II-5/W2, 313–318. [Google Scholar] [CrossRef][Green Version] - Fischler, M.A.; Bolles, R.C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM
**1981**, 24, 381–395. [Google Scholar] [CrossRef] - Yang, L.; Cheng, J.C.P.; Wang, Q. Semi-Automated Generation of Parametric BIM for Steel Structures Based on Terrestrial Laser Scanning Data. Autom. Constr.
**2020**, 112, 103037. [Google Scholar] [CrossRef] - Hong, S.; Jung, J.; Kim, S.; Cho, H.; Lee, J.; Heo, J. Semi-Automated Approach to Indoor Mapping for 3D as-Built Building Information Modeling. Comput. Environ. Urban Syst.
**2015**, 51, 34–46. [Google Scholar] [CrossRef] - Jung, J.; Hong, S.; Jeong, S.; Kim, S.; Cho, H.; Hong, S.; Heo, J. Productive Modeling for Development of As-Built BIM of Existing Indoor Structures. Autom. Constr.
**2014**, 42, 68–77. [Google Scholar] [CrossRef] - Thomson, C.; Boehm, J. Automatic Geometry Generation from Point Clouds for BIM. Remote Sens.
**2015**, 7, 11753–11775. [Google Scholar] [CrossRef][Green Version] - Hough, P.V.C. Method and Means for Recognizing Complex Patterns. U.S. Patent 3069654, 18 December 1962. [Google Scholar]
- Díaz-Vilariño, L.; Conde, B.; Lagüela, S.; Lorenzo, H. Automatic Detection and Segmentation of Columns in As-Built Buildings from Point Clouds. Remote Sens.
**2015**, 7, 15651–15667. [Google Scholar] [CrossRef][Green Version] - Bosché, F.; Ahmed, M.; Turkan, Y.; Haas, C.T.; Haas, R. The Value of Integrating Scan-to-BIM and Scan-vs-BIM Techniques for Construction Monitoring Using Laser Scanning and BIM: The Case of Cylindrical MEP Components. Autom. Constr.
**2015**, 49, 201–213. [Google Scholar] [CrossRef] - Tarsha-Kurdi, F.; Landes, T.; Grussenmeyer, P. Hough-Transform and Extended RANSAC Algorithms for Automatic Detection of 3D Building Roof Planes from Lidar Data. IAPRS
**2007**, 36, 407–412. [Google Scholar] - Schnabel, R.; Wahl, R.; Klein, R. Efficient RANSAC for Point-Cloud Shape Detection. Comput. Graph. Forum
**2007**, 26, 214–226. [Google Scholar] [CrossRef] - Loh, W.-Y.; Shih, Y.-S. Split Selection Methods for Classification Trees. Stat. Sin.
**1997**, 7, 815–840. [Google Scholar] - Manuel, A.; Véron, P.; De Luca, L. 2D/3D semantic annotation of spatialized images for the documentation and analysis of cultural heritage. In Proceedings of the 14th Eurographics Workshop on Graphics and Cultural Heritage, Genova, Italy, 5–7 October 2016; pp. 101–104. [Google Scholar]
- Manuel, A.; M’Dahri, A.A.; Abergel, V.; Rozar, F.; De Luca, L. A Semi-Automatic 2D/3D Annotation Framework for the Geometric Analysis of Heritage Artefacts. In Proceedings of the 2018 3rd Digital Heritage International Congress, Digital Heritage 2018, San Francisco, CA, USA, 26–30 October 2018; pp. 1–7. [Google Scholar]

**Figure 2.**RGB (

**a**) and intensity (

**b**) values for the 3D point cloud of the Grand-Ducal Cloister dataset. The red line of figure (

**a**) displays the area of integration with the UAV survey.

**Figure 5.**Structure of the 3D point cloud matrix. $n$ is the total number of detected points, $m$ is the number of geometric features and $\varrho $ is the radius of the local neighborhood.

**Figure 8.**Annotated point cloud for combinations n. 2 (

**a**) and n. 3 (

**b**) and comparison of the two outputs (

**c**).

**Figure 11.**Selection of a subset of points to classify according to combination n. 3 rather than combination n. 2 (

**a**) and resulting annotated point cloud (

**b**).

**Figure 14.**Label 7—Column shaft imported into Recap upfront (

**a**) and later selected on the overall point cloud in Revit (

**b**).

**Figure 15.**Label 5—Vault selected on the overall point cloud (

**a**), then isolated (

**b**) by acting on the visibility and graphic override parameters in Revit (

**c**).

**Figure 16.**Selection of a single class (

**a**), primitive fitting and shape detection (

**b**), parametric object reconstruction and export to IFC format (

**c**).

**Figure 17.**Reconstruction of parametric shapes in the general case, based on architectural treatises and constructive rules: column capital (

**a**), column base (

**b**), vaults (

**c**).

**Figure 19.**Examples of classes of parametric shapes displayed over the parametric model: Class 6—Column capital (

**a**); Class 11—Window (

**b**); Class 15—Buttress capital (

**c**). The original point cloud is also shown over the H-BIM model.

**Figure 21.**Visual correspondence between annotated point cloud (

**a**) and reconstructed H-BIM model (

**b**) in terms of semantic segmentation.

**Table 1.**Geometric features considered [66].

Feature Name | Expression | |
---|---|---|

Linearity | ${L}_{\lambda}\left(\varrho \right)=\frac{{\lambda}_{1}-{\lambda}_{2}}{{\lambda}_{1}}$ | (4) |

Planarity | ${P}_{\lambda}\left(\varrho \right)=\frac{{\lambda}_{2}-{\lambda}_{3}}{{\lambda}_{1}}$ | (5) |

Sphericity | ${S}_{\lambda}\left(\varrho \right)=\frac{{\lambda}_{3}}{{\lambda}_{1}}$ | (6) |

Omnivariance | ${O}_{\lambda}\left(\varrho \right)$$=\sqrt[3]{{\lambda}_{1}{\lambda}_{2}{\lambda}_{3}}$ | (7) |

Eigenentropy | ${E}_{\lambda}\left(\varrho \right)=-{\displaystyle \sum _{i=1}^{3}}{\lambda}_{i}\mathrm{ln}\left({\lambda}_{i}\right)$ | (8) |

Surface variation | $S{V}_{\lambda}\left(\varrho \right)=\frac{{\lambda}_{3}}{{{\displaystyle \sum}}_{i=1}^{3}{\lambda}_{i}}$ | (9) |

Sum of eigenvalues | ${\Sigma}_{\lambda}\left(\varrho \right)={\displaystyle \sum _{i=1}^{3}}{\lambda}_{i}$ | (10) |

Anisotropy | ${A}_{\lambda}\left(\varrho \right)=\frac{{\lambda}_{1}-{\lambda}_{3}}{{\lambda}_{1}}$ | (11) |

Verticality | ${V}_{\lambda}\left(\varrho \right)=1-\left|\left(\left[001\right],{\mathrm{e}}_{3}\right)\right|$ | (12) |

**Table 2.**Comparison of recall, precision, F-measure and overall accuracy for the 5 feature combinations after 5-fold cross validation. The x sign indicates the features chosen per each combination.

Combination n. | Geometric Features | Z-Coordinate | RGB | Intensity | Avg. Precision | Avg. Recall | Avg. Overall Accuracy | Avg. F-Measure |
---|---|---|---|---|---|---|---|---|

1 | x | - | - | - | 97.83% | 98.14% | 99.20% | 97.98% |

2 | x | x | - | - | 98.73% | 98.90% | 99.50% | 98.81% |

3 | x | x | x | - | 98.56% | 98.79% | 99.40% | 98.68% |

4 | x | x | - | x | 98.54% | 98.77% | 99.40% | 98.65% |

5 | x | x | x | x | 98.47% | 98.47% | 99.40% | 98.61% |

Classes | Combination n. 2 Geometric Features + Z | Combination n.3 Geometric Features + Z + RGB | |||||
---|---|---|---|---|---|---|---|

Precision | Recall | F-Measure | Precision | Recall | F-Measure | ||

1—Roof | 99.04% | 99.33% | 99.18% | 99.01% | 99.15% | 99.08% | |

2—Roof moldings | 99.45% | 99.18% | 99.31% | 99.28% | 99.14% | 99.21% | |

3—Façade | 99.75% | 99.60% | 99.67% | 99.72% | 99.56% | 99.64% | |

4—Arch 1st floor | 96.88% | 98.10% | 97.49% | 96.51% | 97.80% | 97.15% | |

5—Vaults | 99.50% | 99.73% | 99.62% | 99.50% | 99.74% | 99.62% | |

6—Column capital | 97.76% | 98.33% | 98.05% | 97.46% | 97.88% | 97.67% | |

7—Column shaft | 99.97% | 99.97% | 99.97% | 99.94% | 99.94% | 99.94% | |

8—Column base | 98.75% | 99.35% | 99.05% | 98.19% | 99.09% | 98.64% | |

9—Sill | 99.54% | 99.15% | 99.34% | 99.37% | 99.17% | 99.27% | |

10—Window frame | 95.05% | 95.04% | 95.04% | 94.52% | 95.50% | 95.01% | |

11—Window | 98.04% | 97.98% | 98.01% | 98.15% | 97.71% | 97.93% | |

12—Pavement | 99.94% | 99.92% | 99.93% | 99.93% | 99.91% | 99.92% | |

13—Arch ground floor | 98.68% | 98.70% | 98.69% | 98.54% | 98.10% | 98.32% | |

14—Buttress shaft | 99.80% | 99.85% | 99.82% | 99.76% | 99.88% | 99.82% | |

15—Buttress capital | 97.43% | 97.91% | 97.67% | 97.24% | 98.09% | 97.66% | |

16—Base | 98.97% | 99.29% | 99.13% | 98.68% | 99.03% | 98.85% | |

17—Ground | 99.89% | 99.87% | 99.88% | 99.86% | 99.81% | 99.83% | |

Average values | 98.73% | 98.90% | 98.81% | 98.57% | 98.79% | 98.68% |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Croce, V.; Caroti, G.; De Luca, L.; Jacquot, K.; Piemonte, A.; Véron, P.
From the Semantic Point Cloud to Heritage-Building Information Modeling: A Semiautomatic Approach Exploiting Machine Learning. *Remote Sens.* **2021**, *13*, 461.
https://doi.org/10.3390/rs13030461

**AMA Style**

Croce V, Caroti G, De Luca L, Jacquot K, Piemonte A, Véron P.
From the Semantic Point Cloud to Heritage-Building Information Modeling: A Semiautomatic Approach Exploiting Machine Learning. *Remote Sensing*. 2021; 13(3):461.
https://doi.org/10.3390/rs13030461

**Chicago/Turabian Style**

Croce, Valeria, Gabriella Caroti, Livio De Luca, Kévin Jacquot, Andrea Piemonte, and Philippe Véron.
2021. "From the Semantic Point Cloud to Heritage-Building Information Modeling: A Semiautomatic Approach Exploiting Machine Learning" *Remote Sensing* 13, no. 3: 461.
https://doi.org/10.3390/rs13030461