Three-Dimensional Point Cloud Semantic Segmentation for Cultural Heritage: A Comprehensive Review

Abstract

In the cultural heritage field, point clouds, as important raw data of geomatics, are not only three-dimensional (3D) spatial presentations of 3D objects but they also have the potential to gradually advance towards an intelligent data structure with scene understanding, autonomous cognition, and a decision-making ability. The approach of point cloud semantic segmentation as a preliminary stage can help to realize this advancement. With the demand for semantic comprehensibility of point cloud data and the widespread application of machine learning and deep learning approaches in point cloud semantic segmentation, there is a need for a comprehensive literature review covering the topics from the point cloud data acquisition to semantic segmentation algorithms with application strategies in cultural heritage. This paper first reviews the current trends of acquiring point cloud data of cultural heritage from a single platform with multiple sensors and multi-platform collaborative data fusion. Then, the point cloud semantic segmentation algorithms are discussed with their advantages, disadvantages, and specific applications in the cultural heritage field. These algorithms include region growing, model fitting, unsupervised clustering, supervised machine learning, and deep learning. In addition, we summarized the public benchmark point cloud datasets related to cultural heritage. Finally, the problems and constructive development trends of 3D point cloud semantic segmentation in the cultural heritage field are presented.

1. Introduction

The use of 3D point cloud data for cultural heritage assets is becoming widespread since 3D models provide various applications of digital documentation [1,2,3,4], interpretation of information and knowledge [5,6], and visual experience [7,8]. Point cloud data include massive 3D geometric information with colour and reflection intensity attributes. However, semi-automatic or automatic, efficient, and reliable solutions for linking point clouds with semantic information still need to be solved. Three-dimensional point cloud semantic segmentation (3DPCSS) has been regarded as a technology that links a single point or a set of points in the point cloud data with semantic labels (e.g., roof, facade, wall of architectural heritage, plants, damaged area, etc.) [9]. Moreover, 3DPCSS is the key step to extract implicit geometric features and semantic information from point cloud data for 3D scene understanding and cognition. Point clouds with semantic and geometric feature information make it easier for non-remote sensing professional users, such as cultural heritage managers, researchers, conservators, and restorers, to apply and understand the content of the data [10]. For example, the scan-to-BIM (building information modeling) [11,12] method, from point cloud to BIM [13] or heritage building information modeling (H-BIM) [14,15], mainly depends on 3DPCSS to extract geometric primitive, semantic information and building structure for parametric modeling. Smart point cloud [16,17] and rich point cloud [18] are new intelligent point cloud data structures that can be used for 3D scene understanding and decision-making, of which 3DPCSS acts as a core step of their technology framework. 3DPCSS can also achieve damaged area investigation, thematic mapping, and surface material analysis of cultural relics [19,20,21]. In addition, the scientometric method has proven that the semantic segmentation of point clouds is one of the hot topics of leveraging point cloud data in the cultural heritage field [22]. Therefore, it is evident that 3DPCSS has become the essential data processing step in cultural heritage.

Cultural heritage is represented by objects of nature, sizes, and complexity, which can be classified into tangible cultural heritage and intangible cultural heritage. Tangible cultural heritage consists of two categories, immovable cultural heritage (e.g., built heritage, monuments, archaeological sites, etc.) and moveable cultural heritage (e.g., paintings, sculptures, furniture, coins, etc.) [23]. The tasks of point cloud semantic segmentation are different at multi spatial scales, from landscape (i.e., immovable heritage and surrounding environments) to small artifacts (i.e., a part of an immovable heritage). In macro geographical space, a cultural heritage landscape is regarded as a complex dynamic environment composed of elements with semantic information such as land, vegetation, water, buildings (modern and historical buildings), and artifacts [24]. These elements can be reconstructed in geographic information systems (GIS) to support landscape and archaeological site annotation [25] and landscape planning [26]. With regard to the immovable heritage, point cloud semantic segmentation can improve the degree of automation of parametric modeling of objects at different levels of detail [27,28]. The common goal is to locate different elements in a three-dimensional point cloud scene, annotate them, and associate them with semantics and attributes and external knowledge databases [29]. At present, the application of point cloud semantic segmentation mainly focuses on immovable cultural heritage, especially historical buildings.

As point cloud semantic segmentation has been applied in various industries fields, such as urban and rural scenes [30], agricultural scenes [31], and railway environment [32], the field of cultural heritage has its unique features, including:

(1)

Point clouds in cultural heritage require a higher point density to express the complex geometric details of the object surface.

(2)

The basic geometric elements of cultural heritage include a lot of non-planar, curved geometrical shapes, irregular shapes, and complex structures [33].

(3)

Before handling point clouds, the segmentation categories always depend on the knowledge of experts in the field of cultural heritage.

(4)

For the same cultural heritage, the segmentation categories can be identified based on different research objectives and practical applications [34].

(5)

The same segmentation categories in different heritages have very significant morphological differences. For example, different historical periods and architectural styles include a variety of vaults supported by pillars of various patterns and shapes.

(6)

A high level of accuracy is required for the semantic segmentation of point clouds for applications such as structural analysis and damage detection [35].

3DPCSS has attracted significant attention in computer vision, remote sensing, and robotics. As a result, similar terms have emerged to describe the same meaning. In existing studies, point cloud semantic segmentation has also been known as classification [36,37] or point labeling [38,39]. In addition, 3D point cloud segmentation (3DPCS) is also an essential task for point cloud data processing. 3DPCS is essentially a process of extracting the specific geometric structures or features in cultural heritage scenes based on geometric constraints and statistical rules rather than explicit supervised learning or prior knowledge [9]. 3DPCS also groups point clouds into subsets with one or more common characteristic, whereas 3DPCSS means defining and assigning points to specific classes with semantic labels according to different criteria [40]. Therefore, 3DPCS algorithms are also included in this article.

As shown in Figure 1, it demonstrates an overall pipeline from point cloud data acquisition to applications in the cultural heritage field. Three-dimensional laser scanning and photogrammetry are the techniques that transform objects in the physical world into point cloud data. The natural characteristics of high-density, incompleteness, massive, and no neighborhood information in point cloud data bring significant challenges to be directly used in the cultural heritage field [41,42]. 3DPCS and 3DPCSS provide a high-level and semantic representation for original point cloud data to be applied in digital orthophoto maps, damaged area investigations, object recognition, BIM/HBIM, etc.

We searched the Web of Science database (WoS) on 20 October 2022 using “(point cloud*) AND (segmentation OR classification)” to conduct queries through “Topic”, which includes the title, abstract, keywords, and keyword plus. We further filtered the documents by checking “review article” under the “document types”, obtaining 96 papers. Finally, four papers related to cultural heritage were selected manually. Specifically, Yang et al. (2020) [43] focused on built heritage documentation using geometry modeling, knowledge management, and BIM/HBIM. Semantic segmentation of 3D point clouds can enhance the ability of knowledge fusion in built heritage modeling, but it is not the main topic of the paper. Bassier et al. (2020) [44] reviewed the interpretation and reconstruction of raw point cloud data in the construction industry, and semantic segmentation was mentioned as a key step. Moyano et al. (2021) [45] reviewed point cloud segmentation for HBIM parameterization in the field of architecture and archaeology, but semantic segmentation algorithms are not the paper’s focus. Rashdi et al. (2022) [46] explained the point cloud data processing in scan-to-BIM methodologies, such as sampling, registration, and semantic segmentation. However, the above review articles focus on BIM and HBIM technology in the construction field without comprehensively reviewing the 3DPCS and 3DPCSS algorithms and extensive applications in the cultural heritage field. In addition, we conducted a supplementary search on Google Scholar using the same search conditions as WoS. We retrieved one review paper about point cloud semantic segmentation algorithm in cultural heritage. Grilli et al. (2017) [40] carried out a short review of point cloud segmentation and classification algorithms in cultural heritage, but it was not entirely exhaustive, especially lacking detailed knowledge and applications.

Each of the five mentioned review papers have explored 3DPCS and 3DPCSS in building heritage from a specific point of view. Therefore, it is necessary to propose a comprehensive review to cover the up-to-date studies and efficiently organize them. It is supposed to contain an introduction to point cloud data acquisition and a detailed summary of the algorithms and applications of point cloud semantic segmentation in the cultural heritage field. This review aims to introduce 3DPCS and 3DPCSS algorithms and comprehensively analyze applications in different segmentation tasks. The various sources (platforms or sensors) of point cloud raw data significantly affect the performance of 3DPCSS algorithms. Therefore, this paper first surveys the existing studies from two perspectives of 3D point cloud data acquisition in cultural heritage, i.e., a single platform with multiple sensors and multi-platform data fusion. Then, we classified 3DPCS algorithms into three types: region growing, model fitting, and clustering-based methods. 3DPCSS algorithms are classified into supervised machine learning and deep learning methods. The criteria for the classification are determined by those general review papers [9,40,47]. Since the edge-based method usually reviewed in the literature has barely been used for processing cultural heritage, we excluded it from this survey. Last, we discussed the current challenges and potential future research directions.

The rest of this paper is organized as follows. Section 2 introduces the acquisition of multi-source point cloud data in cultural heritage. Section 3 summarizes the three types of 3DPCS methods in detail. The learning-based method of 3DPCSS and public benchmark datasets are summarized in detail in Section 4. Section 5 discusses some existing problems and development trends in 3DPCSS. Finally, some conclusions are drawn in Section 6.

2. Three-Dimensional Point Cloud Data in Cultural Heritage

This section first introduces point cloud data acquisition in the field of cultural heritage and the performance of various data acquisition technologies of different types of cultural heritage at different spatial scales. Then, we summarize the characteristics of data acquisition and data fusion from two perspectives, i.e.:

(1)

A single platform with multiple sensors: a point cloud data acquisition platform equipped with various sensors can obtain additional information in a single acquisition task. This additional information can improve the effect of 3D point cloud segmentation and semantic segmentation.

(2)

Multi-platform data fusion: combining the advantages of point cloud data acquisition of different platforms, a more complete and multi-resolution point cloud can be obtained by data fusion.

2.1. Point Cloud Data Acqusition Technologies

Photogrammetry and 3D laser scanning have become the best approaches for 3D point cloud data acquisition of cultural heritage. Salonia et al. (2009) [48] presented a quick photogrammetric system for surveying archaeological and architectural artifacts at different spatial scales. Multi-image photogrammetry with UAV has become the most economical and convenient 3D survey technology for cultural heritage landscapes, archaeological sites, and immovable heritage (e.g., historical buildings) [49,50,51]. Jeon et al. (2017) [52] compared the performance of image-based 3D reconstruction from UAV photography using different commercial software, e.g., Context Capture, Photo Scan, and Pix4Dmapper. Kingsland (2020) [53] used three types of digital photogrammetry processing software including Metashape, Context Capture, and Reality Capture for small-scale artifact digitization. In addition, computer vision (CV) is a mathematical technique for reconstructing 3D models in imagery [54]. CV together with photogrammetry is known as an image-based point cloud data acquisition method. Aicardi et al. (2018) [55] reviewed the definitions, similarities and differences between the two technologies in detail. In the last decade, software, hardware, and algorithms proposed in the CV field have improved photogrammetric solutions in terms of workflow automation and computational efficiency. It is not easy nowadays to underline the actual break point between the photogrammetry and CV approaches. Dense matching [56,57,58], multiple-view stereovision (MVS), and structure from motion (SFM) [59,60,61,62,63] as CV algorithms have been widely used in structural engineering and conservation and maintenance restoration of sites and structures belonging to the cultural heritage field. Photometric stereo is another popular method for reconstructing small objects, which can recover detailed surface shapes, even on the objects that are without texture. This is an advantage of photometric stereo over the geometry methods [64,65].

Three-dimensional laser scanning is another technology used to acquire cultural heritage point cloud data. A variety of 3D laser scanning platforms can be used to obtain 3D point cloud data of cultural heritage at different spatial scales, such as airborne laser scanning (ALS), terrestrial laser scanning (TLS), mobile laser scanning (MLS), and handheld laser scanning. For example, Risbøl et al. (2014) [66] used ALS data for change detection of the detailed information of a landscape and individual monuments automatically. Damięcka-Suchocka et al. (2020) [67] used TLS point cloud data at the millimeter level to investigate historical buildings, walls, and structures for conducting inventory activities, documentation, and conservation work. Di Filippo et al. (2018) [68] proposed a wearable MLS for collecting indoor and outdoor point cloud data of complex cultural heritage buildings. Lou et al. (2022) [69] proposed a methodology to digitize, extract, and classify cave features of rockeries in Chinese classical gardens by a handheld laser scanner with a camera. Ramm et al. (2022) [70] used a structured-light 3D sensor and a photo camera to capture 3D models of museum objects with a resolution of 0.1 mm.

The requirements of documentation and application determine the choice of technologies, platforms, and sensors. Gomes et al. (2014) [71] reviewed the 3D digitization of cultural heritage in the past and the technical workflow of 3D reconstruction. Maté-González et al. (2022) [72] compared the performance of three lidar techniques (TLS, ALS, and MLS) in surveying built heritage in vegetated areas. Ruiz et al. (2022) [73] made a comparative analysis between the main 3D scanning techniques: photogrammetry, TLS, and a structured light scanner in sculpture heritage.

Table 1. An overview of various point cloud acquisition technologies for different types of cultural heritage.

Technology	Point Density	Advantages	Disadvantages	Spatial Scales
Photogrammetry	Depends on the resolution of the camera sensors	Including colour and spectral information, can be installed on different platforms,	Influenced by light and shadows	Landscape, immovable heritage, and movable heritage
3D laser scanning—ALS	Low density	rapid acquisition of a wide range (meters to centimeters resolution), and able to penetrate occlusion	2.5D point clouds	Landscape scale (heritage landscape, large site)
3D laser scanning—TLS	High desity	High accuracy (centimeter-to-millimeter resolution), access to a geometric surface, and structural details	Expensive and time-consuming, object occlusion	Immovable heritage scale (archaeological site, small landscape, and historical building)
3D laser scanning—MLS	Middle density	High accuracy (centimeter resolution), larger measurement range, and higher efficiency than TLS	Expensive and time-consuming, object occlusion	Landscape and immovable heritage scale
3D laser scanning—Handheld	From high desity to very high density	Very high accuracy (centimeter to submillimeter)	Expensive and time-consuming	Immovable and movable heritage scale (artefacts, objects, a part that is immovable)

provides the basic information about various point clouds, including technology, point density, advantages, disadvantages, and spatial scales.

2.2. A single Platform with Multiple Sensors

In the field of cultural heritage, the equipment used for acquiring point cloud data is mainly based on 3D lidar and photogrammetric cameras equipped with unmanned aerial vehicles (UAV), mobile vehicles, robots, terrestrial methods, handheld devices, and other platforms. The purpose of multiple sensors equipped with one platform is to obtain more information in one data acquisition process. For example, Nagai et al. (2009) [74] proposed a UAV 3D mapping platform consisting of charge-coupled device cameras, a laser scanner, an inertial measurement unit, and a global positioning system (GPS). Erenoglu et al. (2017) [75] used digital, thermal, and multi-spectral camera systems on a UAV platform to collect visible, thermal, and infrared radiations of the electromagnetic spectrum. That multi-source information was employed to produce a highly accurate geometric model of an ancient theater and it revealed the material features because of the spectral classification. Rodríguez-Gonzálvez et al. (2017) [76] employed the Optech LYNX Mobile Mapper platform on a vehicle to acquire 3D point clouds of enormous cultural heritage sites. The platform includes two lidar sensors, four RGB cameras, and an inertial navigation system. During data collection, the system can simultaneously acquire point cloud data, colour information, and spatial geographic references. Milella et al. (2018) [77] assembled a stereo camera, a visible and near-infrared camera, and a thermal imager on a robot with an inertial measurement unit (IMU) to identify soil characteristics and detect changes through the integration of data from different sensors. Hakala et al. (2012) [78] designed a prototype of a full waveform hyperspectral terrestrial laser scanner to produce hyperspectral 3D point clouds, which can provide visualization and automate classification of the point cloud and calculation of spectral indices for extraction of target physical properties. Zlot et al. (2014) [79] developed a handheld mobile mapping system called Zebedee for cultural heritage applications. The system consisted of a laser scanner, a camera, and an IMU, suitable for 3D data acquisition of a large range of complex cultural heritage scenes and capable of collecting data from inside buildings.

To sum up, two main trends exist in using a single platform for scientific observation of cultural heritage. The first is a mobile platform consisting of a global positioning system and inertial navigation, which can achieve the global geospatial reference of cultural heritage without ground control points. The mobile platform can reduce the time required for planning sensors’ network and installing instrumentation. This approach can quickly obtain point cloud data in a large area with a complex spatial structure, such as cultural heritage landscapes and historical buildings. Planning sensor network deployment by non-professional researchers may not be justified due to a lack of expertise [80]. However, cultural heritage authorities may prohibit UAVs and mobile vehicles from accessing cultural heritage sites. Sometimes, there is not enough space and road for flying drones and driving through. In addition, the accuracy of the mobile platform is lower than the ground-station-based method. The second trend is that the content of point cloud data evolves from geometric to synchronous acquisition with the spectrum and texture information in one data acquisition.

Table 2. Three-dimensional point cloud data acquisition using a single platform with multiple sensors.

Paper	Platform	Sensors	Data Characteristics
Nagai et al. [74]	UAV	● Charge-coupled device cameras ● Laser scanner ● Inertial measurement unit ● GPS	Terrain shapes, detailed textures, and global geospatial references.
Erenogl et al. [75]	UAV	● Stereo camera ● Visible, thermal, and infrared radiations sensor	Geometric features and material classification information
Rodríguez-Gonzálvez et al. [76]	Mobile vehicle	● Laser scanner ● RGB cameras ● Applanix POS LV 520 IMU ● GPS	Colour information and spatial geographic reference
Milell et al. [77]	All-terrain vehicle	● Stereo camera ● Visible and near infrared camera ● Thermal imager	Colour, geometry, spectral, and mechanical properties of soil
Hakala et al. [78]	Ground station	● Full waveform hyperspectral ● Laser scanner	Hyperspectral point clouds
Zlot et al. [79]	Handheld	● Laser scanner ● IMU ● Cameras	Both site context and building detail comparable in accuracy

summarizes the data acquisition cases of a single platform with multiple sensors.

2.3. Multi-Platform Data Fusion

In cultural heritage, it is difficult for a single platform to meet the requirements of point cloud integrity and accuracy within its limited observation perspective [81]. Multi-platform collaborative observation can effectively integrate the advantages of different platforms. Multi-source point cloud fusion can make the scene have a more complete spatial scale and geometric information.

Multi-platform data fusion can make up for the problems of incomplete data and excessive data errors that exist in a single platform. For example, Fassi et al. (2011) [82] proposed that in order to understand the global structure of complex artifacts (e.g., Milan Cathedral’s main spire, Italy), together with its reconstruction accuracy, connections, and topological and geometrical logic, different instruments and modeling methods must be used and integrated. Achille et al. (2015) [83] constructed a 3D model of interior and exterior buildings with complex structures by integrating UAV photogrammetry and TLS data. Galeazzi (2016) [84] combined 3D laser scanning and photogrammetry to reconstruct the archaeological record of cave microtopography in extreme environments, such as extreme humidity, difficulty of access, and challenging light conditions. Zaragoza et al. (2017) [85] integrated UAV photography with TLS for 3D documentation in a hazardous situation. The UAV photography method has higher accuracy at the roof surface area, while the TLS method is more suitable for obtaining the facade area (such as walls). Herrero-Tejedor et al. (2020) [86] used the UAV-TSL model in a densely vegetated cultural heritage area (an ancient garden). In their data acquisition process, densely vegetated areas hindered the penetration of airborne lidar, making it challenging to detect near-ground objects. UAV photography destroys the image effect due to ground shadow and vegetation occlusion, resulting in point cloud noise in the near-ground area.

Another purpose is to reconstruct a multi-spatial resolution point cloud model for cultural heritage. For example, Guidi et al. (2009) [87] reported on a multi-resolution and multi-sensor approach developed for the accurate and detailed 3D modeling of the entire Roman Forum. Abate et al. (2018) [88] provided a 3D resolution spanning from a few centimeters in the landscape digital terrain model (DTM) to a few millimeters in the layer-by-layer archaeological site. Young et al. (2019) [89] used a combination method of TLS and UAV photogrammetry to establish a 3D model of a temple. In the study, UAV photogrammetry yielded a higher planar data acquisition rate on the roof of a building than TLS. However, laser scanning was observed to provide higher positional accuracy than photogrammetry. A few studies of multi-platform data fusion are shown in

Table 3. The multi-platform data fusion.

Case	Platform and Main Sensors	Application
Fassi et al. [82]	● UAV Photogrammetry (Canon 5D Mark II) ● TLS (Leica HDS6000)	Integrating different instrumentation and modeling methods to surveying and modeling very complex architecture (Main spire of MILAN CATHEDRAL)
Achille et al. [83]	● UAV Photogrammetry (Canon EOS 5D Mark III with a 35 mm lens) ● TLS (Leica HDS 7000)	Integration of the building’s interior and exterior 3D model with a tall and complex façade.
Galeazzi [84]	● Ground Photogrammetry (Nikon D90 at 12 MPixel with a 60 mm Nikkor lens) ● TLS (Faro Focus 3D) ● DeWALT DC020 fluorescent light	3D documentation of archaeological stratigraphy in extreme environments characterized by extreme humidity, access difficulty, and challenging light conditions.
Zaragoza et al. [85]	● UAV Photogrammetry (Canon IXUS 220HS with 35 mm lens) ● TLS (Riegl Vz1000)	Integrate the survey of roofs, gardens, and inner courts.
Herrero-Tejedor et al. [86]	● UAV Photogrammetry (RGB Phantom 4 camera and Micasense RedEdge multi-spectral camera) ● TLS (Faro Focus S330)	3D documentation for the management and conservation of cultural landscapes with unique biogeographical features
Guidi et al. [87]	● UAV Photogrammetry (Zeiss RMK A 30/23) ● TLS (Leica HDS3000 and Leica HDS6000) ● Ground Photogrammetry(Canon 10DCanon 20D, Kodak DCS Pro)	Multi-resolution 3D modeling of the complex area of Roman Pompeii (150 m × 80 m), DSM (25 mm), medium resolution 3D model (5–20 mm), ground photogrammetry (0.5–10 mm)
Abate et al. [88]	● UAV Laser scanner ● UAV Photogrammetry (Canon 5D Mark v)	Centimeter to millimeter multi-resolution 3D model of Treblink concentration camp (3.75 square kilometers)
Young et al. [89]	● TLS (Leica, Scan Station C10) ● Fisheye camera (SIGMA, F3.5 EX DG Circular fisheye) ● UAV Photogrammetry (SONY, Alpha 6000)	Establish a 3D model and the associated digital documentation of the Magoksa Temple, Republic of Korea.

.

3. 3D Point Cloud Segmentation

This section introduces the main methods and applications of 3DPCS in cultural heritage. 3DPCS is mainly used in the recognition and extraction of basic geometric shapes from point clouds, especially immovable cultural heritage. The 3DPCS algorithms is divided into three parts: region growing, model fitting, and clustering-based. Among them, the region growing method is used to detect and segment planes from point clouds of historic buildings. The model fitting method can segment more basic geometric or irregular geometric shapes. These geometric elements are mainly used for the parametric modeling of BIM and HBIM. The model fitting and clustering-based method can provide surface defect detection and deformation analysis of immovable heritage by calculating the distance between the real point and the fitted plane.

3.1. Region Growing

The region growing method is a classic algorithm in 3DPCS. This algorithm is widely used in the geometric segmentation of plane structures [90,91]. In addition, the region growing algorithm is also applied to segment the plane elements from the point cloud of historic buildings. [92,93]. The basic idea of the region growing algorithm is to merge two spatial points or two spatial regions when they are close enough in a particular geometric measure.

Three critical factors need to be considered when constructing a region growing algorithm with different strategies. The first key factor is selecting the appropriate seed point or region [94,95,96]. The second key factor is dividing the region growth unit to improve the computational efficiency using region unit division or hybrid units division, such as voxel [97], super voxel [98], KD-Tree [94], and Octree [99] structures, etc. For example, Xiao et al. (2013) [100] used a single point and a sub-region as growth units to detect planes. Dong et al. (2018) [101] adopted a hybrid region-growing algorithm based on a single point and super voxel units to achieve coarse segmentation before global energy optimization. The third key factor is determining the appropriate criteria for measuring the similarity, including the normal vector, the distance between adjacent points and the adjustment plane, and the distance between the current point and the candidate point [90,92].

The region growing method is mainly applied to segmenting walls and roofs of historic buildings. For example, Grussenmeyer et al. (2008) [92] selected the gravity point of each voxel unit as the seed point for region growing and then extracted the plane for parameter modeling from the TLS point cloud of a medieval castle. Paiva et al. (2020) [93] extracted plane elements from point cloud data of five different styles and periods of historical buildings. Their method combines hierarchical watershed transform and curvature analysis with region growing to obtain more suitable growing seeds. The method can apply to multi-source point clouds from drones and terrestrial laser scanners. Pérez-Sinticala et al. (2019) [102] used a hybrid region growing algorithm and primitive fitting by sample consensus appears, simplifying the point clouds into a simpler model based on geometric primitives, such as walls, towers, segmentation, and automatic recognition of roofs and slopes.

The region growing method has been proven to be able to segment point cloud data for ancient historical buildings effectively. However, there are still some problems with its weak generalization ability and complex computing power [103]. For example, the reliability of the segmentation results depends on the growth criteria of the seed. Selecting parameters (such as the number of nearest points) and determining region growth units require prior knowledge, leading to poor generalization of the algorithm. The point-wise calculation is time-consuming for massive point clouds (millions, tens of millions of points).

3.2. Model Fitting

Model fitting is a shape detection method that can match point clouds to different original geometries. Therefore, it also could also segment regular geometric shapes from point clouds. The two most important model fitting algorithms are Hough transform (HT) and random sample consensus (RANSAC).

3.2.1. Hough Transform (HT)

Hough transform can detect parametric geometric objects such as lines [104], planes [105], cylinders [106], and spheres [107] in point clouds. There have been several review articles about 3D Hough transform [108,109,110]. Concerning HT, a sample extracted from the 2D images and 3D point cloud data in an original space is mapped into a discretized parameter space, where an accumulator with a cell array is put. A vote is then cast for each sample’s basic geometric element, represented by parameter coordinates in the original space. The cell with the local maximal score is selected as the output.

One of the applications of the HT algorithm in cultural heritage is to extract planner features. For example, Lerma and Biosca (2005) [111] extracted planner surfaces from the point cloud of a monument through the HT algorithm to remove irrelevant points and reduce the data volume. Another application is to create a digital orthophoto map of cultural relics. A digital orthophoto map is an essential documentation of two-dimensional reference systems in cultural heritage. Selecting the appropriate projection surface is the key to generating orthoimages with sufficient geometric accuracy and visual quality [112]. Markiewicz et al. (2015) [113] utilized extended random Hough transform (ERHT) to detect horizontal and vertical planes in point clouds for automatically generating orthophotos from TLS data and digital images. Maltezos et al. (2018) [114] proposed adaptive point random Hough transform (APRHT) to extract planes from polyhedral cultural heritage. The advantage of APRHT is that it automatically selects subregions around the adaptive center point, an automatic descent adjustment process for parameters. It removes points that meet distance tolerances and additional normal tolerances concerning the probed plane. Therefore, the geometric constraint criteria used by APRHT and the new properties of the accumulator parameters make this algorithm widely applicable, scalable, and general. Alshawabkeh [115] proposed a depth-image-based method for linear feature extraction to detect façade surface features such as edges and cracks from TLS point clouds. This work clearly explains and quantifies weathering processes and dangerous fissures in a survey of the Treasury monument in the ancient city of Petra, Jordan.

The advantage of the HT algorithm is that all points are processed independently and they are not affected by outliers. It is robust to noise and can recognize multiple geometric shapes (such as multiple different planes). The disadvantages of the HT algorithm are large computation, high computational complexity, and the choice of parameter length [110].

3.2.2. Random Sample Consensus (RANSAC)

In 1981, Fischler and Bolles [116] proposed the RANSAC model fitting method, which has been widely used in computer vision to detect simple shapes. Since then, many classical RANSAC-based algorithms have been proposed for detecting planar features [117,118,119] and irregular geometric shapes [120,121] from point clouds.

The RANSAC algorithm supports the segmentation of historic buildings to create a subset of point clouds belonging to building components, including basic geometric shapes such as planes, spheres, cylinders, cones, and tori. For example, Aitekadi et al. (2013) [122] extracted the principal planes of classical architecture from coloured point cloud data combined with RGB values, laser intensity, and geometric data. Chan et al. (2021) [123] used the RANSAC algorithm to separate individual planar features, which is the first step of the point cloud colourization method based on a point-to-pixel orthogonal projection. Kivilcim and Zaide (2021) [124] extracted the geometries of architectural façade elements from airborne lidar and ground lidar data that contain noise. The extracted geometric elements facilitate transfer to BIM following Industry Foundation Classes (IFC) standards. Macher et al. (2014) [125] segmented geometric elements based on the RANCAS paradigm, such as planes, cylinders, cones, and spheres from the TLS point cloud data of churches and fortresses in the 11th and 12th centuries. Andrés et al. (2012) [126] used the least median of squares (LMedS) and improved the RANSAC to segment the surface of the Antioch Gate in an ancient building in Aleppo, Syria. The segmented results were used to construct a parametric geometric model to calculate structural deformation through finite element analysis (FEA).

The RANSAC algorithm can also detect defects on the surface of cultural relics by calculating the spatial distance between the original point cloud and the fitted plane. For example, Nespeca and Luca (2016) [127] created a deviation map between the point cloud of a wall and the RANSAC-fitted plane in the Saint-Maurice church in Carrom, southern France. The deviation map can show areas of material loss on the artifact’s surface. The roughness of the material surface of the cultural relic shows the degree of surface corrosion and can indicate stormwater runoff. Poux et al. [128] used RANSAC and the convex hull algorithm to extract contour polygons of each tesserae (a small piece of stone, glass, ceramic, or other hard material cut in a cubical or some other regular shape in mosaic work). The tesserae are classified by domain knowledge of size, geometry, and spatial distribution.

Due to high computational requirements, the HT method exhibits poor runtime performance when applied to large-scale and complex geometric structures of cultural heritage. At the same time, the RANSAC algorithm significantly improves massive point cloud data processing. The RANSAC algorithm shows strong robustness and effectiveness when dealing with point cloud models [129] and can even handle data containing more than 50% outliers robustly [130].

3.3. Unsupervised Clustering Based

Point cloud unsupervised classification algorithms mainly include K-means [131], mean shift [132], and fuzzy clustering [133]. Cluster-based 3DPCS methods are more suitable for irregular geometric objects than region growing and model fitting methods because they do not require the basic geometry of objects to be predefined. Cultural heritage surface or structural diseases often manifest as irregular geometric features such as biological diseases, weathering, cracks, and partial defects. Therefore, this algorithm is very suitable for extracting surface defects.

Segmenting damaged areas with similar geometric, colour, and reflection intensity features from the 3D point cloud data and then making a map of cultural heritage disease investigation have become an efficient method for diagnosing conservation status [134,135]. Armesto-González et al. (2010) [136] used TLS with unsupervised classification methods to produce a thematic map of damage affecting building materials. This work tested three types of 3D laser scanning equipment (FARO Photon, TRIMBLE GX200, and RIEGL-Z390i) with different wavelengths. The best result for the classification is the fuzzy K-means algorithm rather than K-means or ISO data. Sánchez-Aparicio et al. (2018) [137] used the fuzzy K-means method to detect certain types of pathological processes (biological colonization, salts, or moisture) from point cloud data of highly affected historical masonry. Wood et al. (2021) [138] selected three damage-sensitive features (covariance-based, normal vector-based, and curvature-based) and used the ordering points to identify the clustering structure (OPTICS) classifier [139] to detect damaged areas (surface damage, defects, and cracks) in frescoes. The advantage of the OPTICS classifier is to detect clusters with different shapes and spatial distributions in a 3D space. This classifier is suitable for the classification of damaged regions with random patterns. An unsupervised clustering method based on point clouds supports multi-dimensional feature classification. Therefore, the reflection intensity and colour information from point cloud data acquisition can improve classification accuracy [140,141]. The disadvantage is the difficulty of choosing suitable predefined parameters to improve the segmentation effect. The abovementioned research [136,137,138] all involve predefined parameters and parameter selection. Hou et al. (2017) [142] compared the efficiency of four clustering algorithms (K-means, fuzzy C-means, subtractive clustering, and density-based spatial clustering of applications with noise (DBSCAN)) in detecting damage areas on modern building surfaces. As a result, K-means and fuzzy C-means’ performance were better than subtractive clustering and DBSCAN. Although the test was applied in modern architecture, the method used provides a reference for the damaged area investigation of historical buildings.

4. Three-Dimensional Point Cloud Semantic Segmentation

This section introduces the 3DPCSS algorithms, the public benchmark dataset, and their application in cultural heritage. Currently, the study case of the 3DPCSS algorithms is mainly based on historical buildings in Europe, with complex structures, diverse roof types, decorated windows, different styles of columns, and complex geometric decorations. The 3DPCSS algorithm can segment building elements and the surrounding environment (the ground and trees) in one task. These elements have different semantic and physical properties and functions in cultural heritage. The result of the segmentations are used for parametric modeling, incorporating digital models of artifacts into the HBIM environment.

4.1. Supervised Machine Learning

3DPCSS based on machine learning has been widely used for the semantic understanding of indoor and outdoor 3D urban scenes [143,144]. This method has also become a trend in classifying complex structures from cultural heritage point clouds, especially ancient architecture [145,146]. The basic process for importing point clouds to outputting semantic annotation 3D point clouds includes four steps [38,147,148]:

(1)

Point cloud neighborhood selection.

(2)

Local feature extraction.

(3)

Salient feature selection.

(4)

Point cloud supervised classification.

Weinmann et al. (2015) [147] discussed seven methods for defining point cloud proximity, twenty-one geometric features, seven feature selection methods, and ten classifiers for supervised classification. They used two standard datasets (urban 3D point clouds) to evaluate the method in a detailed assessment (Figure 2).

The applications of 3DPCSS based on supervised machine learning involves investigating damaged areas [149,150] and extracting building structures [34,151,152]. Depending on the data source, point density, object type, and application, we have different choices for conducting the segmentation steps. The detailed summary of semantic segmentation in

Table 4. Application of unsupervised machine learning 3DPCSS in the cultural heritage field.

Case	Object	Classification	Neighborhood Selection	Feature Extraction	Geometry Feature Selection	Classifier
Grilli et al. [149]	European historical buildings, ancient ruins, and stone cultural relics	Damaged areas	-	-	Orthophoto or UV map 2D supervised machine learning projection onto 3D data	j48, random tree, RepTREE, LogitBoost, random forest, fast random forest (16), and fast random forest (40)
Valero et al. [150]	Ancient ruins wall	Wall structure and damage information (erosion, delamination, mechanical, damage, and non-defective)	-	17 colour-related features, 16 geometric features	Ten geometric features	Logistic regression multi-class classifier, and binary classifier
Grilli et al. [151]	European historical buildings	Nine structures	0.1–0.8 m	Radiometric features and 77 multi-scale geometric features	Seven geometric features	Random forest classifier
Croce et al. [152]	European historical buildings	19 structures	0.2 m, 0.4 m, and 0.6 m	27 geometric features, RGB values, laser scanner intensity, and point cloud Z coordinate	Nine geometric features	Random forest classifier
Grilli et al. [34]	European historical buildings and Temple	Building: 15 structures Temple: 15 structures	-	Decentralized coordinates, radiometric values, and geometric features	Seven geometric features	Random forest one-versus-one (OvO) classifier
Teruggi et al. [153]	European historical buildings	Building structures, subdivision structures, and detailed structures	0.2 cm–3 m	Geometric features	Six geometric features	Random forest

is based on the process in Figure 2.

In order to improve the effect of semantic segmentation, some scholars [34,150,151,152] have used radiation features and colour information as additional features. Since the density of cultural heritage point cloud data generated by the fusion of various sensors is not uniform, it is necessary to select the point cloud neighborhood according to the actual situation of the point cloud data. There is a correlation between the point cloud neighborhood and its local feature extraction. For example, neighborhood selection differs at different spatial scales when calculating local features in the feature selection step [34,151,152]. Teruggi et al. (2020) [153] adopted the hierarchical machine learning method for semantic segmentation of multi-level and multi-resolution point clouds, which realizes the main structure of the building (such as roofs, the ground, and walls), to the basic elements of the building (such as doors and windows), to architectural details (such as ornaments and sculptures on pillars), etc. These cases demonstrate the success of machine learning algorithms in extracting cultural heritage disease information and the refined segmentation of structures. However, there still needs to be a gap between application in the cultural heritage field and the 3D urban scene. For example, the machine learning classifier used for segmentation is dominated by random forests. Such segmentation algorithms build local geometric feature descriptors based on a single point or a set of points in the point cloud for classification and do not consider point cloud’s contextual features. Although individual classifiers have high computational efficiency, they are prone to noise and affect classification accuracy.

4.2. Deep Learning

With the emergence of deep learning techniques, point cloud semantic segmentation has improved tremendously. In recent years, many deep learning models have emerged to achieve semantic segmentation of 3D point clouds [154].Compared with traditional segmentation algorithms, deep-learning-based model technology has a higher performance of multi-scale spatial 3D information with semantic information at a different level of granularity, such as semantic segmentation (scene level), instance segmentation (object level), and part segmentation (part level) [155,156]. According to the irregularity of point clouds, deep-learning-based point cloud semantic segmentation models are divided into indirect and direct methods [157]. The indirect method converts the irregular point cloud into a regular structure to achieve segmentation. For example, Pellis et al. (2022) [158] adopted the DeepLabv3+ deep learning model (one of the CNN models) for the 2D image semantic segmentation algorithm, used a pre-trained version of the network on the ImageNet database, and used ResNet-18 as the primary classification architecture. Finally, the semantic information was reverse mapped onto the point cloud through the direct linear transform (DLT) algorithm. The method was validated using four European architectural heritages to classify and label architectural 3D point clouds (Arch, column, moldings, floor, door, wall, stairs, vault, roof, other, and none). However, there are some unavoidable defects in the semantic segmentation of point clouds indirectly. Since the picture is not 3D information, the invisible areas cannot be classified.

PointNet [159] is a pioneering deep learning framework that directly uses the 3D information of the point cloud for classification. In a further study, Qi et al. (2017) [160] improved the basic PointNet model using a hierarchical neural network to capture local geometric features. Malinverni et al. (2019) [161] chose PointNet++ for the semantic segmentation of 3D point clouds of historical buildings and achieved remarkable results in terms of effect. Three-dimensional deep learning research focuses on enhancing features, especially local features, and relationships between points, using knowledge from cultural heritage fields to improve the performance of the basic algorithms of PointNet++. For that purpose, Wang et al. (2019) [162] designed a procedure called edgeconv to extract edge features while maintaining permutation invariance called dynamic graph CNN (DGCNN). Pierdicca et al. (2020) [29] proposed a basic framework to recognize historical architectural elements (Figure 3), which adopted an improved DGCNN (dynamic graph convolutional neural network), adding meaningful features. Morbidoni et al. (2020) [163] presented an improved version of the dynamic graph convolutional neural network (DGCNN) named RadDGCNN, which exploits the radius distance. Francesca et al. (2020) [164] made a comparison between machine learning methods (k-nearest neighbor, naive Bayes, decision trees, and random forests) and deep learning methods (DGCNN, DGCNN-Mod, and DGCNN-3Dfeat) on European architectural heritage point cloud data. Then, they proposed an architecture named DGCNN-Mod+3Dfeat for large 3D cultural heritage semantic segmentation. Chen et al. (2021) [165] proposed the ring grouping neural network architecture with an attention module (RGAM) to enhance complex scene recognition. Lee et al. (2021) [166] proposed a graph-based hierarchical DGCNN (HGCNN) model for representing 3D objects such as bridge components in the BIM framework. Yin et al. (2021) [167] proposed a deep-learning-based approach, i.e., ResPointNet++, by integrating deep residual learning with a conventional PointNet++ network to automatically create as-built BIM models from point clouds.

However, deep learning methods also have certain limitations. In order to build a suitable deep learning model, many human resources are required to label and train the data in the early stage. Many point clouds by manual means are very time-consuming. Moreover, compared with the urban 3D point cloud scene, the point cloud annotation related to cultural heritage requires specific relevant knowledge (a correct understanding of the cultural heritage scene and structure) and non-professional annotation is prone to errors.

4.3. Public Benchmark Dataset

The public standard benchmark dataset helps train the 3DPCSS models, based on which the automation of point cloud data processing and 3D reconstruction of cultural heritage can be improved. An intensive search indicated only four benchmark datasets that include cultural heritage, while most of them concern architectural heritage. An architectural cultural heritage (ArCH) dataset was collected by 3D laser scanning and oblique photography, including 17 annotated and ten unlabeled scenes [168]. The WHU-TLS point cloud benchmark dataset [169,170,171] is not a dataset dedicated to the cultural heritage field but includes a small part of architectural heritage. SEMANTIC3D.NET [172] presents a 3D point cloud classification benchmark dataset with manually labeled points. It covers a few historic buildings. Pepe et al. (2022) [173] shared a point cloud for 3D reconstruction of Temple of Hera (Italy) based on a photogrammetric approach and georeferenced through a UAV survey. The 3D pottery dataset [174] includes 1012 digitized, hand-modeled, and semi-automatically generated 3D models. Due to the wide variety of cultural heritage artifacts, such a small number of benchmark datasets for point clouds in the cultural heritage field hinders the automatic interpretation of cultural heritage scenes using point cloud data.

5. Discussion

5.1. Multi-Source Point Cloud Data

Multi-source point cloud data fusion aims at obtaining more complete data (TSL has difficulty observing building roofs), multi-resolution data (UAV and TLS data fusion), and additional information (colour, spectral information, etc.), as described in Section 2. Regarding point clouds from difference observation modes, spatial scales, and accuracy among platforms and sensors, multi-source point clouds have a series of problems, such as uneven point density, different colour spaces (e.g., RGB or HSV), multiple spatial scales, and additional heterogeneous sensor information. The characteristics of multi-source point clouds also bring new challenges to 3DPCSS. For example, 3D laser scanning point clouds contain information on laser reflection intensity. A dense matching point cloud of a photogrammetric image has colour information. Both reflection intensity and pixel colour contribute to point cloud semantic segmentation. However, in the same scene, the semantic segmentation of point cloud data obtained by 3D laser scanning and photogrammetry may lead to different results.

5.2. Over-Segmentation Results in Useless Classes

Over-segmentation is a common problem in semantic segmentation, resulting in many meaningless classifications. It is necessary to further structure regularization and smooth the segmentation results. Hao et al. (2022) [175] designed a self-supervised pretext task to improve the poor performance of point cloud semantic segmentation in detail processing, especially in boundary regions. Yang et al. (2022) [176] built an encoder–decoder network based on CRF (conditional random fields) graph convolution (CRFConv) to enhance the location ability of the network, thereby benefiting segmentation. However, there is no similar processing for 3DPCSS in cultural heritage. A potential solution to overcome the over-segmentation problem is introducing the contextual information or domain knowledge strategy in the 3DPCSS process [177]. For example, doors and windows of buildings often do not need more detailed semantic elements. Therefore, if a new category appears in the area of doors and windows, it needs to be merged into a major category. The 3DPCSS algorithm for cultural heritage can be improved through those known conditions, making the segmentation results more aligned with the human cognition. Colucci et al. (2021) [178] used an ontological scheme to guide the segmentation of point clouds from cultural heritage, then generated parametric geometries to be used in a historical building information model (HBIM).

5.3. Supervised Machine Learning Versus Deep Learning

Learning-based 3DPCSS algorithms require a large number of labeled point cloud data samples. Manual sample labeling methods are inefficient. Compared with typical 3D urban scenes, the annotation of point cloud samples in cultural heritage relies more on professional knowledge and domain experts. The weakly supervised learning method that has been applied in urban scenarios is an alternative approach to avoid this exhausting annotation [179,180].

Deep learning can automatically learn features from massive point cloud data without relying on manual design. However, there are few point cloud datasets of cultural heritage landscapes, immovable heritage, or objects, and the content needs to be more prosperous. The diversity of artifact geometries also results in an unbalanced sample of different artifact types. Especially in the field of cultural heritage, there are always some extremely complex geometric forms and unique appearances, such as decorative objects and sculptures. Furthermore, since point cloud data are unstructured data (discrete points in space), convolutional neural network (CNN) filters cannot be directly used in deep learning methods. Most deep learning models require extensive preprocessing of point clouds, such as voxelization, which increases algorithmic complexity and loses local details.

The classifiers and feature selections commonly used in machine learning methods are shown in Table 4. The training of these classifiers depends on the selection of valuable features. When using machine learning methods for classification, only the training data and the data to be classified have the same characteristics, and a better classification effect can be achieved. From the contributions of the literature [29,34,149,151,164], the learning-based method has mainly been applied to ancient European buildings achieving, satisfactory research progress and results, and it has accumulated some datasets. However, 3DPCSS algorithms are not used in other types of cultural heritage, such as grottos, caves, and ancient ruins.

5.4. The Application of 3DPCSS in Cultural Heritage

Point cloud data have been used for the 3D reconstruction of cultural heritage as digital documentation and visualization. As important spatial information in cultural heritage, point cloud data can express complex geometric shapes through sufficiently dense points. As shown in Section 2, point cloud data will result in uneven density through data fusion, and complex internal structures will lead to incomplete point clouds. In addition, point cloud data lack semantic, attribute, and functional information. These problems mean that point cloud data are difficult to use directly by cultural heritage experts. The user must identify collections of points that belong to individual surfaces and then fit surfaces and solid geometry objects appropriate for the analysis [181]. The point clouds of cultural heritage landscapes and archaeological sites need to generate vector data through semantic segmentation of elements for thematic mapping in GIS to realize spatial retrieval, semantic annotation, and heritage management. For the immovable heritage of historical buildings, semantic segmentation is also needed to build parametric models, such as 3D CAD or HBIM, to realize numerical simulation and restoration engineering management. However, those tasks are prohibitively time-consuming and, in response, 3DPCSS has focused on developing algorithms for automating vector data extraction and parametric or solid modeling processes.

The particularity of cultural heritage makes 3DPCSS face greater challenges in industrial application. For example, deformation analysis can provide results for structural analysis (stress and deformation) of wooden building construction. The algorithm needs to extract building units from complex structures and irregular geometric shapes [182]. The classification and analysis of surface damage areas of cultural heritage also require high segmentation. Ancient architectural heritage undergo deformation (straight boundary becomes circular arc) and local defects. Damage detection of a heritage surface is irregular, with geometric and small deformations. It is necessary to accurately calculate the distance between the area with high roughness and the actual surface. 3DPCSS can effectively assist the generation of 3D geometric models and solve key problems when rehabilitating a heritage building, such as controlling the deformations related to the static and dynamic structure behavior [183]. Three-dimensional geometric models can be further used for 3D numerical finite element analysis to study the structural response, such as providing information for construction rules, geometry, and connection between different structural elements [33,184]. In the reconstruction process, the point cloud can only cover the surface of the structure within the structure that is missing. Therefore, the result of segmentation cannot be directly converted to a solid model and it even requires knowledge related to the architectural structure or historical drawings. In the construction of HBIM and HBIM, the point cloud or mesh of 3D components can be inserted into the HBIM environment; in particular, parametric modeling makes it difficult to express complex geometric objects and structures (such as decorations and sculptures) [185].

5.5. Understanding and Cognition of Cultural Heritage 3D Scenes

3DPCSS promotes the 3D virtual reconstruction of cultural heritage with semantic information. An autonomous understanding of 3D scenes can be achieved by identifying the physical meaning of each point or a set of points in the point clouds. Furthermore, 3DPCSS results are widely used in 3D GIS, HBIM, and other 3D spatial information systems or models for complex calculations such as spatial and structural analysis. These calculations and analyses can help understand the spatial relationships and spatial patterns between physical entities/elements in cultural heritage 3D scenes. However, forming spatial cognition still depends on the complex data processing process to transform point clouds into raster or vector data for a 3D spatial information system. The geometric details of point clouds as a data source in presenting 3D scenes with complex geometric shapes and structures are unmatched by other parametric models. More refined three-dimensional representations are needed, especially in cultural heritage. For example, irregular surfaces may be disease areas. Point clouds still face significant challenges in forming autonomous cognitive abilities without relying on 3D spatial information systems.

The smart point cloud is a more potential development trend in the field of cultural heritage (a detailed discussion on the concept of the smart point cloud has been included in the papers published by Poux et al. [6,17,18,186]). However, each point in the point cloud data after point cloud semantic segmentation has spatial coordinates, physical attributes, semantic information, etc. These tasks transform the point cloud data into a high-dimensional point cloud. High-dimensional point clouds require constructing new data structures for individual points in the data. The structure requires standardized meta-data or conceptual models to describe it. Defining the new data structure is highly related to cultural heritage knowledge. In a cultural heritage point cloud 3D scene, a structured presentation of the scene is required to express the spatial relationship between single points or point sets (point cloud patch) with the same semantic division. Convert discrete and disordered point clouds into geometric primitives with topology to support computation and analysis of spatiotemporal patterns. For example, the spatial topological relationship expression method, the calculation method, and the spatial query method between point cloud patches must be detailed.

6. Conclusions

This paper reviews and analyzes current 3DPCS and 3DPCSS algorithms and applications in cultural heritage. Since 3DPCS and 3DPCSS algorithms are highly dependent on the density and additional information of point cloud data, this paper first introduces point cloud data acquisition and two development trends in the cultural heritage field: single platforms with multi-sensors and multi-platform data fusion. 3DPCS algorithms are mainly used to extract the basic geometry elements from point cloud data and can be used for surface damage detection and deformation analysis. Supported by machine learning and deep learning technologies, 3DPCSS algorithms have become the focus of current research. Learning-based approaches can separate different categories of elements in a point cloud scene in one task. According to the existing studies, the main applications of 3DPCSS in the field of cultural heritage consist of landscapes and immovable cultural heritage, especially historical buildings and their surrounding environment. The extracted geometric primitives can be directly associated with semantic information and external knowledge and improve the efficiency of parametric models, such as BIM and HBIM. The development of the HBIM platform illustrates the strengthening of the connection between geometrical entities and the physical domain. The combination of 3D digitization, semantic segmentation, reconstruction of geometric entities, and knowledge engineering enables computers to understand scene content expressed by digital models, rather than just visualizing scenes. More importantly, based on semantic annotation, technical terms, qualitative attributes, morphological characteristics, and the correlation between physical data, digital twin technology can realize the mapping and feedback between the physical world and the real world to support the preventive conservation of heritage places [187,188,189,190]. However, the diversity and geometric complexity of cultural heritage pose challenges for point cloud semantic segmentation. Many methods have produced adequate results for automatically segmenting cultural heritage geometric structures and damaged areas but they are mainly limited to historic buildings. More publicly available datasets are needed to ensure more discussion of algorithms in more types of cultural heritage. The main problems of the 3DPCSS of cultural heritage include the semantic consistency of multi-source point cloud data, semantic coherence under different spatial scales, and the over-segmentation problem. Learning-based methods are the primary development trend in the future, but public benchmark datasets and the generalization ability of the algorithms limit those methods. Semantic segmentation technology will also be used for scene cognition to identify the implicit spatial knowledge and patterns in cultural heritage scenes.