Automatic Recognition of Common Structural Elements from Point Clouds for Automated Progress Monitoring and Dimensional Quality Control in Reinforced Concrete Construction

: This manuscript provides a robust framework for the extraction of common structural components, such as columns, from terrestrial laser scanning point clouds acquired at regular rectangular concrete construction projects. The proposed framework utilizes geometric primitive as well as relationship-based reasoning between objects to semantically label point clouds. The framework then compares the extracted objects to the planned building information model (BIM) to automatically identify the as-built schedule and dimensional discrepancies. A novel method was also developed to remove redundant points of a newly acquired scan to detect changes between consecutive scans independent of the planned BIM. Five sets of point cloud data were acquired from the same construction site at di ﬀ erent time intervals to assess the e ﬀ ectiveness of the proposed framework. In all datasets, the framework successfully extracted 132 out of 133 columns and achieved an accuracy of 98.79% for removing redundant surfaces. The framework successfully determined the progress of concrete work at each epoch in both activity and project levels through earned value analysis. It was also shown that the dimensions of 127 out of the 132 columns and all the slabs complied with those in the planned BIM.


Introduction
In construction projects, as-designed vs. as-built dimensional incompliances result in rework, which can cost up to 25% of the contracted construction cost [1]. In concrete structures, rework has shown to be a dominant factor in increasing concrete waste with the highest impact on project cost over-run due to waste, compared to other construction materials [2]. Concrete is in fact the most widely used material in the construction industry, used almost twice as much as other construction materials in the United States of America [3]. Not only does rework contribute to a direct cost over-run due to concrete waste, but it also causes an implicit cost to the environment since cement manufacturing accounts for about 5-7% of the global CO 2 emissions annually [4]. Other than the cost, time, and possible environmental impact associated with rework in concrete construction, dimensional errors and structural damages may impact structural integrity, diminishing safety during and after construction. One prime example of structural failure during construction is the 2018 pedestrian bridge collapse in Florida [5], which resulted in several fatalities. Early and accurate identification and

State of the Art in Semantic Extraction of Structural Components from Point Clouds
Comprehensive reviews of the recent developments in processing of point clouds acquired from construction sites and indoor environments can be found in [8][9][10][11][12][13][14]. Since the focus of this manuscript is the automated semantic extraction of structural components in regular rectangular concrete construction, the review of previous work is restricted to that addressing the specific problem of automated semantic labeling of objects with predominantly planar and linear facades. The presentation of the previous work in semantic feature extraction from point clouds is divided into the following three research categories: 1.
Scan vs. BIM, which is used only when a reliable as-planned 4D BIM exists; 2.
Supervised learning, which is used when an object template or library of preclassified similar objects exist for training/matching; 3.
Spatial and contextual relationship, which uses unique a prior knowledge of an object and its relationship to other objects.

Scan vs. BIM
Scan vs. BIM, initially proposed by Bosché [15][16][17], utilizes the as-planned 4D BIM to assign points to a BIM element in close spatial proximity. First, synthetic as-planned point clouds are generated by decomposing the planned BIM into points with the same spatial resolution of the point cloud. The as-planned and as-built point clouds are then registered through an iterative closest point (ICP) method, and corresponding points are matched by satisfying some spatial similarity criteria [8]. Once matched, the as-built point cloud is labeled as the element representing the as-planned point cloud. The scan vs. BIM method has been widely implemented in the previous literature for applications such as progress monitoring and reporting [18,19], extraction of formwork/rebars [20], and completion of rectangular concrete columns [21]. Scan vs. BIM is easy to implement and enables semantic labeling of key objects directly from BIM when a detailed planned BIM is available. The approach is, however, unreliable when the distance between the as-built and as-planned locations of an object is larger than the predefined spatial similarity criteria. In other words, the method works well when the planned and 1.
Robust extraction of planar and linear features from registered point clouds ( Figure 1b); 2.
Semantic labeling of point clouds into floors, columns, and rebars using contextual and spatial information ( Figure 1c); 3.
Identification and visualization of deviations between as-built and planned BIM ( Figure 1g); and 5.
Removal of redundant points of previously modeled surfaces for newly acquired point clouds ( Figure 1f). For prospective scans, the processes described in steps 1-4 will only be carried out for the new points (shown in green; Figure 1f).
Remote Sens. 2019, 11, x FOR PEER REVIEW 5 of 23 concrete structures are targeted here specifically, since they are commonly employed in the building construction industry [3,46,47]. The overview of the methodology is as follows: 1. Robust extraction of planar and linear features from registered point clouds ( Figure 1b); 2. Semantic labeling of point clouds into floors, columns, and rebars using contextual and spatial information ( Figure 1c); 3. Surface intersection and modeling ( Figure 1d); 4. Identification and visualization of deviations between as-built and planned BIM ( Figure 1g); and 5. Removal of redundant points of previously modeled surfaces for newly acquired point clouds ( Figure 1f). For prospective scans, the processes described in steps 1-4 will only be carried out for the new points (shown in green; Figure 1f).

Figure 1:
Step-by-step process for semantic extraction of regular rectangular reinforced concrete components: (a) point cloud of first epoch; (b) robust planar and linear segmentation using the method described in [22]; (c) extraction of semantic features, here concrete columns, using contextual constraints; (d) 3D model generation through surface intersection; (e) point cloud of the second (to last) epoch; and (f) removal of redundant points that had been modeled in the previous epoch.

Target-Based Point Cloud Registration
The convention proposed in [8,22] is used to register point clouds to a reference coordinate system (the coordinate system of the planned BIM). This approach uses signalized TLS targets on presurveyed site control points to register coordinate systems of scans at each epoch to a reference coordinate system. The centers of the signalized targets are measured through high-precision surveying and the TLS instrument. The centers of targets are matched to estimate the exterior orientation parameters along with registration precision through rigid body transformation.

Robust Planar and Linear Segmentation
As illustrated in Figure 1b, planar and linear segmentation is the first step towards semantic extraction of concrete elements. Here, the method developed in [22] is employed for robust planar and linear segmentation. The method is specifically adopted since it is robust to common outliers of construction site point clouds and the segmentation results are not a function of a subjectively predefined threshold. According to this method, horizontal planes are first extracted using the Step-by-step process for semantic extraction of regular rectangular reinforced concrete components: (a) point cloud of first epoch; (b) robust planar and linear segmentation using the method described in [22]; (c) extraction of semantic features, here concrete columns, using contextual constraints; (d) 3D model generation through surface intersection; (e) point cloud of the second (to last) epoch; and (f) removal of redundant points that had been modeled in the previous epoch.

Target-Based Point Cloud Registration
The convention proposed in [8,22] is used to register point clouds to a reference coordinate system (the coordinate system of the planned BIM). This approach uses signalized TLS targets on presurveyed site control points to register coordinate systems of scans at each epoch to a reference coordinate system. The centers of the signalized targets are measured through high-precision surveying and the TLS instrument. The centers of targets are matched to estimate the exterior orientation parameters along with registration precision through rigid body transformation.

Robust Planar and Linear Segmentation
As illustrated in Figure 1b, planar and linear segmentation is the first step towards semantic extraction of concrete elements. Here, the method developed in [22] is employed for robust planar Remote Sens. 2019, 11, 1102 6 of 23 and linear segmentation. The method is specifically adopted since it is robust to common outliers of construction site point clouds and the segmentation results are not a function of a subjectively predefined threshold. According to this method, horizontal planes are first extracted using the histogram of point height (similar to [37,43]) to promote computational efficiency. The horizontal plane extraction method used in [22] is robust to Type II errors and was shown to be effective in extracting floor objects from construction site environments. The remaining points are then classified into planes and lines using a robust principal component analysis (PCA) method to determine local surface curvature. The local surface curvature values are then matched to that obtained by Monte Carlo point cloud simulations, subjected to random measurement errors, to determine the final set of planar and linear points. The classified planar and linear features are then segmented into surfaces with similar geometrical attributes using a new iterative and robust variation of the complete linkage hierarchical clustering method.

Semantic Object Extraction Using Relationship-Based Reasoning
Here, the objective is to extract flat slab floors, rectangular columns, and rebars from the segmented planar and linear features. For floors, the method developed in [22] is used since it was consistently able to extract floors from other possible horizontal objects in various construction site settings. The problem is now reduced to the semantic extraction of rectangular columns and rebars from other elements. Figure 2 shows the point cloud of a typical column before and after the slab of the top floor is poured. The exposed rebars on the top of the columns in Figure 2a enable the seamless transfer of stresses and bending moments between floors. After the concrete for the top floor slab is poured, these rebars are no longer visible (Figure 2b). From Figure 2, it can be observed that rebars are linear features on top of columns. Figure 2 also shows that rectangular columns are confined by a floor object at the bottom and either linear objects (rebars) or a floor object (ceiling) on the top.  [37,43]) to promote computational efficiency. The horizontal plane extraction method used in [22] is robust to Type II errors and was shown to be effective in extracting floor objects from construction site environments. The remaining points are then classified into planes and lines using a robust principal component analysis (PCA) method to determine local surface curvature. The local surface curvature values are then matched to that obtained by Monte Carlo point cloud simulations, subjected to random measurement errors, to determine the final set of planar and linear points. The classified planar and linear features are then segmented into surfaces with similar geometrical attributes using a new iterative and robust variation of the complete linkage hierarchical clustering method.

Semantic Object Extraction Using Relationship-Based Reasoning
Here, the objective is to extract flat slab floors, rectangular columns, and rebars from the segmented planar and linear features. For floors, the method developed in [22] is used since it was consistently able to extract floors from other possible horizontal objects in various construction site settings. The problem is now reduced to the semantic extraction of rectangular columns and rebars from other elements. Figure 2 shows the point cloud of a typical column before and after the slab of the top floor is poured. The exposed rebars on the top of the columns in Figure 2a   Another important characteristic of structural columns is its responsibility to transfer the bending moments and vertical stresses from top floors to the foundation. For regular rectangular buildings, columns are almost always oriented in the direction of the two orthogonal main axes of the rectangular plan to accommodate consistent load transfer [3,47,48]. The consistent column orientation is also desirable to preserve symmetry. Therefore, by examining the orientation as well as the objects surrounding the boundaries of planar surfaces, planar surfaces representing columns can be uniquely identified. These two criteria are used to distinguish columns from other planar objects on sites. Once the columns are identified, rebars objects are linear segments on top of the columns. The solution to the column extraction is formulated as follows: • Algorithm 1: First, the two main orthogonal orientation directions of planar surfaces, excluding the floor objects, are identified. The planar surfaces whose normal vectors are in the same direction of these two vectors are selected as potential column candidates (i.e., planes whose normal vector follows the direction of the main orthogonal site axes). Another important characteristic of structural columns is its responsibility to transfer the bending moments and vertical stresses from top floors to the foundation. For regular rectangular buildings, columns are almost always oriented in the direction of the two orthogonal main axes of the rectangular plan to accommodate consistent load transfer [3,47,48]. The consistent column orientation is also desirable to preserve symmetry. Therefore, by examining the orientation as well as the objects surrounding the boundaries of planar surfaces, planar surfaces representing columns can be uniquely identified. These two criteria are used to distinguish columns from other planar objects on sites. Once the columns are identified, rebars objects are linear segments on top of the columns. The solution to the column extraction is formulated as follows: • Algorithm 1: First, the two main orthogonal orientation directions of planar surfaces, excluding the floor objects, are identified. The planar surfaces whose normal vectors are in the same direction of these two vectors are selected as potential column candidates (i.e., planes whose normal vector follows the direction of the main orthogonal site axes). Assign the normal vector associated with the planar surface to each point of that segment.

3.
Estimate the mode of the bivariate x−y components of the normal vectors. In this study, the mean-shift mode detection [49] with a normal kernel and optimized fixed-bandwidth proposed by [50] was adopted.

4.
For every two identified modes, calculate the allowable standard deviation of the inner product of two modes, σ n i , n j , derived by applying the law of variance propagation, using Equation (1): where n i and n j are the normal vectors of the i th and j th mode, respectively; σ n i , n j is the allowable tolerance of the inner product of vectors n i and n j ; σ θ is the allowable angular tolerance in radians; θ x i , θ y i , θ z i are the angles of the normal vector of the i th mode to the x, y and z axes, respectively; and n i , n j is the inner product of vectors n i and n j . In this study, the allowable plumb tolerance (σ θ ) is set to approximately 0.52 • , derived from ACI 117 [51,52].

5.
Select the two modes whose absolute value of the inner product satisfies the orthogonality criteria of Equation (2): The threshold 3σ n 1 , n 2 is used to account for approximately 99% confidence.

6.
For the pair of normal vectors satisfying Equation (2), find the normal vectors of the planar surfaces whose angles are within ±3σ θ in each direction.
The surfaces satisfying step 6 are the surfaces whose normal follows the direction of the main site axes. Other than identifying surfaces following the main site axes, the algorithm can also identify surfaces that are not built to the specified tolerances in relation to the majority of the built surfaces before a plan vs. actual comparison is even performed. The two main orthogonal axes can also be used to improve registration of point clouds in the absence of reliable target-based registration.

1.
Select the planar candidates obtained from Algorithm 1.

2.
Calculate the first and third quartile of the height of all planar candidates.

3.
Identify planar surfaces whose minimum height is smaller than the first quartile, and maximum height is larger than the third quartile of height (to ensure removal of shorter clutters).

5.
Perform connected components region growing (Algorithm 5 of [8]) on the identified boundaries to group together potential columns. Here, the neighborhood size is set to r √ 2, where r is the radius of the neighborhood used for robust PCA classification [22]. This neighborhood size was chosen since the local neighborhood of points within r from the edge of two intersecting surfaces Remote Sens. 2019, 11, 1102 8 of 23 are prone to misclassification using classical PCA (see Figure 3b,c). Since robust PCA classifies more planar points closer to the boundaries than classical PCA [22,43], the defined threshold will be large enough to group together surfaces of the same column. Once the columns are extracted using Algorithms 1 and 2, the enclosed linear segments are labeled as rebar objects using Algorithm 3: Semantic Rebar Extraction, as follows: 1. Select the column segments from Algorithm 2 that satisfy the condition of step 7b. 2. For each column segment, identify all linear segments whose minimum height is larger than the column's height. 3. Project all identified linear segments onto the x−y plane. 4. The linear segments whose boundaries in the x−y plane are enclosed by the boundaries of the columns are considered as rebars.

Parametric Surface Representation
After the floors, rebars, and columns have been automatically extracted from the segmented point cloud, it is important to provide a parametric model to represent the as-built for comparison with the planned BIM. The parametric representations of the identified objects are as follows: 1. Floors: every floor is represented by a normal vector (estimated through robust PCA), and a point on the plane (robust center of the points) [22]. The boundary of the floors is identified using the modified convex hull algorithm and boundary regularization presented in [56] to define the extents of the floor planes. 2. Rebars: each rebar is represented by a point (e.g., robust center of the segmented rebar), length of the rebar, cylinder's axis, and radius. The radius and cylinder's axis are estimated through Algorithms 1 through 3 of [8] to provide an accurate and robust estimation. To define the length of the cylinder, rotate the cylinder's axis to the z direction using Rodrigues rotational formulation. The length of the rebar is then the difference between the maximum and minimum heights of the rotated rebar.

6.
Select the connected segments that contain a floor object within r from its minimum height (r is used for the same reasons given in step 5 and Figure 3c). 7.
From the remaining connected segments satisfying step 6, a connected segment is labeled a column if one of the following two criteria is satisfied: The largest height of the segment is within r from the median height of a floor object; or b. A linear segment within (2r) 2 + (rebar cover size) 2 (derived from the Pythagorean theorem) of the boundaries of the connected segment exists. The cover size, schematically shown in Figure 3d, is set to 50 mm following ACI 318 (2014) [55].
Once the columns are extracted using Algorithms 1 and 2, the enclosed linear segments are labeled as rebar objects using Algorithm 3: Semantic Rebar Extraction, as follows: 1.
Select the column segments from Algorithm 2 that satisfy the condition of step 7b.

2.
For each column segment, identify all linear segments whose minimum height is larger than the column's height.

3.
Project all identified linear segments onto the x−y plane. 4.
The linear segments whose boundaries in the x−y plane are enclosed by the boundaries of the columns are considered as rebars.

Parametric Surface Representation
After the floors, rebars, and columns have been automatically extracted from the segmented point cloud, it is important to provide a parametric model to represent the as-built for comparison with the planned BIM. The parametric representations of the identified objects are as follows:

1.
Floors: every floor is represented by a normal vector (estimated through robust PCA), and a point on the plane (robust center of the points) [22]. The boundary of the floors is identified using the modified convex hull algorithm and boundary regularization presented in [56] to define the extents of the floor planes.
Remote Sens. 2019, 11, 1102 9 of 23 2. Rebars: each rebar is represented by a point (e.g., robust center of the segmented rebar), length of the rebar, cylinder's axis, and radius. The radius and cylinder's axis are estimated through Algorithms 1 through 3 of [8] to provide an accurate and robust estimation. To define the length of the cylinder, rotate the cylinder's axis to the z direction using Rodrigues rotational formulation. The length of the rebar is then the difference between the maximum and minimum heights of the rotated rebar. 3.
Columns: the extents of the rectangular columns are defined by the eight vertices of the rectangular prism ( Figure 2b). Each planar façade of the column is represented by the four plane parameters (see floor objects above). The bottom vertices are estimated through the intersection of planar surfaces and the floor object on the bottom. The process is identical in cases where a floor object also exists on the top (i.e., ceiling; see Figure 2b). In cases where only rebars exist on top, a virtual plane parallel to the bottom floor plane with distance of the maximum height of the column segment from the bottom floor is generated. The top four vertices are calculated accordingly through planar intersection.

Planned vs. As-Built Comparison
As explained in Section 3.1, the scans are registered to the reference coordinate system of the planned BIM following the convention proposed in [8,22]. To compare the planned and the as-built, the corresponding objects between the plan and as-built are identified using a distance threshold. Following the convention set forth by [15][16][17][18], a 50 mm distance threshold is suggested as expected construction errors. Figure 4a represents the planned 4D model of a construction site at a given baseline. Figure 4b,c show the superimposition of the planned model and the automatically identified columns at the baseline. The blue crosses in Figure 4c represent the generated edges (8 vertices) of the identified columns using the method presented in Section 3.4. The objects whose minimum distance is smaller than 50 mm are then identified. The progress can then be visually presented through a color-coding scheme. In Figure 4d, blue represents on-schedule activities (identified objects), the red color represents behind-schedule activities (not found), and green objects (not shown in the presented example) are ahead of schedule activities.

Planned vs. As-Built Comparison
As explained in Section 3.1, the scans are registered to the reference coordinate system of the planned BIM following the convention proposed in [8,22]. To compare the planned and the as-built, the corresponding objects between the plan and as-built are identified using a distance threshold. Following the convention set forth by [15][16][17][18], a 50 mm distance threshold is suggested as expected construction errors. Figure 4a represents the planned 4D model of a construction site at a given baseline. Figures 4b and 4c show the superimposition of the planned model and the automatically identified columns at the baseline. The blue crosses in Figure 4c represent the generated edges (8 vertices) of the identified columns using the method presented in Section 3.4. The objects whose minimum distance is smaller than 50 mm are then identified. The progress can then be visually presented through a color-coding scheme. In Figure 4d, blue represents on-schedule activities (identified objects), the red color represents behind-schedule activities (not found), and green objects (not shown in the presented example) are ahead of schedule activities.

Redundant Point Removal of Prospective Scans
Since the process of monitoring and control is carried out in a continuous manner, it is possible that a newly acquired scan contains points of objects that were modeled from previous scans. It is, hence, desirable to remove these points first before point cloud processing so that only the new changes in structural components are detected.
Consider a newly acquired scan registered to the reference coordinate system (object space), following the convention proposed in Section 3.1. The objective is to identify if point A (Figure 5a) of scan is close enough to a presegmented object surface P to be considered as a point on surface P. To this end, an error ellipsoid (Figure 5b) is estimated around point A that accounts for both scanner observational as well as registration uncertainties. Point A of scan is considered redundant if a presegmented surface P in the object space exists such that the error ellipsoid has an intersection with surface P (Figure 5c). Point A is then semantically labeled as the object represented by surface P.

Redundant Point Removal of Prospective Scans
Since the process of monitoring and control is carried out in a continuous manner, it is possible that a newly acquired scan contains points of objects that were modeled from previous scans. It is, hence, desirable to remove these points first before point cloud processing so that only the new changes in structural components are detected.
Consider a newly acquired scan j registered to the reference coordinate system (object space), following the convention proposed in Section 3.1. The objective is to identify if point A (Figure 5a) of scan j is close enough to a presegmented object surface P to be considered as a point on surface P. To this end, an error ellipsoid (Figure 5b) is estimated around point A that accounts for both scanner observational as well as registration uncertainties. Point A of scan j is considered redundant if a presegmented surface P in the object space exists such that the error ellipsoid has an intersection with surface P (Figure 5c). Point A is then semantically labeled as the object represented by surface P. To determine the error ellipsoid, the covariance of the radiated point from scanner space to the object space (reference coordinate system) is calculated through the law of variance propagation as follows: = sec 0 0 where is the observed scanner space vector of point in space , is the object space vector of point , is the object space vector of scanner (translation vector), is the rotation matrix from object space to scanner space , is the covariance matrix of observation in scanner space , is the instrumental range error variance at normal incidence, and are the instrumental angular variances, is the incidence angle of observation collected from scan , is the set of registration parameters, and is the covariance matrix of registration parameters for scan . Using the covariance matrix of Equation (5), an error ellipsoid is constructed with 95% confidence and three degrees of freedom (three-dimensional data). Using the derived equations, the redundant surfaces are removed following Algorithm 4: Redundant Surface Extraction: 1. For every new scan point, , calculate the covariance matrix, using Equation (5). 2. Calculate the eigenvalues ( ) and eigenvectors ( ) of the covariance matrix ( ). 3. Construct error ellipsoid using Equation (6): To determine the error ellipsoid, the covariance of the radiated point i from scanner space j to the object space (reference coordinate system) is calculated through the law of variance propagation as follows: where r ij is the observed scanner space vector of point i in space j, R i is the object space vector of point i, R c j is the object space vector of scanner j (translation vector), M j is the rotation matrix from object space to scanner space j, C r ij is the covariance matrix of observation i in scanner space j, σ 2 ρ is the instrumental range error variance at normal incidence, σ 2 θ and σ 2 α are the instrumental angular variances, β ij is the incidence angle of observation i collected from scan j, x e is the set of registration parameters, and C x j is the covariance matrix of registration parameters for scan j. Using the covariance matrix of Equation (5), an error ellipsoid is constructed with 95% confidence and three degrees of freedom (three-dimensional data). Using the derived equations, the redundant surfaces are removed following Algorithm 4: Redundant Surface Extraction:

1.
For every new scan point, i, calculate the covariance matrix, C R i using Equation (5).

2.
Calculate the eigenvalues (λ) and eigenvectors (v) of the covariance matrix (C R i ).

3.
Construct error ellipsoid using Equation (6): where R i is the vector of coordinates of point i in the object space and X 2 0.95,3 is a chi-squared probability with 95% confidence and 3 degrees of freedom (X 2 0.95,3 = 7.8147).

4.
Find all planar and cylindrical (rebars are modeled as cylinders; see Section 3.4) objects from Algorithms 1-3 that intersect the error ellipsoid. 5.
If more than one surface meets the conditions of step 4, the point is assigned to the closest surface. The point is then semantically labeled to the corresponding object represented by the segmented surface.
Step 4 of Algorithm 4 requires a procedure to find the intersection between an ellipsoid and a plane as well as a cylinder, which is not trivial. Here, two original and generic algorithms (Algorithms 5 and 6) are developed to identify the intersection of a plane/cylinder and ellipsoid in space.

1.
Calculate the distance of the point to the planar segments (ρ A P of Figure 5a).

2.
Identify all surfaces where ρ A P is smaller than λ max X 2 0.95,3 , the semimajor axis.

3.
Calculate the linear transformation matrix λ − 1 2 v T that transforms the error ellipsoid of Equation (6) into a sphere with radius X 2 0.95,3 . This transformation reduces the problem to finding the intersection between a sphere and a plane, since planes are affine equivariant.

4.
Calculate the distance of point i from each transformed planar segment (p d ).
Project the transformed error sphere onto the surfaces satisfying condition 5 to construct an error circle with the point's projection as the center and radius X 2 0.95,3 − p 2 d (Pythagorean theorem). 7.
The point is assigned to the surface if and only if its projected circle intersects with the boundary of that surface.

1.
Calculate the distance of the point to the axis of the cylindrical segments ρ A L .

2.
Identify all cylinders where ρ A P is smaller than λ max X 2 0.95,3 + r cyl , where r cyl is the radius of the cylindrical segment.

3.
Find the rotation matrix, M cyl , that orients the cylinder's axis parallel to the z-axis following Rodrigues rotation formula.

4.
Rotate the error ellipsoid (Equation 6) and the cylindrical segment using M cyl .

5.
Project the rotated ellipsoid and cylinder onto the x−y plane. This reduced the problem to finding the intersection between a circle and an ellipse in a 2D plane. To this end, we first identify the closest point, P Closest , from the center of the circle (O cyl ) to the ellipse using the following steps.

6.
Calculate the transformation matrix λ − 1 2 v T M T cyl of the newly rotated error ellipse. 7.
Transform the error ellipse into an error circle with radius X 2 0.95,3 and center, O elp .

8.
Transform the center of the circle, O cyl , into O Trans using the same transformation matrix as step 6. This transformation further reduces the problem to identifying the closest point between the newly transformed point (O Trans ) and error circle of step 7. 9.
Calculate the point of intersection, P Trans , between the line segment O Trans − O elp and the error circle. 10. Identify P Closest by transforming the point of intersection, P Trans , back to the correct coordinate system (i.e., before the affine transformation of step 6). 11. Identify all segments where the distance between P Closest and O cyl is smaller than the radius, r cyl (the condition for the intersection of the ellipse and circle). 12. Project the rotated error ellipsoid of step 4 onto the z axis. Identify the maximum and minimum height of the projected ellipsoid. The point is assigned to the surface if and only if its projected height intersects with the height of the cylindrical segment.

Method of Validation of Results
To assess the effectiveness of the object classification as well as the redundant surface extraction, the precision, recall, and accuracy are estimated following the definitions presented in [57]: where T P , T N , F P , and F N are the number of true positive, true negative, false positive, and false negative counts, respectively. The ground truth is determined through manual extraction. Planned vs. as-built comparison for progress monitoring and dimensional compliance is also performed. For progress reporting, the well-established earned value management (EVM) is used to determine the project's performance at different epochs. For dimensional quality control, the dimensions of structural components such as columns were compared to the design as well as ground truth measurements. For each epoch, the percentage of columns passing the standard dimensional tolerances is also determined. The ground truth measurements for the required object dimensions were manually estimated using measuring tape. To calculate the horizontal (two-dimensional) accuracy, the distance root mean squared (DRMS) [58] is used. To calculate the one-dimensional accuracy, such as in the case of slab thickness, the absolute deviation is used.

Experiment Description
Five sets of TLS point clouds were acquired from the Graduate Student Hall of Residence (GSHR) construction site (Figure 6) at the University of Calgary. The objective of this experiment was to automatically monitor the progress of concrete work on a specific portion of the site (Figure 6b). The building structure is comprised of cast-in-place reinforced concrete with flat slab floors, steel rebars, and rectangular columns. The planned schedule for the completion of the concrete work for each floor was one week; hence, the site was monitored roughly once every week for a duration of five weeks. For each floor, one dataset was collected after the scheduled completion of the columns of the floor, and another just after the completion of the slab of the floor above. The point cloud data was acquired using the Leica HDS6100 TLS [59]. The scans were registered to the planned model reference coordinate system using signalized targets on presurveyed site control points as described in Section 3.1. The number of points, number of scan stations, and registration precision per epoch is provided in Table 1. rebars, and rectangular columns. The planned schedule for the completion of the concrete work for each floor was one week; hence, the site was monitored roughly once every week for a duration of five weeks. For each floor, one dataset was collected after the scheduled completion of the columns of the floor, and another just after the completion of the slab of the floor above. The point cloud data was acquired using the Leica HDS6100 TLS [59]. The scans were registered to the planned model reference coordinate system using signalized targets on presurveyed site control points as described in Section 3.1. The number of points, number of scan stations, and registration precision per epoch is provided in Table 1.

Extraction of Columns from Segmented Planar and Linear Features
The results of the planar and linear segmentation of the GSHR construction site using robust PCA classification and robust complete linkage segmentation were given in [22]. Here, the planar and linear segmentation results are used to extract structural columns through Algorithms 1 and 2. Figure 7a,b illustrates the point cloud and segmented planar and linear features for epoch 1. In Figure 7b, the red ovals represent two instances where the rebars were not effectively identified due to low point density of the collected data. As can be seen in Table 1, the scan resolution was increased after epoch 1 to collect more points and prevent the misclassification problem due to low point density [22]. Figure 7c shows the bivariate histogram of the x−y components of the normal vector of the identified planar surfaces. Illustrated in red ovals, the histogram consists of two main modes, which represent the planar surfaces corresponding to the columns. The two modes with the highest frequency satisfying the orthogonality criteria of Algorithm 1, Equations (1) and (2), were identified. To provide some perspective, the absolute value of the dot product of the two main axes was 0.0036 (Equation 2), whereas the threshold of Equation (1) was 0.035, almost an order of magnitude larger. The boundaries of the planar segments whose normal vectors comply with the identified modes (the final output of Algorithm 1) are presented in Figure 7d. Figure 7e shows the boundaries of the segmented columns after the application of Algorithm 2 (i.e., segments satisfying the height and boundary conditions). In Figure 7d, the planar surfaces that followed the main site axes but were unable to satisfy the height restriction or boundary conditions are also marked. The planar surface shown in the blue dashed oval is the same surface shown in Figure 7b, where low point density prevented the correct extraction of the rebars. Since the rebars were not extracted, the surface was incorrectly eliminated following Algorithm 2. This shows the importance of the point cloud density on the planar and linear segmentation results and consequently the column extraction procedure. Once the point density was increased (epochs 2 through 5), all surfaces representing columns were consistently extracted correctly. The results of the column extraction for all epochs is summarized in Table 2. As observed, the overall precision, recall, and accuracy for the presented column extraction is 99.24%, 100.00%, and 99.31%, respectively. The visual results of the column extraction for epochs 2 through 5 can be illustrated in Figures 8 and 9.
PCA classification and robust complete linkage segmentation were given in [22]. Here, the planar and linear segmentation results are used to extract structural columns through Algorithms 1 and 2. Figure 7a,b illustrates the point cloud and segmented planar and linear features for epoch 1. In Figure  7b, the red ovals represent two instances where the rebars were not effectively identified due to low point density of the collected data. As can be seen in Table 1, the scan resolution was increased after epoch 1 to collect more points and prevent the misclassification problem due to low point density [22].   the project's planned schedule, the ceiling and surrounding columns of the first floor were expected to be completed. Before the point cloud of epoch 2 was processed, Algorithms 4-6 were applied to remove possible redundant surfaces so that the remaining processes (planar segmentation and object extraction) would only be carried out on points of newly added components. The results of the redundant surface removal for epoch 2 are shown in Figure 8c. The newly added points, redundant points between epochs 1 and 2, and points available in epoch 1 but not covered in epoch 2 are shown in green, blue, and red colors, respectively. The redundant points (shown in blue Figures 1, 8, and 9) are semantically labeled as the objects represented by the matched surface (e.g., column represented by a plane). The processes described in Sections 3.2 and 3.3 are then only carried out for the newly added points (green points in Figures  1, 8, and 9). Figure 8e shows the output of Algorithm 1, i.e., the planar surfaces following the main orthogonal orientations of the site. Figure 8f shows the extracted columns, shown in different colors following the height and boundary conditions imposed by Algorithm 2. As shown in Table 2, all columns for epoch 2 were correctly extracted. The results of the redundant surface removal as well as column extraction (Algorithms 1 through 3) for epochs 3 through 5 are shown in Figure 9. Table 3 presents the precision, recall, and accuracy of the redundant surface extraction algorithm for epochs 2 through 5. The redundant surface removal method achieved an overall precision, recall, and accuracy of 97.09%, 98.04%, and 98.79%, respectively. The recall rate shows that a very small percentage of the new points were incorrectly added to the presegmented surfaces of previous epochs. Upon closer examination, 95.2% of the new points that were incorrectly identified as redundant were points close to two intersecting planar surfaces, where one of the surfaces was not covered in the previous scan; hence, the point was assigned to the closest presegmented surface. The rest (4.8%) were outlier points, such as mixed pixels, commonly present in datasets acquired from construction sites that satisfied the conditions presented in Algorithms 4-6.

Results of Redundant Surface Removal
The generated as-built model of the final set of extracted columns for epoch 1 is shown in Figure 7f. Approximately one week after epoch 1, epoch 2 was acquired from the construction site. Based on the project's planned schedule, the ceiling and surrounding columns of the first floor were expected to be completed. Before the point cloud of epoch 2 was processed, Algorithms 4-6 were applied to remove possible redundant surfaces so that the remaining processes (planar segmentation and object extraction) would only be carried out on points of newly added components. The results of the redundant surface removal for epoch 2 are shown in Figure 8c. The newly added points, redundant points between epochs 1 and 2, and points available in epoch 1 but not covered in epoch 2 are shown in green, blue, and red colors, respectively.
The redundant points (shown in blue Figures 1, 8 and 9) are semantically labeled as the objects represented by the matched surface (e.g., column represented by a plane). The processes described in Sections 3.2 and 3.3 are then only carried out for the newly added points (green points in Figures 1,  8 and 9). Figure 8e shows the output of Algorithm 1, i.e., the planar surfaces following the main orthogonal orientations of the site. Figure 8f shows the extracted columns, shown in different colors following the height and boundary conditions imposed by Algorithm 2. As shown in Table 2, all columns for epoch 2 were correctly extracted.
The results of the redundant surface removal as well as column extraction (Algorithms 1 through 3) for epochs 3 through 5 are shown in Figure 9. Table 3 presents the precision, recall, and accuracy of the redundant surface extraction algorithm for epochs 2 through 5. The redundant surface removal method achieved an overall precision, recall, and accuracy of 97.09%, 98.04%, and 98.79%, respectively. The recall rate shows that a very small percentage of the new points were incorrectly added to the presegmented surfaces of previous epochs. Upon closer examination, 95.2% of the new points that were incorrectly identified as redundant were points close to two intersecting planar surfaces, where one of the surfaces was not covered in the previous scan; hence, the point was assigned to the closest presegmented surface. The rest (4.8%) were outlier points, such as mixed pixels, commonly present in datasets acquired from construction sites that satisfied the conditions presented in Algorithms 4-6. Table 3. Precision, recall, and accuracy of the automated redundant surface extraction.

Epochs
Precision Recall Accuracy Using the method described in Section 3.5, the corresponding elements between those planned and as-built were identified. Since the planned BIM did not contain 3D information related to the rebars, only the correspondence of columns and floor slabs were examined. Figure 10 (right) shows the result of the superimposition of the planned and the as-built elements. The object correspondences between the planned and as-built elements are visualized through color-coding. On-schedule, behind-schedule, and ahead-of-schedule objects are shown in blue, red, and green colors, respectively.

Dimensional Compliance Control
Since no information about the rebars' placements was available in the planned BIM, only the conformity of the as-built dimensions of the columns and the slab thickness of those planned are presented. The planned cross-sectional dimensions of the columns (width and length) and thickness of the slabs were 350 mm, 600 mm, and 175 mm, respectively. The width and length of these columns as well as the slab thickness were also measured by means of a measuring tape as ground truth. Based on the information presented in [52], the tolerance of the cross section of a rectangular column with planned dimensions between 305 mm and 914 mm is between +13 mm and −10 mm (i.e., the column's dimensions can exceed more than they can be reduced due to designed strength limitations). The tolerance of the thickness of the concrete slabs is ±6 mm. For each epoch, the DRMS [58] of the dimensions of the columns' cross sections from those planned were calculated. The DRMS was also calculated for the estimated dimensions compared to ground truth measurements (dimensions The colors presented in Figure 10 help with visual identification and reporting of the progress of specific construction elements (i.e., in activity level). To determine the performance of the whole project at each baseline, EVM is commonly employed [18]. Using the basic principles of EVM, the budgeted cost of work scheduled (BCWP), budgeted cost of work performed (BCWP), and schedule performance index (SPI) for each epoch were calculated. The result of the EVM is presented in Table 4. Using the calculated SPI, the schedule performance of the whole project at each epoch was determined (behind, ahead of, and on schedule). According to earned value analysis, SPI smaller than, equal to, or larger than 1 demonstrates that the project is behind, on, or ahead of schedule, respectively.

Dimensional Compliance Control
Since no information about the rebars' placements was available in the planned BIM, only the conformity of the as-built dimensions of the columns and the slab thickness of those planned are presented. The planned cross-sectional dimensions of the columns (width and length) and thickness of the slabs were 350 mm, 600 mm, and 175 mm, respectively. The width and length of these columns as well as the slab thickness were also measured by means of a measuring tape as ground truth. Based on the information presented in [52], the tolerance of the cross section of a rectangular column with planned dimensions between 305 mm and 914 mm is between +13 mm and −10 mm (i.e., the column's dimensions can exceed more than they can be reduced due to designed strength limitations). The tolerance of the thickness of the concrete slabs is ±6 mm. For each epoch, the DRMS [58] of the dimensions of the columns' cross sections from those planned were calculated. The DRMS was also calculated for the estimated dimensions compared to ground truth measurements (dimensions obtained by measuring tape) for comparison. The results of the column dimensional compliance are presented in Table 5. The percentage of columns passing the cross-sectional tolerance criteria for each epoch is also provided in Table 5. A column is considered compliant if it satisfies the tolerance check criteria in both width and length. As illustrated, 96.21% of the columns (127 out of 132 identified columns) in all datasets complied with their planned dimensions (i.e., their cross-sectional dimensions were within +13 mm and −10 mm from the planned dimensions). Accordingly, the difference in the slab thickness from the designed as well as the ground truth for floors 2 and 3 is calculated and presented in Table 6. The first floor was on the ground (slab on grade), and hence, calculation of the slab thickness is not relevant. As illustrated, the thickness of the slab of both floors 2 and 3 are within 6 mm of the planned dimensions.

Summary of Findings and Discussion
The objective of the experiment was to assess the effectiveness of the column extraction and redundant point removal methods (Algorithms 1-6) in real-world point cloud data acquired from cast-in-place regular rectangular concrete construction. It was shown that 132 out of 133 columns were extracted correctly using Algorithms 1 and 2 as presented in the manuscript. In all five datasets, only one column was not correctly extracted. This was attributed to the low point density of the scan in epoch 1, preventing the linear classification and segmentation of [22] to identify the rebars on top of this particular column. Once the point density of the TLS instrument was increased for prospective scans, the column extraction was able to correctly extract the remaining columns consistently. The recall rate of the column extraction was 100%, which demonstrates the robustness of the method to type II errors (i.e., no other object was incorrectly identified as a column).
The redundant surface removal was also applied to epochs 2 through 5 with an overall precision, recall, and accuracy of 97.09%, 98.04%, and 98.79%, respectively. The recall rate suggests that only a small portion of the points of the new scan was incorrectly identified as redundant. Upon closer examination, it was observed that 95.2% of the incorrectly classified points were points of an adjacent surface where the surface was not covered in the previous scan.
The success of the algorithms in the removal of redundant as well as the semantic labeling of points enabled the automatic comparison of as-built vs. planned elements. Here, two applications, namely progress monitoring as well as dimensional compliance control, using the proposed methods, were presented. It was shown that the system identified and visualized the progress of construction work in activity level, which is one of the limitations of current progress-monitoring practices in the industry [60]. It was also shown that the produced results can be used to determine the performance of the whole project at a given baseline using some project controls method, such as EVM.
The as-built dimensions of the cross section of the rectangular columns and the thickness of the floor slabs were compared to that of the planned to determine potential dimensional discrepancies. It was shown that the thickness of the flat slabs of floors 2 and 3 complied with the planned dimensions (i.e., within standard tolerances). It was also shown that 127 out of the 132 identified columns were also within the acceptable tolerances from their planned dimensions. The ground truth of the dimensions of the columns and slabs was collected using a measuring tape to a millimeter precision. The DRMS of all the dimensions of columns to the ground truth was approximately 1 mm, which shows good agreement since most quality standards in industrial construction allow up to 5 mm deviation in each direction [8].

Conclusions
This manuscript provides a robust framework for the semantic labeling of common reinforced structural concrete components from unorganized point clouds acquired at regular rectangular buildings during construction. The framework first classifies and segments registered point clouds into planar and linear features using robust PCA, Monte Carlo simulation, and the robust variation of the complete linkage hierarchical clustering method, as proposed by [22]. Columns, floors, and rebars are then extracted through relationship-based reasoning derived from the specific characteristics of reinforced concrete structures and regular rectangular buildings. The framework also incorporates a novel redundant point removal method to remove points of prospective scans that were classified into objects in previous scans.
Five sets of point cloud data were acquired from the GSHR construction site at the University of Calgary to assess the effectiveness of the proposed methods for semantic labeling and redundant point removal. The results substantiated the effectiveness and the ability of the column extraction to extract 132 out of 133 columns in all datasets with an overall object extraction accuracy of 99.31%. The redundant point removal also achieved an overall extraction accuracy of 98.79%, which demonstrates its applicability in redundant point removal of point clouds acquired from construction sites.
Two applications, namely progress monitoring and dimensional quality control, of the proposed column extraction and redundant point removal were presented. It was observed that these methods enable the automated color-coding for visual representation of progress in activity level. An EVM was also carried out to determine the overall project's performance at each epoch. The dimensions of the extracted columns and slabs were also compared to the planned and ground truth. It was shown that 127 out of the identified 132 columns and both floor slabs (levels 2 and 3) passed the tolerance criteria set within the standard code for concrete structures.
The methods presented in this manuscript showed great promise for the automated extraction of common structural elements from reinforced concrete structures with applications to automated progress monitoring and dimensional compliance control. The following provide additional avenues for future research and expansion: • Examination of the methods proposed in this manuscript for progress monitoring and dimensional conformity control of rebars in reinforced concrete projects where a detailed planned BIM, containing the complete details of the rebars, exists.

•
The simultaneous application of scan vs. BIM, supervised learning, and the methods proposed in our study for the extraction of structural components with complex geometries. Additionally, the application of novel methods used to reduce the dependency of semantic labeling on new training data, such as those presented in [61], for TLS acquired from construction sites is an interesting research topic for future investigations.

•
The extraction of temporary objects, such as scaffolds and formwork, from TLS acquired from construction sites using validated methods applied to photogrammetric point clouds, such as those proposed in [62].

•
Evaluations of methods proposed by [63] for surface flatness assessment to generate a standardized surface flatness metric. • Development of a fuzzy logic-based uncertainty model for the estimation of the location of structures, similar to the method proposed by [64] for the prediction of the locations of utility data.