Windows and Doors Extraction from Point Cloud Data Combining Semantic Features and Material Characteristics

Abstract

Point cloud data have become the primary spatial data source for the 3D reconstruction of building engineering, where 3D reconstructed building information models can improve construction efficiency. In such applications, detecting windows and doors is essential. Previous research mainly used red-green-blue (RGB) information or semantic features for detection, where the combination of these two features was not considered. Therefore, this research proposed a practical approach to detecting windows and doors using point cloud data with the combination of semantic features and material characteristics. The point cloud data are first segmented using Gradient Filtering and Random Sample Consensus (RANSAC) to obtain the 3D indoor data without intrusions and protrusions. As input, the 3D indoor data are projected to horizontal planes as 2D point cloud data. The 2D point cloud data are then transformed to 2D images, representing the indoor area for feature extraction. On the 2D images, the 2D boundary of each potential opening is extracted using an improved Bounding Box algorithm, and the extraction result is transformed back to 3D data. Based on the 3D data, the reflectivity of building material is applied to differentiate windows and doors from potential openings, and the number of data points is used to check the opening condition of windows and doors. The abovementioned approach was tested using the point cloud data representing one campus building, including two big rooms and one corridor. The experimental results showed that accurate detection of windows and doors was successfully reached. The completeness of the detection is 100%, and the correctness of the detection is 90.32%. The total time for the feature extraction is 22.8 s for processing 2 million point cloud data, including time from reading data of 10.319 s and time from showing the results of 4.938 s.

1. Introduction

When addressing building engineering problems, such as indoor localization and navigation systems [1,2], disaster management [3,4], building structure maintenance [5,6], and building information modeling (BIM) [7,8], it often requires establishing a three-dimensional (3D) visual model that can be read and monitored by computer programs [9]. Such 3D visual models can be used as building information models that can improve construction efficiency such as conducting virtual conflict detection. However, it is always challenging to efficiently obtain accurate spatial data for generating 3D models because of traditional surveying methods’ time-consuming measurement and interpolation errors [10]. Laser scanning, also known as Light Detection and Ranging (LiDAR), is a prominent remote-sensing technology for point cloud data collection [11]. By integrating a digital camera, the laser scanner can acquire a massive number of 3D points with the information, including the Cartesian coordinates in x-, y-, and z-directions, the intensity of laser beams, and red-green-blue (RGB) color. These 3D points are called point cloud data and are accurate points that can be used for generating 3D models. Compared with traditional surveying methods, terrestrial laser scanning can conduct spatial data collection efficiently and cost-effectively [12]. Nevertheless, information extraction of the 3D model from the point cloud data compatible with computational software is also problematic due to the unstructured [13] and high-density [14] characteristics of point cloud data and operation imperfections of laser scanners such as occlusions and data noise [9]. But it is worth to construct computation related models to facilitate the construction of civil engineering area, like railway [15,16] and recycled water infrastructure [17].

The extraction of building façade from point cloud data mainly includes two steps, segmentation and feature extraction. Segmentation is the precondition of feature extraction and is the process of segregating a group of points belonging to a plane [9], such as segregating a wall from the roof or segregating building polyhedrons into different planes. By contrast, feature extraction extracts building elements such as windows, doors, and walls from the façade planes.

For segmentation, the mainstream method is the model fitting approach [18,19], such as the Random Sample Consensus (RANSAC) method proposed by Fischler and Bolles [20]. In 2009, Boulaassal et al. [21] applied this RANSAC algorithm to wall segmentation and successfully extracted planar surfaces such as walls, roofs, and doors from building façade. Another popular technique is the octree indexing structure proposed by Wang and Tseng [22]. This method can only perform rough segmentation as the principle is to split points into small planes based on average characteristics such as height, intensity, and shape orientation, where complicated shapes cannot be handled. Then in 2014, Lari and Habib [23] adopted principal component analysis to identify whether a point belongs to a linear, a planar, or a cylindrical surface and improved the model by estimating the local points’ intensity variation. The model is robust, but the result is not accurate enough for opening detection. In 2016, Mahmoudabadi et al. [24] proposed a computer vision algorithm by exploiting the angular structure of point cloud data including laser intensity, range, normal vectors and color information. The method showed less computation time, more qualitative segmentation and less user intervention than RANSAC. After that, in 2018, Che and Olsen [25] implemented NORmal VAriation ANAlysis (Norvana) for extracting edge points and applied region growing to grouping the points on a surface to obtain the segmentation result. The method enabled exploiting angular grid structure from unorganized point cloud for multiple registered scans simultaneously. Then in 2020, Luo et al. [26] proposed a deep learning method for conducting unsupervised scene adaptation semantic segmentation, including data alignment using pointwise attentive transformation module (PW-ATM) and feature alignment using a maximum classifier discrepancy-based (MCD-based) adversarial framework. The method focuses on cross-scene semantic segmentation of urban mobile laser scanning (MLS) point clouds. In 2021, Chen et al. [27] applied a learnable region-growing method for segmenting a class-agnostic point cloud. A deep neural network was trained to add or remove points from the point cloud region to morph into a more complete object instance, and the method can process arbitrary shape. After that, Zhang and Fan [28] adopted an improved multi-task pointwise network (RoofNet) for roof plane segmentation. The method did not depend on human intervention and had good performance on both instance and semantic segmentation.

Feature extraction is the second step after segmentation. In 2004, Wang and Tseng [22] applied a simple classification method using planar attributes to classify roofs, walls, and ground. Then in 2006, Bendels et al. [29] tried calculating the distance from each point to the neighboring points by combining different angles and shapes to detect the holes in point cloud data. This method successfully detected holes, but intensive computation and human intervention are required. Shortly after, Pu and Vosselman [30] introduced a fully automatic method for extracting building façade features. The method works well for doors but not for windows because of a lack of data, and a set of rules and constraints necessitated to be followed. Then in 2012, the Façade Delaunay (FD) method was proposed by Truong-Hong et al. [31] based on Delaunay Triangulation (DT) for extracting boundary points of façade and openings. The process is highly time-consuming, and a preset length threshold is needed. After that, Truong-Hong et al. [32] applied angle criteria for detecting boundary points and kd-tree for accelerating the selection of k-nearest neighbor (KNN) points for candidate boundary points. Still, the computation is tedious as all the data points need to be checked. To reduce time consumption, the Façade Voxel (FX) method was introduced in 2014 by Truong-Hong and Laefer [33], where only data points around the façade boundary and openings will be considered as candidate points. This method significantly improves the time consumption, but sufficient data and a predefined voxel size threshold are still needed. Then in 2016, Ni et al. [34] proposed Analysis of Geometric Properties of Neighborhoods (AGPN) for detecting edges and tracing feature lines. The method enables to eliminate preprocessing such as segmentation and object recognition and is insensitive to the density of input data. After that, in 2017, Li et al. [35] applied least-squares fitted normalized gray accumulative curves to regular structures detection and used a binarization dilation extraction algorithm for façade elements partition. The method is robust with respect to noise and point cloud densities and is reliable when experimenting with different façade shape. In 2019, Shi et al. [36] proposed PointRCNN for 3D object detection from point cloud using bottom-up 3D proposal generation and proposal refinement in canonical coordinates. Different from projecting point cloud to bird’s view or voxels, the method directly generates canonical 3D boxes of detected objects. Then in 2021, Shao et al. [37] implemented a top-down strategy based on cloth simulation to detect seed point set with semantic information and applied a region growing method for further feature extraction. Different from the bottom-up method, this top-down strategy with help of a region-growing algorithm can directly extract roofs with little error and fast speed.

As a specific branch of feature extraction, extracting windows and doors has multiple methods based on point cloud data, resembling normal vector components and spectral band information [38]. In 2014, Previtali et al. projected the 3D point cloud data onto a 2D plane along the normal vector orientation, followed by automatic extraction of the break lines of the façades from the 2D planes [39] for the detection of windows and doors. They used the method of transferring 3D point cloud data to 2D images and, by extracting façade break lines, potential openings with rectangular shapes were segmented from the building planes. However, the research did not distinguish windows, doors, and walls, where all the potential openings were detected simultaneously, and further work needs to be performed on differentiation. Malihi et al. adopted the Gestalt principles for window frame detection, in which the differences in data densities between window frames and other building elements were considered [40]. The approach can avoid the most influence from intrusions and protrusions on the window detection, but several doors with glass and parts of scaffolding were detected as windows. Then in 2018, Previtali M. applied the ray-tracing algorithm to detect indoor openings from the point clouds collected by Indoor Mobile Mapping System [41]. The differentiation of windows and doors was taken by applying semantic features of windows and doors, where the assumption was made that windows and doors had only single shapes and these two shapes were different. The differentiation result was compelling, but the assumption is not suitable for all the buildings, as windows may have different shapes in one building (same for doors). Furthermore, windows and doors may have similar shapes. Jung et al. adopted noise filtering, subsequent segmentation and regularization using the inverse binary map to reconstruct building interiors [8]. Less-constrained input is needed in this method, but the time taken on segmentation was 105.7 s, which caused expensive computation work. In 2020, Jarząbek-Rychard et al. applied TIR (thermal infrared) and RGB information for façade classification [42], but the differentiation of windows and doors was not conducted as well. Zhao et al. applied point cloud slices and minimum bounding rectangles to extract windows and doors, and the process can be finished in 8 s for 1.6 million point cloud data [43]. The processing is efficient, but windows are still not differentiated from doors. Then in 2021, Cai and Lei implemented a moving window to detect potential openings on 2D planes of buildings and intensity histograms of data points were used to differentiate doors and walls [44]. The generated model is accurate, but the time consumption is high because of the large amount of point cloud data, where further research can be taken on improving the efficiency.

In summary, previous studies usually adopt semantic features of windows and doors or intensity of building materials for windows and doors detection from point cloud data. Generally, previous studies have the limitation that differentiation of windows and doors and computational efficiency cannot be satisfied simultaneously. In this paper, a new approach utilizing both semantic features and building material property is proposed, based on Gradient Filtering [45] and RANSAC for the segmentation process and an improved Bounding Box algorithm in MATLAB for the feature extraction process. The proposed approach has three main advantages: (1) it is a fast method in windows and doors’ extraction with only 7.543 s needed for 2 million point cloud data; (2) windows and doors can be differentiated; and (3) the opening condition and the material property of the openings can be detected.

2. Materials and Methods

2.1. Study Site and Data

One campus building, including two big rooms and one corridor, was selected as the study site. The data were surveyed from multiple scan stations by a Leica RTC360 scanner. The point cloud data obtained from each station were firstly aligned by the laser scanner during the surveying process, and further refined registration was taken by Leica Cyclone software. The registered point cloud data has a large amount (476 million data points). The positional accuracy of the data points is calculated from the point-to-plane distance using the RANSAC algorithm to fit the plane containing the maximum data points of each dataset. After calculation, the mean and standard deviation of the point-to-plane distance is 0.11 mm and 0.26 mm, respectively. For further data processing, the registered point clouds for the site are approximately 2 million data points after subsampling. The result in Cloudcompare is displayed in Figure 1.

2.2. Methodology

The method mainly includes four steps, as shown in Figure 2. The first step is to subsample the data from 476 million data points to 2 million data points. The second step is the segmentation process, which includes the Gradient Filtering method for removing the horizontal structure part of buildings (ceilings and floor) and the RANSAC algorithm for fitting wall planes from the building structure. The third step is to extract windows and doors (potential openings) from the wall planes, including the projection of 3D data to 2D binary images, and apply the improved Bounding Box algorithm for detecting potential openings. The fourth step is to transform 2D detection results back to 3D, thus differentiating windows and doors and identifying their opening conditions in 3D data.

The improved Bounding Box algorithm is written as codes in MATLAB. By running the codes in MATLAB to process the point cloud data, the improved Bounding Box algorithm can be applied for detecting potential openings in this paper. Generally, using artificial intelligence approach in windows and doors detection can be performed using Convolutional Neural Network (CNN) for image processing, which uses amounts of images for learning. Since the images for this study are limited, the artificial intelligence approach is not chosen.

This paper introduces several methods for processing the data points and reducing the time consumption, including subsampling the dataset, using the Gradient Filtering method, and using an improved Bounding Box algorithm. Subsampling the dataset can reduce the data density. It can reduce the further processing time as the number of data points has been reduced. The Gradient Filtering method can directly discard the roof and floor part and remain the façade part. Similarly, as the number of data points have been reduced, the further processing time will be reduced. Improved Bounding Box algorithm can minimize the computational resources so as to reduce the processing time.

2.2.1. Data Preprocessing (Step 1)

As the original data is too dense for further processing (476 million data points), subsampling is used to preprocess data points to decrease the number of points to 2 million. The subsampling process is conducted in Cloudcompare software. The ‘spatial mode’ is used as the subsampling method, and the minimum distance between two points is 20 mm. The minimum distance of 20 mm has been tested in Cloudcompare using a wide range of minimum distances for test samples so that the obtained point cloud data density can be successfully fitted to separate planes using RANSAC in step 2.

The schematics of subsampling has been shown in Figure 3, where $\mathrm{d}$ is the minimum distance between two points. When a candidate point (blue point) is used to conduct the distance-based subsampling, a minimum distance will be set by the user to select points having distance to the candidate point less than the minimum distance. These points will be selected points (red points) and the other points having a distance to the candidate point larger than the minimum distance will be unselected points (black points). All the selected points will be eliminated. The process will be conducted for all the points so that the minimum distance between two arbitrary points will be $\mathrm{d}$.

2.2.2. Segmentation Using Gradient Filtering and RANSAC Algorithm (Step 2)

Different from Previtali’s [41] research work using the ray-tracing algorithm, this study used Gradient Filtering, which can discard the floor and ceiling part directly and obtain the wall part of the building within a shorter time. The principle of the Gradient Filtering method is to calculate the gradient value of the scalar field and then filter the data points by gradient value [45,46]. The method for calculating the gradient vector of one point in 3D coordinates is shown in Equation (1).

$$\nabla \mathrm{f}=\frac{\partial f}{\partial x}i+\frac{\partial f}{\partial y}j+\frac{\partial f}{\partial z}k\left(1\right)$$

(1)

where $i$, $j$, $k$ are the standard unit vectors in the direction of x, y, z coordinates, respectively. $\nabla \mathrm{f}$ is the resultant vector. The method to calculate the gradient value here is to select the gradient value in z direction, which is $\frac{\partial f}{\partial z}$. The ceiling and roof part will have small gradient value in z direction as they are generally perpendicular to z direction. The façade part will have large gradient value as it is generally parallel to z direction. So, the ceiling and roof part can be selected and discarded by using this method and the façade part can be remained.

Then, by setting RANSAC parameters appropriately, the plane fitting is satisfactory, and the process is efficient (2.424 s from Cloudcompare). The minimum support points per primitive are set to be 500, and the maximum distance of samples to the ideal shape is set to be 0.2 m. The sampling resolution, which represents the distance between neighboring points, is set as 0.1 m. The maximum deviation from the ideal shape average vector is set to be 5°. The overlooking probability, indicating the probability that no other candidate was overlooked during sampling, is set as 0.01 [46].

The principle of RANSAC is as follows: (1) Randomly selecting a subset of data set; (2) Fitting a model to the selected subset; (3) Determining the number of outliners; (4) Repeating steps 1–3 for a prescribed number of iterations. By conducting such steps, the RANSAC can perform a robust estimation of the model parameters.

2.2.3. Windows and Doors Extraction in the 2D Image with Improved Bounding Box Algorithm (Step 3)

Each plane has been rotated to planes parallel to the XOZ plane or the YOZ plane using a rotating matrix. For planes supposed to be parallel to the y-y axis, the rotating matrix should be as follows:

$$\begin{array}{ccc}\cos \left(0.5\pi -\theta \right)& -\sin \left(0.5\pi -\theta \right)& 0\\ \sin \left(0.5\pi -\theta \right)& \cos \left(0.5\pi -\theta \right)& 0\\ 0& 0& 1\end{array}$$

For planes supposed to be parallel to the x-x axis, the rotating matrix should be as follows:

$$\begin{array}{ccc}\cos \left(\pi -\theta \right)& -\sin \left(\pi -\theta \right)& 0\\ \sin \left(\pi -\theta \right)& \cos \left(\pi -\theta \right)& 0\\ 0& 0& 1\end{array}$$

The rotation is based on the theory of the Manhattan World assumption [47] that the common planar primitives and straight lines are intersected orthogonally. Therefore, all the façade planes are assumed to be perpendicular to the XOY plane, which is supposed to be the ground level. Then, the 3D point cloud data will be regulated to 2D point cloud data by rotating the fitted planes to the orientation parallel to the XOZ plane or the YOZ plane. θ is the rotating angle, as shown in Figure 4. After rotating all the planes parallel to the XOZ plane or the YOZ plane, the 3D point cloud data can be directly projected to 2D point cloud data.

Then, the 2D point cloud data can be transformed into 2D images, as shown in Figure 5. The method here is to define a rectangular box divided into several small square boxes. For instance, the whole 2D point cloud data range from [0, 20] in the X direction and [0, 10] in the Y direction. Then, by dividing the whole 2D area into 200 small squares (20 portions in the X direction and 10 portions in the Y direction), each 2D point cloud data can be sorted into one specific small square box according to the position of the 2D point cloud data. The spectral information (reflectivity) of the small square box is to be determined by averaging the spectral value of the 3D point cloud data sorted into this tiny square box. If there is 2D point cloud data in the small square box, the binary value should be 1. If there is no 2D point cloud data to be classified into this tiny square box, the binary value of this box will be 0.

After all the 3D point cloud data have been transformed into 2D images, the extraction of windows and doors can be implemented on the 2D images by using an optimized Bounding Box algorithm. The original Bounding Box algorithm is shown in Equation (2) [48], which returns the location of the fitted rectangle.

$$\left(x,y,w,h\right)= regionprops\left(fil{l}_{hole}{,}^{\prime }boundingbo{x}^{\prime }\right).Boundingbox$$

(2)

where $x$ indicates the x location of the initial point; $y$ indicates the y location of the initial point; $w$ indicates the width of the fitted rectangle; $h$ indicates the height of the fitted rectangle for determining the location of windows and doors. The improved Bounding Box algorithm has been used by setting specific parameters based on the building semantic features. The process of setting specific parameters is as follows.

Ameliorate the Boundary Condition of the Bounding Box Algorithm

At the first stage, the detection result of the potential openings is not satisfactory, as most of the bottom boundaries of the doors were not detected. The solution is to eliminate the white space (noise) at the bottom of each plane’s binary image so that the image’s margin can approach the doors’ bottom boundary. To eliminate the image noise, “bwareaopen” removes the minimum area smaller than the threshold value, as shown in Equation (3).

$$BW2=bwareaopen\left(BW,P\right)$$

(3)

where BW2 means another binary image removing all the connected components with fewer than P pixels from the binary image BW. After testing the result of the “bwareaopen” command, the connect-area parameter P is determined to be set to P = 100.

Elimination of Nonstandard Shape

The nonstandard shapes of windows and doors include shapes with too small or big areas and too high aspect ratio. The elimination of the nonstandard rectangles will be conducted by setting an area threshold value in the programming code that is not to be smaller than 0.9 m² and not to be larger than 3 m². In addition, the shape with an aspect ratio (height:width) larger than 3 (3:1) will be considered not to be windows and doors and eliminated.

Region Union Method

The region union method is developed to solve the specific condition where two rectangles intersect, as shown in Figure 6. Then, the effect of the region union method is to merge the two rectangles to be one rectangle. For instance, if one rectangle with four points $\left(x_{1,1},y_{1,1}\right)$, $\left(x_{1,2},y_{1,2}\right)$, $\left(x_{1,3},y_{1,3}\right)$, $\left(x_{1,4},y_{1,4}\right)$ intersects with another rectangle with four points $\left(x_{2,1},y_{2,1}\right)$, $\left(x_{2,2},y_{2,2}\right)$, $\left(x_{2,3},y_{2,3}\right)$, $\left(x_{2,4},y_{2,4}\right)$, then the merged rectangle will be with four points $\left(x_{1,1},y_{1,1}\right)$, $\left(x_{1,2},y_{2,2}\right)$, $\left(x_{2,3},y_{1,3}\right)$, $\left(x_{2,4},y_{2,4}\right)$.

Segmentation Method

The effect of the segmentation method is to segment the area composed of rectangles with different levels, which is developed by adding a segmenting line at the juncture of the two modules that are supposed to be partitioned from each other, as shown in Figure 7.

Method for Window Detection

In most cases, doors have similar bottom levels, while windows have similar top levels, as shown in Figure 8. Therefore, it is more reasonable and practical to detect the windows from the top of the building façade. The idea of this study for window detection is to reverse the image plane up and down so that the detection from the bottom can be transversed to detection from the top. Thus, we used the “flipud” command in MATLAB to reverse the image plane to flip vertically.

After all the potential openings have been detected, the result of the 2D image will be transformed back to 3D point cloud data by using the inverse rotating matrix, where angle $\theta$ will be the same value as the rotating matrix.

2.2.4. Identification of Windows and Doors and Their Opening Condition (Step 4)

To further conduct the differentiation of windows and doors and identification of their opening condition, this study combines building semantic features and material properties:

(1)

If the bottom boundary of the opening is on the ground level, then the opening is supposed to be a door, and the reflectivity of the area needs to be verified.

(2)

If the bottom boundary of the opening does not fit the ground level, the material properties will be taken as the primary factor to define whether the detected area is a window, door, or wall. Timber material has reflectivity ranging from 0.2 to 0.35 [46]. White paint on the non-filed concrete has a reflectivity of about 0.6 [49]. Glass has a transmittance of over 0.95 [49]. The scalar field value in Cloudcompare software represents the reflectivity value proportionally. The object having a scalar field value larger than timber material and smaller than white-painted concrete will be regarded as a whiteboard. The overall value relationship will be $S_{white painted concrete}>S_{whiteboard}>S_{wood}>S_{glass}$, where $S$ stands for the scalar field value.

For the opening condition, the detection result is based on the rate of the number of the small squares without point cloud data divided by the number of all the small squares (Equation (4)), where one small square is one pixel of the binary image.

$$opening rate= \frac{number of small squares without point cloud data}{number of all small squares set for binary image}$$

(4)

If the opening rate is smaller than 0.5, the opening is defined to be closed; if the opening rate is larger than 0.5 and smaller than 0.65, the opening is defined to be half open; if the opening rate is larger than 0.68, the opening is defined to be open.

3. Results

3.1. Results of Gradient Filtering and RANSAC Algorithm

The steps mentioned in Section 2 were realized in MATLAB codes that run in a computer with Windows 10 system (64-bits), an Intel Core i5-3317U processor, and 16 GB RRD3 RAM. Gradient Filtering was applied to the site data to discard the ceilings and floors of the building. The post-filtering dataset is shown in Figure 9.

The RANSAC shape detection results are shown in Figure 10. Using RANSAC, indoor obstacles (desks and chairs) have been directly discarded. After deleting indoor noise data (Figure 10b) manually, the planes left can be used for further analysis, as shown in Figure 10c. The maximum distance of samples to the ideal shape has been set to larger than the offset of windows and doors from the wall plane so that the data of windows and doors can be reserved within the wall plane.

3.2. Results of Windows and Doors Detection on 2D Image

3.2.1. Ameliorate Boundary Condition and Adopt Nonstandard Shape Elimination and Region Union Method

The same grammar rules were applied to all the façade planes for door detection, and one specific rule was applied to Plane 6 and Plane 27 for windows detection. For ease of demonstration, Plane 1 was used as an example to present each step’s output of door detection. The primary result is shown in Figure 11a, where the detected box selected the whole area as the potential opening. The main reason for the unsatisfactory result was that the boundary condition of the image was not ideal and needed to be improved on the ground level. After improving the boundary condition of the Plane 1 binary image, the detection results are shown in Figure 11b. There is still a small-sized rectangle that is not the normal size of windows and doors, which need to be eliminated. After applying the elimination of rectangles with nonstandard shapes, the detection results can be shown in Figure 11c. The normal size of the building openings is determined. However, the detected openings intersected, which needs to be tackled. Then, the union region method was proposed, and the satisfactory post-processing result is shown in Figure 11d. Since the segmentation method and elimination of openings with a high aspect ratio do not affect Plane 1’s detection, the demonstration of the two methods will be conducted on Plane 8.

3.2.2. Adopt Segmentation Method and Elimination of High Aspect Ratio Openings

Figure 12a shows the detection result on Plane 8 after using the same grammar rules as Plane 1, where two potential openings were detected as one. Figure 12b displays the result after adopting the segmentation method, where two potential openings were segmented successfully. Still, one rectangular shape with high aspect detected as a potential opening is not correctly detected. By eliminating of high aspect ratio of 3 (height:width = 3:1), the final detection result is shown in Figure 12c. All planes except for Plane 5 and Plane 27 applied the same grammar rules for door detection, and Planes 5 and 27 used the inverse method for windows detection.

3.2.3. Adopt Specific Method for Windows Detection

Plane 5 was taken as an example to present the rule for windows detection. The result of using the same grammar rules of door detection is displayed in Figure 13a, where none of the windows have been detected. After reversing the binary image from up to down, the detection result is shown in Figure 13b, where all the potential openings have been detected.

3.3. Results of Windows and Doors Differentiation and Opening Condition on 3D Point Cloud Data

Building material properties and semantic features were applied to 3D point cloud data detection for further differentiating windows and doors. Moreover, the opening condition is judged by calculating the opening rate defined in Section 2.2.4.

3.3.1. Windows and Doors Detection Result on 3D Point Cloud of Each Plane

Figure 14 shows the detection result of Plane 1. The detected location of the first door on the plane’s left side is correct, but the opening condition is wrong. The door should be closed, but the model’s detection opens it. The missing data points probably cause the detection mistake.

The detection result of Plane 2 is displayed in Figure 15. The third door from the left should be detected as closed, but the detection result is half open. The main reason for the mistake is similar to Plane 1, for missing data points. The material type is all correct. The rest of the results are shown in Figure 16 and Figure 17, where all the material type and opening conditions are correct, except for that Plane 6’s framed size of the whiteboard is larger than the real size.

3.3.2. Final Result Shown in MATLAB Window

The perspective views of the overall windows and doors detection of the building façade have been displayed in Figure 18. To have a good view of the results, doors are marked yellow; windows are marked blue; whiteboards are marked white; and walls are marked red.

To evaluate the performance of our framework, we computed two final evaluation indexes, completeness (Equation (5)) and correctness (Equation (6)), related to the quality of feature extraction.

$$Completeness=\frac{N_d}{N_s}$$

(5)

$$Correctness=\frac{N_c}{N_s}$$

(6)

where $N_d$, $N_c$, $N_s$ are the number of detected potential openings, correctly detected potential openings, and potential openings that should be detected. The corresponding quality performance is shown respectively in

Table 1. Table of completeness.

	Plane 1	Plane 2	Plane 3	Plane 4
Completeness	8/8 (7 doors and 1 whiteboard)	6/6 (6 doors)	1/1 (1 door)	0/0 (no opening)
	Plane 5	Plane 6	Plane 7	Plane 8
Completeness	4/4 (4 windows)	1/1 (1 whiteboard)	0/0 (no opening)	4/4 (4 doors)
	Plane 9	Plane 10	Plane 11	Plane 12
Completeness	4/4 (4 doors)	1/1 (1 door)	0/0 (no opening)	0/0 (no opening)
	Plane 13	Plane 27
Completeness	1/1 (one door)	1/1 (one window)

and

Table 2. Table of correctness.

	Plane 1	Plane 2	Plane 3	Plane 4
Correctness	8/7	6/5	1/1	0/0
	Plane 5	Plane 6	Plane 7	Plane 8
Correctness	4/4	1/0	0/0	4/4
	Plane 9	Plane 10	Plane 11	Plane 12
Correctness	4/4	1/1	0/0	0/0
	Plane 13	Plane 27
Correctness	1/1	1/1

.

$$Completeness=\frac{8+6+1+0+4+1+0+4+4+1+0+0+1+1}{8+6+1+0+4+1+0+4+4+1+0+0+1+1}=100\%$$

$$Correctness=\frac{7+5+1+0+4+0+0+4+4+1+0+0+1+1}{8+6+1+0+4+1+0+4+4+1+0+0+1+1}=\frac{28}{31}=90.32\%$$

The completion of the model detection reaches 100%, and the correctness of the model reaches 90.32%. The programming speed is fast for only 22.8 s, including the time cost by reading the point cloud data of 10.319 s and the time cost by showing the data result of 4.938 s. The time cost for the segmentation process is 40.862 s and the time cost for the feature extraction process is 7.543 s. The processing time and the differentiation effect is generally more efficient than other existing method in the literature.

Table 3. Comparison of computational efficiency with other method.

Ref. Year	Number of Data Points (Million)	Segmentation and Feature Extraction Time (s)	Differentiation of Windows and Doors
2016 [9]	2.6	3	No
2018 [8]	31–66	58.5–140.79	No
2020 [43]	1.6	8	No
2021 [44]	469.1	284	No
Our approach	2.3	7.543	Yes

shows a comparison of computational efficiency with other existing methods in the literature.

4. Conclusions

This paper introduced the improved Bounding Box algorithm to detect potential openings from building façade, where the detection of potential openings can help build 3D building information models that can be used to improve construction efficiency. The method involves three main steps. Firstly, each façade is roughly segmented using the Gradient Filtering method and RANSAC algorithm. Secondly, the segmented façade planes are transformed into 2D images. Methods including eliminating nonstandard shape, region union, and segmentation are adopted to extract potential openings on 2D images. Thirdly, the detection result from each image is transformed back to 3D data. Material reflectivity is used to differentiate windows and doors, and the opening ratio is calculated to identify the opening conditions. The improved Bounding Box algorithm represents advantages with respect to the following: (1) accurately extracting and differentiating windows and doors; (2) minimizing the computational resources needed; (3) detecting the opening condition and the material property of the openings. The speed of the programming model is generally fast. The preprocessing process for subsampling the dataset and segmentation process totally takes 40.862 s. The time cost for feature extraction process is only 7.543 s, excluding the time cost of reading data of 10.319 s and the time cost of showing data of 4.938 s. By calculation, the completeness of the feature extraction is 100% and the correctness is 90.32%. In future research, the number of read-in points needs to be tested to obtain the optimal result for each plane of the dataset so that the parameters of the Bounding Box used for detecting windows and doors can be set to be in a more robust range.