Improve Building Façades in Open Lidar Data using 2 Ground Imagery

Recent advances in open data initiatives allow us to free access to a vast amount of open 8 LiDAR data in many cities. However, most of these open LiDAR data over cities are acquired by 9 airborne scanning, where the points on façades are sparse or even completely missing due to the 10 viewpoint and object occlusions in the urban environment. Integrating other sources of data, such 11 as ground images, to complete the missing parts is an effective and practical solution. This paper 12 presents an approach for improving open LiDAR data coverage on building façades by using point 13 cloud generated from ground images. A coarse-to-fine strategy is proposed to fuse these two 14 different sources of data. Firstly, the façade point cloud generated from terrestrial images is initially 15 geolocated by matching the SFM camera positions to their GPS meta-information. Next, an 16 improved Coherent Point Drift algorithm with normal consistency is proposed to accurately align 17 building façades to open LiDAR data. The significance of the work resides in the use of 2D 18 overlapping points on the outline of buildings instead of limited 3D overlap between the two point 19 clouds and the achievement to a reliable and precise registration under possible incomplete 20 coverage and ambiguous correspondence. Experiments show that the proposed approach can 21 significantly improve the façades details of buildings in open LiDAR data and improving 22 registration accuracy from up to 10 meters to less than half a meter compared to classic registration 23 methods. 24


Introduction
In recent years, there has been a significant push from the open data initiatives in many North American cities [1][2][3] or the large projects such as Infrastructure for Spatial Information in the European Community (INSPIRE) [4,5] proposed by European Commissions to provide vast amounts of open datasets that also include open LiDAR data [6,7].Nowadays, open LiDAR data covering most parts of Europe and North America are already available for the public.Due to the free access to these open LiDAR data, new avenues of research for students, researchers, and other LiDAR data user community have been opened [8][9][10].However, these open LiDAR data are often sparse and incomplete, or even entirely void on the façades due to the viewpoint and occlusions in the urban environment.This problem makes it difficult to achieve fine building reconstruction with high levels of detail (LoD) [11].
Recently, ground imagery capture devices such as off-the-shelf digital cameras, smartphones with GPS and digital compass have become ever prevalent.They allow us to acquire a number of high-resolution images of the building façade through crowd-sourcing at low cost.Additionally, the state-of-the-art Structure-from-Motion (SfM) and Multi-View Stereo (MVS) reconstruction techniques [12][13][14][15][16] allow us to process these ground images so as to recover façades information in Various researches have been studied on the fusion of multiple sources of data to reconstruct buildings with rooftops and façades information.Boulaassal et al. [17] combined airborne LiDAR scanning (ALS), terrestrial LiDAR scanning (TLS) and vehicle LiDAR scanning (VLS) data to produce reliable 3D building models, however, the high costs of using several kinds of laser scanners limited the applications of this technique.Besides, the success of this combination relied on a controlled and corrected geo-referencing of GPS before processing.Shan et al. [18] addressed this problem using a viewpoint-dependent matching method so that the aerial and the ground images could be accurately matched to generate high-quality multi-view stereo models.However, this depends on the quality of ground imagery and its similar appearance with the aerial images.Wang et al. [19] proposed a system for aligning 3D SfM point clouds produced from Internet imagery to existing Google Earth 3D models and Google Street View photos.Their method relied on the quality of Google Earth 3D models which often is not very credible and may vary in different cities.
Essentially, the fusion of façade point cloud and open LiDAR data is a process of point set registration that maps one point set to the other according to their correspondence.Point set registration is a crucial step in many computer vision and photogrammetry tasks including stereo matching [20], medical imaging [21], heritage reconstruction [22], shape retrieval [23] and industrial applications [24].Iterative Closest Point (ICP) [25] is the most widely used and classic point sets registration algorithm due to its simplicity and low computational complexity.It iteratively assigns correspondence based on the closest distance criterion and finds the least-squares transformation between the pair of point sets until a local minimum is reached.A major drawback of ICP algorithm is that it demands an accurate initial guess of the correspondence between two point sets.Otherwise, it may fall into a local minimum or even be non-convergent.A lot of ICP-based variants have been proposed to address the weaknesses [26][27][28][29][30]. Myronenko et al. proposed a probabilistic-based point set registration algorithm [31] which is called Coherent Point Drift (CPD).CPD considers the alignment of a pair of point sets as a probability density estimation problem where one point set represents the Gaussian Mixture Model (GMM) centroids, and the other one represents the data points.The rigid transformation that aligns GMM centroids to data points is obtained by maximizing the GMM posterior probability for data points at the optimum.The CPD algorithm, which exhibits a linear computational complexity, outperforms most state-of-the-art algorithms and achieves promising results with respect to conditions of noise, outliers, and missing points.This paper presents a novel method for improving open LiDAR data on the building façade using the façade point cloud generated from ground images.Firstly, to reduce the significant differences in rotation, scale, and translation between the two kinds of point cloud, we achieve initial geolocation of façade point cloud by matching the SfM camera positions to their GPS imaging metadata.Then, a modified CPD algorithm with normal consistency is proposed to achieve precise registration by making full use of similarity on 2D outlines of buildings.The significance of the work resides in the best use of the most likely overlap between the two point clouds and the achievement to a reliable and precise registration under possible incomplete coverage and ambiguous correspondence.The overview of the proposed method is illustrated as Figure 1.The remainder of this paper is structured as follows.In section 2, we describe our approach for aligning façade point cloud generated from ground images to open LiDAR data.Section 3 presents the test results and discusses its performance.Finally, we conclude in section 4.

Methodology
Given ground image set {  | = 1,2, … }, COLMAP [32]   A RANSAC-like 2D similarity transformation is estimated between the camera positions' x-y coordinates in local SfM coordinate and their corresponding longitude and latitude of GPS: Given the local coordinates {  } and the geo-referenced coordinates {  } of the ground cameras, the minimal subset (size 3) of the ground cameras for point sets registration is selected from {  } and {  } at random.Then, the 2D-2D similarity transformation is estimated using the leastsquares solver, and the similarity transformation parameters are calculated.The inlier set of the estimated transformation is obtained with the inlier threshold ε.This process is repeated to obtain the maximal consensus set, which has the maximal number of inliers.Finally, the similarity transformation {  ,   ,   } for geolocating the cameras (images) as well as the facade point cloud into the geo-referenced coordinate system is estimated with this maximal consensus set using the least-square method again.This procedure is formulated in Equation 2.
Then, a vertical translation   is calculated by matching the mean value of z coordinate in {  } and mean value of altitude in {  } .Finally, apply transformation {  ,   ,   } to (x, y) coordinate of  and transformation {  ,   } to z coordinate of  , initial geolocated facade point clouds  can be obtained.
Scale, translation and rotation differences are greatly relieved after initial alignment as described above, although there are meter-level [33] alignment errors between the initial geolocated facade point cloud  and the open LiDAR point cloud  due to the GPS location uncertainties.

Extended Coherent Point Drift with Normal Consistency (NC-CPD)
To further reduce the initial alignment errors, accurate alignment is necessary for obtaining and the other one represents the data points ( × ).The rigid transformation ( , , ) that aligns GMM centroids  × to data points  × are obtained by maximizing the GMM posterior probability for the data point  × at the optimum.The GMM probability density function of CPD can be written as Equation 2: where (|) = ( ) /  ‖ ‖ and a uniform distribution (| + 1) = 1  ⁄ is used to account for outliers and the weight of it is donated as  (0 ≤  ≤ 1).() = 1  ⁄ for all GMM components.Then, the mixture model takes the form: GMM centroids locations are re-parametrised by rigid transformation parameters (, , ).We can estimate them by maximizing the negative likelihood function: The correspondence probability is defined between two points  and  as the posterior probability of the GMM centroids given the data points: The estimation of parameters (, , ,  ) can use the Expectation Maximization (EM) algorithm.
The first step (E step) : predicts the value of parameters based on previous values ( , , ) and then Bayes' theory is used to calculate a posteriori probability distributions as the following equation: The second step (M step): obtain new parameters by minimising negative logarithm likelihood function of Equation 4. The EM algorithm proceeds by alternating between E and M-steps until convergence.After ignoring the constants independent of (, , ,  ), it can be written as: where  = ∑ ∑  (| ) .For detailed solution process, please refer to [31].In the original CPD algorithm, a Gaussian distribution is used to model the likelihood of each centroid (|).

Coherent Point
where  is the standard deviation of all   •   − 1 .Then, the likelihood of centroids is modified as following: When  = 1, the corresponding priority of each centroid is same, and NC-CPD degenerated to the original CPD algorithm.

Accurate alignment using NC-CPD
Although overlaps between  and  in 3D space is hard to be found, 2D façade point overlaps can be accurately extracted at most of the time.We decompose the accurate alignment into a horizontal transformation and a vertical transformation, as shown in Figure 4.

Datasets Description
So far, there are currently no available benchmark datasets for fusing airborne LiDAR data and facade point cloud generated from images.The proposed method is evaluated on a combined dataset: 1. Open LiDAR data of Dortmund in German which contain three experimental buildings (Rathaus, Lohnhalle, Verwaltung) are downloaded from a Germany open data download portal [7].
These open LiDAR data have been geolocated in the ETRS89 reference system with a UTM projection with a point density of 25 points/m 2 .
2. Ground images of the three buildings come from a part of a benchmark dataset named "ISPRS benchmark on multi-platform photogrammetry" [35] which can be downloaded from the official website of ISPRS.These images are captured around the buildings using high-resolution digital cameras on the ground.Due to the use of GPS locating accessories, image shooting positions are recorded in these JPEG format images as GPS meta-data.Global coordinates of targets centers distributed on the façade of the three buildings are provided for accuracy estimation.The details of these image collections are listed in Table 1.

  
With GCP   

Qualitative Analysis
As shown in Figure 7

Quantitative Analysis
As shown in section 2.2, an iteration process is performed in the EM process to find the optimal alignment result.After about less than 30 times iteration, the ratio of  to initial  quickly decline to 1%, as shown in Figure 10.Consequently, a relatively large error appeared in dataset Rathaus using the proposed method.For Lohnhalle dataset, we believe the disclosure of images captured around the target building cause the relatively apparent errors in SfM process, even though the mean capture distance is much closer than that of Rathaus dataset.However, the quality of the façade point cloud generated from images is uncontrollable during the registration process, and how to improve it is beyond the scope of this article.

Robustness Analysis
It is known that different point densities and degrees of noise in the point clouds have a significant impact on the performance of registration.We take several experiments on point clouds mixed with different degrees of noise as well as point densities to test the robustness of the proposed method.as ICP and NDT cannot handle our datasets in which limited overlaps can be found.Comparing with ICP and NDT, the proposed method reduces the registration errors from up to 10 meters to less than half a meter.We believe that our approach achieves good accuracy for the following reasons:(1).
Using Two-step strategies.Scale, translation and rotation differences are greatly relieved after coarse alignment using GPS information of images.Both completeness and structural details of buildings in the open LiDAR data are significantly improved after accurate alignment so that a complete and full resolution city building modelling and other applications can be achieved.Our method can be extended to acquire images of different buildings via crowdsourcing for improving façades details for open LiDAR data at city-scale.In the future, we intend to carry larger trials with more terrestrial images of buildings via crowdsourcing.
Despite working well on many datasets, our method relies heavily on high-quality façade point cloud generated from the SfM and MVS process in order to use the 2D outline information, which is the only similarity between the two building point clouds.As such, the completeness and correctness of façade point cloud require continuous improvement in image matching.
fine detail and precision.Considering that ground images are complementary to open LiDAR data in terms of façade details, fusing façade point cloud generated from ground images into open LiDAR data is a promising way to improve open LiDAR data on the details of façade.
Nonetheless, the alignment between façade point cloud and open LiDAR data remains a problem because of (1) inevitable noise points in facade point cloud, including noise points from SfM and MVS procedure and noise points of other ground objects such as trees, lamp-posts, and passersby; (2) limited overlaps between the two point clouds; (3) their large density difference; and (4) their large initial offset.All these issues lead to a challenge to traditional point sets registration algorithms.

Figure 1 .
Figure 1.The overview of the proposed method.
Geolocation of Leveling Facade Point Cloud using GPS meta-data.Since the SfM procedure has recovered the facade point cloud as well as cameras (images) shooting positions in the same local coordinate system, the problem of geolocating the facade point cloud can be converted into the problem of locating the SfM camera positions to the geo-referenced coordinate system, as shown in

Figure 2 .Figure 2 .
Figure 2. Due to GPS positioning features a dramatic accuracy difference between horizontal and altitude direction, a planar transformation and a vertical translation is calculated respectively.
reliable and precise correspondences between  and  .Because of inevitable noise points in facade point cloud, including noise points generated in the SfM and MVS procedure and noise points of other ground object such as trees, lamp-posts and passers-by, an improved CPD algorithm with normal consistency is used to register the two point clouds with noise and structure ambiguities.Coherent Drift Algorithm.This algorithm was first introduced in [31] for considering the alignment of two point sets as a probability density estimation.Given two D-dimensional point sets  × = ( , … ,  ) and  × = ( , … ,  ), CPD method considers the alignment of the two point sets as a probability density estimation problem where one point set represents the GMM centroids ( × ) Drift with Normal Consistency.Though the original CPD algorithm achieves promising registration results in the situation of some noise and missing points, it may fail to handle the situation of ambiguities induced by repetitive and symmetric scene elements of buildings, as shown in Figure 3.(c).To resolve this problem, in other words, to avoid facade point cloud from registering to the ambiguities part, we introduce normal consistency into the original CPD algorithm to suppress aligning to ambiguities part by considering the normal direction of corresponding points.The normalized normal   of 2D boundary points (detail introduced in section 2.2.3) extraction from open LiDAR data can be estimated according to their neighbor points (normal direction is toward the exterior of buildings), as shown in Figure 3. (a).The normalized normal   of 2D façade points (detail introduced in section 2.2.3) extraction from the facade point cloud is generated as a byproduct in the MVS process so that it can be obtained without estimation, as shown in Figure 3. (b).We assume that the facade point cloud is correctly aligned to the actual part of the open LiDAR data only if   •   > 0 is satisfied, in other words, the included angle between   and   should be less than 90 degrees, as shown in Figure 3. (d).

Figure 3 .
Figure 3. Illustration of the wrong alignment caused by ambiguities.Fig (a) shows the 2D boundary points (red points) and their normal   (green lines with arrow).(b) shows the 2D façade points (blue points) and their normal   (blue lines with arrow).The alignment result in (c) is wrong for the big difference of normal direction in overlapping part of the two point clouds.The alignment results in (d) is correct for the high similarity in the normal direction of points in the overlapping part.

Figure 4 .
Figure 4. Overview of the accurate alignment process.

Figure 5 .
Figure 5. Results of 2D façade points extraction and 2D boundary points extraction.Figure (a) and (c) show the open LiDAR data and facade point cloud respectively.Figure (b) shows the boundary points (top view) extracted from open LiDAR data and (d) shows the façade points (top view) extracted from the facade point cloud.
, facade point clouds of Rathaus, Lohnhalle, Verwaltung (Figure.7 B1, B2, B3) are generated from ground images (Figure.7 A1, A2, A3, image sample) using SfM and MVS algorithms in COLMAP [32].The open LiDAR data of the three buildings are visualized by height rendering map in Figure.7 C1, C2, C3.It can be seen that there are no overlaps between open LiDAR data and facade point clouds on their façade part except for Verwaltung, which has a small number of points on the façades.But from another perspective, open LiDAR data and facade point clouds are complementary, the former lack structural details on the façades, while the latter lack roof information.

Figure 7 .Figure 8 .
Figure 7. Datasets for evaluating the proposed method.From top to bottom, the different rows respectively show the ground images (A), facade point clouds (B), open LiDAR (C), coarse alignment result (D) and accurate alignment result (E) of Rathaus, Verwaltung and Lohnhalle.

Figure 9 .
Figure 9. Poisson Surface reconstruction results.(a) Poisson Surface reconstruction from open LiDAR data.(b).Poisson Surface reconstruction from façade point cloud generated from ground images.(c).Poisson Surface reconstruction from fusion point cloud of open LiDAR data and façade point cloud.
photogrammetry', accurate geographical coordinates { } of target centers distributed on the façade are used for quantitative evaluations of the alignment results, as shown in Figure10.(b).We estimate RMSE (root mean square error), mean errors and standard deviation between provided global coordinates of target centers and their coordinates in the aligned results from different methods, as shown in Table2.

Figure 10 .
Figure 10.(a) Change of  after each iteration.(b) Illustration of ground targets.
Fig. 12a-b illustrates the registration results achieved by the proposed method under different degrees of noise and point densities.Robustness to Point Densities.To evaluate the robustness of the proposed method to point density, we randomly down-sample facade point cloud of Rathaus, Verwaltung and Lohnhalle from their original point density to various densities.Different RMSEs evaluated between target centre coordinates at different point density are given in Figure.12a.It can be seen that the proposed method performed well even at 1% of the original point density, indicating the robustness of the proposed method to different point densities.We attribute the robustness of our approach to different point densities to the use of 2D similarity of building outlines in the registration between two sources of point clouds.Robustness to Noise.To evaluate the robustness of the proposed method to noise, Gaussian noise with different standard deviations (1, 2, 3, 4, and 5 cm) was added to the point cloud data.The different RMSE evaluated between target center coordinates under different levels of noise are shown in Figure.12b.Even when Gaussian noise with a standard deviation of 5 cm is added to the point cloud, the proposed method achieved fine and stable accuracy.It indicates that the proposed method is very robust to different levels of noise.We attribute the robustness of our approach to different degrees of noise to the use of the probabilistic method in the accurate alignment.

Figure 11 .
Figure 11.Robustness analysis of the proposed method.

4 .
Conclusion and Future Work This paper presents an accurate and efficient framework for improving building façade details of open LiDAR data using terrestrial images without ground control point.The critical step of this framework is the alignment between limited overlapped facade point cloud generated from ground images and open LiDAR point cloud.Experiments have shown that classic registration methods such

( 2 )
. Decompose registration into a horizontal transformation and a vertical transformation instead of 3D registration directly.2D overlapping points on the contour of buildings are more stable for registration of the façade point cloud and airborne point cloud than 3D overlapping points which can hardly be found between the two different sources of point clouds.(3).The NC-CPD inherits the noise robust property of original CPD algorithm.At the same time, it can handle the registration with structural ambiguities of buildings by introducing normal consistency into the original CPD algorithm.
, a general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline, is used to generate the facade point cloud  and camera positions {  | = 1, 2 , … } in SfM local coordinate system.Additionally, the GPS meta-information {  | = 1, 2 , … } of these images are extracted from the EXIF information of {  }.Corresponding to the capture area of {  }, the open LiDAR data To avoid aligning facade point cloud to ambiguities part of open LiDAR data, corresponding priority based on normal consistency is introduced to decrease the likelihood when they are aligned to the ambiguities.To tolerate errors in estimating      , we assign the dot product of   and   to 1 if the angle between   and   is less than 45 degree, or   • axis direction are obtained.The registration process of NC-CPD is described in detail in Figure 6.EM optimization.repeat until convergence to obtain the final   ,  ,   ,   E-step: Update (|  ) with  , , ,  ,   M-step: solve for the new  , , ,  by minimizing Equation.(6)  The aligned 2D façade points is  =     +   Figure 6.The algorithm of horizontal alignment using CPD with normal consistency.
can also be calculated by using new   =   •  .Repeat the previous steps until Q does not change too much or a certain number of iterations is reached.After applying final transformation parameters (, , ) to x and y coordinates of  , accurately aligned facade point cloud  on X and Y Then, after updating Z coordinates of façade points by applying    , 3D façade point cloud  is obtained.Vertical Alignment.Façade point cloud has been accurately aligned to open LiDAR data on X and Y axis direction by the horizontal alignment described in the previous section.A translation  on the vertical direction between  and  remain to be calculated.We calculate optimal  by matching corresponding boundary points respectively from  and  on the Z axis direction, as following steps: (1).For one point ̅ ( ,  ) in 2D boundary points  , we find 2D neighbour point set { , . . .,  } and { , . . .,  } of ̅ with radius 0.1 meter, respectively from  and  .(2).Find the point  and  with maximum value on Z axis, respectively from { , . . .,  } and { , . . .,  }, then calculate height difference  =  −  .(3).For other point in  , repeat step (1) (2) to obtain height difference set { , . . .,  }.Then, the optimal  is calculated by fitting height difference set { , . . .,  }.Finally, applying translation  to z coordinate of  , accurately aligned facade point cloud  is obtained in the end.

Table 2 .
[37], ME and SD of the proposed method compared with other methods.By analysing the errors of the different methods in Table2, we can see that the proposed method significantly improves the accuracy of test datasets comparing with ICP and NDT.It is well known that Iterative Closest Point (ICP)[25]and Normal-Distributions Transform (NDT)[37]are effective methods of point sets registration with large overlaps.Experiments have shown that ICP and NDT methods cannot handle our datasets in which almost no overlaps can be found.So, errors up to 10 meters are obtained except for dataset of Verwaltung, which have a small number of points on the Although not as good as the result of TCR methods, the proposed method achieves the best accuracy compared to ICP and NDT methods due to the use of similarity on 2D outlines of buildings.The accuracy of the proposed method is highly correlated with quality of façade point cloud generated from images.For Rathaus, the farthest mean capture distance leads to an apparent worst quality of façade point cloud from captured images (most significant errors in TCR method).