Registration of Vehicle-Borne Point Clouds and Panoramic Images Based on Sensor Constellations

Abstract

A mobile mapping system (MMS) is usually utilized to collect environmental data on and around urban roads. Laser scanners and panoramic cameras are the main sensors of an MMS. This paper presents a new method for the registration of the point clouds and panoramic images based on sensor constellation. After the sensor constellation was analyzed, a feature point, the intersection of the connecting line between the global positioning system (GPS) antenna and the panoramic camera with a horizontal plane, was utilized to separate the point clouds into blocks. The blocks for the central and sideward laser scanners were extracted with the segmentation feature points. Then, the point clouds located in the blocks were separated from the original point clouds. Each point in the blocks was used to find the accurate corresponding pixel in the relative panoramic images via a collinear function, and the position and orientation relationship amongst different sensors. A search strategy is proposed for the correspondence of laser scanners and lenses of panoramic cameras to reduce calculation complexity and improve efficiency. Four cases of different urban road types were selected to verify the efficiency and accuracy of the proposed method. Results indicate that most of the point clouds (with an average of 99.7%) were successfully registered with the panoramic images with great efficiency. Geometric evaluation results indicate that horizontal accuracy was approximately 0.10–0.20 m, and vertical accuracy was approximately 0.01–0.02 m for all cases. Finally, the main factors that affect registration accuracy, including time synchronization amongst different sensors, system positioning and vehicle speed, are discussed.

1. Introduction

1.1. Background

The Mobile Mapping System (MMS) is an advanced system used to collect the environmental geospatial and texture data and consists of three main parts: mapping sensors, a positioning and navigation unit for spatial referencing, and a time referencing unit [1]. Since the first MMS was created by the Center for Mapping at Ohio State University [2,3], MMS has developed rapidly and become an important data source of 3D city modeling, indoor mapping and urban mapping and planning [1,4,5]. Other advanced MMSs have been developed and are used for real-time air pollution monitoring and health risk management [6,7].

Vehicle-borne MMSs are usually installed with a laser scanner and digital camera or video as mapping sensors [1]. Therefore, the integration of the data collected by different sensors, especially the fusion of point clouds and digital imagery, including depth imagery, has become an important research topic [8,9,10,11]. Panoramic cameras have been commonly utilized to replace the traditional digital camera in collecting texture information [12,13,14]. Compared with traditional digital cameras, panoramic cameras collect environmental data with a better field of view and accurate positioning module. Therefore, the fusion of panoramic images and other data from sensors, especially the point clouds from a laser scanner, is an important issue.

The following subsection presents several previous studies on the fusion of a point clouds and images, including panoramic images.

1.2. Previous Studies

Previous studies relating to the registration of panoramic images and the LiDAR (Light Detection And Ranging) point clouds describe four methods. The first method is a non-rigid ICP (Iterative Closest Point) algorithm proposed by [15], which incorporates a bundle adjustment for ICP processing and conducts a refinement using SIFT (Scale-Invariant Feature Transform) features detected from both kinds of datasets. The second method is a sensor-alignment based method, which extracts each CCD (Charge-Coupled Device) camera from the panoramic imaging system and obtains the accurate internal orientation parameters through calibration [16]. Third, a complex model from a world coordinate system to single CCD images was established to find the precise relationship between point and pixel. A collinear principle, which establishes the relationship between the center of the panoramic camera, the image point on the sphere, as well as the object point, was proposed by [17]. This method also uses the accurate relationship between GPS, POS (Positioning and Orientation System), and LiDAR sensors. The final method, the feature-line based registration model proposed by [18], extracts the linear features of the terrain from both LiDAR data and the panoramic images to establish the transformation model. All these methods were successfully applied to the registration of panoramic images and TLS (Terrestrial Laser Scanning) or MLS (Mobile Laser Scanning) point clouds. As the panoramic camera is always working during data collection, this will lead to situations where the registration between panoramic images with point clouds is a 1:N problem, as a point of the point clouds will find more than one pixel during registration.

Although only a few studies have addressed the registration between point clouds and panoramic images, the methods for the registration of point clouds and images can also be used if the panoramic images are separated into individual ones. These methods can be divided into two types.

The first type of registration method is based on the point clouds generated by laser scanners and digital cameras. An important parameter to describe this method is the overlapping rate of the images acquired by digital cameras. When the overlapping rate of digital images exceeds a set threshold, dense registration [19] or structure from motion (SFM) [20] algorithms can be applied to calculate the point clouds pixel by pixel for adjacent or unorganized images. Registration of the point cloud and digital imagery is transformed to registration between the point clouds from a laser scanner and point clouds from digital imageries by SFM or other methods. Several traditional or recently developed methods can be used in this situation, of which iterative closest point (ICP) [21] is the most common method. Several extensions and additional conditions based on the basic ICP algorithm, such as random sample consensus [22] and the least median of squares (LMS) estimators [23], have been developed to increase robustness and convergence and improve computational efficiency and performance. Various other methods based on the features extracted from both sources of the point clouds, such as corner points of buildings [24,25], polyline and polygon features [26,27,28], normal vectors of polygon features [29,30] and urban road networks [31], can also be applied. An important feature of this registration approach is that the acquisition view of the point clouds is different for the two technologies. For example, a terrestrial laser scanner acquires point clouds data on a terrain, and an unmanned aerial vehicle (UAV) acquires images from a low-altitude space. Good complementarity exists between the point clouds collected through these two methods, therefore, this registration method is commonly used to integrate point clouds, especially from complex buildings or other such infrastructure.

The second type of method is based on the features extracted from a point clouds and images. Before extracting and applying features for registration, the accurate relationship amongst sensors, such as digital cameras, GPS and IMU, should be calibrated and computed. Another pre-processing procedure for digital cameras is the calibration of the different lenses of cameras. Shift and rotation relationship data are important in the registration of different sensors. The authors of [32] calibrated the fixed mathematical relationship between laser scanners and digital cameras and synchronously acquired point clouds and image data to complete the registration. POS data is usually used to integrate a 3D point clouds and a 360° linear array panoramic camera [33]. The authors of [34] reviewed relevant feature-based methods for direct registration of point clouds and digital images. Current studies indicate that, compared with natural targets, man-made features are used more frequently [35]. Several easy-to-find, man-made features, such as linear edges [36,37], connected line segments [38], and planar features [39], are often used for registration. Several other invisible features, such as vanishing point [40] and mutual information [41,42], are also used. Amongst these methods, visible and linear feature extraction algorithms, including scale-invariant feature transform, are consistently employed [43,44]. UAVs have also been utilized to capture aerial images, and the registration of aerial-based point clouds and images has been studied [45].

1.3. Present Work

Several registration and data integration methods have been developed for the fusion of point clouds and digital images. However, from our perspective, these methods cannot be directly used to combine point clouds and panoramic images. For example, given that the overlap rate amongst the adjacent lenses of a panoramic camera is small, the dense registration method for point clouds generated by the photogrammetry method can only achieve a few point clouds. In addition, feature-based methods are usually developed to register point clouds and traditional digital images, and the application of these methods to different situations, such as panoramic images, requires further verification. Furthermore, as more than four lenses exist in a panoramic camera, the selection of corresponding imagery for a certain point is vital for registration. Therefore, a new method to register point clouds and panoramic images is required.

To address this issue, a new registration approach based on sensor constellation was developed in this study. This approach makes full use of the GPS and the panoramic camera’s position and orientation relationship. A segmentation feature point was computed based on the real-time sensor constellation. Using such segmentation feature points, point clouds acquired through different laser scanners can be divided into small blocks. Finally, points of each block were introduced to the geometric conversion model to compute the corresponding pixels in panoramic images. After that, the color or texture information of the panoramic images was extracted and fused with the 3D point clouds. Compared with other methods, considering the position of sensors was fixed after installation, the proposed method is simple, and suitable for large volume calculation.

2. Sensor Constellation of MMS

The sensor constellation of an MMS differs according to its application purpose. In this study, an MMS was designed to collect data from urban roads and extract the visible features, including symbols and markers for transportation and terrain objects around the roads. This MMS contained a panoramic camera, sectional scanners, GPS, IMU and other necessary sensors.

Three laser scanners were installed in different locations. Figure 1 shows the relationship between the panoramic camera, sectional laser scanners and GPS antenna. The first sectional laser scanner, see LS-1 in Figure 1, was installed in the rear of the vehicle to collect data from the top view. The other two laser scanners were installed on either sides of the roof of the vehicle to collect data on both sides of the road. LS-2 collected data from the left-front direction, and LS-3 acquired data from the right-front direction. Both LS-2 and LS-3 collected point clouds in a vertical plane. As the orientation and position relationship was accurately calibrated, the point clouds captured by these three laser scanners was treated with the same coordinate system for subsequent registration.

The panoramic camera used in this study was a LadyBug^® (FLIR Integrated Imaging Solutions, Inc., Richmond, BC, Canada), and was installed in the center of the vehicle roof (see Figure 1a and the green arrow in Figure 1c). The camera contained six high-definition lenses and captured images from six directions simultaneously. The sampling rate was 1.0 Hz. NMEA (National Marine Electronics Association) GPS signals were input into the panoramic camera in real-time to record the position of the panoramic camera.

The sectional laser scanners were SICK^® LMS 511 Pro. (SICK Vertriebs-GmbH, Düsseldorf, NRW, Germany). The maximum scanning distance of this kind of laser scanner was 80 m, and the field of view reached 190°. The scanning frequency was 100 Hz, and the angle resolution was 0.166°. The main original data collected by the laser scanner were time, distance, intensity, and scan angle.

According to the data-processing model, an integration system was developed to collect sensor data and compute point clouds data from the original IMU, GPS and laser scanner so that a 3D point clouds of surrounding objects could be obtained. The MMS comprised of three sectional laser scanners where two were installed in the front part of the vehicle, and the other one was installed in the rear. The point clouds from different scanners were separated from the original point clouds. Figure 2 shows the spatial distribution of the point clouds captured by the different laser scanners. The green points were captured by LS-1 and are located in the road and go along with the movement of the vehicle. The other points in yellow or brown were located on the roadside and were captured by LS-2 and LS-3. The scan planes of LS-2 and LS-3 were perpendicular to the horizontal plane.

3. Registration of Mobile Point Clouds and Panoramic Images Based on Sensor Constellations

In this section, the registration procedures of the mobile point clouds and the panoramic images are introduced based on the sensor constellation of the MMS. Section 3.1 outlines the flowchart of the proposed method and Section 3.2, Section 3.3 and Section 3.4 introduce the detailed processing steps.

3.1. Flowchart of Proposed Method

Three main steps were included in the sensor-constellation based method (Figure 3). First, in order to separate the whole road’s point clouds into small blocks, the sensor constellation (mainly the panoramic camera and the GPS) were analyzed and a segmentation feature point extraction model was proposed. As both the panoramic camera and GPS contained a positioning module, the position of the panoramic camera and the GPS could always be obtained. This ensured that the feature point could always be acquired while MMS was travelling. This step will be explained in Section 3.2 in detail.

After the segmentation feature point was obtained, a polygon area was extracted and defined as the central block for LS-1 (discussed in Section 3.3.1). The corner points of the central block were also computed, and the side blocks were fixed (discussed in Section 3.3.2). The central and side blocks were used to segment the whole point clouds into small blocks.

Finally, each point in the small blocks was selected to find the corresponding pixel in the panoramic image using the sensors’ relationship matrixes introduced in Section 3.4. An image search strategy is also provided in this subsection.

3.2. Segmentation Feature Point Extraction Based on Sensor Constellation

In this subsection, a feature point was extracted according to the sensor constellation between the GPS and panoramic camera. As the position of the GPS and panoramic camera can always be obtained, the relationship between the two sensors, as well as the vehicle and the ground is easy to rebuild. A diagram of the principle of segmentation feature point extraction is shown in Figure 4.

$G\left(x_{gps}^T,y_{gps}^T,h_{gps}^T\right)$ is assumed to be the coordinate of GPS antenna at a certain time (T), and $P\left(x_{cam}^T,y_{cam}^T,h_{cam}^T\right)$ is the coordinate of a panoramic camera at the same time. These positions can be obtained according to GPS observation and the relationship between the GPS and the panoramic camera.

Let $\beta$ be the road inclination angle at a certain position, which can be obtained by GPS according to the adjacent epochs or the point clouds captured nearby. In this study, previous epochs of GPS were utilized to calculate the inclination angle.

$h_{cam}$ is the distance from the centre of the panoramic camera to the road (see line P-P1 in Figure 4) and will stay the same after the panoramic camera has been installed. The horizontal plane is the projection plane with a certain elevation h and is also the computing side.

i is the intersection point between line PG and the road. L is the line between point j and Q, which is the projection point of i and P to the horizontal plane. In this study, point j was regarded as the segmentation feature point.

Therefore, the vertical angle, $\theta$, can be obtained with

$$\theta +\beta =arcsin\left(\frac{z_{cam}^T-z_{gps}^T}{\sqrt{{\left(x_{cam}^T-x_{gps}^T\right)}^2+{\left(y_{cam}^T-y_{gps}^T\right)}^2+{\left(z_{cam}^T-z_{gps}^T\right)}^2}}\right).$$

(1)

The distance of line i-P2, $d_{i-P2}$, can be obtained with

$$d_{i-P2}=d_{i-P1}-d_{P1-P2}=\frac{h_{cam}}{\tan \theta }-h_{cam}\times \tan \beta .$$

(2)

Given that L is the projection of line i − P, which is also the projection of line i − P2, it can be calculated with

$$L=d_{j-Q}=d_{i-P2}\times \cos \beta =\left(\frac{h_{cam}}{\tan \theta }-h_{cam}\times \tan \beta \right)\times \cos \beta .$$

(3)

Figure 5 shows the relationship of the segmentation feature point (j) and mobile mapping vehicle.

The coordinate of segmentation feature point $j\left(x_j^T,y_j^T,h_j^T\right)$ can be calculated according to azimuth angle $\mathsf{\gamma }$ and projected panoramic camera coordinate $Q\left(x_{cam}^T,y_{cam}^T,h\right)$ at a certain time (T) as follows:

$$\{\begin{array}{c}{x}_j^T=x_{cam}^T-L\times \cos \gamma =x_{cam}^T-\left(\frac{h_{cam}}{\tan \theta }-h_{cam}\times \tan \beta \right)\cos \beta \cos \gamma \\ y_j^T=y_{cam}^T-L\times \sin \gamma =y_{cam}^T-\left(\frac{h_{cam}}{\tan \theta }-h_{cam}\times \tan \beta \right)\cos \beta \sin \gamma \\ h_j^T=h\end{array}$$

(4)

3.3. Division of Original Points into Blocks

3.3.1. Division of the Point Clouds Captured by LS-1

After the location of the segmentation feature point was calculated, the road’s point clouds was divided to separate the large volume of points into small blocks.

Based on the data collection frequency of the panoramic camera, each segmentation point during the time the panoramic camera was operational was computed. A threshold, ${\mathrm{W}}_{back}$, which indicates the width of the block shape, was used to determine the width of the polygon (Figure 6), and the length of the block shape was calculated by the adjacent segmentation feature points (j^T and j^T+1). Next, the block shape was fixed (see green box in Figure 6) and the point clouds contained in the polygon was stored as a small block for further processing.

Considering that the field angle of the sectional laser scanner used in this study was more than 180°, the distance of the scan line is extremely large in theory. However, the threshold ${\mathrm{W}}_{back}$ limits the area to the most central part of the scan lines as a point’s accuracy in this area is better than that of other points. Furthermore, the points located at the side were overlapped by the point clouds from the other laser scanner. Therefore, the points that were out of the block shape were not used in the registration technique described in Section 4.

3.3.2. Division of the Point Clouds Captured by LS-2 and LS-3

The block shape for LS-1 can be accurately determined by the segmentation points and the parameter ${\mathrm{W}}_{back}$; however, the method introduced in Section 3.3.1 cannot be used for the two other laser scanners (LS-2 and LS-3) as no obvious target exist at the side of the vehicle. Therefore, a two-step method was introduced to divide the point clouds captured by LS-2 and LS-3 into small blocks.

First, the coordinates of four corner points were obtained at each time point (see A, B, C and D in Figure 7). Given that the coordinates of the feature points, $j^T\left(x_j^T,y_j^T,h\right)$ and $j^{T+1}\left(x_j^{T+1},y_j^{T+1},h\right)$, were obtained with Equation (4), the corner points could also be calculated. For example, the coordinate of point $A\left(x_A^T,y_A^T,h_A^T\right)$ at time T can be calculated as

$$\{\begin{array}{c}{x}_A^T=x_j^T+\frac{W_{back}}{2}\times \sin \left(\gamma -90°\right)\\ y_A^T=y_j^T+\frac{W_{back}}{2}\times \cos \left(\gamma -90°\right)\\ h_A^T=h\end{array}$$

(5)

where $\mathsf{\gamma }$ is the azimuth angle of line jQ (Figure 5).

Second, the shape of the block was determined as per the corner points and the scan direction of the laser scanner (LS-2 and LS-3). The scan direction of the laser scanner was determined when the sensor was installed.

Similar to the point clouds captured by LS-1, most of the point clouds captured by LS-2 or LS-3 were inserted into blocks for registration. Few points were removed as they were overlapped by the points of the central block captured by LS-1. The blocks from a different laser scanner were adjacent, albeit without overlap.

3.4. Registration of a Block’s Point Clouds and Panoramic Images

The point clouds of MMS are continuously collected and represent the locations of surrounding objects. The frequency of the laser scanner is higher than that of the panoramic camera, so the point clouds are continuous while panoramic images are discontinuous. After processing via the methods introduced in Section 3.2 and Section 3.3, the point clouds were divided into small blocks. The points in each block were used to register the images collected by the panoramic camera.

As the purpose of the registration between the point clouds and panoramic images was to fuse the texture information of the images with the geometric information of the point clouds, each point extracted from the blocks was selected and converted into the pixel coordinate of the panoramic camera via the local coordinate system of the MMS and the local coordinate system of the panoramic camera. If a proper pixel was found in the relevant lens’ imagery, the color information was stored. Otherwise, if no proper pixel was found, then the point was removed. As these steps are quite simple using photogrammetry principles and equations in textbooks, this part of the processing is omitted in this paper.

Before applying the point cloud coordinate conversion, some pre-processing, such as distortion correction for the panoramic camera, interior orientation element calibration for each lens of the panoramic camera and geometric relationship calibration between the local coordinate system of the panoramic camera and MMS, needs to be conducted. In this study, several traditional methods including [46,47,48,49] were used.

$\left(X_k^w,Y_k^w,Z_k^w\right)$ is assumed to be the coordinate of point (k) in the world coordinate system, $\left(X_k^{pan},Y_k^{pan},Z_k^{pan}\right)$ denotes the coordinate of point (k) in the panoramic camera coordinate system and $\left(X_k^{mms},Y_k^{mms},Z_k^{mms}\right)$ denotes the corresponding coordinate of point (k) in the MMS coordinate system. According to the point cloud calculation method, the relationship between $\left(X_k^w,Y_k^w,Z_k^w\right)$ and $\left(X_k^{mms},Y_k^{mms},Z_k^{mms}\right)$ is

$$\left(\begin{array}{c}{X}_k^w\\ Y_k^w\\ Z_k^w\end{array}\right)=\left(\begin{array}{c}\Delta X_{mms}^w\\ \Delta Y_{mms}^w\\ \Delta Z_{mms}^w\end{array}\right)+R_{mms}^w\times \left(\begin{array}{c}{X}_k^{mms}\\ Y_k^{mms}\\ Z_k^{mms}\end{array}\right).$$

(6)

Therefore,

$$\left(\begin{array}{c}{X}_k^{mms}\\ Y_k^{mms}\\ Z_k^{mms}\end{array}\right)={\left(R_{mms}^w\right)}^{-1}\left(\begin{array}{c}{X}_k^w-\Delta X_{mms}^w\\ Y_k^w-\Delta Y_{mms}^w\\ Z_k^w-\Delta Y_{mms}^w\end{array}\right)$$

(7)

where $R_{mms}^w$ and ${\left(\Delta \mathrm{X},\Delta \mathrm{Y},\Delta \mathrm{Z}\right)}_{mms}^w$ are the transformation parameters from the MMS coordinate system to the world coordinate system, which can be achieved during point cloud computation processing. The coordinate of point k in the MMS coordinate can be achieved using Equation (7).

Next, $\left(X_k^{mms},Y_k^{mms},Z_k^{mms}\right)$ was transferred to the coordinate system of the panoramic camera via

$$\left(\begin{array}{c}{X}_k^{pan}\\ Y_k^{pan}\\ Z_k^{pan}\end{array}\right)=\left(\begin{array}{c}\Delta X_{mms}^{pan}\\ \Delta Y_{mms}^{pan}\\ \Delta Z_{mms}^{pan}\end{array}\right)+R_{mms}^{pan}\times \left(\begin{array}{c}{X}_k^{mms}\\ Y_k^{mms}\\ Z_k^{mms}\end{array}\right)$$

(8)

where $\left(\Delta X_{mms}^{pan},\Delta Y_{mms}^{pan},\Delta Z_{mms}^{pan}\right)$ and $R_{mms}^{pan}$ were determined after the panoramic camera was installed and can be obtained via the calibration of the MMS system. Finally, in the panoramic camera coordinate system, the pixel coordinate (x, y) of an image can be computed according to the following collinear equation.

$$\{\begin{array}{c}x=-f\frac{a_1\left(X_k^{pan}-X_s\right)+b_1\left(Y_k^{pan}-Y_s\right)+c_1\left(Z_k^{pan}-Z_s\right)}{a_3\left(X_k^{pan}-X_s\right)+b_3\left(Y_k^{pan}-Y_s\right)+c_3\left(Z_k^{pan}-Z_s\right)}\\ y=-f\frac{a_2\left(X_k^{pan}-X_s\right)+b_2\left(Y_k^{pan}-Y_s\right)+c_2\left(Z_k^{pan}-Z_s\right)}{a_3\left(X_k^{pan}-X_s\right)+b_3\left(Y_k^{pan}-Y_s\right)+c_3\left(Z_k^{pan}-Z_s\right)}\end{array}$$

(9)

where ($X_s,Y_s,Z_s$) refer to the linear elements of the exterior orientation of the panoramic camera. These parameters can be obtained as per the GPS and IMU observations, as well as the installation relationship of the panoramic camera and GPS. $\left(a_1,b_1,c_1,a_2,b_2,c_2,a_3,b_3,c_3\right)$ are the elements of the direction cosine matrix, which can also be fixed at a certain time.

Considering that the panoramic camera used in this study contained six different lenses, for a normal registration process, the corresponding pixel should be searched for each image captured by each lens. However, this procedure reduces calculation efficiency so a registration strategy was adopted in this study.

Figure 8 shows the observation field of view of each lens of the panoramic camera. According to the installation relationship between the panoramic camera and laser scanner, the field of view of a lens is always associated with a certain laser scanner. Therefore, we performed search processing according to improve search efficiency (

Table 1. Image search range of each laser scanner.

Laser Scanner	Lens of Panoramic Camera
LS-1	Lens-0
LS-2	Lens-1, Lens-2, Lens-5
LS-3	Lens-3, Lens-4, Lens-5

).

Using the coordinates of the point and the parameters listed in Equations (6)–(9), the corresponding pixel can be directly found and there is an obvious difference when registering the different laser scanners. However, when applying Equation (9), the lens’ direction cosine matrix should be given as per Table 1.

4. Case Studies

4.1. Case Area

Four areas in Shanghai were selected to validate the proposed method. Given that the main purpose of the MMS was to obtain the symbols and markers as well as the relevant objects of roads, these test cases corresponded to four different types of roads: overpass, freeway, tunnel, and surface roads with intersections. The main information of these test cases and the data collection environments and parameters are listed in

Table 2. Environment and main parameters of data collection.

Case Type	Overpass	Freeway	Tunnel	Surface Roads
Environment complexity	Complex	Simple	Simple	Complex
GPS signal	Good	Good	None	Average
Length (km)	30.8	27.5	2.0	11.6
Average speed (km/h)	30	40	30	22
Time span (min)	61.50	41.27	3.34	31.75

.

4.2. Registration Results

4.2.1. Efficiency Evaluation for Different Laser Scanner’s Point Clouds

To evaluate the efficiency of the registration between different laser scanners and the different lenses of a panoramic camera, the total points, valid matched points, match rate, and computation time were summarized after registration processing. During the processing, the ${\mathrm{W}}_{back}$ was 160.0 m, which was twice the maximum distance of the laser scanner. Therefore, most of the point clouds was divided into small blocks and used for registration. The total points captured by different laser scanner, matched points as well as the match rate and computation time are shown in

Table 3. Efficiency evaluation of the different laser scanner’s point clouds in each dataset.

Type	Laser Scanner	Total Points	Matched Points	Match Rate (%)	Computation Time (s)
Overpass	LS-1	120,210,560	120,099,095	99.91	3163
	LS-2	46,040,486	45,750,431	99.37	1972
	LS-3	48,009,700	47,625,623	99.20	2060
Freeway	LS-1	72,612,193	72,601,383	99.98	1910
	LS-2	26,218,503	26,207,957	99.95	1092
	LS-3	28,495,605	28,486,135	99.96	1187
Surface roads	LS-1	61,756,177	61,695,284	99.90	1625
	LS-2	23,249,154	23,238,371	99.95	968
	LS-3	26,393,008	26,381,647	99.95	1099
Tunnel	LS-1	8,582,302	8,511,419	99.79	199
	LS-2	4,797,171	4,787,142	99.17	225
	LS-3	6,002,796	5,971,450	99.47	250

.

From the information presented in Table 3, we see that the number of points of LS-1 is higher than LS-2 and LS-3 as LS-1 was designed to collect points of the road and most of the laser beam was reflected. However, the other two laser scanners were designed to collect points around the road, and some of the laser beam was missing, as no targets exist in the air.

As seen in Table 3, most of the points were successfully matched (with an average of 99.7%) with the corresponding pixels in the panoramic images. Very few points were unmatched during the registration procedures. After summarizing the total points and computation time for each laser scanner, the average computation efficiency was 38,155 points/s, 23,870 points/s and 24,006 points/s for LS-1, LS-2 and LS-3, respectively (

Table 4. Average computation efficiency evaluation.

Laser Scanner	Total Points	Matched Point	Total Computation Time (s)	Average Computation Efficiency
LS-1	263,161,232	262,907,161	6897	38,155
LS-2	101,305,314	99,983,901	4257	23,870
LS-3	108,901,109	108,464,855	4596	24,006

). The efficiency of LS-1 was 1.59 times higher than other two laser scanners, as the image search range was different (shown in Section 3.4).

4.2.2. Visualization of Registration Results

To evaluate the registration results for different objects around the road completed by our proposed method, Figure 9 shows the visualization of four different types of roads. As the light conditions within the tunnel environment was poorer than the other roads, the brightness of the tunnel point clouds (see Figure 9d) after matching was lower than the other three examples. The main objects and the symbols around the roads—for example, solid/dash line, lights, central isolation belt, acoustic panels, crosswalk, left turn lane, trees, and some vehicles—could be accurately identified using the fused point clouds. This also enhanced the possibility of automatically extracting the objects and symbols around the roads.

4.3. Accuracy Evaluation

4.3.1. Evaluation Method

The distance of checkpoints before and after registration was used to evaluate registration accuracy. Given that the main purpose of this study was to fuse the texture information of an image with the geometric information of the point clouds, the feature points were manually selected based on the fused point clouds. For example, as the arrow can be manually found in both the original (rendered by intensity) and fused point clouds (rendered by color), the arrow in the road was then selected, and the top point of the arrow was used for comparison (Figure 10).

Supposing that the coordinate of the arrow point in the fused point clouds is $\left({\mathrm{X}}^f,{\mathrm{Y}}^f,{\mathrm{Z}}^f\right)$ and in the original point clouds is $\left({\mathrm{X}}^o,{\mathrm{Y}}^o,{\mathrm{Z}}^o\right)$, three indices can be obtained to evaluate the geometric accuracy of registration. $d_H$ indicates the horizontal offset for the arrowhead after registration.

$$d_H=\sqrt{{\left(X^f-X^o\right)}^2+{\left(Y^f-Y^o\right)}^2}$$

(10)

$$d_V=\left|Z^f-Z^o\right|$$

(11)

$$d=\sqrt{d_H{}^2+d_V{}^2}.$$

(12)

4.3.2. Evaluation Result

Twenty checkpoint pairs for each different road type were manually selected, and the relative indicators (described in Section 4.3.1) were computed to evaluate accuracy. The coordinates and the differences of checkpoints of each case are listed in Appendix A. The statistical results are shown in

Table 5. Average geometric accuracy evaluation results (m).

Index	Case Area	${\mathit{d}}_{\mathit{H}}$			${\mathit{d}}_{\mathit{V}}$			$\mathit{d}$
		Min.	Max.	Avg.	Min.	Max.	Avg.	Min.	Max.	Avg.
1	Overpass	0.000	0.340	0.178	0.000	0.030	0.009	0.000	0.352	0.179
2	Intersection	0.031	0.393	0.139	0.000	0.020	0.011	0.033	0.394	0.140
3	Tunnel	0.030	0.228	0.135	0.000	0.020	0.011	0.030	0.228	0.135
4	Freeway	0.000	0.160	0.079	0.000	0.010	0.020	0.000	0.161	0.112

. $d_H$ indicates the horizontal offset, while $d_V$ indicates the horizontal offset. d is the total offset between before and after registration.

Table 5 indicates that plane geometric accuracy was approximately 0.10–0.20 m. Accuracy varied across the different environments, which may have been caused by several factors and will be analyzed in Section 4.4.

Although the tunnel elevation varied from −4.4 m to −40.6 m, the vertical accuracy of the tunnel was approximately 0.00–0.03 m, which is considered as relatively stable. The vertical accuracy of the overpass, intersection and freeway was also stable. Hence, the proposed method can be used to achieve stable and good accuracy in the vertical direction.

4.4. Discussion of the Main Factors Influence Registration Accuracy

Section 4.3.2 shows the registration accuracy evaluation results for each case area. In this subsection, the main influence factors are analyzed and discussed, including time synchronization error, GPS signal and vehicle speed.

4.4.1. Time Synchronization for Different Sensors

The time system of an MMS is an important parameter as it determines the main accuracy of the MMS. Therefore, a vital step in data processing is time synchronization amongst the different sensors.

Table 6. Time system used in the MMS.

Sensor	Time System
GPS	GPS time
IMU	GPS time
Panoramic camera	GPS time
Laser scanner	Windows time

shows the time system used in the proposed MMS. The GPS, IMU and panoramic camera adopted GPS time, whereas the laser scanner adopted the time of the operating system (Windows time). GPS time is generally accurate, and Windows time is characterized by relatively low accuracy. Therefore, the error between the GPS time and Windows time will affect the registration results. During data collection, the synchronization error accumulates.

The time synchronization error will affect accuracy in the driving direction according to the following equation.

$$ds=v\times dt$$

(13)

where $ds$ denotes the affect distance and$dt$ denotes the time synchronization error. $v$ is the vehicle speed. As per the preceding equation, registration accuracy is affected by the time synchronisation error. When the time synchronisation error was 1 ms (0.001 s) and the vehicle speed was 40 km/h, accuracy decreased by 1.1 cm. Therefore, prior to data collection, the system error between GPS time and Windows time needs to be calibrated; and after data collection, the same operation should also be conducted to improve the time accuracy of the laser scanner data.

4.4.2. Vehicle Speed

As shown in Equation (13), the affect distance is associated with the time synchronization error as well as the speed of the vehicle. Vehicle speed was an important parameter when the MMS collected the data as it directly determined the efficiency of the MMS. The collection parameter of the other sensors was fixed during data collection, and only the speed of the vehicle was not determined. Therefore, the resolution of the point clouds will be affected by the vehicle’s speed and when compared with the dense point clouds, registration with sparse point clouds will obviously lead to lower accuracy. Therefore, for the data acquisition system concerned, the vehicle speed should be not as fast as possible.

Given that the vehicle speed was 30 km/h and the frequency of the laser scanner was 100 Hz, the resolution of the point clouds along the driving direction was 0.083 m. Once the speed increased, the resolution decreased rapidly and led to less registration accuracy.

4.4.3. Positioning Error

GPS was one of the main positioning sensors installed in the MMS. This sensor determined the location of the vehicle and provided it to the IMU to obtain accurate orientation parameters of the vehicle. Thus, a lack of GPS signal will affect MMS positioning and registration accuracy. Although vehicle position can be obtained by other sensors, including the angular sensor for wheels and IMU data, when the GPS signal is unlocked, the positioning error accumulates. In our studies, the GPS signal was unlocked only in the tunnel case. The spatial distribution of the checkpoints’ accuracy associated with mileage is shown in the Figure 11. The mileage can be regarded as the time after the vehicle entered the tunnel.

As seen in Figure 11, checkpoint accuracy decreased with mileage. Therefore, if a tunnel is too long, registration accuracy cannot be guaranteed. The vertical accuracy as relatively stable and varied from 0.00 to 0.02 m as the road area around the symbols was flat. Even when the horizontal difference is large, given that the road is flat, the vertical difference remains stable.

5. Discussion and Conclusions

An invisible feature point, which was calculated based on the sensor constellation, was hired for registration between the road’s point clouds with the panoramic imagery. The feature point, which is the intersection of the connecting line between the GPS antenna and the panoramic camera with a horizontal plane, was utilized to separate the large volume point clouds into blocks. This invisible feature point was fixed after the sensor constellation was determined and was non-relevance with the environment. Therefore, it can always be calculated during data processing. This ensures the 1:1 matching and thus increases the possibility of successful registration.

Four typical road types—overpass, freeway, tunnel, and surface roads—were selected to verify the proposed method. Our results show that most of the point clouds (with an average of 99.7%) successfully registered with the panoramic images with a high efficiency. Geometric evaluation results indicate that horizontal accuracy was approximately 0.10–0.20 m, and vertical accuracy was approximately 0.01–0.02 m for all cases.

The novelty of the proposed method is that an invisible feature point was calculated according to the sensor constellation, mainly the GPS and panoramic camera. As the constellation of the GPS and panoramic camera was fixed after the sensors were installed, and both sensors contained positioning modules, the feature point can always be found during travel. This improved the stability and reduced the complexity of registration computing.

In this paper, the segmentation feature point was computed based on the real-time position of the GPS and panoramic camera. However, if the sensor constellation is different to the MMS proposed in this paper, users may alter a different feature point to segment the point clouds. The selection of the segmentation feature should satisfy two conditions: first, the feature point should be fixed and easy to find after the sensors are installed; and second, the calculation of the feature point should be simple and with no iteration required. These will ensure the accuracy and efficiency of registration.

Although the calculation method in this study is relatively simple and uses only one uncertain parameter (${\mathrm{W}}_{back}$), this parameter is important for registration as it determines the width of the central blocks. An unmatched part will exist if too small a value is selected, and a mixture part will be available if a large value is used during the registration. We recommend a selection of ${\mathrm{W}}_{back}$ according to the urban environment. For example, if the MMS operates in a freeway, then the ${\mathrm{W}}_{back}$ parameter can be determined by the width of the lanes, including the emergency lane. This condition means that the first laser scanner, LS-1, will always be used to capture the point clouds in the pavement of the road.

Another important parameter in the proposed method is the road inclination angle $\mathsf{\beta }$, which is computed during the registration for each moment when the panoramic camera operates. The road’s point clouds around the area are used to fit the inclination angle at that time and location. Furthermore, the adjacent epochs of GPS can be used to calculate the inclination angle. Therefore, the proposed method cannot be directly used in a real-time system; however, once the slopes of the roads are available for a city, the proposed method can be used to register the point clouds with panoramic images in real-time.