Next Article in Journal
Restoration of Reflection Spectra in a Serial FBG Sensor Array of a WDM/TDM Measurement System
Previous Article in Journal
Tree Height Growth Measurement with Single-Scan Airborne, Static Terrestrial and Mobile Laser Scanning
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Benchmarking the Performance of Mobile Laser Scanning Systems Using a Permanent Test Field

1
Department of Remote Sensing and Photogrammetry, Finnish Geodetic Institute, P.O. Box 15, FI-02431 Masala, Finland
2
Helsinki Metropolia University of Applied Sciences, FI-00079 Metropolia, Finland
3
School of Science and Technology, Aalto University, FI-00076 Aalto, Finland
*
Author to whom correspondence should be addressed.
Sensors 2012, 12(9), 12814-12835; https://doi.org/10.3390/s120912814
Submission received: 25 June 2012 / Revised: 14 August 2012 / Accepted: 10 September 2012 / Published: 19 September 2012
(This article belongs to the Section Remote Sensors)

Abstract

: The performance of various mobile laser scanning systems was tested on an established urban test field. The test was connected to the European Spatial Data Research (EuroSDR) project “Mobile Mapping—Road Environment Mapping Using Mobile Laser Scanning”. Several commercial and research systems collected laser point cloud data on the same test field. The system comparisons focused on planimetric and elevation errors using a filtered digital elevation model, poles, and building corners as the reference objects. The results revealed the high quality of the point clouds generated by all of the tested systems under good GNSS conditions. With all professional systems properly calibrated, the elevation accuracy was better than 3.5 cm up to a range of 35 m. The best system achieved a planimetric accuracy of 2.5 cm over a range of 45 m. The planimetric errors increased as a function of range, but moderately so if the system was properly calibrated. The main focus on mobile laser scanning development in the near future should be on the improvement of the trajectory solution, especially under non-ideal conditions, using both improvements in hardware and software. Test fields are relatively easy to implement in built environments and they are feasible for verifying and comparing the performance of different systems and also for improving system calibration to achieve optimum quality.

Graphical Abstract

1. Introduction

Mobile mapping is currently an emerging technology, which is used to enhance the attractiveness of mobile phone ecosystems by collecting large data sets covering the biggest cities. Automatic techniques to process the data into 3D models is a topic of increasing importance since accurate and intelligent up-to-date 3D roadside information will be needed in the future, especially for vehicle and pedestrian navigation and location-based services. Most mobile mapping systems are based on images taken from mobile systems, but the use of mobile laser scanning (MLS) together with images is also increasing. The advantage of laser scanning is in its high level of automation in creating the geometry for virtual models using point cloud data.

MLS is a multi-sensor system that integrates various navigation, laser scanning, and other data acquisition sensors on a rigid, moving platform (typically a van or a car) for acquiring road-side data. The navigation sensors typically include Global Navigation Satellite System (GNSS) receivers and an Inertial Measurement Unit (IMU), while the data acquisition sensors typically include terrestrial laser scanners and imaging systems.

Recent studies on MLS systems and their accuracy can be found in Barber et al. [1], Brenner [2], Clarke [3], El-Sheimy [4], Früh and Zakhor [5], Graham [6], Haala et al. [7], Hassan and El-Sheimy [8], Jaakkola et al. [9], Kukko [10], Kukko et al. [1113], Kukko and Hyyppä [14], Lehtomäki et al. [15], Manandhar and Shibasaki [16], Petrie [17], Shen et al. [18], Steinhauser et al. [19], Tao and Li [20], Weiss and Dietmayer [21], Yu et al. [22] and Zhao and Shibasaki [2325]. Mobile laser scanning systems are being developed both in the field of robotics and surveying. Relatively complete lists of MLS systems have been presented by Petrie [17] and Narayana [26].

The experiences gained in earlier remote sensing research have shown that permanent test fields with accurate ground truth are valuable tools for analyzing the performance of remote sensing systems and methods. To be able to compare various systems or methods, the test data should be from a common test field; this has also been demonstrated in an earlier EuroSDR project “Building Extraction” [27] and in the EuroSDR/ISPRS project “Tree Extraction” [28]. The task of comparing mobile laser scanning systems is challenging as the accuracy of the georeferenced point cloud is highly dependent on the GNSS visibility during data acquisition and the satellite geometry is constantly changing. Also, the seasons of the year can affect satellite visibility if there are tall deciduous trees close to the trajectory.

Prior to the present study, there have been only a few studies that have focused on MLS comprehensively in combination with test fields. System manufacturers have carried out and published their own tests, but only a few publications exist where system performance has been examined using an established test field and where the results have been analyzed by an independent actor.

Barber et al. [1] used RTK-GPS measurements to collect reference data on two test sites to validate the geometric accuracy of the Streetmapper MLS system. The main focus then was on elevation accuracy, while only a few control points, measured on white line markings on the road, were used for the analysis of planimetric accuracy.

Researchers of the University of California at Davis, United States, used total station and static TLS data to analyze the accuracy of MLS systems (Streetmapper 360, Optech Lynx and Ambercore Titan) when producing digital terrain models of pavement surfaces [29]. Then only elevation accuracy was the subject of concern.

Haala et al. [7] demonstrated that the StreetMapper system could produce dense 3D measurements with an accuracy level of 30 mm in good GNSS conditions. Furthermore, the remaining differences between the point clouds from different scanners, due to the imperfect boresight calibration of the upward looking scanner, could be corrected during post processing. Under degraded GNSS conditions, they reported a georeferencing error of up to 1 m for the horizontal position. They also reported that despite the limited absolute accuracy, 3D point measurements during bad GNSS conditions are still useful, especially if the purpose is mainly to exploit their relative positions. As an example, they presented that the standard deviation of such data is only 5 cm if the points from two scanners are combined and 2.6 cm if the points are separated for each scanner. Thus, such data are feasible for the extraction of the features of windows or passages if a certain error as to their absolute position is acceptable. Since the best laser systems in MLS are capable of estimating the range with an accuracy of 2 mm, and as direct georeferencing dominates in error propagation, an improvement is needed in the of georeferencing solution. The options in improving the georeferencing solution include more accurate calibration of the relative orientation of the MLS system components, automatic/manual detection of those objects (the position of which is known) from the road sides that may be used to improve georeferencing, and development of new data fusion approaches for MLS. The most in-depth analysis of MLS quality thus far is that presented by Haala et al. [7].

This paper concentrates on evaluating the geometrical properties of laser point clouds collected by various commercial and research-based MLS systems in good GNSS conditions on an established urban test field.

2. Benchmarking Mobile Laser Scanning Systems

2.1. Materials

2.1.1. Test Field

The test field was implemented in Espoonlahti, about 16 km west of Helsinki. The test field covers one block around the Lippulaiva shopping mall covering 1,700 m of road environment (Figure 1). The test field was divided into four sections separated by intersections as shown in Figure 1. GNSS visibility from Sections A, B and D is good, although some trees and higher buildings may restrict the visibility of lower satellites. As can be seen from the digital surface model in Figure 2, Section C has large trees standing close to the road, thus making the GNSS conditions far more challenging. There are many types of buildings and other constructions, such as stairs and walls, in the area, as well as hundreds of pole type objects, such as lamp posts, traffic signs and trees. The road area, as well as most of the terrain close to the road, is very flat in Section A. The other sections are more variable in regard to terrain elevation, both on the road area and in the surroundings. The height difference between the lowest and highest points along the road is 12 m (Figure 2).

2.1.2. Reference Data for Accuracy Evaluation

Dense terrestrial laser scanner point clouds were used to obtain the reference targets for the analysis of geometric accuracy. The reference point clouds were collected on 7 May 2009 using FGI's mobile mapping system called the ROAMER [11], Road Environment Mapper in static mode. The ROAMER was installed on the roof of a car, and the car was kept standing static on the road during each 360° scanning performed using of the FARO Photon 80 terrestrial scanner (Figure 3). The scan resolution was set to 0.0013 rad point separation. The georeferencing of individual scannings was computed during post-processing: the scanner position and heading were obtained from the ROAMER's SPAN navigation system and the scannings were leveled using the scanner's built-in inclinometer. The virtual GPS reference station data used in GPS post-processing were downloaded from the GPSNet.fi service. The SPAN data were processed using the Waypoint Inertial Explorer software, which gave the estimated accuracies of 11 mm in 3D-position and 0.027° in heading (RMS) for the ROAMER's inertial measurement unit (IMU) during the measurements. The offset between the IMU and the scanner origin, as well as the offset between the SPAN and the scanner heading, were determined during system calibration.

The ROAMER's TLS/static data were validated against 150 check points measured using a total station (Trimble 5602S DR200+). A total of nine ground control points (GCPs) were measured for the total station setup around the Espoonlahti test field using repeated real-time GPS measurements (Leica SR530). Eight individual measurements were taken at each point using different reference data sources (RTK-GPS using its own reference station and VRS-GPS using a virtual reference station) and different satellite constellations (a few hours passing between the measurement sessions). For each GCP, the first two sessions were measured using RTK-GPS (with an expected accuracy of 1 cm + 1–2 ppm in horizontal plane and 1.5–2 cm + 2 ppm in height [30]), and then two sessions using VRS-GPS (with an expected accuracy of 2 cm in horizontal plane and 4 cm in height [31]). A new GPS initialization was acquired between each session. This procedure was carried out for a second time after a few hours. The GCP coordinates were then computed as a mean of the obtained eight coordinates. The maximum standard deviation of the eight “original” coordinates was 23 mm in horizontal plane and 30 mm in elevation (the averages were 13 mm and 20 mm, respectively).

With most of the ROAMER scannings, the comparisons showed that the check points and point clouds matched one another within the standard deviation of the GCPs, i.e., a couple of centimeters, but with a few scans there was error in the leveling of the point cloud. In these cases there had been a passing bus visible in the scanning data, and so it is obvious that a large vehicle had caused a disturbance in the functioning of the scanner's inclinometer. These point clouds were then re-leveled by using neighboring point clouds and check points.

Following the point cloud validation, the targets for accuracy analysis were measured along a 350 m length of the test field's Section A (Figure 1) with the best GNSS visibility. The TerraScan-software by TerraSolid Ltd. was used for all point cloud operations. Firstly, the ground points were classified and a regular grid with a point spacing of 5 cm was computed to achieve an even distribution of the ground points. This grid was then thinned by selecting every 1,000th point, and these thinned points were compared to the original ground points. Every thinned point deviating more than 5 mm from the original data was deleted, and the remaining points were selected as the reference points for analysis of the elevation accuracy. The complete ground reference data for the elevation consisted of 3,283 points, and also the distance and direction to all possible driving trajectories were determined for these points.

The ground reference data were used to separate all laser points within 10 cm below and 50 cm above the ground, and these close-to-ground points were then used to measure the reference targets for the evaluation of planimetric accuracy (Figure 4). The targets included centers of poles, building corners and curb corners. Another slice of laser points, 1 m thick, was taken at approximately 5 m above the ground, and these laser points were used to measure more building corners and centers of poles. Altogether 273 planimetric reference targets were measured. The pole coordinates were measured by visually fitting a circle to the point cloud in the top view, and the centre of the circle was used as the reference coordinate.

2.1.3. Benchmarked Mobile Laser Scanning Data

Mobile laser scanning data were collected from the test field using five different systems (Tables 1 and 2). The test field was driven in both clockwise (CW) and counter-clockwise (CCW) direction at a speed of about 30–40 km/h with all of the systems.

Examples of the acquired point clouds are shown in Figures 59. The laser points are visualized by their intensity value, but with the exception that Sensei does not record intensity.

FGI's ROAMER system, developed in-house [11], has been operational since the summer of 2007. Mobile mapping data from test field using this system were acquired in June 2009. At that time, the ROAMER consisted of a FARO Photon 80 terrestrial laser scanner and a NovAtel SPAN positioning system (NovAtel DL-4 plus GPS-receiver, a NovAtel GPS-702-GG antenna and a Honeywell HG1700 AG58 inertial measurement unit (IMU) with ring laser gyros). Later on, the laser scanner was updated, and currently a FARO Photon 120 terrestrial laser scanner is used. The maximum point measurement rate of the Photon 80 scanner was 120 kHz and its range was 76 m (Photon 120: 976 kHz and 153 m, respectively). The laser profiling was carried out using a scanning frequency of 48 Hz. The ROAMER has an adjustable scanning angle, and in the Espoonlahti exercise the scanner was operated for measuring profiles by having it tilted 45° below the horizontal; see Figure 10. The ROAMER is the only system in this comparison, which utilized a laser scanner with continuous wave laser and phase-shift-based distance measurement. The beam size of the scanner was also the smallest in the test.

The direct georeferencing of the ROAMER point clouds was computed using the Waypoint Inertial Explorer™ GPS-IMU post-processing software. The GPS reference station data were acquired from the Finnish virtual reference station (VRS) network GPSNet.fi. After georeferencing, the dark points were deleted by filtering out points with intensity value of less than 8,000 (range 0–20,470), and isolated points were deleted by filtering out points that had less than 50 points within a 2 m radius around them. TerraScan by Terrasolid Ltd was used for the filtering.

Two point clouds were analyzed for the ROAMER. In the first one, the georeferencing of the point cloud was computed using the calibration values between the instruments determined only in laboratory calibration. The laboratory calibration was based on measuring the physical offsets and rotations between the scanner, IMU and the GPS antenna. This laboratory calibration was fine-tuned using the measured data, e.g., by utilizing the data acquired by driving the same location in two directions and using some control targets. These fine-tuned calibration values were applied in recomputing the data, and in producing the second set of point clouds for analysis.

The RIEGL VMX-250 (Figure 11) was introduced at the beginning of 2010 and the test field data were acquired in March 2010. The system consists of two RIEGL VQ-250 scanners and a navigation unit with IMU, GNSS and odometer instruments. Each of the scanners measures up to 300,000 points and 100 profiles per second. The maximum measurement range is 500 m. In September 2011, RIEGL announced another mobile mapping system, the VMX-450, with VQ-450 scanners, and the capability to measure up to 550,000 points and 200 profiles per second.

RIEGL delivered two point clouds for analysis. The first batch of data was received in June 2010. Later on, RIEGL announced that they have developed their system calibration further and wish to implement their latest expertise also in the test field data. A small set of control points from the test field was delivered to RIEGL to assist in the calibration procedure. The second batch of data was received in May 2011.

The FGI Sensei is a low-cost modular measurement system consisting of a number of measurement instruments. These include a GPS/IMU positioning system, two laser scanners, a CCD camera, a spectrometer and a thermal camera.

The above GPS/IMU system is a NovAtel SPAN-CPT integrated GPS/INS receiver system, embodying NovAtel's OEMV GNSS precision receiver technology with three fiber optic gyros and three MEMS accelerometers in a single unit. The SPAN-CPT delivers 3D position, velocity and attitude solutions. The measurements of the different INS subsystems are combined using Waypoint Inertial Explorer and GPS virtual reference station (VRS) data.

An Ibeo Lux laser scanner was used on the Espoonlahti test field. The Ibeo Lux simultaneously measures points from four different layers and it is theoretically capable of measuring up to 38,000 points/second if only one return per pulse per layer is assumed. The scanner is able to record up to three returns per pulse per layer, thus enabling it to get hits from building walls or the ground even when these are covered by nearby trees or other vegetation. Its distance measurement range is 0.3 m to 200 m (50 m for targets with 10% remission), its ranging accuracy is 10 cm, its angular resolution is 0.25° and the divergence of its laser beam is 1.4 mrad horizontally and 14 mrad vertically with respect to the scanner body, meaning that objects may appear elongated in the vertical direction. When mounted on the roof of a car (Figure 12), this vertical elongation turns into a horizontal error as the scanner is mounted vertically. For this reason, most of the reference targets for analysis of planimetric accuracy could not be reliably measured, and the performance analysis was completed only for elevation accuracy. When the Sensei is used on the roof of a car, the instruments point towards the side of the car, and the laser scanner scans vertically, and thus covers only one side of the trajectory at a time [39].

The Streetmapper 360 (Figure 13) was launched in October 2008 and test site data were acquired in June 2011. The system consists of two RIEGL VQ-250 scanners and a navigation unit with a fiber-optic gyro-based IMU, GPS and Direct Inertial Aiding (DIA) to assist in areas of poor GPS reception. Each scanner measures up to 300,000 points and 100 profiles per second. The maximum measurement range is 500 m.

The Optech Lynx Mobile Mapper was introduced in September 2007. The test field data were collected in June 2011 by TerraTec AS (Figure 14) using a system with two Optech scanners, which produce 200,000 points per second each with ranging up to 200 m. The navigation system is an Applanix POS LV 420 system with IMU, GPS and odometer instruments. The newest version Lynx Mobile Mapper M1 can produce up to 500,000 points per second per scanner.

2.2. Methods for Accuracy Evaluation

Firstly, the received point clouds were checked by comparing them with the reference data (Section 2.1.2) to detect any gross errors either in elevation or plane. If there was a larger systematic shift than one of a few centimeters, this was compensated to ensure validity in the comparison; especially a large systematic shift in plane can lead to distorted elevation accuracy results and it is a common practice to use some ground control points in laser scanning surveys to eliminate the bias.

Comparison between the elevation reference points and the received MLS point clouds was carried out using the Output control report tool in TerraScan-software [40]. It reads in the reference points and loads every laser point within a given search radius from the individual reference points. Then a small triangulated surface model is created from the laser points and laser elevation is computed for each of the reference points' easting-northing locations from the triangulated model surface. This effectively interpolates the laser elevation from the three laser points closest to the reference point to be compared. The search radius used for the densest point clouds (RIEGL, Streetmapper and Optech Lynx) was 20 cm, for the others it was 50 cm. The maximum permitted slope in the triangulated model was set to 20°.

The planimetric accuracy was evaluated by measuring the reference targets in the received MLS point clouds and then computing the differences in easting and northing.

The most deviating values were checked against the ground truth and removed from the analysis if there was any doubt that the error was due to the target, not due to the system. These errors were mainly detected in the analysis of elevation accuracy and they were due to parked cars or changes in vegetation. Following this ‘gross error filtering’, the systematic errors were removed, in plane separately for easting and northing, and the accuracy values were computed. The minimum, maximum and standard deviation values were computed for both elevation and planimetric accuracy. The mean and root mean squared errors (abbreviated to RMSE, Equation (1)) were determined for the description of the planimetric accuracy:

RMSE = i = 1 n ( d i ) 2 n
where di is the distance between the reference target and the point cloud target, and n is the number of samples.

3. Results

3.1. Elevation Accuracy

The results of the analysis of elevation accuracy are shown in Table 3 and Figure 15. Two results are given for the ROAMER and the RIEGL, Column I before fine-tuning and Column II after fine-tuning (see Section 2.1.3 for details).

The RIEGL data were acquired while there still was a lot of snow on the ground, and so the number of reference points that could be used in the RIEGL analysis was lower than when using the other high-density systems. The Sensei covered only one side of the trajectory, which explains the lower number of used reference points. Elevation accuracy as a function of distance from the trajectory is shown in Figure 16. Fine-tuning has a significant effect on the ROAMER system performance.

Even though Figure 16 suggests that the elevation accuracy improves in some cases when the distance from the trajectory increases, this is unlikely to be so. This phenomenon is most propably caused by the accuracy of the reference data having reached its limits and not being available for analysis of sub-centimeter accuracy. Nonetheless, this proves that the elevation accuracy of the best MLS systems can reach values of 1–2 cm up to a range of 35 m.

3.2. Planimetric Accuracy

The results of analysis of planimetric accuracy are shown in Table 4 and Figure 17. Two results are given for the ROAMER and the RIEGL, Column I before fine-tuning and Column II after fine-tuning (see Section 2.1.3 for details).

The planimetric accuracy was not analyzed for the Sensei as the reference targets could not be reliably measured from the Sensei point clouds (details in Section 2.1.3). Planimetric accuracy as a function of distance from the trajectory is shown in Figure 18. Fine-tuning has a significant effect on the ROAMER system's performance.

With properly calibrated high-end MLS systems, the planimetric accuracy in good GNSS conditions is high, and in the present study it was within the limits of the used reference data accuracy, i.e., about 2 cm. Accuracy deteriorates as the distance to the trajectory increases, but very moderately, when the system calibration is in order.

The results for the ROAMER in Figures 16 and 18 show very clearly how accuracy is affected when there are problems with calibration (ROAMER I) and how re-calibration using collected point cloud data can improve the performance (ROAMER II). Similar performance improvement can be expected when errors in trajectory, caused by satellite signal outtakes or IMU disturbances, for example, are compensated for using strip adjustments or control targets, for instance.

4. Discussion

As the results show, the tested MLS systems are capable of acquiring accurate point cloud data under good GNSS coverage conditions. Buildings, trees and other structures often cause disturbances in satellite visibility. Moreover, the performance of other navigation instruments, such as IMUs and odometers as well as post-processing algorithms, defines the achievable accuracy. Tools for trajectory accuracy improvement are being developed and new satellites are being launched, which should improve accuracy in areas where the current systems have problems.

Even though the computation of the sensor's driving path and orientation results in observation (GNSS, IMU) errors being minimized, there are still errors in laser distance measurement, in scanning mirrors, in position (GNSS) and in orientation (IMU). Consequently, there are systematic offsets and random variations both in plane and height [41]. These errors can be minimized by means of strip adjustment, which is earlier known from ALS (e.g., TerraMatch), and which requires repeated measurements of the same surfaces and objects. With MLS, the possible objects and surfaces, which can be used in the correction process, include elevation model, paintings in the pavement, vertical poles and building corners.

Imperfect boresight calibration, in addition to the navigation errors, between the scanners on a multi-scanner systems leads to multiple reproductions of objects, as in Figure 19. These kinds of errors in the relative orientation of the instruments lead to errors in the measured point clouds, which can cause problems in the continued processing of the data, such as extraction and modeling of objects. Systematic offset errors between the sensors (ΔX, ΔY, ΔZ) can be detected using observations of common objects close to them. For example, paintings on the road surface are feasible for such analysis. The Δroll error can be detected using the elevation model acquired with multiple surveys; Δpitch and Δheading errors can be detected with vertical objects such as poles and building corners. These systematic errors can be corrected appropriately by even manual processing of the data. The time-dependent variation of these data (random part) needs larger numbers of observations for corrections, which calls for the development and use of more automated techniques.

Since the results produced by all of the systems were adequate for high level of roadside mapping, the main focus in MLS quality research should be on the improvement of the trajectory solution under non-ideal (and also with ideal) conditions using both improved hardware (additional sensors) and software solutions (post-processing). As regards the hardware, the commercially available systems already include odometers. Additional sensors improving the georeferencing/localization solution during GNSS signal loss should be included. Examples of these additional sensors are cameras, whose data are automatically processed to solve the position changes together with IMU and odometer when GNSS signal is not appropriate. Alternative solutions include the use of signals, e.g., painted signals on the road surfaces, as has been demonstrated by Soininen [42] in connection with the Helsinki tram MLS survey.

The MLS data for comparison was obtained by applying the standard process of the data providers. Laboratory calibration of offsets and orientations of system components can be fine-tuned by the real-live data. In Optech and Streetmapper both of these calibrations are assumed to have been done already by the data provider. With RIEGL and ROAMER, the calibration was done first with by using only the laboratory-type calibration, which showed that there was a need for further calibration in the field. The second point cloud was provided taking the aspects mentioned in Section 2.1.3 into account, and (for example) in ROAMER a significant roll error was found with overlapping point cloud data. For future reference, the improvement of georeferencing solutions via MLS strip adjustment process needs more automation. The corrections to the offsets and orientations need to be made to the trajectory data.

The improvements in georeferencing could also include improvements due to abrupt jumps and other errors in the data. Schwarz and El-Sheimy [43] have discussed the use of three post-processing techniques, such as denoising, auto regression modeling and smoothing. Denoising is noise reduction done directly to the sensor measurements. Auto regression modeling reduces noise in the data fusion step, and forward and backward Kalman filters are example of this method type and they have already been implemented in the typical GNSS/IMU integration. Also, numerical smoothing on the post-processed trajectories can be performed.

5. Conclusions

It was within the context of the European Spatial Data Research project “Mobile Mapping—Road Environment Mapping Using Mobile Laser Scanning” that most of the leading mobile mapping manufactures, among them RIEGL, Optech and Streetmapper, together with some research systems participated in the benchmarking of the performance of mobile laser scanning systems. The verification was done using a permanent test field established in Espoonlahti (Finland). The comparison revealed that high-quality point clouds can be generated by all systems under good GNSS conditions. With all professional systems properly calibrated, the elevation accuracy was better than 3.5 cm up to a range of 35 m. The best system had a planimetric accuracy of 2.5 cm even with range of 45 m. The planimetric error increases as a function of range, but moderately so if the system is properly calibrated. Proper calibration can be achieved, for example, by using the test field concept also for mobile laser scanning. The main focus in mobile laser scanning development in the near future should be on the improvement of the trajectory solution, especially under non-ideal conditions, using both improvements in hardware and computational solutions.

Acknowledgments

All participants, i.e., Gerald Zach representing RIEGL Laser Measurement Systems GmbH, Iain Lorraine representing 3D Laser Mapping, Halvor Holvik, Morten Taraldsten Brunes, Heikki Luukkonen and Jan Biström representing TerraTec AS, are gratefully acknowledged for their cooperation in this project. The Espoonlahti test field will be maintained by the Mobile Mapping Group at the FGI ( www.fgi.fi/mobimap) for further performance and calibration studies on the MLS systems and data processes.

References

  1. Barber, D.; Mills, J.; Smith-Voysey, S. Geometric validation of a ground-based mobile laser scanning system. ISPRS J. Photogramm. Remote Sens. 2008, 63, 128–141. [Google Scholar]
  2. Brenner, C. Extraction of features from mobile laser scanning data for future driver assistance systems. Adv. GIScience 2009. [Google Scholar] [CrossRef]
  3. Clarke, K.C. Mobile mapping and geographic information systems. Cartogr. Geogr. Inform. 2004, 31, 131–136. [Google Scholar]
  4. El-Sheimy, N. An Overview of Mobile Mapping Systems. Proceedings of From Pharaohs to Geoinformatics FIG Working Week 2005 and GSDI-8, Cairo, Egypt, 16–21 April 2005.
  5. Früh, C.; Zakhor, A. An automated method for large-scale, ground-based city model acquisition. Int. J. Comp. Vis. 2004, 60, 5–24. [Google Scholar]
  6. Graham, L. Mobile mapping systems overview. Photogramm. Eng. Remote Sens. 2010, 76, 222–228. [Google Scholar]
  7. Haala, N.; Peter, M.; Kremer, J.; Hunter, G. Mobile LiDAR mapping for 3D point cloud collection in urban areas—A performance test. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2008, 37, 1119–1127. [Google Scholar]
  8. Hassan, T.; El-Sheimy, N. Common adjustment of land-based and airborne mobile mapping system data. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2008, 37, 835–842. [Google Scholar]
  9. Jaakkola, A.; Hyyppä, J.; Hyyppä, H.; Kukko, A. Retrieval algorithms for road surface modelling using laser-based mobile mapping. Sensors 2008, 8, 5238–5249. [Google Scholar]
  10. Kukko, A. Road Environment Mapper—3D Data Capturing with Mobile Mapping. Licentiate's Thesis, Helsinki University of Technology, Espoo, Finland, 2009; p. 158. [Google Scholar]
  11. Kukko, A.; Andrei, C.-O.; Salminen, V.-M.; Kaartinen, H.; Chen, Y.; Rönnholm, P.; Hyyppä, H.; Hyyppä, J.; Chen, R.; Haggrén, H.; Kosonen, I.; Čapek, K. Road Environment Mapping System of the Finnish Geodetic Institute—FGI Roamer. Proceedings of the ISPRS Workshop “Laser Scanning 2007 and SilviLaser 2007”, Espoo, Finland, 12–14 September 2007; pp. 241–247.
  12. Kukko, A.; Jaakkola, A.; Lehtomäki, M.; Kaartinen, H.; Chen, Y. Mobile Mapping System and Computing Methods for Modelling of Road Environment. Proceedings of the 2009 Joint Urban Remote Sensing Event, Shanghai, China, 20–22 May 2009.
  13. Kukko, A.; Kaartinen, H.; Hyyppä, J.; Chen, Y. Multiplatform Mobile laser scanning: Usability and performance. Sensors 2012, 12, 11712–11733. [Google Scholar]
  14. Kukko, A.; Hyyppä, J. Small-footprint laser scanning simulator for system validation, error assessment and algorithm development. Photogramm. Eng. Remote Sens. 2009, 75, 1177–1189. [Google Scholar]
  15. Lehtomäki, M.; Jaakkola, A.; Hyyppä, J.; Kukko, A.; Kaartinen, H. Detection of vertical pole-like objects in a road environment using vehicle-based laser scanning data. Remote Sens. 2010, 2, 641–664. [Google Scholar]
  16. Manandhar, D.; Shibasaki, R. Auto-extraction of urban features from vehicle-borne laser data. ISPRS J. Photogramm. Remote Sens. 2002, 34, 650–655. [Google Scholar]
  17. Petrie, G. Mobile mapping systems: An introduction to the technology. GeoInformatics 2010, 13, 32–43. [Google Scholar]
  18. Shen, Y.; Sheng, Y.; Zhang, K.; Tang, Z.; Yan, S. Feature Extraction from Vehicle-Borne Laser Scanning Data. Proceedings of International Conference on Earth Observation Data Processing and Analysis, Wuhan, China, 28–30 December 2008.
  19. Steinhauser, D.; Ruepp, O.; Burschka, D. Motion Segmentation and Scene Classification from 3D LIDAR Data. Proceedings of IEEE Intelligent Vehicles Symposium, Eindhoven, The Netherlands, 4– 6 June 2008; pp. 398–403.
  20. Tao, C.; Li, J. Advances in Mobile Mapping Technology; Taylor & Francis Group: London, UK, 2007. [Google Scholar]
  21. Weiss, T.; Dietmayer, K. Automatic Detection of Traffic Infrastructure Objects for the Rapid Generation of Detailed Digital Maps Using Laser Scanners. Proceedings of the IEEE Intelligent Vehicles Symposium 2007, Istanbul, Turkey, 13– 15 June 2007; pp. 1271–1277.
  22. Yu, S.J.; Sukumar, S.R.; Koschan, A.F.; Page, D.L.; Abidi, M.A. 3D reconstruction of road surfaces using an integrated multi-sensory approach. Opt. Lasers Eng. 2007, 45, 808–818. [Google Scholar]
  23. Zhao, H.; Shibasaki, R. Reconstructing a textured CAD model of an urban environment using vehicle-borne laser range scanners and line cameras. Mach. Vis. Appl. 2003, 14, 35–41. [Google Scholar]
  24. Zhao, H.; Shibasaki, R. A vehicle-borne urban 3-D acquisition system using single-row laser range scanners. IEEE T. Syst. Man Cybern. B. 2003, 33, 658–666. [Google Scholar]
  25. Zhao, H.; Shibasaki, R. Updating a digital geographic database using vehicle-borne laser scanners and line cameras. Photogramm. Eng. Remote Sens. 2005, 71, 415–424. [Google Scholar]
  26. Narayana, K. Solutions for the localization of Mobile Mapping Systems in Structured Environments. Ph.D. Thesis, MINES Paris Tech, Paris, France, 2011; p. 267. [Google Scholar]
  27. Kaartinen, H.; Hyyppä, J. EuroSDR Project, Commission III “Evaluation of Building Extraction”; Final Report; Official Publication no 50; EuroSDR (European Spatial Data Research): Dublin, Ireland, 2006; pp. 9–77. [Google Scholar]
  28. Kaartinen, H.; Hyyppä, J. EuroSDR/ISPRS Project, Commission II “Tree Extraction”; Final Report; Official Publication no 53; EuroSDR (European Spatial Data Research): Dublin, Ireland, 2008. [Google Scholar]
  29. Yen, K.S.; Akin, K.; Lofton, A.; Ravani, B.; Lasky, T.A. Using Mobile Laser Scanning to Produce Digital Terrain Models of Pavement Surfaces; Final Report of the Advanced Highway Maintenance and Construction Technology Research Center Research Project. University of California at Davies: Pittsburgh, PA, USA, 2010. Available online: http://ahmct.ucdavis.edu/pdf/UCD-ARR-10-11-30-01.pdf (accessed on 13 November 2010).
  30. Bilker, M.; Kaartinen, H. The Quality of Real-Time Kinematic (RTK) GPS Positioning; Reports of the Finnish Geodetic Institute; FGI: Masala, Finland, 2001; pp. 1–25. [Google Scholar]
  31. Häkli, P. Practical Test on Accuracy and Usability of Virtual Reference Station Method in Finland. Proceedings of FIG Working Week, Athens, Greece, 22–27 May 2004.
  32. FARO Photon 80 specifications; FARO Technologies Inc.: Stuttgart, Germany, 2008.
  33. FARO Photon 120 specifications; FARO Technologies Inc.: Stuttgart, Germany, 2009.
  34. Ibeo Lux specifications. Available online: http://www.ibeo-as.com/ (accessed on 10 December 2011).
  35. Lynx Mobile Mapper Spec Sheet. Available online: http://www.optech.ca/pdf/Lynx_SpecSheet_110909_web.pdf (accessed on 19 September 2012).
  36. Datasheet RIEGL VQ-250. Available online: http://riegl.com/uploads/tx_pxpriegldownloads/10_DataSheet_VQ250_13-04-2010.pdf (accessed on 13 November 2010).
  37. TerraTec. Report 16.09.2011; Mobile Mapping; Geodetic Institute: Kyrkslätt, Finland, 2007. [Google Scholar]
  38. RIEGL image gallery. Available online: http://www.riegl.com/nc/products/mobile-scanning/gallery/ (accessed on 2 February 2012).
  39. Jaakkola, A.; Hyyppä, J.; Kukko, A.; Yu, X.; Kaartinen, H.; Lehtomäki, M.; Lin, Y. A low-cost multi-sensoral mobile mapping system and its feasibility for tree measurements. ISPRS-J. Photogramm. Rem. Sens. 2010, 65, 514–522. [Google Scholar]
  40. TerraScan User's Guide 03.10.2005. Available online: http://www.terrasolid.fi (accessed on 17 September 2012).
  41. Pfeifer, N.; Oude Elberink, S.; Filin, S. Automatic Tie Elements Detection for Laser Scanner Strip Adjustment. In International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Proceedings of the ISPRS Workshop Laser scanning 2005, Enschede, The Netherlands, 12–14 September 2005; ISPRS: Vienna, Austria, 2005; Volume 36, Part 3/W19. pp. 174–179. [Google Scholar]
  42. Soininen, A. Presentation in Terrasolid European User's Event 2012 in Levi, Finland. Available online: http://www.terrasolid.fi/system/files/helsinki_tram_survey.ppt (accessed on 15 March 2012).
  43. Schwarz, K.P.; El-Sheimy, N. Mobile Mapping Systems – State of The Art and Future Trends. In International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Proceedings of the 20th International ISPRS Congress, Istanbul, Turkey, July 2004; ISPRS: Vienna, Austria, 2004; 35, Part 5. pp. 759–768. [Google Scholar]
Figure 1. The Espoonlahti test field for mobile laser scanning covers 1,700 m of road environment. The driving route is marked by the red line, and the various sections are marked by the red letters A–D. The parking spaces are marked by the letter P. The map data were provided by the courtesy of the City of Espoo.
Figure 1. The Espoonlahti test field for mobile laser scanning covers 1,700 m of road environment. The driving route is marked by the red line, and the various sections are marked by the red letters A–D. The parking spaces are marked by the letter P. The map data were provided by the courtesy of the City of Espoo.
Sensors 12 12814f1 1024
Figure 2. A digital surface model of the Espoonlahti test field (based on MLS data). The map data were provided by the courtesy of the City of Espoo.
Figure 2. A digital surface model of the Espoonlahti test field (based on MLS data). The map data were provided by the courtesy of the City of Espoo.
Sensors 12 12814f2 1024
Figure 3. An example of reference point clouds. The scanning locations can be seen as shadows on the road that by the car was driven along.
Figure 3. An example of reference point clouds. The scanning locations can be seen as shadows on the road that by the car was driven along.
Sensors 12 12814f3 1024
Figure 4. The reference targets: orange points are elevation reference points and blue (targets 5 m above ground), red (large poles) and green (small poles) points are planimetric reference points (the map data provided by the courtesy of the City of Espoo). The small figure in the top right corner shows a detail of the measured poles from close-to-ground laser points.
Figure 4. The reference targets: orange points are elevation reference points and blue (targets 5 m above ground), red (large poles) and green (small poles) points are planimetric reference points (the map data provided by the courtesy of the City of Espoo). The small figure in the top right corner shows a detail of the measured poles from close-to-ground laser points.
Sensors 12 12814f4 1024
Figure 5. Optech Lynx data.
Figure 5. Optech Lynx data.
Sensors 12 12814f5 1024
Figure 6. Streetmapper data.
Figure 6. Streetmapper data.
Sensors 12 12814f6 1024
Figure 7. RIEGL VMX-250 data. There were still patches of snow on the ground during the data acquisition, and these resulted in voids in the data.
Figure 7. RIEGL VMX-250 data. There were still patches of snow on the ground during the data acquisition, and these resulted in voids in the data.
Sensors 12 12814f7 1024
Figure 8. ROAMER data.
Figure 8. ROAMER data.
Sensors 12 12814f8 1024
Figure 9. Sensei data. Due to the scanning angle, the Sensei covered only one side of the trajectory at a time.
Figure 9. Sensei data. Due to the scanning angle, the Sensei covered only one side of the trajectory at a time.
Sensors 12 12814f9 1024
Figure 10. The ROAMER.
Figure 10. The ROAMER.
Sensors 12 12814f10 1024
Figure 11. The RIEGL VMX-250 [38].
Figure 11. The RIEGL VMX-250 [38].
Sensors 12 12814f11 1024
Figure 12. The Sensei mounted on the roof of a car.
Figure 12. The Sensei mounted on the roof of a car.
Sensors 12 12814f12 1024
Figure 13. The Streetmapper 360 mapping the roads on the Espoonlahti test field.
Figure 13. The Streetmapper 360 mapping the roads on the Espoonlahti test field.
Sensors 12 12814f13 1024
Figure 14. TerraTec AS mapping the Espoonlahti test field with an Optech Lynx Mobile Mapper.
Figure 14. TerraTec AS mapping the Espoonlahti test field with an Optech Lynx Mobile Mapper.
Sensors 12 12814f14 1024
Figure 15. The elevation accuracy values (std), in cm, for the tested MLS systems for the two driving directions. The left column stands for counter-clockwise, CCW, and the column on the right for clockwise, CW, direction.
Figure 15. The elevation accuracy values (std), in cm, for the tested MLS systems for the two driving directions. The left column stands for counter-clockwise, CCW, and the column on the right for clockwise, CW, direction.
Sensors 12 12814f15 1024
Figure 16. Elevation accuracy as a function of distance from the trajectory; with linear trend lines fitted to the observed errors in the two driving directions.
Figure 16. Elevation accuracy as a function of distance from the trajectory; with linear trend lines fitted to the observed errors in the two driving directions.
Sensors 12 12814f16 1024
Figure 17. The planimetric accuracy of the tested MLS systems in two driving directions. The left column stands for counter-clockwise, CCW, and the one on the right for clockwise, CW, direction.
Figure 17. The planimetric accuracy of the tested MLS systems in two driving directions. The left column stands for counter-clockwise, CCW, and the one on the right for clockwise, CW, direction.
Sensors 12 12814f17 1024
Figure 18. Planimetric accuracy as a function of distance from the trajectory; with the linear trend lines fitted to the observed errors in the two driving directions.
Figure 18. Planimetric accuracy as a function of distance from the trajectory; with the linear trend lines fitted to the observed errors in the two driving directions.
Sensors 12 12814f18 1024
Figure 19. A large pole (reference target no. 52) seen double by a dual-scanner system (white points).
Figure 19. A large pole (reference target no. 52) seen double by a dual-scanner system (white points).
Sensors 12 12814f19 1024
Table 1. The tested MLS-systems.
Table 1. The tested MLS-systems.
MLS systemOperated byData acquisition date
ROAMERFinnish Geodetic InstituteJune 2009
RIEGL VMX-250RIEGL Laser Measurement Systems GmbHMarch 2010
SenseiFinnish Geodetic InstituteMay 2011
Streetmapper 3603D Laser MappingJune 2011
Optech LynxTerraTec ASJune 2011
Table 2. The MLS scanner specifications [3237].
Table 2. The MLS scanner specifications [3237].
Optech Lynx Mobile Mapper/TerraTec ASSenseiROAMERRIEGL VMX-250 and Streetmapper 360
Laser scannerOptechIbeo LuxFARO Photon 80RIEGL VQ-250
Laser wavelengthN/A905 nm785 nmNear infrared
Distance measurement principleTime-of-flight, max four returnsTime-of-flight, max three returnsPhase-shiftTime-of-flight, no. of returns selectable
Points/s (×1000) max2 × 200381202 × 300
Range200 m200 m76 m500 m
Profiles/c max2 × 20050612 × 100
Beam divergenceN/A1.4 × 14 mrad0.16 mrad0.3 mrad
Beam size at exitN/AN/A3.3 mm7 mm
Distance measurement accuracy8 mm100 mm2 mm@25 m10 mm@150 m
Angular resolutionN/A0.25°0.009°0.018°
Table 3. The elevation accuracy values, in cm, for the tested MLS systems. Driving directions: counter-clockwise CCW and clockwise CW. N is the number of used reference points.
Table 3. The elevation accuracy values, in cm, for the tested MLS systems. Driving directions: counter-clockwise CCW and clockwise CW. N is the number of used reference points.
ROAMER IROAMER IISenseiRIEGL IRIEGL IIStreetmapperOptech Lynx
CCWCWCCWCWCCWCWCCWCWCCWCWCCWCWCCWCW
N2,9362,9462,8192,8161,4181,4501,5661,6001,5851,5852,7902,5492,6932,527
Min−12.9−17.1−6.8−7.5−11.1−10.1−5.7−5.3−4.6−5.7−5.0−5.0−7.1−6.4
Max10.412.66.05.014.315.06.14.16.55.25.45.18.19.1
Std3.93.92.02.03.32.91.21.31.21.51.61.62.72.6
Table 4. The planimetric accuracy values, in cm, for the tested MLS systems. Driving directions: counter-clockwise CCW and clockwise CW. N is the number of used reference targets.
Table 4. The planimetric accuracy values, in cm, for the tested MLS systems. Driving directions: counter-clockwise CCW and clockwise CW. N is the number of used reference targets.
ROAMER IROAMER IIRIEGL IRIEGL IIStreetmapperOptech Lynx
CCWCWCCWCWCCWCWCCWCWCCWCWCCWCW
N124120136120169178193192190177160160
Mean3.43.52.21.82.31.51.62.01.71.83.23.7
Min0.40.50.20.30.10.00.10.00.00.20.10.3
Max8.711.86.75.111.711.711.510.69.49.712.313.2
Std1.82.21.10.91.71.31.41.51.51.31.82.3
RMSE3.94.22.52.02.82.02.22.52.22.23.74.3

Share and Cite

MDPI and ACS Style

Kaartinen, H.; Hyyppä, J.; Kukko, A.; Jaakkola, A.; Hyyppä, H. Benchmarking the Performance of Mobile Laser Scanning Systems Using a Permanent Test Field. Sensors 2012, 12, 12814-12835. https://doi.org/10.3390/s120912814

AMA Style

Kaartinen H, Hyyppä J, Kukko A, Jaakkola A, Hyyppä H. Benchmarking the Performance of Mobile Laser Scanning Systems Using a Permanent Test Field. Sensors. 2012; 12(9):12814-12835. https://doi.org/10.3390/s120912814

Chicago/Turabian Style

Kaartinen, Harri, Juha Hyyppä, Antero Kukko, Anttoni Jaakkola, and Hannu Hyyppä. 2012. "Benchmarking the Performance of Mobile Laser Scanning Systems Using a Permanent Test Field" Sensors 12, no. 9: 12814-12835. https://doi.org/10.3390/s120912814

Article Metrics

Back to TopTop