Quality Analysis of Direct Georeferencing in Aspects of Absolute Accuracy and Precision for a UAV-Based Laser Scanning System

Abstract

The use of UAV-based laser scanning systems is increasing due to the rapid development in sensor technology, especially in applications such as topographic surveys or forestry. One advantage of these multi-sensor systems is the possibility of direct georeferencing of the derived 3D point clouds in a global reference frame without additional information from Ground Control Points (GCPs). This paper addresses the quality analysis of direct georeferencing of a UAV-based laser scanning system focusing on the absolute accuracy and precision of the system. The system investigated is based on the RIEGL miniVUX-SYS and the evaluation uses the estimated point clouds compared to a reference point cloud from Terrestrial Laser Scanning (TLS) for two different study areas. The precision is estimated by multiple repetitions of the same measurement and the use of artificial objects, such as targets and tables, resulting in a standard deviation of <1.2 cm for the horizontal and vertical directions. The absolute accuracy is determined using a point-based evaluation, which results in the RMSE being <2 cm for the horizontal direction and <4 cm for the vertical direction, compared to the TLS reference. The results are consistent for the two different study areas with similar evaluation approaches but different flight planning and processing. In addition, the influence of different Global Navigation Satellite System (GNSS) master stations is investigated and no significant difference was found between Virtual Reference Stations (VRS) and a dedicated master station. Furthermore, to control the orientation of the point cloud, a parameter-based analysis using planes in object space was performed, which showed a good agreement with the reference within the noise level of the point cloud. The calculated quality parameters are all smaller than the manufacturer’s specifications and can be transferred to other multi-sensor systems.

1. Introduction

1.1. Motivation

Nowadays, the use of mobile mapping systems has become increasingly popular due to the rapid technological development and the resulting increase in the range of applications. As a result, these multi-sensor systems are used on mobile platforms, whether on the ground, on water or in the air, and have been adopted in a wide range of application areas. Typical fields for the use of mobile mapping systems are topographic and bathymetric surveys, road and power line inspections, mining, forestry or precision agriculture [1,2,3,4,5].

The recent development of sensor technology for mobile mapping systems in the field of airborne laser scanning offers the possibility to operate compact profile line scanners on an Unmanned Aerial Vehicle (UAV) [6]. The advantages of using a combination of UAV and laser scanner are numerous compared to traditional airborne laser scanning, and benefit, in particular, from a higher level of detail and more flexible flight planning. In comparison to conventional Terrestrial Laser Scanners (TLS), it also offers a faster acquisition of survey objects with higher coverage. In addition, one of the major benefits of UAV-based laser scanning is the possibility of direct georeferencing, which describes the calculation of a 3D point cloud in a global reference frame [7]. The direct georeferencing relies on the UAV’s trajectory estimation with positions and orientations, using the fusion of sensors such as IMU (Inertial Measurement Unit) and GNSS (Global Navigation Satellite System) receiver. Since the trajectory estimation is the most critical component in the error budget of the UAV-based laser scanning, the overall accuracy of the georeferenced point cloud is highly dependent on the quality of the UAV trajectory [7]. In the following, the error budget is used as a summary of errors included in the measurements or processing of the UAV-based laser scanning system. If direct georeferencing with a sufficient accuracy is provided, different advantages are obvious. The need for overlapping flight strips in data collection is less important, which is a great advantage for applications such as corridor mapping, also resulting in larger areas being covered in the same amount of time. Furthermore, the need for Ground Control Points (GCPs) is less important and potentially not needed anymore [8]. Finally, most of the benefits lead to time and cost savings in the execution of a project. This study aims to provide a reliable strategy for the quality assessment of the direct georeferencing of a UAV-based laser scanning system. The main aspects are the analysis of the absolute accuracy in the horizontal and vertical direction and the precision of the system considering repetitions of the same measurement and, thus, the same flight planning.

This paper is organized as follows: Section 1 focuses on the methodology of UAV-based laser scanning, related work and the research scope. This is followed by Section 2 about materials and methods, which describes the used sensor system, the evaluation strategy and the study areas. Section 3 presents and discusses the results of all experiments conducted in this study. Section 4 concludes the paper.

1.2. Methodology of UAV-Based Laser Scanning

For the analysis of the system accuracy, the influencing parameters and the basic principle of UAV-based laser scanning are relevant. The concept is always based on sensors to localize the UAV with position and orientation, as well as sensors to acquire the environment with a laser scanner and an optional camera. The entire processing, starting with the raw data (blue boxes) and ending with the 3D point cloud (green box), is shown in Figure 1 with additional processing steps (orange boxes) and potential error sources, discussed later.

For the trajectory estimation with positions and orientations of the system, a combination of GNSS and IMU measurements is used. The GNSS observations from the receiver on the UAV are usually processed in differential mode with the observations from an additional GNSS master station, and then fused with the IMU data using a Kalman filter or similar algorithm [9]. The result is the UAV trajectory described by a set of estimated positions and orientations. With subsequent georeferencing of time-synchronized laser scans based on the estimated trajectory and laser scanner measurements, the georeferenced point cloud is created. Additional information needed in this processing chain are the system calibration parameters describing the relative positions and orientations between the sensors on the platform. The calculation of each 3D point ${\left[x_e,y_e,z_e\right]}^T$ can further be described with the equation

$$\begin{array}{c}\hfill \left[\begin{array}{c}{x}_e\hfill \\ y_e\hfill \\ z_e\hfill \end{array}\right]=\left[\begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\right]+{\mathbf{R}}_n^e(L,B){\mathbf{R}}_b^n(\varphi ,\theta ,\psi )·\left(\left[\begin{array}{c}\Delta x\\ \Delta y\\ \Delta z\end{array}\right]+{\mathbf{R}}_s^b(\alpha ,\beta ,\gamma )·\left[\begin{array}{c}{x}_s=0\\ y_s=d·sinb\\ z_s=d·cosb\end{array}\right]\right),\end{array}$$

(1)

including the parameters for the trajectory, sensor calibration and laser scanner measurements, which contribute to the calculation and, thus, the accuracy [10]. The calibration parameters of the system are defined with the lever arm ${\left[\Delta x,\Delta y,\Delta z\right]}^T$ and the bore-sight angles $\alpha ,\beta$ and $\gamma$ between the 2D laser scanner and GNSS/IMU unit. The measurements of the 2D laser scanner are introduced with ${\left[0,d·sinb,d·cosb\right]}^T$, with d the measured distance and b the scan angle. Further parameters are the orientation angles of the platform with roll $\varphi$, pitch $\theta$ and yaw $\psi$ and its position with translation vector ${\left[t_x,t_y,t_z\right]}^T$ and ellipsoidal longitude L and latitude B. Based on these parameters and the additional rotation matrices $\mathbf{R}$ between the sensor $\left(s\right)$, body $\left(b\right)$, navigation $\left(n\right)$ and earth $\left(e\right)$-frame, the coordinates of each 3D point can be derived. For a detailed description of the included coordinate frames, calibration parameters and the functional model we refer to [1,11].

For system evaluation, the assumed uncertainty of the point cloud, or rather of the single 3D points, is crucial, which, in turn, is a combination of several systematic and random errors that influence the estimation process [7]. As mentioned before, the major error sources influencing the accuracy of the point cloud are the trajectory estimation of the UAV, the system calibration, the accuracy of the laser scanner itself and further miscellaneous errors like the ones coming from sensor time synchronization [12]. A more detailed list of errors can be found in

Table 1. Major error sources for UAV-based laser scanning systems (refer to [7]).

	Errors
Trajectory estimation	Errors in position and orientation-sensor platform
System calibration	error in lever arm (GNSS antenna and IMU)
	error in lever arm (laser scanner and IMU)
	bore-sight angle error between IMU body and laser scanner
Laser scanner	range measurement error
	object characteristics
	atmospheric refraction
Miscellaneous errors	time synchronisation
	sensor mounting rigidity

, which is assigned to the included parameters in Equation (1).

Typically, the largest errors of mobile mapping systems as well as UAV-based laser scanning come from the GNSS/IMU-based trajectory estimation and they can be expressed in terms of platform position and orientation errors. Both position and orientation errors systematically propagate to errors in the point cloud [12]. Further errors are related to the system calibration with lever arms and the bore-sight angles already described above. These errors influence the point cloud uncertainty also in a systematic manner. In addition, the measurements of the laser scanner may contain errors due to atmospheric refraction, object characteristics, and the range measurement error itself. The error due to object characteristics is given due to different materials of the objects, the reflectivity or even the scan geometry. The remaining errors are caused by the time synchronization and the rigidity of the sensor mounting. To make a qualitative specification of the UAV-based laser scanning error budget and quantify it more concretely, the different sensors of the system used must be considered, which will be done in Section 2.

1.3. Related Work

The assessment of the accuracy of UAV-based laser scanning systems can typically be divided into theoretical and empirical studies. One theoretical approach is described by Pilarska et al. [13], who focus on the propagation of errors in the UAV’s position and orientation on the point cloud. In addition, Glennie [14] also considers the errors introduced by the calibration of the system with bore-sight angles and lever arm. Further studies about the error budget for airborne or UAV-based laser scanning systems are [7,12,15]. The critical aspect for the theoretical investigations is mainly the consideration of all errors that are included in any estimation process for mobile mapping systems with laser scanner [16]. For this reason, most of the quality assessments so far have been done on the empirical approach based on the resulting product, which is usually a georeferenced 3D point cloud with optional RGB information coming from a camera. Examples for the empirical assessment of point cloud accuracy can be found in Tulldahl et al. [17] or investigations with additional optimization strategies by Mandlburger et al. [18].

For the empirical evaluation of point clouds different approaches are recommended, which can be divided into (1) point-based, (2) area-based and (3) parameter-based strategies [11]. Point-based evaluation requires the identification of identical points with a given reference. These reference points are determined by traditional surveying methods such as total stations, GNSS, leveling or TLS. Typical objects for point-based evaluation can be natural objects (e.g., building corners, road markings or manholes) or artificial objects (e.g., targets or checkerboard patterns) [19,20,21,22,23,24,25]. These enable the assessment of the absolute accuracy and the separation of errors into horizontal and vertical components. By measuring points multiple times with the same system, the precision can also be quantified. One challenge with point-based evaluation methods and UAV-based laser scanning is due to the comparatively low point density with increasing flight height and speed, which has to be considered during flight planning.

Besides the point-based evaluation, the area-based strategies are often used for comparison with a known reference point cloud (e.g., TLS) [23,26,27,28]. Also, 3D city models have been used as a reference for evaluation [29]. For the calculation of differences, various algorithms such as the cloud-to-cloud distance or the Multiscale Model to Model Cloud Comparison (M3C2) method are used, which are integrated into common software tools. Since UAV-based laser scanning measurements are typically made with overlapping flight strips to correct the estimated UAV position and orientation [30], area-based comparisons between flight strips can indicate the precision during one flight [31].

The third possibility of the above-mentioned evaluation strategies is the parameter-based evaluation, in which derived parameters from the point cloud are used for the comparison [1,23]. Potential parameters are typically estimated from objects with a defined geometric shape. Fuad et al. [32] for example analyzed the accuracy of estimated DEM models with different flight parameters. Furthermore, several studies investigated the quality for the calculation of forest structural attributes (e.g., stem diameter, volume) in forestry [33,34,35]. In addition, Kucharcyzk et al. [36] analyzed the accuracy of point clouds in vegetated terrain with a focus in the vertical direction. Similar to the previous evaluation methods, parameter-based evaluation can also serve for the analysis of precision.

1.4. Research Scope

The proposed evaluation strategy used in this work intend to evaluate the direct georeferencing quality of the UAV-based laser scanning system by using the calculated point cloud without any additional information or optimization. Since the processing chain recommended by the manufacturer includes point cloud optimization techniques, such as strip adjustment or insertion of GCPs, it is difficult to conclude the accuracy of the direct georeferencing itself. Therefore, the strategy adopted for quality assessment is chosen to evaluate the absolute accuracy and precision of a single measurement without optimization, for example, cross-flight measurements or additional point cloud processing interventions, similar to Torresan et al. [37]. This analysis to derive absolute accuracy is performed by comparing the point cloud obtained with the UAV-based laser scanning system with a TLS reference point cloud using point-based and parameter-based evaluation strategies. The objects for comparison are mainly TLS targets adopted for use with the UAV-based laser scanning system. Furthermore, additional objects are used for the analysis of the height component and, therefore, the accuracy in vertical direction. In addition to absolute accuracy, the precision of the system is also evaluated by performing repeated flights with identical flight planning. This approach allows a proper quantification of the precision of the UAV-based laser scanning system, especially, when the GNSS conditions change, particularly the satellite geometry. Another important factor affecting direct georeferencing is the GNSS master station used. To investigate this aspect in detail, four different master stations are employed: one own master station, one Continuously Operating Reference Station (CORS) and two Virtual Reference Stations (VRS). In addition to the analysis of single strip measurements, a second data set is considered with a cross-flight pattern and extended post-processing as it is commonly done.

2. Materials and Methods

2.1. UAV-Based Laser Scanning System

The UAV-based laser scanning system used for all experiments within this work is shown in flight and on the ground in Figure 2a,b. The platform is based on the DJI Matrice 600 Pro [38] and the lightweight RIEGL miniVUX-SYS designed for the use on UAVs [39]. This system consists of the IMU/GNSS combination Trimble APX-20 UAV [40] for trajectory estimation in post-processing and the full waveform 2D laser scanner RIEGL miniVUX-2UAV for object acquisition.

The scanning mechanism is based on a rotating mirror resulting in a field of view of up to ${360}^{\circ }$ and a maximum laser pulse repetition rate of 200 kHz. Furthermore, the laser scanner’s absolute accuracy at 50 m distance is given with $1.5$ cm and the precision with $1.0$ cm [41]. To ensure a precise trajectory estimation and the calculation of the 3D point cloud, the sensor calibration between IMU, GNSS antenna and laser scanner is accurately estimated by the manufacturer RIEGL. Referring to the manufacturer’s specifications in

Table 2. Error budget of the UAV-based laser scanning system based on RIEGL miniVUX-SYS according to manufacturer’s specification [39,40,41].

Accuracy	Values
trajectory estimation-position vertical	<0.10 [m]
trajectory estimation-position horizontal	<0.05 [m]
trajectory estimation-roll & pitch	0.015 [deg]
trajectory estimation-heading	0.035 [deg]
laser scanner	0.015 [m]

, it is apparent that also for this UAV-based laser scanning system, the major influence on the accuracy of the estimated point cloud comes from the accuracy of the trajectory.

The stated accuracy values are typical values that can be expected for trajectory estimation with this platform given short baseline GNSS processing and a sufficient satellite constellation. The sensor and system specifications listed so far are given without trajectory improvements, which are typically provided by strip adjustment approaches or GCPs [30]. To assess the quality of the point cloud in the following, the assumed accuracy of the final product should be within the range of the specified trajectory accuracy.

2.2. Evaluation Strategy

The evaluation approach to analyze the absolute accuracy and precision in this study is generally separated into the analysis based on single strip measurements without additional multi-strip or GCP corrections, and the analysis of measurements with a cross-flight pattern, referred to as multiple strip measurements. Single strip measurements are performed to focus the analysis of system performance without additional GCP input or corrections like strip adjustment, whereas the cross-flight pattern with multiple strips is used to evaluate the full benefit of the post-processing provided by the manufacturer’s software, such as strip adjustment. The limiting factor for the absolute accuracy in the investigation of a single strip is clearly given by the GNSS processing within trajectory estimation. The accuracy achieved in the differential processing of GNSS observations is further dependent on the GNSS master station, the satellite geometry, and the measurement conditions. Since it is almost impossible to evaluate the accuracy of estimated UAV trajectory directly, the calculated point clouds are used for the evaluation of the system. The individual research aspects for the evaluation approach are detailed in the following with the assessment parameters and strategies used, which are investigated for the single strip and the multiple strip measurements.

(1)

Noise

The noise level is analyzed based on different parts of the resulting point clouds. This parameter primarily gives a quantification of the noise of the laser scanner measurements (included in Equation (1)) and an understanding of how much it contributes to the error budget of the system. The noise evaluation parameter is based on a plane fit and the corresponding residuals. Therefore, planar objects with good reflective properties are necessary with an appropriate scan geometry. The noise level is investigated only at the given flight height without a detailed evaluation of the systematic dependence with increasing distance to the object.

(2)

Absolute accuracy

The absolute accuracy is the main part of this study. For this, the UAV-based laser scanning point clouds are analyzed in comparison to a georeferenced point cloud measured with TLS. This evaluation is supposed to determine the absolute accuracy of the system in the vertical and horizontal direction using a point-based evaluation approach. Since the accuracy in the vertical direction is in most cases lower than in the horizontal direction [41], a special focus is given on the height component. Besides the point-based evaluation, the orientation of the point cloud in the global reference frame is analyzed using a parameter-based approach. Concerning the parameters in Equation (1), trajectory estimation, laser scanner and system calibration are evaluated simultaneously in this investigation.

(3)

Precision

Additionally, the precision is evaluated with repeated measurements using the same flight plan for each repetition, spread over several hours. This approach is intended to indicate the precision of the system and, in particular, its performance under changing GNSS conditions and constellations. This investigation uses point-based as well as parameter-based evaluation strategies.

(4)

GNSS master station

This aspect examines the importance of the GNSS master station used for the trajectory estimation of the UAV. In principle, there are different possibilities of master raw data which are used for the relative GNSS processing. The analysis provides a result of the impact of different master stations on the point cloud’s accuracy and a recommendation for the application of UAV-based laser scanning in practice.

2.3. Data Acquisition and Reference Point Cloud

The data acquisition environment was selected according to the different study aspects described in Section 2.2 to generate suitable data sets for the analysis. Since the main part of the investigation relies on a reference point cloud, we selected areas near an existing network of control points that can be used for georeferencing a TLS point cloud. Since the evaluation is performed using artificial objects such as targets or planes, the experiment can be easily implemented in other areas. The two main evaluation strategies are divided into single strip and multiple strip measurements, for which two different study areas were chosen. Since the single strip measurements are the focus, they are discussed in more detail. In the following, the two study areas and the objects contained therein (Section 2.3.1), the procedure for generating the TLS reference (Section 2.3.2) and the UAV-based measurements (Section 2.3.3) are explained.

2.3.1. Study Areas and Objects

The study area used for the single strip measurements is located at the Campus Klein-Altendorf near Bonn and is shown in Figure 3a with additional objects that were arranged over the entire area with a length of about 300 m.

The location was chosen based on an existing network of high accuracy control points [1], which are shown in Figure 3a and used to georeference the TLS point cloud. The measurement setup was designed in a way that flight planning will be similar to the corridor mapping application with a fixed flight direction, in this case parallel to the road. A measurement with the UAV system, shown in Figure 3b, includes the path along the road in the north–east direction (flight direction 1) and back along the same path (flight direction 2), which means that each object is always measured twice for one data set. The objects within the area used for the quality analysis are six tables (1.2 m × 0.8 m) and four targets (0.3 m × 0.3 m), shown in Figure 4b,c.

The tables have a white and plain surface with high reflective properties and were placed directly below the trajectory. In addition, the tables were aligned horizontally using a spirit level so that each position on the table had the same height.

The second study area used for the evaluation of the multiple strip measurements is a typical urban environment with buildings, streets and vegetation. The principle approach of analyzing the UAV-based laser scanning system is similar by the use of an accurately measured TLS reference point cloud and the following assessment of the UAV point clouds using point-based and parameter-based evaluation techniques. Compared to the data set before, a cross-flight pattern, which is typical for UAV-based laser scanning, is performed.

The study area used for the multiple strip measurements is shown in Figure 5a, where additional targets distributed over the whole area as indicated. The 12 targets are in the same TLS design as before but with a size of 0.8 m × 0.8 m (Example shown in Figure 5b) that targets center could be estimated also with a flight height of 25 m.

2.3.2. Reference Point Cloud from Terrestrial Laser Scanning

A reference point cloud is used for most of the evaluation aspects described in Section 2.2. The point cloud for the first study area was acquired with a Leica ScanStation P50 TLS and a target-based registration using BOTA8 (Bonn Target 8) laser scanning targets [42], shown in Figure 4a. To cover the entire study area, 16 viewpoints with 29 target locations were required, which were measured with the scanner resolution setting of 3.1 mm @ 10 m. In addition, the viewpoints were planned next to the objects along the road and near the control points, which are defined by attached targets (pillar or building) shown in Figure 3. These control points are used for georeferencing the point cloud and are part of an evaluation environment [1], which consists of a dense network of 37 control points in total. The control points are determined in a network adjustment based on measurements from total station, GNSS and leveling, which results in an estimated standard deviation of <0.1 cm. Measurements of the network were carried out in two consecutive years to verify the stability of the control points included. Further information about the measurements and methodology can be found in Heinz [1]. The coordinates of all control points are given in the official German coordinate system ETRS89/UTM32 (European Terrestrial Reference System 1989, Universal Transverse Mercator, Zone 32) for positions. Furthermore, the heights are given in physical heights (DHHN16, German Main Height Network 2016) and also ellipsoidal heights (GRS80, Geodetic Reference System 1980). Subsequently, the control point coordinates are used in the registration of TLS viewpoints to process the georeferenced point cloud. The processing of TLS measurements was done using the Leica Cyclone processing software. The registration error related to control points is given with a RMSE (root mean square error) of 0.5 cm. To control the reference point cloud, additional RTK-GNSS measurements were performed at two different epochs on this day, which showed a RMSE of the reference point cloud <1 cm for the height component. For the second study area with the multiple strip measurements, the creation of the reference point cloud is done quite similar as for the first study area, using a Leica ScanStation P50 and a georeferencing based on known control points as already described. The registration error related to the control points is given with an RMSE of 0.8 cm for the second study area and is further verified by RTK-GNSS measurements of the distributed targets, which are again in a range <1 cm. Therefore, the absolute accuracy described by the RMSE of the georeferenced point cloud in both study areas is much higher than the expected accuracy of the UAV-based laser scanning system measurements and can be used as a reliable reference.

2.3.3. UAV-Based Laser Scanning Measurements and Processing

The UAV measurements are divided into the flights for single and multiple strip measurements. For both approaches we collected data from four identical flights, using the UgCS software for flight planning. Right before and after the measurements, a recommended flight pattern is included, which is later used for sensor initialization in the trajectory estimation process. For the single strip measurements, the flight path is shown in Figure 3 with a straight flight from the start in the north–east direction and on the same path back. The flight plan was chosen due to the challenging conditions for the trajectory estimation with a constant heading of the UAV and, thus, performance evaluation under less optimal conditions similar to corridor mapping. For the measurements with multiple strips, the corresponding flight path is shown in Figure 4. The flight height and speed were chosen according to the buildings and objects to cover the entire area with one flight and the highest possible point density. The corresponding flight parameters for the individual approach are listed in

Table 3. UAV flight parameters for single and multiple strip measurements.

	Single Strip	Multiple Strip
flight repetitions	4	4
flight height [m]	10	25
flight speed [m/s]	1	3
laser scanner line speed [lps]	55	$58.92$
avg. point distance [m]	$0.022$	$0.055$
point density [pts/m${}^2$]	2495	333

.

The flight height for the single strip measurements was 10 m, the flight speed was 1 m/s and the line speed of the laser scanner was 55 lines per second. With this configuration, the average point spacing is about 2.2 cm leading to about 2495 points per square meter. With this setting of the flight parameters, the evaluation of the point cloud should mainly include the errors from the trajectory estimation and the system calibration, since the influence of the errors due to the laser scanner is expected to be within the manufacturer’s specification of 1.5 cm. Based on this, the evaluation of the system mainly refers to the aspects of trajectory and system calibration. Thus, the most important aspect is the trajectory accuracy and especially the absolute accuracy resulting from the differential GNSS processing. To evaluate the influence of the choice of the GNSS master station on the result, we compare the different GNSS master stations listed in

Table 4. GNSS master stations used for trajectory estimation of single strip measurements for experiment (4).

	GNSS	Baseline Length [km]
Own master station (pillar)	GPS/GLO/GAL/BDS	1
Virtual Reference Station 1 (VRS1)	GPS/GLO/GAL/BDS	1
Virtual Reference Station 2 (VRS2)	GPS/GLO/GAL/BDS	2
CORS station (SAPOS NRW)	GPS/GLO/GAL/BDS	16

.

The master station mainly used within all experiments is the own master station set to pillar $P_2$ (Figure 3) next to both study areas. The pillar is part of the network of control points and, therefore, known with an accuracy of a few millimeters. The receiver used as the master station is a Leica GS10 with a Leica AS10 multi-GNSS antenna. Besides this own master station, two additional VRS next to the study area and a CORS station from the SAPOS NRW network in 16 km distance are included, which are only used in the comparison between master stations.

The explained measurements are afterward processed using the calculation scheme described in Figure 1 divided in trajectory estimation and the following georeferencing of laser scanner data. The first step of trajectory estimation is done using the software POSPac UAV providing the smoothed trajectory in the same coordinate system as the reference point cloud (ETRS89). The following processing of the point cloud is done with RIEGL’s software RiPROCESS without any additional optimization step or integration of GCPs. The outputs are the georeferenced 3D point clouds for the first study area, which are used in the following section for the evaluation of the system’s performance.

For the second study area, the full processing using RiPRECISION within RIEGL’s RiPROCESS software is used for the optimization of the point cloud product based on overlapping flight strips (Figure 1). This processing corrects the initial UAV trajectory by the alignment of multiple laser scans from the same area and a following reprocessing of the optimized point cloud. The corrections are performed with the point clouds from multiple strips, derived geometric point cloud features, and sensor information such as amplitude or reflectance used in various types of distance minimization to reduce modeling disparities. Within RiPRECISION all flight strips are selected for the optimization with automatic feature detection and a non-rigid adjustment mode with translation. The capability of this additional processing step with RiPRECISION is analyzed shortly with the noise parameter for parts of the point cloud. If the correction of overlapping scans from different flight directions would fail, scan strips are misaligned and the estimated residuals for a plane surface within the point cloud will increase systematically. Since the general accuracy and especially the georeferencing of the UAV system is investigated, no additional GCPs are used.

3. Results

The results are presented in the same order as described in Section 2.2 with the four research aspects related to the single strip measurements separated in Experiment (1)–(4) and the second part of multiple strip measurements described in Experiment (5)–(6). The first results of Experiment (1)–(3) are all based on processing with the own GNSS master station located next to the study area.

3.1. Experiment (1): Noise for Single Strip Measurements

The first experiment analyses the noise of the system using planar surfaces in the study area. The data sets utilized for this are the four repetitions of the single strip measurements. The objects used for the noise analyzes are the six tables shown in Figure 3, which are extracted from the four data sets. For noise evaluation, the residuals with respect to a plane fit are taken, as described in [43]. For the plane estimation, a RANSAC algorithm is first used for plane identification and extraction of the corresponding 3D points ${\left[x_e,y_e,z_e\right]}^T$. Afterwards, the individual plane can be described with equation

$$\begin{array}{c}\hfill n_x·{\mathbf{x}}_{\mathbf{e}}+n_y·{\mathbf{y}}_{\mathbf{e}}+n_z·{\mathbf{z}}_{\mathbf{e}}-p=0\end{array}$$

(2)

in the three dimensional space. The parameters for the estimation using a Gauß-Helmert model are the orthogonal distance p between origin and plane, and the components of the unit normal vector $n_x,n_y$ and $n_z$[44]. Since the tables are placed horizontally and directly under the flight path, it can be assumed that the angle of incidence for all measurements is close to ${90}^{\circ }$ and deviations due to different incidence angles are therefore negligible. Based on this assumption, the residuals to the plane in z-direction ${\mathbf{v}}_{z_i}$ obtained from the adjustment are used for the estimation of the range precision ${\sigma }_d$ with

$$\begin{array}{c}\hfill {\sigma }_d=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\mathbf{v}}_{z_i}^2},\end{array}$$

(3)

where n defines the number of points referring to the plane. An example of the distribution of residuals with respect to the plane fit is shown in Figure 6a for a table from the first data set.

Considering all data sets for the single strip measurements, the range precision is estimated for all tables included in each data set and plotted summarized by a histogram in Figure 6b. A total of 48 planes were used for noise evaluation with a mean number of points per plane fit of 1327. The resulting mean range precision for the 10 m flight height is 0.35 cm, which is within the manufacturer’s specification of 1.0 cm [41]. Furthermore, this result describes the impact of scanner precision on the subsequent quality analysis, which is comparatively negligible taking the uncertainty due to trajectory estimation into account. The absolute accuracy of the laser scanner cannot be analyzed with this approach because a systematic offset cannot be separated from errors due to trajectory estimation. For investigations that consider the absolute accuracy of the laser scanner, the sensor should be analyzed without the UAV, while the uncertainties due to trajectory estimation have no influence.

3.2. Experiment (2): Point-Based Absolute Accuracy and Precision for Single Strip Measurements

The second experiment analysis the precision and absolute accuracy based on the comparison to the TLS reference point cloud. The first quantification of accuracy is derived from the target coordinates of four TLS targets divided over the whole study area. The target centers are computed for TLS with ${\left[x_{TLS},y_{TLS},z_{TLS}\right]}^T$ and the four flight repetitions with the algorithm by Janßen et al. [42]. This algorithm uses a robust plane estimation for the selected target and subsequent template correlation to detect the target center coordinates with a standard deviation in the millimeter range. The TLS point cloud as well the UAV-based laser scanning point clouds are given in the official German coordinate system ETRS89/UTM32 with ellipsoidal heights. The used relations for quality assessment are shown in Figure 7 simplified for one target and only two dimensions.

The coordinates for the target centers from TLS with ${\left[x_{TLS},y_{TLS},z_{TLS}\right]}^T$ are assumed to be the reference. Since each target is included multiple times in the four repetitions with the UAV, the precision $\sigma$ can be estimated based on the mean target coordinates ${\left[{\mu }_x,{\mu }_y,{\mu }_z\right]}^T$. The differences between TLS and the UAV-derived target center coordinates are used for the description of the absolute accuracy described with $\delta$. Considering the differences for all target centers from the UAV data set, a corresponding root mean square error ( RMSE) is calculated.

The histograms in Figure 8 combine the results for four UAV flights and the differences for target coordinates in East, North and height direction. The results are shown together for all targets indicated with the index i for the corresponding data set and index j for the target number.

The mean absolute difference for the east direction ${\delta }_x$ is 0.9 cm with a RMSE of 1.1 cm and for the north direction ${\delta }_y$ it is 1.9 cm with 1.2 cm RMSE. The mean absolute difference of target coordinates describes the absolute accuracy of a single measurement without any additional information or correction within the processing. Based on the shown results a systematic error can be seen in the height coordinate by ${\delta }_z$ with 3.3 cm, which should be analyzed also by the use of additional objects in the study area. Furthermore, the RMSE for the vertical direction is only 1.2 cm and of a similar order of magnitude compared to the horizontal direction. The precision of the system is determined with the standard deviation using $d{x}_i$, $d{y}_i$ and $d{z}_i$. It is lower than 1.2 cm for all directions. Both results are better than the manufacturer’s system specification. Especially the precision shows good results in position and height direction.

Since the uncertainty in the vertical direction is typically worse than the horizontal direction, the tables within the study area are used for the additional analysis of the height component. Therefore, the described plane estimation from Experiment (1) is used for the determination of table heights for a given position on the table. The estimated ellipsoidal heights are compared with the reference from the TLS point cloud. The six tables within the study are measured again twice for one data set as before described for the target evaluation. The results for the individual flights are shown in Figure 9 with a summary of the whole data set in the histogram on the right.

The mean absolute height difference for all table heights are 3.4 cm with a RMSE of 1.3 cm. The systematic offset of about 3 cm is equal to the target-based analysis before, which in general describes a better and consistent absolute accuracy than expected. Also, the RMSE with 1.3 cm provides a good result based on the error budget assumed for the UAV-based laser scanning system. Overall the analysis compared to the TLS reference has shown an absolute accuracy defined by mean difference of <3 cm for the horizontal direction and <4 cm for the vertical direction. In addition, the precision of the system analyzed based on the repeated flights described with the standard deviation is <1.2 cm for the horizontal and vertical direction. These results should serve as a good benchmark for the transfer to other settings and primarily analyze the accuracy of the UAV’s trajectory.

3.3. Experiment (3): Parameter-Based Absolute Accuracy for Single Strip Measurements

The third experiment is an assessment based on the estimated table planes from Experiment (2). Compared to the point-based evaluation of the point clouds, this approach uses a parameter-based evaluation with the estimated plane parameters given in Equation (2). The normal vector defining the orientation of each plane in space is compared to the reference planes from the TLS point cloud. For better interpretation, the tilt is divided into a tilt along and one across the direction of flight. The computation is done using the known heading during the flight. With this tilt parameter, any misalignment of the UAV point cloud could be detected, especially due to errors in the bore-sight angles. The summarized results for the four flights and all included tables are shown in Figure 10 with the tilt across and in the direction of flight.

The mean tilt across flight direction is $-0.{014}^{\circ }$ with a RMSE of 0.085${}^{\circ }$ and the mean tilt in flight direction is $-0.{052}^{\circ }$ with a RMSE of $0.{052}^{\circ }$. The tilt estimates for the precision are nearly the same and the standard deviation is within the same magnitude.

With these results, various conclusions can be drawn about the precision and absolute accuracy. Due to the scan configuration, the roll angle and the corresponding calibration parameter can be analyzed perfectly. A systematic error in the roll angle would lead to an offset respectively tilt of the point clouds from the flights in the opposite direction. With the histogram of the tilt parameter across the flight direction, a systematic error can be excluded. The small systematic offset with the mean value of $-0.{052}^{\circ }$ can be classified as non-significant. The standard deviation for both tilt directions is another measure of the high precision of the system for multiple flights. Based on the table size, a tilt of $0.1^{\circ }$ would result in a height offset of less than 0.2 cm, which is within the noise level of the UAV-based laser scanning system and also the limitation imposed by the accuracy of the TLS reference.

3.4. Experiment (4): GNSS Master Station

The fourth experiment based on the single strip measurements data set is investigating the impact of different GNSS master stations on the quality of the georeferenced point cloud. In addition to the own master station mounted on a pillar and used for all experiments before, a CORS station and two different VRS are used within the processing. The evaluation parameters are identical to Experiment (2) based on the difference of target center coordinates compared to the TLS reference and the corresponding RMSE of targets centers from four flight repetitions. The calculated differences are shown in Figure 11 for the four different master stations each for the coordinate difference in the east (blue), north (red) and height (yellow) direction.

In addition, the mean absolute difference is indicated in each histogram and in combination with the RMSE given in

Table 5. Target center difference between TLS reference and UAV point cloud with four different GNSS master stations. The difference is described using the mean value and standard deviation for the East, North and height direction.

	Mean Absolute Difference [cm]			Root Mean Square Error [cm]
	${\mathit{\delta }}_{\mathit{x}}$	${\mathit{\delta }}_{\mathit{y}}$	${\mathit{\delta }}_{\mathit{z}}$	${\mathbf{RMSE}}_{\mathit{x}}$	${\mathbf{RMSE}}_{\mathit{y}}$	${\mathbf{RMSE}}_{\mathit{z}}$
Own master station (pillar)	$1.0$	$1.9$	$3.3$	$1.1$	$1.3$	$1.2$
Virtual Reference Station 1 (VRS1)	$0.9$	$1.6$	$2.7$	$1.1$	$1.4$	$1.4$
Virtual Reference Station 2 (VRS2)	$0.9$	$1.6$	$2.6$	$1.1$	$1.4$	$1.7$
CORS station (SAPOS NRW)	$0.4$	$1.7$	$2.9$	$1.2$	$1.4$	$1.3$

. The parameters are estimated based on eight targets for each of the four flights which leads to a total number of 32 targets used for the computation.

In the visual evaluation with Figure 11, the results for the estimated coordinates for different master stations are very similar without much variation. This is stated also with the calculated mean difference as quantification of the absolute accuracy, which only differs in the mm range between different solutions. The expectation beforehand for absolute accuracy was, that results based on own master station would perform best since the pillar coordinates are well known and the conditions for GNSS observations are the same as for the UAV. Comparing the mean difference of the own master station with both VRS and the CORS station, the difference is slightly higher and the results for the RMSE can be rated as equal. Summarizing the absolute accuracy for the comparison of different GNSS master stations, the results are equal on the cm level. The estimated offset is about 1 cm in the horizontal direction and about 3 cm in the vertical direction for all variations of processing. Additionally, the RMSE is used as a measure of the quality, where results are similar or even identical for different master stations. In this case, the own master station provides the best results but without a substantial deviation from the other ones. The conclusion from the evaluation of the Experiment (4) leads to the statement that the use of a VRS and also a CORS station in 16 km distance delivers similar results. Since setting up an own master station always involves additional effort, the two other variants offer a viable alternative that can be recommended. In particular, the use of a VRS is a frequently used approach anyway and is probably the best variant for many applications.

3.5. Experiment (5): Point-Based Absolute Accuracy and Precision for Multiple Strip Measurements

The last Experiments (5) and (6) are investigations based on the second data set already mentioned in Table 3 as multiple strip measurements. Since the analysis of different GNSS master stations in Experiment (4) has shown equal results for all master stations, the processing is only done using a VRS. The first parameter for the precision of the point cloud is given with the noise of different parts in the point cloud. The approach for noise estimation is used as in Experiment (1) and the areas of the 12 targets are extracted for this estimation. The result for the four repetitions and all targets is a mean noise of 0.6 cm. Here, several factors influencing the accuracy of the point cloud can be evaluated simultaneously. First of all, the influence of the laser scanner on the error budget can again be classified as small, since the manufacturer accuracy is even worse specified than the total noise of the resulting point cloud. Of course, the specified distance for the given accuracy is not yet reached with 25 m flight altitude, but the calculated value is still much lower than the given 1.0 cm. Furthermore, the performance of the optimization by RiPRECISION is good for this data set, since no major offsets remain within the point clouds. The further results in this experiment are derived from the estimated target centers and the comparison with the target centers from the TLS reference point cloud. The difference divided in the east (blue), north (red) and height (yellow) coordinates are shown in Figure 12 in histograms summarising all four flight repetitions.

The determined mean absolute difference result in $-1.1$ cm for the East, $1.1$ cm for the North and $4.3$ cm for the height direction with corresponding RMSE of $1.8$, $1.0$ cm, and $2.2$ cm. Therefore, the absolute accuracy described by these differences is consistent with the results from Experiment (2) and even a bit smaller for the horizontal direction. The systematic effect in the vertical direction is again similar with a magnitude of 4 cm, which was already recognized in the analysis of the single strip measurement. Since this is only presented as a shift of the point cloud, a simple solution in practice would be the use of one or multiple GCPs. By using a single GCP with precise height information, the entire point cloud can be shifted to the correct height without great effort. The precision described by the standard deviation is given with $0.8$ cm for the East, $0.5$ cm for North and $2.1$ cm for the height component. Both components, horizontal and vertical, are again small and are in accordance with the specification of the manufacturer. Overall, the second data set based on the typical cross-flight pattern with the use of the full processing capability except for the inclusion of ground control information has shown again good results with absolute accuracy described by the mean difference around 1 cm for the horizontal direction and around 4 cm for the vertical direction. Furthermore, the additional processing with RiPRECISION provides a reliable adjustment of the trajectory and so an optimization of the point cloud. The standard deviation as a parameter for the precision provides even better results for the horizontal direction compared to the single strip measurements. This can be explained by the improvement of the point cloud by using multiple measurements from the same area, which are used in the optimization of the point cloud.

3.6. Experiment (6): Parameter-Based Absolute Accuracy for Multiple Strip Measurements

The sixth experiment analyses the orientation of planes within the point cloud according to the parameter-based evaluation as also done in Experiment (3). For the single strip measurements, this approach is capable to detect orientation errors in the trajectory estimation or calibration of the system. The evaluation with the multiple strip measurements in this part will give a general assessment of how well the point cloud is orientated absolutely in space without the ability to trace errors back to their cause due to the complexity of the calculation steps performed. The surfaces of the targets have a size of 0.8 m × 0.8 m and are used for the GHM plane estimation with the normal vector as the parameter. With this normal vector the tilt of the planes can be estimated in relation to the plane parameter derived from the reference point cloud. The tilt is further divided into a tilt between the zenith and north direction and perpendicular between zenith and east direction, which is shown in Figure 13.

The resulting mean values for the tilt in zenith–north direction is $-0.{017}^{\circ }$ with a RMSE of $0.{158}^{\circ }$ and in zenith–east direction $0.{015}^{\circ }$ with a RMSE of $0.{116}^{\circ }$. The mean values for both tilt directions indicate that there is no systematic trend or tilting of the entire point cloud compared to the reference. Since the histogram summarizes the estimations based on four separated flights, the distribution can also be used as a measure for the precision. With both RMSE values the precision is a bit higher compared to the first data set but this is explained due to the 25 m flight height. Furthermore, the histogram includes also larger tilt estimates around $0.4^{\circ }$, which might come from a misalignment in the UAV point cloud but could also be enhanced by the small uncertainty of the TLS point cloud. The tilt of a magnitude of $0.4^{\circ }$ would lead to an error of about 0.6 cm in the height direction considering the object size, which is within the noise level of the point cloud from the UAV-based laser scanning system. The output of the last experiment is a consistent and well orientated point cloud over larger areas and multiple flights. Furthermore, the additional processing for point cloud optimization performs well without manual interaction in the estimation.

4. Conclusions and Outlook

This paper presents an evaluation strategy for the quality assessment of single measurements with a UAV-based laser scanning system and, in particular, the capability for direct georeferencing and the corresponding absolute accuracy and precision. Since the evaluation of a UAV-based laser scanning system is non-trivial due to the complex processing chain and included error sources, the evaluation is performed on the derived point cloud using parameter-based and point-based evaluation strategies. The first study area was designed similar to the challenges within corridor mapping. With this flight planning, systematic errors due to trajectory estimation and especially system calibration are detectable. With the use of a second study area and the different flight plan based on a cross-flight pattern, the capability of RIEGL’s software RiPRECISION was analyzed. The comparison for both study areas is performed with a TLS point cloud reference and the use of different objects like tables and targets. The following results are structured in the same order as the research aspects in Section 2.2 and summarize the results from both study areas.

(1)

The first parameter describing the noise and therefore the range precision of the laser scanner resulted in 0.4 cm for the flight height of 10 m for the single strip measurements and 0.6 cm for the flight height of 25 m for the multiple strip measurements. With the second data set, the performance of RiPRECISION is evaluated as well, which performs a proper alignment of several strips from the cross-flight pattern.

(2)

The absolute accuracy using the target-based evaluation for four separated flights was given with 0.9 cm in east, 1.9 cm in north and 3.4 cm in height direction for the single strip measurements. The additional focus to the vertical direction using table heights provided nearly the identical height offset of 3.4 cm with a RMSE of 1.3 cm. The results for the multiple strip measurements are consistent with everything concluded from the first data set with a mean difference in the horizontal direction around 1 cm and in vertical direction around 4 cm. The additional parameter-based evaluation was performed for the two different study areas and the tilts between estimated planes are always within the noise level of the point cloud and no systematic effects are observed.

(3)

The precision of the UAV system resulted in a standard deviation of 0.7 cm for the east, 1.1 cm for the north and 1.2 cm for height direction. The system provides excellent results over the whole study area for the single strip measurements. The same analysis resulted in a precision of 0.8 cm in the east, 0.5 cm in the north and 2.1 cm in height direction for the multiple strip measurements.

(4)

The impact of different master stations is assessed by the use of an own master station, a CORS station and two different VRS. The output of the performed point-based evaluation leads to nearly identical results for the four different GNSS data sets and therefore the conclusion, that strategy should be chosen, which is easiest to handle for the user. Overall, the VRS and the CORS station have shown good results considering the mean absolute difference and are therefore an appropriate alternative for the processing.

Besides the investigations of absolute accuracy, this study shows a new investigation approach for analyzing the precision of UAV-based laser scanning systems. By the use of identical flight planning and repeated measurements, the independent data sets are usable for a precision investigation. This approach can also be performed on a larger scale with different flight parameters and the use of natural objects for similar quality analysis. Overall, the results for the used UAV-based laser scanning system are better than expected based on the manufacturer’s specification. The larger difference in height direction for both study areas might be explained by the uncertainty of the system calibration and especially a systematic error due to the lever arm between sensors. This has to be investigated if a higher accuracy level should be achieved with direct georeferencing. However, it should again be emphasized that for all investigations GCPs were still not used or further manual adjustment in the point cloud optimization is performed. The additional processing step by RiPRECISION has shown a conceivable easy optimization step, which should be part of a standard procedure, but is specifically skipped in parts of this work to analyze the possibilities of the system.