The use of unmanned aerial vehicles (UAVs) for surveying purposes such as mapping, 3D modeling, point cloud extraction and orthophoto generation has become a standard operation in recent years. The large availability of so-obtained 3D models force the scientific community to explore and develop new tools for visualizing and sharing the achieved products and, as a consequence, open new scenarios strictly related to the use of virtual and augmented reality that for sure need to be considered as an important research topic even more related to the geospatial information [1
]. Within this framework, the high quality of commercial off-the-shelf cameras, easily implemented in UAV platforms and the development of new generation software packages created an important revolution in the geomatics field. Mapping and 3D modeling are increasingly exploiting UAV potentialities in several applications, many of which, as the present paper, related to morphological landscape surveying. The topographic characteristics of these sites are significantly different, since they include rivers [3
], shore regions [6
], gullies [8
], pits [9
] or glaciers [11
]; nevertheless UAV photogrammetry has always produced satisfactory results, thanks mainly to the capability of the Structure from Motion (SfM) approach to take advantage of the high spatial resolution of images to recognize textures at the ground even for uniform surfaces [6
The SfM technique operates under the same basic principle of traditional photogrammetry in which the 3D position of a point can be resolved from a series of overlapping images [14
]. What differentiates SfM from conventional photogrammetry is the use of a new generation of image-matching algorithms, developed within the computer-vision community, capable to automatically extract a database of features from a set of multiple overlapping images [15
]. We have then assisted to a transformation in the photogrammetric community that has adopted this new technique, demonstrating its potentiality in several contributions such as [13
The SfM approach is particularly of great interest in the processing of images acquired by UAV, where a series of characteristics such as path irregularity and varying altitude, which produce variation in overlapping, image scale and resolution, can cause problems to the classical photogrammetric method but where SfM algorithms are instead able to recognize a very large number of homologous features [5
]. This capacity is even more useful when a well-defined flight plan cannot be prepared for emergency reasons [20
] or for morphological constrains [21
In the first step, SfM aligns the imagery solving the collinearity equations in an arbitrary scaled coordinate system without any initial requirements of external information (camera location and attitude or ground control points (GCPs)). To frame the block to the ground reference system, two alternatives are possible: a seven-parameters rigid transformation or a bundle blocks adjustment. The former, simpler, can introduce significant distortions in the final 3D product if the SfM-automated matching points present not neglectable errors [5
]; the latter, more complex, allows to exploit photogrammetric experience to compute external orientation parameters but also to perform camera calibration and evaluate block precision and accuracy [13
]. For the second strategy, several studies deal with performance of UAV survey under several block configuration and, nowadays, when the UAV and SfM combination has led out on the photogrammetric marketplace different software solutions and also package performances; a review can be found in [22
Block configuration influences BBA results under several point of views related to flight schema, flying height, overlapping and the number and distribution of GCPs. A correct flight configuration must be planned according to the area of interest (AOI), the required ground resolution (expressed by the ground sampling distance) and acquisition geometry [23
]. Flying rather high with a high side overlap guarantees an optimal acquisition geometry and a ground resolution enough for most of the common medium-scale applications. For instance, [25
] gives an exhaustive overview of the influence of altitude, overlap and weather conditions for a forestry survey. Several blocks were acquired at a flying height between 20 and 80 m and with a side overlap range of 20% to 80%. Optimal configuration is reached with a side overlap at least of 60% and a flying height of about 80 m. Some examples of similar conclusions can be found in [26
] for landslide monitoring purposes, in [27
] for olive orchards, who suggest also a high forward overlap (95%) and in [28
], which analyze the influence of highly overlapping imagery in a 3D point cloud generation for a large test site of about 40 ha. Number and distribution of GCPs have been explored by several authors too. In [29
], the influence of six different GCP configurations (from a minimum of 4 to a maximum of 9 GCPs, located only on the block edges or also in the central area) have been studied for digital orthophoto and digital elevation model (DEM) production, using a fixed-wings UAV in Malaysia. Besides, [19
] evaluated the influence of GCP numbers and distributions in terms of DEM quality for two case studies located in Morocco and France, while [30
] have conducted a similar experience to produce a multispectral orthomosaic for crop management; in this case, the main key factor was represented by the spatial distribution of the GCPs. Finally, [31
] evaluates the horizontal and vertical accuracy for 13 cases of GCP configurations. Generalizing the results, it is confirmed that a small number of GCPs is useful only when data transformation (seven-parameters rigid transformation) is required while a larger number is necessary for camera self-calibration. For the second aim, spatial distribution is important too, as ground points should cover the whole AOI. Precisely because only a large GCPs number is useful for the stabilization of interior orientation, it is necessary to evaluate different approaches based on more accurate camera distortion models and, at the same time, on a reduction of markers number. However, since complex distortion models can over-parameterize the system, an independent ground control becomes mandatory, and the leave-one-out (LOO) cross-validation method can be applied to evaluate it. LOO is a statistical estimation technique where the original data are partitioned in k
-training subsets of equal size; the overall accuracy is obtained averaging the accuracy values computed on each subset. This method, originally applied in other fields like machine learning [32
], is more recently adopted in geomatics applications such as accuracy assessment of satellite imagery orientation [33
] or as outliers’ detector for least squares compensation in geodesy [34
After establishing the ideal configuration, software solutions performance must be evaluated too. In [35
], five software packages (Microsoft Photosynth - University of Washington, USA, Bundler - University of Washington, USA, PMVS2 – University of Illinois, USA, Agisoft PhotoScan - St. Petersburg, Russia and ARC3d – Ku Leuven, Belgium) are compared for 3D point cloud generation, and results are strongly influenced by the program choice in terms of point density, distribution and accuracy. Additionally, in [36
], five software packages are compared, some of which not based on the SfM approach: Erdas-LPS – Heerbrugg, Switzerland, EyeDea -University of Parma, Parma, Italy, PhotoScan, Pix4D – Prilly, Switzerland and PhotoModeler – Vancouver, Canada. Even if the SfM software can provide results in an automatic way (the photogrammetric ones have required an operator intervention in some phases), they present worst results in terms of the root mean square error (RMSE) of control points. Besides, [21
] compares the geometric accuracy of the DSM (digital surface model) computed using different scenarios, number of images and GCPs and two software packages: PhotoScan and IGN MicMac – Saint-Mandè, France. Both software packages provide satisfying results, even if PhotoScan seems to provide better results (limited deviation in the DSM and better reconstruction) outside the control region (GCPs’ bounding box). Other examples can be found in [37
Starting from these assumptions, the aim of the paper is to analyze the performance of five software packages which are today generally employed for processing the data acquired by UAVs using the SfM-oriented approach. The work deals with the data acquired by a UAV flight performed over a sandpit where several points were measured to use within the BBA operation to perform an independent check afterwards. The different followed strategies are accurately described in terms of weights of the observations used in the adjustment, strategies for tie point extraction and number of GCPs and check points (CPs). Besides, an accurate analysis of the achieved results is reported to understand which are the problems that could be founded during the data-processing and which strategy should be the most suitable for the survey purpose. Finally, the LOO cross-validation method is adopted to assess the block accuracy for one of the proposed configurations.
4. Discussion and Further Activities
A significant part of a sandpit was surveyed by a UAV equipped with a Sony A6000 camera. A set of ground points were measured and used either for block orientation or quality assessment. Five software packages were compared: PhotoScan, UAS Master, Pix4D, ContextCapture and MicMac. They were used to perform BBA in three configurations characterized by a different ratio between GCPs and CPs. In Configuration 1, markers were all used as GCPs to perform robust camera calibration, Configuration 2 dealt with an intermediate setup with strong ground control and some check points and Configuration 3 was the more realistic one and simulated a routine surveying.
For each program, the BBA strategy was carefully studied, and final settings were tuned to optimize results. Section 3
contained an extensive discussion on the BBA parameters, since the processing choices significantly influenced the final accuracy. In literature, other authors have worked on this topic; [20
] reported that appropriate camera models and tie/control points weighting can improve the results, while [36
] showed that better values can be found in programs that have the capability to set the computation parameters in comparison with those that follow a black box approach. The software packages tested in the present paper have different levels of configurability: PhotoScan and MicMac (remembering that this last is an opensource project) were the most flexible programs, because they allowed to configure, contrary to the other three, whose interior parameters must be optimized or image coordinates must be weighted.
Residuals between the photogrammetrically obtained object coordinates of markers and those determined by surveying were formed and analyzed. Results for PhotoScan, Pix4D, UAS Master and MicMac were always good and comparable, less than 1 GSD for the horizontal components and less than 1.5 GSD, at worst, for the vertical one. Therefore, the decreasing number of GCPs influenced results but less than expected. The only exception was ContextCapture for the vertical component, as better discussed in the following.
The obtained accuracies were extremely good and lower than the well-accepted practical accuracy of 1−2 GSD in planimetry and 2−3 GSD in altimetry. This outcome can be attributed to the BBA strategies, optimized for the tested block, and to the high quality of the ground truth (GCPs/CPs had a precision of 0.5 cm for the horizontal components and 1 cm for the vertical one). The general trend followed that reported by other authors, such as [30
], who showed as the horizontal accuracy was better than the vertical one, and both are better when the number of GCPs increased (even if only slightly). In literature are also present papers in which GCP numbers and distributions assume key roles in the final accuracy. The authors of [51
] show that, to achieve optimum results in planimetry, the GCPs must be placed on the edge of the study area; besides, it is advisable to create a well-distributed configuration with a density of around 0.5–1 GCP per hectare to minimize altimetry errors; this conclusion was achieved using more than 70 markers. This outcome, even if interesting from a scientific point of view, refers to a nonrealistic routine surveying, and the question arises as to whether it can be reached also with a more traditional configuration, like that proposed by us and other authors [30
]. As previously reported, the only exception is ContextCapture, which presented larger values in the vertical component for Configuration 3. This result was not an isolated case for this program, and a similar one can be found in [52
]. This paper reported the accuracy of some software packages, among them, ContextCapture, in the processing of data acquired by UAVs on a coastal area. In this case study, ContextCapture performed worse than other softwares, such as PhotoScan and Pix4D, both for horizontal and vertical components. In our experience, only the altimetry suffered by the ground configuration, while planimetry obtained always good results and was comparable with the other programs. Moreover, excluding the anomalous residuals shown by ContextCapture, the values obtained by the five programs were fully comparable and can be summarized in a unique table (Table 5
Finally, an innovative use of the leave-one-out cross-validation was proposed to assess how each single GCP influenced the results and if, among them, some outliers were present. The analysis was conducted only for the two best performer software packages: PhotoScan and MicMac but did not highlighted anomalies in any GPCs, meaning the values discussed depended only on programs instead of on ground truth quality.
Further activities will follow different directions. On one hand, the other flights described in Table 1
will be processed with attention to oblique blocks in order to investigate their influence on final accuracy. On the other, final products, such as dense point clouds, will be assessed to explore the influence of BBA parameters in their generation. Several check points (more than 250) were already measured with a topographic total station on the upper flat area and on the scarp of the sandpit. An accurate comparison between the achieved point clouds and these points will be performed; an evaluation of point density will also be realized, comparing the clouds obtained in flat or scarp areas. LOO cross-validation analysis will be also extended, taking into consideration other information such as tie-points.
A significant part of a sandpit was surveyed by a UAV equipped with a Sony A6000 camera. A set of ground points were measured and used either for block orientation or quality assessment. Five software packages were compared: PhotoScan, UAS Master, Pix4D, ContextCapture and MicMac.
Residuals between the photogrammetrically obtained object coordinates of markers and those determined by surveying were formed and analyzed. Results for PhotoScan, Pix4D, UAS Master and MicMac were always good and comparable, less than 1 GSD for the horizontal components and less than 1.5 GSD, at worst, for the vertical one. The only exception was ContextCapture which presented some outliers in vertical component; excluding these anomalous residuals, values obtained by the five programs were fully comparable. Besides, the decreasing number of GCPs influenced results but less than expected. This outcome can be attributed to the BBA strategies, optimized for the tested block, and to the high quality of the ground truth. Finally, an innovative use of the leave-one-out cross-validation was proposed to assess how each single GCP influenced the results; no anomalies were found meaning that results depends only on programs instead of on ground truth quality.