3.1. Influence of the Standard Deviation Assigned to the Observations
The results of varying the standard deviation of the camera coordinates on the accuracy of block orientation in the RTK
configuration are shown in the graphs of Figure 2
, where the RMSE for the horizontal coordinates and for the elevation has been computed on all the 26 available checkpoints. Note that the standard deviation value read on the abscissa is always the one assigned to the XY coordinates of a particular BBA, as well as for the plots referring to the Z coordinate.
The variety of the trends from program to program and from test flight to test flight is large. As the XY and Z graphs do not necessarily show their minimum at the same value of standard deviation, an optimization of the standard deviation should be performed for XY and Z coordinates, as GNSS measurements are less accurate in height than in latitude and longitude. Overall, with all packages the RMSE is barely affected by the precision assigned to the horizontal coordinates (especially in MicMac), though generally the larger the standard deviation the larger the error. With PhotoScan, however, in one case (the RTK1 flight) the RMSE range in XY is larger than 3 cm. The accuracy in elevation, to the contrary, is strongly affected by changes in the standard deviation assigned to the camera Z coordinate: the RMSE range is up to 8 cm with PhotoScan and Pix4D and much larger for MicMac. A common trend is hard to identify, even for the same package: in some cases, the RMSE shows a well-defined minimum, in some other it decreases with increasing standard deviation, in some other the opposite happens.
As far as the software packages are concerned, with PhotoScan the best results in Z are obtained for standard deviations of 2–4 cm in RTK1 and RTK4; then they deteriorate with larger standard deviations. In the other two cases, though there is no clear minimum, larger standard deviations produce smaller errors.
In Pix4D in three cases the Z error improves the larger the standard deviation; only in RTK4 there is a clear minimum at around 3 cm of standard deviation. In all cases an RMSE in Z of 4 cm or better is reached.
MicMac is almost insensitive to changes in the horizontal coordinate’s standard deviation in the whole range from 1 to 10 cm. In Z, three types of behavior are apparent. In RTK2 and RTK4, the error drops from 25 to 35 cm at 1 cm standard deviation to a clear minimum around 3.5 cm; then, the RMSE grows slowly and steadily (asymptotically) towards a constant error (of 8 and 5 cm respectively). In RTK1, the error decreases monotonically towards a constant error of 2 cm, while, in RTK3, there is a minimum at 1.5 cm and then an asymptotic increase to about 8 cm.
Looking for hints on how to operate practically, inner quality measurements such as the mean reprojection error are useless (or even wrong) guides in the search for such optimum. In principle the smaller the standard deviation assigned to the camera stations in the BBA, the more the solution will be constrained, so the reprojection error should increase. On the contrary, if the standard deviations are large, the contribution of camera station observations gets smaller, i.e., they will not affect the residuals of collinearity equations. In practice, however, the variations of the reprojection error will be very small, even if mm level standard deviations are assigned to the camera stations (see Table A1
in Appendix A
). Simulations with CALGE show that only the weighted average of the residuals of the least squares adjustment, sigma zero, is affected to some extent as, unlike the reprojection error, it accounts for all observations. To the best of our knowledge, none of the software packages coming from Computer Vision provides this information, however.
If available, the estimates of camera precision provided by the RTK data processing should be used, perhaps with a penalty factor as they are generally too optimistic. For periodic surveys of targets, a sensitivity analysis is advisable, looking at the changes of the ground coordinates for different standard deviations.
As far as the four flights are considered, with a penalty factor of 2 applied to the nominal precision of the camera position encoded in the image EXIF (1 to 1.5 cm in X and Y and from 1.8 to 2.5 cm in Z), all of the software packages produce satisfactory, although not necessarily optimal, results in most cases.
The results of the second set of test on the accuracy of block orientation as a function of different standard deviations assigned to tie points and camera stations are summarized in Figure A1
in Appendix A
, where the RMSE in elevation obtained by PhotoScan is plotted for RTK1 and RTK3 (notice that the curves in Figure 2
represent a section of the surface plotted). As long as the ratio between the two standard deviations is constant, the least squares solution should be unchanged and so the RMSE. Therefore, one would expect that the RMSE remain constant over an angle with vertex on the origin. This is indeed roughly the case for RTK1, where it seems that the tie point precision is less critical than the camera station precision as far as accuracy in concerned. However, with RTK3 the pattern looks quite different, as the best accuracy is found far from the “reasonable” set of tie point and camera station precision. The reason why this happens needs to be further investigated.
3.2. eBee RTK Survey Accuracy Assessment
The mean errors and the standard deviation of the errors, computed at the 14 CP for all blocks, configurations and software packages are reported in Appendix A
in Table A3
and Table A4
. Figure 3
depicts the RMSE of each coordinate for all the combinations while Table 3
contains the mean error for the Z coordinate. In all flights and with all packages the RMSE obtained in the traditional block orientation by GCPs is in the range from 1.1 to 2.0 cm for the combined horizontal coordinates (equivalent to 0.5–0.8 GSD), while the elevations are about 1.5 times less accurate (from 0.7 to 1.3 GSD). There is little or no bias, since the mean error is always better than 1 cm in XY and never larger than 1.1 cm in Z (see Table A2
Looking at the results of RTK
, it is apparent that the accuracy of GNSS-assisted AT with the eBee RTK is on the one hand remarkable and often on the par with that obtained with ground control, especially in horizontal coordinates, where the combined RMSE in XY ranges from 0.6 to 1.2 GSD. On the other hand, comparing the 12GCP
results with those of RTK
, it is evident that there might be problems with heights: here the RMSE in Z ranges from 0.8 GSD (almost as good as the best 12GCP
) to 4.0 GSD. In two cases (RTK1 and RTK4) both the XY and the Z accuracies of RTK
are very satisfying, with average residuals no more than 5 mm larger than those of 12GCP
. In other words, the RTK solution practically achieves the same accuracy found using GCP as far as the horizontal coordinates are concerned. In elevation, to the contrary, there are three cases with very good agreement (with RMSE better than 2.5 cm) and cases where the accuracy is worse, up to 9.5 cm in PhotoScan, 6.8 cm in MicMac and 8.5 cm in Pix4D, see Table 2
. Here, a bias is apparent, since the mean error is markedly different from zero, with either positive or negative values for the same flight, depending on the software package.
in Appendix A
shows the plots of the errors obtained with PhotoScan on all 26 available checkpoints for the RTK configuration and for all four flights. In the horizontal coordinates, the error spatial distribution does not show any global systematic pattern, though in three of the test flights, in the eastern side of the block, a residual component parallel to strip direction is apparent on most points. As far as the elevation is concerned, it is hard to find any similarity between the four flight patterns as well as any pattern at all within a particular flight.
As the error in elevation turned out to be higher than expected (in particular in RTK2 and RTK3), in RTK+1GCP
an extended BBA has been computed, using as control information the projection center coordinates and those of GCP No. 13 (see Figure 1
) located in the middle of the block. As shown in Table 3
and in the diagrams of Figure 3
, the elevation accuracy improves considerably, down in the worst case to slightly more than 3 cm. Where the statistics in elevation were good already, such as for the Pix4D and PS solutions in RTK1 and RTK4, adding the GCP brings small changes (less than 1 cm) to all coordinates (some for the better, some for the worse). With the additional GCP, in the RTK1 block with MicMac a previous elevation RMSE of 8.5 cm is down to 1.4 cm. In other words, with at least a GCP the bias is removed or greatly reduced in all cases: the RMSE in Z now ranges from 0.6 to 1.6 GSD. As far as horizontal coordinates are concerned, the changes are very small, in the order of a few mm, except for RTK2 with PhotoScan and Pix4D, where they reach 1.2 cm; the combined RMSE in XY now ranges from 0.7 to 1.2 GSD.
MicMac is almost always the most accurate in XY in every configuration; PhotoScan is the best in elevation in the 12GCP and RTK cases, while in the RTK+1GCP case there is no clear winner.
The accuracy obtained on checkpoints is finally summarized in Table 4
, where the best and worst RMSE in each configuration are reported for all packages and all flights, expressed both in metres and in GSD units.
As stated before, the statistics in Figure 3
and Table A3
and Table A4
refer to the 14 checkpoints available. However, the accuracy of RTK
configurations has also been calculated on all available CP, i.e., 26 and 25, respectively. The changes to all the RMSE shown in Figure 3
are in both cases negligible, since they amount to just a few mm at most.
3.3. Interaction between Interior and Exterior Orientation Parameters
As far as the 12GCP configuration is concerned, the correlations between the principal distance and the camera heights, as can be expected, are very strong (on average 70% with maximum value 87%). This can produce restitution errors, if the scene has large differences in depth (e.g., high-rise buildings). Cross strips at different height would have limited the correlations, though the different image scale would probably limit the number of good tie-points between the sub-blocks. On the other hand, the analysis for RTK and RTK+1GCP shows much lower correlations, respectively, 5% (with 14% maxima) and 12% (with 33% maxima).
reports the estimated camera principal distance for the three configurations and for each software package in the RTK2 flight, where the bias in elevation was remarkable with all packages.
From Table 5
it can be noticed that the estimated values are remarkably different (by several micrometers) from package to package, though the standard deviation of the principal distance from the BBA varies between 0.2 and 1 µm depending on the flight and on the software. These differences might be due to some extent to the different camera models (see Table A4
in Appendix A
) and the estimation procedures of the three software packages; they remark, however, the already mentioned unreliability of interior orientation parameters determined using self-calibration in nadiral blocks.
Moving the block control from ground to the projection centres (RTK) in the RTK2 block there is a sharp increase in principal distance (between 5 and 11 µm) with every software package. Adding 1 GCP, Pix4D keeps the principal distance unchanged, PhotoScan almost reverts to the 12GCP value and MicMac settles for a value in between.
The spatial plots of the camera location residuals (i.e., of the differences between the estimated and GNSS-measured position) of the RTK2 block are shown for RTK
, left) and RTK+1GCP
, right) configurations. Table 6
reports the average of such residuals, in elevation only, for all blocks.
Cross examination of Table 3
, Table 5
and Table 6
and of the projection center residual plots in Figure 4
highlights the different behaviors of the software packages when RTK-based georeferencing is constrained by the introduction of the GCP.
, in Pix4D the mean of the projection center coordinates is fixed; in the plot, even though some systematic displacements in elevation can be seen in two strips, the general pattern is random, with a balance of positive and negative signs in elevation. When the ground point is included in the BBA the projection centers are systematically shifted upwards; at the same time, as shown in Table 3
, the principal distance is kept fixed.
On the contrary, the two plots of PhotoScan in RTK
differ very little and show no systematic patterns as far as the sign of the elevation changes is concerned. It is the principal distance which changes to adapt the block to the ground constraint of the single GCP. Considering the stereo normal case, if
is the depth (vertical distance from camera centre),
the principal distance and
the image scale, a variation in principal distance produce a change in depth
. Since the average image scale is about 1:18,000, a principal distance change of 1 µm is equivalent to a movement of camera height of 1.8 cm. Therefore, in RTK2, with a 3.5 µm change in principal distance, a variation
in depth of about 6.3 cm can be expected. The mean error change on the CP between RTK
is equal in PhotoScan to 7.3 cm, i.e., most of the 9.3 cm bias reported in Table 2
Summarizing, it seems that the balance between the weights of the different observations (GCP and camera center constraints) and the interaction between interior and exterior orientation parameters have different patterns in the three packages: as can be seen in Table 6
, the behavior found for the RTK2 flight also applies to the other blocks.
With MicMac, the behavior is different, because both principal distance and camera elevations are affected. In the RTK case, the RMSE in Z on CP is about 6 cm, with an estimated principal distance significantly higher than those calculated by PhotoScan and Pix4D. This excess in the estimated principal distance is compensated by an average upwards shift of the camera projection centers with a mean value of 26 cm. When one GCP is introduced, the average shift with respect to the measured positions is 15 cm and the principal distance change with respect to the 12GCP case (8.4 µm) is equivalent to a depth change of 15 cm.