Experimental Tests and Simulations on Correction Models for the Rolling Shutter Effect in UAV Photogrammetry
Abstract
:1. Introduction
1.1. State-of-the-Art in RS Modelling
1.2. Experimental Tests on Rolling Shutter Correction
1.3. Paper Goals
- (i).
- Following the results presented in [23] over flat terrain, the paper aims primarily to investigate the performance of the 10-parameter Fraser model over rough terrain, where the image scale varies considerably also within a single frame and therefore may result in less effective results. In this respect, evidence will also be sought on whether it is better to estimate a single set of affine parameters for the whole block or to work on an image-by-image basis. Alongside these new contributions, as the photogrammetric processing will be performed with Metashape, an evaluation of the effectiveness and a comparison with the 10-parameter camera calibration model of the two RS correction methods available in the 1.8.0 version will be performed on the test flight results. Besides the evaluation of the accuracy on the ground, an analysis of the estimated correction parameters and their correlations with flight or drone characteristics has been performed. Likewise, an analysis of the interior and exterior orientation parameters estimates from the BBA shows their correlations (particularly between principal distance and camera elevation a.g.l.) to be even stronger than usual with RS-distorted images.
- (ii).
- Adding the RS correction parameters as unknowns in the BBA might weaken the stability of the solution, introducing correlations that may require a denser ground control. To this aim, an analysis of the optimal number of GCP necessary, their density and their spatial distribution will be performed over the experimental test fields.
- (iii).
- The costs and operational benefits for drone surveys of hosting on-board global navigation satellite systems (GNSS) receivers capable of measuring with cm-level accuracy the camera stations are today largely acknowledged. On the one hand, drone manufacturers are recognising that the RS technology is an objective obstacle to the metric use of images and switching to global shutters in their latest products. On the other hand, apart from the turn-key RTK plug-in modules offered by virtually all main drone manufacturers, many kits are available on the market that allow users to render their RS platforms RTK-capable. Though, therefore, the problem of RS may fade in the medium or long term, the paper investigates with a series of simulations whether using drones with RS sensors and such enhanced-performance receivers helps to contrast the RS effect by reducing the number of GCP.
2. Materials and Methods
2.1. Equipment and Test Site Characteristics
2.2. Image and Reference Data Acquisition
2.3. Test Overview
2.3.1. Analysis of Check Point Accuracy as a Function of the Number of GCP
- -
- without RS modelling and with an 8-parameter camera calibration model;
- -
- without RS modelling and with a 10-parameter camera calibration model;
- -
- with Metashape 2-parameter RS model and with an 8-parameter camera calibration model;
- -
- with Metashape 6-parameter RS model and with an 8-parameter camera calibration model.
- RMSE(all): RMSE on CP of the BBA with all GCP fixed
- RMSE(i): RMSE on CP of the BBA with i GCP fixed, i = {22, 18, 14, 9, 6}
2.3.2. Analysis of Rolling Shutter Compensation Model Performance
2.3.3. A Simulation Study on RS Modelling with GNSS-Assisted Block Orientation
3. Results
3.1. CP Accuracy as a Function of the Number of GCP
3.2. Rolling Shutter Compensation Strategies
Parameters’ Value Evaluation
3.3. Rolling Shutter and GNSS-Assisted BBA
4. Discussion
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A
Appendix B
References
- Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision, 2nd ed.; Cambridge University Press: Cambridge, UK, 2000; ISBN 9780521540513. [Google Scholar]
- Remondino, F.; Spera, M.G.; Nocerino, E.; Menna, F.; Nex, F. State of the art in high density image matching. Photogramm. Rec. 2014, 29, 144–166. [Google Scholar] [CrossRef]
- Maltezos, E.; Doulamis, N.; Doulamis, A.; Ioannidis, C. Deep convolutional neural networks for building extraction from orthoimages and dense image matching point clouds. J. Appl. Remote Sens. 2017, 11, 1. [Google Scholar] [CrossRef]
- Eisenbeiß, H. UAV Photogrammetry; ETH ZURICH: Zurich, Switzerland, 2009. [Google Scholar]
- Bigas, M.; Cabruja, E.; Forest, J.; Salvi, J. Review of CMOS image sensors. Microelectron. J. 2006, 37, 433–451. [Google Scholar] [CrossRef]
- Fossum, E.R.; Hondongwa, D.B. A review of the pinned photodiode for CCD and CMOS image sensors. IEEE J. Electron Devices Soc. 2014, 2, 33–43. [Google Scholar] [CrossRef]
- Nakamura, J. Image Sensors and Signal Processing for Digital Still Cameras; CRC Press: Boca Raton, FL, USA, 2006; ISBN 9781420026856. [Google Scholar]
- Vautherin, J.; Rutishauser, S.; Schneider-Zapp, K.; Choi, H.F.; Chovancova, V.; Glass, A.; Strecha, C. Photogrammetric Accuracy and Modeling of Rolling Shutter Cameras. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, III-3, 139–146. [Google Scholar] [CrossRef]
- Zhou, Y.; Rupnik, E.; Meynard, C.; Thom, C.; Pierrot-Deseilligny, M. Simulation and analysis of photogrammetric UAV image blocks-influence of camera calibration error. Remote Sens. 2020, 12, 22. [Google Scholar] [CrossRef]
- Grundmann, M.; Kwatra, V.; Castro, D.; Essa, I. Calibration-free rolling shutter removal. In Proceedings of the 2012 IEEE International Conference on Computational Photography (ICCP), Seattle, WA, USA, 29 April 2012; pp. 1–8. [Google Scholar]
- Hedborg, J.; Ringaby, E.; Forssen, P.-E.; Felsberg, M. Structure and motion estimation from rolling shutter video. In Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 7 November 2011; pp. 17–23. [Google Scholar]
- Baker, S.; Bennett, E.; Kang, S.B.; Szeliski, R. Removing rolling shutter wobble. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; IEEE Computer Society: Washington, DC, USA, 2010; pp. 2392–2399. [Google Scholar]
- Oth, L.; Furgale, P.; Kneip, L.; Siegwart, R. Rolling Shutter Camera Calibration. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June2013; pp. 1360–1367. [Google Scholar]
- Meingast, M.; Geyer, C.; Sastry, S. Geometric Models of Rolling-Shutter Cameras. arXiv 2005, arXiv:abs/cs/0503076. [Google Scholar]
- Forssen, P.-E.; Ringaby, E. Rectifying rolling shutter video from hand-held devices. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; pp. 507–514. [Google Scholar]
- Wang, K.; Fan, B.; Dai, Y. Relative Pose Estimation for Stereo Rolling Shutter Cameras. Proc.-Int. Conf. Image Process. ICIP 2020, 2020-Octob, 463–467. [Google Scholar]
- Fan, B.; Dai, Y.; Zhang, Z.; Wang, K. Differential SfM and image correction for a rolling shutter stereo rig. Image Vis. Comput. 2022, 124, 104492. [Google Scholar] [CrossRef]
- Albl, C.; Sugimoto, A.; Pajdla, T. Degeneracies in rolling shutter SfM. Lect. Notes Comput. Sci. (Incl. Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinform.) 2016, 9909 LNCS, 36–51. [Google Scholar]
- Hedborg, J.; Forssen, P.E.; Felsberg, M.; Ringaby, E. Rolling shutter bundle adjustment. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 1434–1441. [Google Scholar] [CrossRef]
- Saurer, O.; Pollefeys, M.; Lee, G.H. Sparse to Dense 3D Reconstruction from Rolling Shutter Images. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 3337–3345. [Google Scholar]
- Im, S.; Ha, H.; Choe, G.; Jeon, H.G.; Joo, K.; Kweon, I.S. Accurate 3D Reconstruction from Small Motion Clip for Rolling Shutter Cameras. IEEE Trans. Pattern Anal. Mach. Intell. 2019, 41, 775–787. [Google Scholar] [CrossRef] [PubMed]
- Shoemake, K. Animating rotation with quaternion curves. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques-SIGGRAPH’85, San Francisco, CA, USA, 22–26 July 1985; ACM Press: New York, NY, USA, 1985; Volume 19, pp. 245–254. [Google Scholar]
- Zhou, Y.; Daakir, M.; Rupnik, E.; Pierrot-Deseilligny, M. A two-step approach for the correction of rolling shutter distortion in UAV photogrammetry. ISPRS J. Photogramm. Remote Sens. 2020, 160, 51–66. [Google Scholar] [CrossRef]
- Agisoft Metashape Documentation Agisoft Metashape User Manual: Professional Edition; Version 1.8; Agisoft LLC: Saint Petersburg City, Russian, 2022.
- Fraser, C.S. Digital camera self -calibration. ISPRS J. Photogramm. Remote Sens. 1997, 52, 149–159. [Google Scholar] [CrossRef]
- Kurczyński, Z.; Bielecki, M. Metric properties of rolling shutter low-altitude photography. Arch. Fotogram. Kartogr. i Teledetekcji 2017, 29, 177–190. [Google Scholar]
- Incekara, A.H.; Seker, D.Z. Rolling Shutter Effect on the Accuracy of Photogrammetric Product Produced by Low-Cost UAV. Int. J. Environ. Geoinform. 2021, 8, 549–553. [Google Scholar] [CrossRef]
- Chun, J.B.; Jung, H.; Kyung, C.M. Suppressing rolling-shutter distortion of CMOS image sensors by motion vector detection. IEEE Trans. Consum. Electron. 2008, 54, 1479–1487. [Google Scholar] [CrossRef]
- Benassi, F.; Dall’Asta, E.; Diotri, F.; Forlani, G.; di Cella, U.M.; Roncella, R.; Santise, M. Testing accuracy and repeatability of UAV blocks oriented with gnss-supported aerial triangulation. Remote Sens. 2017, 9, 172. [Google Scholar] [CrossRef]
- Zhang, H.; Aldana-Jague, E.; Clapuyt, F.; Wilken, F.; Vanacker, V.; Van Oost, K. Evaluating the potential of post-processing kinematic (PPK) georeferencing for UAV-based structure-from-motion (SfM) photogrammetry and surface change detection. Earth Surf. Dyn. 2019, 7, 807–827. [Google Scholar] [CrossRef]
- Roncella, R.; Forlani, G. UAV Block Geometry Design and Camera Calibration: A Simulation Study. Sensors 2021, 21, 6090. [Google Scholar] [CrossRef] [PubMed]
- Stott, E.; Williams, R.D.; Hoey, T.B. Ground control point distribution for accurate kilometre-scale topographic mapping using an rtk-gnss unmanned aerial vehicle and sfm photogrammetry. Drones 2020, 4, 55. [Google Scholar] [CrossRef]
- Fraser, C. Camera Calibration Considerations for UAV Photogrammetry. In Proceedings of the ISPRS TC II Symposium: Towards Photogrammetry, Riva del Garda, Italy, 4–7 June 2018; pp. 3–7. [Google Scholar]
Drone | Pixel Size [μm] | Resolution [pix] | Shutter Type | Sensor | Focal Length [mm] |
---|---|---|---|---|---|
Air2S | 2.4 | 5472 × 3648 | Rolling | 1″ CMOS | 22 * |
Mavic Mini | 1.8 | 4000 × 2250 | Rolling | 1/2,3″ CMOS | 24 * |
Phantom 4 Pro | 2.4 | 5472 × 3648 | Global | 1″ CMOS | 24 * |
Site | Number of Images | Height | Forward Overlap | Side Overlap |
---|---|---|---|---|
A | 68 | 45 m a.g.l. | 70% | 60% |
B | 108 | 50 m a.g.l. | 80% | 70% |
Site | Air2S | Mavic Mini | Phantom P4 Pro |
---|---|---|---|
A | 11 mm | 13 mm | 10 mm |
B | 15 mm | 17 mm | 12 mm |
Site | Air2S | Mavic Mini | Phantom P4 Pro | |
---|---|---|---|---|
A | 1 m/s | 0.93 | 0.47 | 0.47 |
2 m/s | 0.93 | 0.48 | 0.51 | |
4 m/s | 0.96 | 0.49 | 0.49 | |
B | 1 m/s | 0.96 | 0.48 | 0.48 |
2 m/s | 0.98 | 0.51 | 0.43 | |
4 m/s | 1.06 | 0.51 | 0.47 |
Case | Camera Calibration Model | Rolling Shutter Model |
---|---|---|
A | 8-parameter (f, cx, cy, k1, k2, k3, p1, p2) | None |
B | 10-parameter (f, cx, cy, k1, k2, k3, p1, p2, b1, b2) | None |
C | 8-parameter + b1 only | None |
D | 10-parameter (f, cx, cy, k1, k2, k3, p1, p2, b1, b2) with b1 and b2 computed image by image | None |
E | 8-parameter | Sx, Sy |
F | 8-parameter | Tx, Ty, Tz, Rx, Ry, Rz |
G | 10-parameter (f, cx, cy, k1, k2, k3, p1, p2, b1, b2) | Tx, Ty, Tz, Rx, Ry, Rz |
B | C | G | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
Site | 1 m/s | 2 m/s | 4 m/s | 1 m/s | 2 m/s | 4 m/s | 1 m/s | 2 m/s | 4 m/s | |
A | Air2S | −4.74 | −8.95 | −17.41 | −4.73 | −8.95 | −17.41 | −10.32 | −14.87 | −21.30 |
Mavic Mini | −2.02 | −3.87 | −7.89 | −2.03 | −3.87 | −7.9680 | −9.13 | −9.95 | −13.03 | |
Phantom | −0.01 | 0.01 | 0.04 | −0.04 | −0.02 | 0.01 | - | - | - | |
B | Air2S | −5.26 | −8.88 | −16.76 | −5.19 | −8.92 | −16.73 | −8.63 | −13.47 | −16.69 |
Mavic Mini | −2.68 | −4.83 | −8.40 | −2.68 | −4.81 | −8.39 | −8.70 | −10.14 | −13.26 | |
Phantom | −0.23 | −0.49 | −0.72 | −0.31 | −0.49 | −0.71 | - | - | - |
Site A | Site B | ||||
---|---|---|---|---|---|
Air2S | Mavic Mini | Air2S | Mavic Mini | ||
b1-Sy | 1 m/s | 0.910 | 0.944 | 0.776 | 0.444 |
2 m/s | 0.953 | 0.757 | 0.713 | 0.809 | |
4 m/s | 0.935 | 0.450 | 0.941 | 0.859 | |
b2-Sx | 1 m/s | −0.940 | −0.982 | −0.747 | −0.896 |
2 m/s | −0.973 | −0.861 | −0.568 | −0.972 | |
4 m/s | −0.980 | −0.666 | −0.686 | −0.908 |
Principal Distance Difference (pix) | Projection Centre Z Difference (m) | ||||||
---|---|---|---|---|---|---|---|
UAV-Site | Case | 1 m/s | 2 m/s | 4 m/s | 1 m/s | 2 m/s | 4 m/s |
Mavic Site A | B | −1.4 | −80.4 | −242.4 | −0.03 | −1.11 | −3.40 |
E | −3.9 | −83.3 | −180.6 | −0.04 | −1.09 | −2.45 | |
F | 18.8 | −76.3 | −191.3 | 0.26 | −1.00 | −2.60 | |
Mavic Site B | B | 9.8 | 27.7 | 39.4 | 0.14 | 0.39 | 0.54 |
E | 0.2 | 8.0 | 10.9 | 0.02 | 0.11 | 0.22 | |
F | −1.8 | 5.5 | 23.0 | −0.03 | 0.09 | 0.36 | |
Air2s Site A | B | −13.4 | −19.3 | −59.1 | −0.17 | −0.25 | −0.75 |
E | −6.1 | −22.9 | −49.1 | −0.04 | −0.20 | −0.44 | |
F | −69.2 | −61.1 | −95.5 | −0.75 | −0.63 | −0.97 | |
Air2s Site B | B | 22.3 | 35.1 | 73.8 | 0.26 | 0.41 | 0.90 |
E | 6.2 | 9.3 | 40.7 | 0.09 | 0.14 | 0.64 | |
F | −1.8 | −1.3 | 28.1 | −0.01 | 0.01 | 0.47 |
PC Set | X (cm) | Y (cm) | Z (cm) | |
---|---|---|---|---|
Original | A | 2.3 | 1.5 | 3.6 |
E | 1.1 | 1.0 | 1.6 | |
F | 2.0 | 1.5 | 2.5 | |
With random errors | A1 | 2.3 | 1.6 | 3.6 |
E1 | 1.1 | 1.0 | 1.7 | |
F1 | 1.2 | 1.1 | 1.6 | |
14 GCP Case B | 1.2 | 0.9 | 1.6 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bruno, N.; Forlani, G. Experimental Tests and Simulations on Correction Models for the Rolling Shutter Effect in UAV Photogrammetry. Remote Sens. 2023, 15, 2391. https://doi.org/10.3390/rs15092391
Bruno N, Forlani G. Experimental Tests and Simulations on Correction Models for the Rolling Shutter Effect in UAV Photogrammetry. Remote Sensing. 2023; 15(9):2391. https://doi.org/10.3390/rs15092391
Chicago/Turabian StyleBruno, Nazarena, and Gianfranco Forlani. 2023. "Experimental Tests and Simulations on Correction Models for the Rolling Shutter Effect in UAV Photogrammetry" Remote Sensing 15, no. 9: 2391. https://doi.org/10.3390/rs15092391