Improved Feature Matching for Mobile Devices with IMU
Abstract
:1. Introduction
- Feature matching and geometry estimation (Structure from Motion (SfM)). Feature matches are computed in couple of images. These correspondences are used in order to estimate the geometry of the scene [21,22,23,24,25,26] (camera poses and point positions). Geometric constraints are exploited to detect wrong feature matches, if any.
- Increase the number (and/or the percentage) of correct feature matches between two camera views (in particular for wide changes of the camera point of view, Section 2). A number of feature descriptors have been proposed in the literature [17,18,19,20]. Despite the fact that they are typically designed to deal with certain image transformations (e.g., scale changes), sometimes they are not effective in dealing with generic changes of the camera point of view. The Affine Scale-Invariant Feature Transform (ASIFT, or affine SIFT [37,38]) has been recently introduced to improve the SIFT performance in this case. The method presented in Section 2 can be considered as a revision of the ASIFT when prior information from the device orientation is available. The goal considered here is similar to that in [34,36]; however, in [36], device orientation is provided by vanishing points (whereas here, orientation is provided by the IMU) and the method in [36] is designed for buildings. Instead, [34] exploits gravity direction information, whereas relative orientation information is used in Section 2: interestingly, the method considered in [34] can be integrated in the procedure of Section 2 (this aspect will be the subject of future investigation). Since the interest of increasing the number of correct feature matches. Since the net effect of the method presented in Section 2 (increasing the number (and/or percentage) of correct feature matches) is of particular interest when dealing with quite large changes of the point of view, it is also related to the wide–baseline stereo (WBS) matching problem: differently from standard approaches for WBS [39,40,41,42], here the problem is addressed by adding information provided by the IMU about the camera orientation.
- Feature matching with geometry constraints: assuming interior camera parameters as known, then an estimate of the essential matrix (and hence the relative camera pose) between two camera views is computed by means of two feature correspondences (this two-point procedure is the same proposed in [35], with only minor changes). This is employed in a two-step RANdom SAmple Consensus (RANSAC, [43]) procedure in order to make the estimation more robust (Section 3): the goal is that of removing wrong feature matches and correctly estimating the geometry of the scene (i.e., the essential matrix). It is well-known that the number of iterations of the RANSAC algorithm (necessary in order to obtain a good estimate with the desired probability) is often underestimated because only certain subsets of the inliers allow for obtaining a correct estimation [44]. Similarly to the locally optimized RANSAC [44], the procedure presented in Section 2 aims at compensating for this issue by preselecting a candidate inlier subset where the probability of drawing a correct feature match is higher than in the original set of feature matches. However, differently from [44], this preliminary step is done by exploiting information provided by the sensors embedded in the device.
2. Similarity Based Feature Matching
3. Feature Matching with Geometry Constraints: Estimation of the Essential Matrix
3.1. Two-Point Estimation of the Essential Matrix
- if , then , and ;
- , then ;
- , then .
Algorithm 1: Essential matrix RANSAC estimation with the two-point algorithm |
|
3.2. Two-Step Algorithm for the Estimation of the Essential Matrix
Algorithm 2: Two-step algorithm |
|
4. Results
4.1. Similarity Based Feature Matching
- The first case study is a set of 11 images of the veterinary hospital of the University of Padova, Italy (Figure 3a).
- The second case study is a set of 17 images of the Pentagono building of the University of Padova, Italy (Figure 3b).
- The third case study is a set of images downloadable on the Internet from the website of [63] (Figure 3c). Since, in this case, IMU measurements are not available, they have been substituted with orientations computed after matching features in the images and adding to the computed orientation angles a Gaussian random noise with standard deviation 0.1 radiants (100 independent Monte Carlo simulations have been considered in order to provide statistically reliable results).
4.2. Feature Matching with Geometry Constraints: Estimation of the Essential Matrix
- In each iteration of the Monte Carlo simulation, 50 feature points have been considered. Feature positions have been randomly sampled in a 10 m × 10 m × 3 m rectangular cuboid (uniform distribution).
- In each iteration of the Monte Carlo simulation, camera positions have been randomly sampled at a mean distance of 10m from the feature points. Camera orientations are obtained by summing a random Gaussian angle (zero-mean, standard deviation of ) to the direction pointing from the camera position to the cuboid center.
5. Discussion
6. Conclusions
Author Contributions
Conflicts of Interest
References
- Sung, C.; Kim, P.Y. 3D terrain reconstruction of construction sites using a stereo camera. Autom. Constr. 2016, 64, 65–77. [Google Scholar] [CrossRef]
- Yang, M.D.; Chao, C.F.; Huang, K.S.; Lu, L.Y.; Chen, Y.P. Image-based 3D scene reconstruction and exploration in augmented reality. Autom. Constr. 2013, 33, 48–60. [Google Scholar] [CrossRef]
- Tsai, H.P.; Lin, Y.H.; Yang, M.D. Exploring Long Term Spatial Vegetation Trends in Taiwan from AVHRR NDVI3g Dataset Using RDA and HCA Analyses. Remote Sens. 2016, 8, 290. [Google Scholar] [CrossRef]
- Prosdocimi, M.; Calligaro, S.; Sofia, G.; Dalla Fontana, G.; Tarolli, P. Bank erosion in agricultural drainage networks: New challenges from structure-from-motion photogrammetry for post-event analysis. Earth Surf. Process. Landf. 2015, 40, 1891–1906. [Google Scholar] [CrossRef]
- Remondino, F. Heritage recording and 3D modeling with photogrammetry and 3D scanning. Remote Sens. 2011, 3, 1104–1138. [Google Scholar] [CrossRef] [Green Version]
- Tanskanen, P.; Kolev, K.; Meier, L.; Camposeco, F.; Saurer, O.; Pollefeys, M. Live metric 3d reconstruction on mobile phones. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013.
- Hesch, J.A.; Kottas, D.G.; Bowman, S.L.; Roumeliotis, S.I. Camera-IMU-based localization: Observability analysis and consistency improvement. Int. J. Robot. Res. 2013, 33, 182–201. [Google Scholar] [CrossRef]
- Kolev, K.; Tanskanen, P.; Speciale, P.; Pollefeys, M. Turning mobile phones into 3d scanners. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014.
- Klingensmith, M.; Dryanovski, I.; Srinivasa, S.; Xiao, J. Chisel: Real time large scale 3D reconstruction onboard a mobile device. In Proceedings of the Robotics Science and Systems, Rome, Italy, 3 July 2015.
- Schöps, T.; Engel, J.; Cremers, D. Semi-dense visual odometry for AR on a smartphone. In Proceedings of the 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany, 10–12 September 2014.
- Prisacariu, V.A.; Kähler, O.; Murray, D.W.; Reid, I.D. Real-time 3d tracking and reconstruction on mobile phones. IEEE Trans. Vis. Comput. Graph. 2015, 21, 557–570. [Google Scholar] [CrossRef] [PubMed]
- Balletti, C.; Guerra, F.; Tsioukas, V.; Vernier, P. Calibration of Action Cameras for Photogrammetric Purposes. Sensors 2014, 14, 17471–17490. [Google Scholar] [CrossRef] [PubMed]
- Remondino, F.; Barazzetti, L.; Nex, F.; Scaioni, M.; Sarazzi, D. UAV photogTrammetry for mapping and 3d modeling-current status and future perspectives. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2011, 38, C22. [Google Scholar]
- Surovỳ, P.; Yoshimoto, A.; Panagiotidis, D. Accuracy of Reconstruction of the Tree Stem Surface Using Terrestrial Close-Range Photogrammetry. Remote Sens. 2016, 8. [Google Scholar] [CrossRef]
- Figueira, W.; Ferrari, R.; Weatherby, E.; Porter, A.; Hawes, S.; Byrne, M. Accuracy and Precision of Habitat Structural Complexity Metrics Derived from Underwater Photogrammetry. Remote Sens. 2015, 7, 16883–16900. [Google Scholar] [CrossRef]
- Piermattei, L.; Carturan, L.; de Blasi, F.; Tarolli, P.; Dalla Fontana, G.; Vettore, A.; Pfeifer, N. Analysis of glacial and periglacial processes using structure from motion. Earth Surf. Dyn. Discuss. 2015, 3, 1345–1398. [Google Scholar] [CrossRef]
- Lowe, D. Object recognition from local scale-invariant features. In Proceedings of the 1999 IEEE International Conference on Computer Vision (ICCV), Kerkyra, Greece, 20–27 September 1999.
- Bay, H.; Ess, A.; Tuytelaars, T.; Van Gool, L. Speeded-up robust features (SURF). Comput. Vis. Image Underst. 2008, 110, 346–359. [Google Scholar] [CrossRef]
- Leutenegger, S.; Chli, M.; Siegwart, R.Y. BRISK: Binary robust invariant scalable keypoints. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011.
- Alahi, A.; Ortiz, R.; Vandergheynst, P. Freak: Fast retina keypoint. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA, 16–21 June 2012.
- Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
- Ma, Y.; Soatto, S.; Košecká, J.; Sastry, S. An Invitation to 3D Vision; Springer: New York, NY, USA, 2003. [Google Scholar]
- Byröd, M.; Aström, K. Conjugate Gradient Bundle Adjustment. In Computer Vision—ECCV 2010; Lecture Notes in Computer Science; Springer: Berlin, Germany, 2010; Volume 6312, pp. 114–127. [Google Scholar]
- Agarwal, S.; Snavely, N.; Seitz, S.; Szeliski, R. Bundle Adjustment in the Large. In Computer Vision—ECCV 2010; Lecture Notes in Computer Science; Springer: Berlin, Germany, 2010; Volume 6312, pp. 29–42. [Google Scholar]
- Masiero, A.; Guarnieri, A.; Vettore, A.; Pirotti, F. An ISVD-based Euclidian structure from motion for smartphones. ISPRS Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2014, XL-5, 401–406. [Google Scholar] [CrossRef]
- Brand, M. Incremental Singular Value Decomposition of Uncertain Data With Missing Values. In Computer Vision—ECCV 2002; Lecture Notes in Computer Science; Springer: Berlin, Germany, 2002; Volume 2350, pp. 707–720. [Google Scholar]
- Furukawa, Y.; Ponce, J. Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 32, 1362–1376. [Google Scholar] [CrossRef] [PubMed]
- Furukawa, Y.; Curless, B.; Seitz, S.; Szeliski, R. Towards Internet-scale Multi-view Stereo. In Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA, 13–18 June 2010.
- Goesele, M.; Snavely, N.; Curless, B.; Hoppe, H.; Seitz, S. Multi-View Stereo for Community Photo Collections. In Proceedings of the 2007 IEEE International Conference on Computer Vision (ICCV), Rio de Janeiro, Brazil, 14–21 October 2007.
- Hartley, R.; Sturm, P. Triangulation. Comput. Visi. Image Underst. 1997, 68, 146–157. [Google Scholar] [CrossRef]
- Triggs, B.; McLauchlan, P.; Hartley, R.; Fitzgibbon, A. Bundle Adjustment—A modern Synthesis. In Vision Algorithms: Theory and Practice; Springer Lecture Notes on Computer Science; Springer Verlag: Berlin, Germany, 1999; pp. 298–372. [Google Scholar]
- Masiero, A.; Cenedese, A. On triangulation algorithms in large scale camera network systems. In Proceedings of the 2012 American Control Conference, ACC 2012, Montreal, QC, Canada, 27–29 June 2012.
- De Agostino, M.; Manzino, A.M.; Piras, M. Performances comparison of different MEMS-based IMUs. In Proceedings of the 2010 IEEE/ION Position Location and Navigation Symposium (PLANS), Indian Wells, CA, USA, 4–6 May 2010.
- Kurz, D.; Benhimane, S. Gravity-aware handheld augmented reality. In Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, 26–29 October 2011.
- Troiani, C.; Martinelli, A.; Laugier, C.; Scaramuzza, D. 2-point-based outlier rejection for camera-imu systems with applications to micro aerial vehicles. In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–7 June 2014.
- Baatz, G.; Köser, K.; Chen, D.; Grzeszczuk, R.; Pollefeys, M. Handling urban location recognition as a 2d homothetic problem. In Computer Vision—ECCV 2010; European Conference on Computer Vision; Springer: Berlin, Germany, 2010; pp. 266–279. [Google Scholar]
- Morel, J.; Yu, G. ASIFT: A New Framework for Fully Affine Invariant Image Comparison. SIAM J. Imaging Sci. 2009, 2, 438–469. [Google Scholar] [CrossRef]
- Morel, J.; Yu, G. Is SIFT Scale Invariant? Inverse Probl. Imaging 2011, 5, 115–136. [Google Scholar] [CrossRef]
- Pritchett, P.; Zisserman, A. Wide baseline stereo matching. In Proceedings of the Sixth International Conference on Computer Vision, Bombay, India, 4–7 January 1998.
- Tuytelaars, T.; Van Gool, L.J. Wide Baseline Stereo Matching based on Local, Affinely Invariant Regions. In Proceedings of the BMVC, Bristol, UK, 11–14 September 2000.
- Matas, J.; Chum, O.; Urban, M.; Pajdla, T. Robust wide-baseline stereo from maximally stable extremal regions. Image Vis. Comput. 2004, 22, 761–767. [Google Scholar] [CrossRef]
- Tola, E.; Lepetit, V.; Fua, P. Daisy: An efficient dense descriptor applied to wide-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 32, 815–830. [Google Scholar] [CrossRef] [PubMed]
- Fischler, M.; Bolles, R. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 1981, 24, 381–395. [Google Scholar] [CrossRef]
- Chum, O.; Matas, J.; Kittler, J. Locally optimized RANSAC. In Pattern Recognition; Springer: Berlin, Germany, 2003; pp. 236–243. [Google Scholar]
- Saeedi, S.; Moussa, A.; El-Sheimy, N. Context-Aware Personal Navigation Using Embedded Sensor Fusion in Smartphones. Sensors 2014, 14, 5742–5767. [Google Scholar] [CrossRef] [PubMed]
- El-Sheimy, N.; Kai-wei, C.; Noureldin, A. The Utilization of Artificial Neural Networks for Multisensor System Integration in Navigation and Positioning Instruments. IEEE Trans. Instrum. Meas. 2006, 55, 1606–1615. [Google Scholar] [CrossRef]
- Barbarella, M.; Gandolfi, S.; Meffe, A.; Burchi, A. Improvement of an MMS trajectory, in presence of GPS outage, using virtual positions. In Proceedings of the ION GNSS 24th International Technical Meeting of the Satellite Division, Portland, OR, USA, 20–23 September 2011.
- Masiero, A.; Guarnieri, A.; Pirotti, F.; Vettore, A. A Particle Filter for Smartphone-Based Indoor Pedestrian Navigation. Micromachines 2014, 5, 1012–1033. [Google Scholar] [CrossRef]
- Fusiello, A.; Irsara, L. Quasi-Euclidean epipolar rectification of uncalibrated images. Mach. Vis. Appl. 2011, 22, 663–670. [Google Scholar] [CrossRef] [Green Version]
- Heyden, A.; Pollefeys, M. Multiple view geometry. In Emerging Topics in Computer Vision; Medioni, G., Kang, S.B., Eds.; Prentice Hall: Upper Saddle River, NJ, USA, 2005; pp. 45–107. [Google Scholar]
- Brown, D. Close-range camera calibration. Photogramm. Eng. 1971, 37, 855–866. [Google Scholar]
- Heikkila, J. Geometric Camera Calibration Using Circular Control Points. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1066–1077. [Google Scholar] [CrossRef]
- Zhang, Z. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of the 1999 IEEE International Conference on Computer Vision (ICCV), Kerkyra, Greece, 20–27 September 1999.
- Fraser, C.S. Digital camera self-calibration. ISPRS J. Photogramm. Remote Sens. 1997, 52, 149–159. [Google Scholar] [CrossRef]
- Remondino, F.; Fraser, C. Digital camera calibration methods: Considerations and comparisons. ISPRS Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2006, XXXVI-5, 266–272. [Google Scholar]
- Habib, A.; Morgan, M. Automatic calibration of low-cost digital cameras. Opt. Eng. 2003, 42, 948–955. [Google Scholar]
- Karel, W.; Pfeifer, N. Range camera calibration based on image sequences and dense comprehensive error statistics. In Proceedings of the IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, San Jose, CA, USA, 19 January 2009.
- Luhmann, T.; Fraser, C.; Maas, H.G. Sensor modelling and camera calibration for close-range photogrammetry. ISPRS J. Photogramm. Remote Sens. 2015, 115, 37–46. [Google Scholar] [CrossRef]
- Beis, J.S.; Lowe, D.G. Shape indexing using approximate nearest-neighbour search in high-dimensional spaces. In Proceedings of the 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, 17–19 June 1997.
- Nistér, D. An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 2004, 26, 756–770. [Google Scholar] [CrossRef] [PubMed]
- Longuet-Higgins, H. A computer algorithm for reconstructing a scene from two projections. Nature 1981, 293, 133–135. [Google Scholar] [CrossRef]
- Hartley, R. In Defense of the Eight-Point Algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 1997, 19, 580–593. [Google Scholar] [CrossRef]
- Lhuillier, M.; Quan, L. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 418–433. [Google Scholar] [CrossRef] [PubMed]
- Vedaldi, A.; Fulkerson, B. VLFeat: An Open and Portable Library of Computer Vision Algorithms. Available online: http://www.vlfeat.org/ (accessed on 2 August 2016).
- Hol, J. Sensor Fusion and Calibration of Inertial Sensors, Vision, Ultra-Wideband and GPS. Ph.D. Thesis, The Institute of Technology, Linköping University, Linköping, Sweden, 2011. [Google Scholar]
- Syed, Z.; Aggarwal, P.; Goodall, C.; Niu, X.; El-Sheimy, N. A new multi-position calibration method for MEMS inertial navigation systems. Meas. Sci. Technol. 2007, 18, 1897. [Google Scholar] [CrossRef]
Noise Level | Five-Point Algorithm | Two-Point Algorithm | Two-Step Algorithm |
---|---|---|---|
0.33 | 0.22 | 0.26 | |
0.18 | 0.21 | 0.15 | |
0.15 | 0.41 | 0.18 |
© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Masiero, A.; Vettore, A. Improved Feature Matching for Mobile Devices with IMU. Sensors 2016, 16, 1243. https://doi.org/10.3390/s16081243
Masiero A, Vettore A. Improved Feature Matching for Mobile Devices with IMU. Sensors. 2016; 16(8):1243. https://doi.org/10.3390/s16081243
Chicago/Turabian StyleMasiero, Andrea, and Antonio Vettore. 2016. "Improved Feature Matching for Mobile Devices with IMU" Sensors 16, no. 8: 1243. https://doi.org/10.3390/s16081243
APA StyleMasiero, A., & Vettore, A. (2016). Improved Feature Matching for Mobile Devices with IMU. Sensors, 16(8), 1243. https://doi.org/10.3390/s16081243