Cross-View Outdoor Localization in Augmented Reality by Fusing Map and Satellite Data
Abstract
:1. Introduction
2. Related Work
2.1. Localization without Deep Learning
2.2. Large-Scale Localization with Deep Learning
2.3. Small-Scale Localization with Deep Learning
3. Methods
3.1. Data Overview
OpenStreetMap Extraction
3.2. Model Overview
Feature Extractors
3.3. Top-Level to Ground-Level Projection
3.4. Levenberg–Marquardt Optimization
3.5. Starting Position
3.6. Loss Functions
4. Results
4.1. Setup
4.2. Results
4.3. Analysis of Model
Model Behavior over Iterations
4.4. Ablation Studies
4.4.1. Importance of Ground-Level Data for Alignment
4.4.2. OSM Ablation
4.4.3. Contribution Ablation
5. Discussion and Conclusions
Author Contributions
Funding
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Sattler, T.; Leibe, B.; Kobbelt, L. Fast image-based localization using direct 2D-to-3D matching. In Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 667–674. [Google Scholar] [CrossRef]
- Li, Y.; Snavely, N.; Huttenlocher, D.; Fua, P. Worldwide pose estimation using 3d point clouds. In Proceedings of the European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2012; pp. 15–29. [Google Scholar]
- Zeisl, B.; Sattler, T.; Pollefeys, M. Camera Pose Voting for Large-Scale Image-Based Localization. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 2704–2712. [Google Scholar] [CrossRef]
- Sattler, T.; Leibe, B.; Kobbelt, L. Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1744–1756. [Google Scholar] [CrossRef] [PubMed]
- Zamir, A.R.; Shah, M. Accurate Image Localization Based on Google Maps Street View. In Proceedings of the Computer Vision—ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5–11, 2010, Proceedings, Part IV 11; Daniilidis, K., Maragos, P., Paragios, N., Eds.; Springer: Berlin/Heidelberg, Germany, 2010; pp. 255–268. [Google Scholar]
- Salarian, M.; Manavella, A.; Ansari, R. Accurate localization in dense urban area using Google street view images. In Proceedings of the 2015 SAI Intelligent Systems Conference (IntelliSys), London, UK, 10–11 November 2015; pp. 485–490. [Google Scholar] [CrossRef]
- Heinly, J.; Schonberger, J.L.; Dunn, E.; Frahm, J.M. Reconstructing the World* in Six Days *(As Captured by the Yahoo 100 Million Image Dataset). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015. [Google Scholar]
- Schonberger, J.L.; Frahm, J.M. Structure-From-Motion Revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar]
- Reinhardt, T. Using Global Localization to Improve Navigation, 2019. Available online: https://ai.googleblog.com/2019/02/using-global-localization-to-improve.html (accessed on 14 May 2023).
- Fervers, F.; Bullinger, S.; Bodensteiner, C.; Arens, M.; Stiefelhagen, R. Uncertainty-aware Vision-based Metric Cross-view Geolocalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023, Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar] [CrossRef]
- Lentsch, T.; Xia, Z.; Caesar, H.; Kooij, J.F.P. SliceMatch: Geometry-guided Aggregation for Cross-View Pose Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023, Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar] [CrossRef]
- Sarlin, P.E.; DeTone, D.; Yang, T.Y.; Avetisyan, A.; Straub, J.; Malisiewicz, T.; Bulo, S.R.; Newcombe, R.; Kontschieder, P.; Balntas, V. OrienterNet: Visual Localization in 2D Public Maps with Neural Matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023, Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar]
- Shi, Y.; Li, H. Beyond Cross-view Image Retrieval: Highly Accurate Vehicle Localization Using Satellite Image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 17010–17020. [Google Scholar]
- Shi, Y.; Yu, X.; Campbell, D.; Li, H. Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020, Seattle, WA, USA, 13–19 June 2020. [Google Scholar]
- Wang, S.; Zhang, Y.; Li, H. Satellite Image Based Cross-view Localization for Autonomous Vehicle. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK, 29 May–2 June 2023. [Google Scholar]
- Xia, Z.; Booij, O.; Kooij, J.F.P. Convolutional Cross-View Pose Estimation. arXiv 2023, arXiv:2303.05915. [Google Scholar]
- Xia, Z.; Booij, O.; Manfredi, M.; Kooij, J.F. Visual Cross-View Metric Localization with Dense Uncertainty Estimates. In Proceedings of the European Conference on Computer Vision; Springer Nature: Cham, Switzerland, 2022; pp. 90–106. [Google Scholar]
- Zhu, S.; Shah, M.; Chen, C. TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022. [Google Scholar] [CrossRef]
- Zhu, S.; Yang, T.; Chen, C. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 3640–3649. [Google Scholar]
- Levenberg, K. A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 1944, 2, 164–168. [Google Scholar] [CrossRef]
- Castaldo, F.; Zamir, A.; Angst, R.; Palmieri, F.; Savarese, S. Semantic Cross-View Matching. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, Santiago, Chile, 7–13 December 2015. [Google Scholar]
- Lin, T.Y.; Belongie, S.; Hays, J. Cross-View Image Geolocalization. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 891–898. [Google Scholar] [CrossRef]
- Mousavian, A.; Kosecka, J. Semantic Image Based Geolocation Given a Map. arXiv 2016, arXiv:1609.00278. [Google Scholar]
- Workman, S.; Jacobs, N. On the location dependence of convolutional neural network features. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, 1–12 June 2015; pp. 70–78. [Google Scholar] [CrossRef]
- Downes, L.M.; Kim, D.K.; Steiner, T.J.; How, J.P. City-wide street-to-satellite image geolocalization of a mobile ground agent. In Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 23–27 October 2022; pp. 11102–11108. [Google Scholar]
- Downes, L.M.; Steiner, T.J.; Russell, R.L.; How, J.P. Wide-Area Geolocalization with a Limited Field of View Camera. arXiv 2022, arXiv:2209.11854. [Google Scholar]
- Hu, S.; Lee, G.H. Image-based geo-localization using satellite imagery. Int. J. Comput. Vis. 2020, 128, 1205–1219. [Google Scholar] [CrossRef]
- Kim, D.K.; Walter, M.R. Satellite image-based localization via learned embeddings. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017; pp. 2073–2080. [Google Scholar]
- Liu, L.; Li, H. Lending orientation to neural networks for cross-view geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 5624–5633. [Google Scholar]
- Cai, S.; Guo, Y.; Khan, S.; Hu, J.; Wen, G. Ground-to-aerial image geo-localization with a hard exemplar reweighting triplet loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 8391–8400. [Google Scholar]
- OpenStreetMap Contributors. 2017. Available online: https://planet.osm.org (accessed on 10 June 2023).
- Geiger, A.; Lenz, P.; Stiller, C.; Urtasun, R. Vision meets robotics: The KITTI dataset. Int. J. Robot. Res. 2013, 32, 1231–1237. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18; Springer International Publishing: Cham, Switzerland, 2015; pp. 234–241. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2015, arXiv:1409.1556. [Google Scholar]
Name | Summary |
---|---|
center | Start with all positional variables set to 0 |
lon - 1 | Longitudinal is set to −1, rest to 0 |
Name | Sat | OSM | Cross-Area | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Lateral | Longitudinal | Azimuth | |||||||||
d = 1 | d = 3 | d = 5 | d = 1 | d = 3 | d = 5 | = 1 | = 3 | = 5 | |||
LM [13] | ✓ | ✗ | 27.82 | 59.79 | 72.89 | 5.75 | 16.36 | 26.48 | 18.42 | 49.72 | 71.00 |
SliceMatch [11] | ✓ | ✗ | 32.43 | 78.98 | 86.44 | 8.30 | 24.48 | 35.57 | 46.82 | 46.82 | 46.82 |
CCVPE [16] | ✓ | ✗ | 44.06 | 81.72 | 90.23 | 23.08 | 52.85 | 64.31 | 57.72 | 92.34 | 96.19 |
OrienterNet (a) [12] | ✗ | ✓ | 51.26 | 84.77 | 91.81 | 22.39 | 46.79 | 57.81 | 20.41 | 52.24 | 73.53 |
AlignNet (b) | ✓ | ✓ | 56.62 | 78.32 | 81.62 | 22.42 | 47.31 | 57.61 | 57.57 | 92.54 | 98.97 |
Name | Cross-Area | ||||||||
---|---|---|---|---|---|---|---|---|---|
Lateral | Longitudinal | Azimuth | |||||||
d = 1 | d = 3 | d = 5 | d = 1 | d = 3 | d = 5 | = 1 | = 3 | = 5 | |
original | 59.96 | 83.81 | 87.24 | 22.25 | 48.28 | 58.13 | 60.94 | 95.90 | 99.48 |
switched | 59.39 | 83.78 | 87.32 | 19.50 | 45.83 | 57.52 | 57.49 | 94.93 | 99.35 |
Name | Sat | OSM | Start | Cross-Area | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Lateral | Longitudinal | Azimuth | ||||||||||
d = 1 | d = 3 | d = 5 | d = 1 | d = 3 | d = 5 | = 1 | = 3 | = 5 | ||||
Ours (a) | ✓ | ✓ | lon - 1 | 56.62 | 78.32 | 81.62 | 22.42 | 47.31 | 57.61 | 57.57 | 92.54 | 98.97 |
✓ | only nodes | lon - 1 | 17.50 | 37.39 | 46.43 | 9.10 | 24.26 | 34.83 | 27.41 | 71.88 | 96.67 | |
✓ | only lines | lon - 1 | 58.71 | 79.66 | 83.49 | 19.89 | 45.32 | 55.33 | 57.96 | 92.85 | 99.00 | |
✓ | only areas | lon - 1 | 15.79 | 35.71 | 45.17 | 9.25 | 24.53 | 35.04 | 26.80 | 72.57 | 96.83 | |
✓ | ✗ | lon - 1 | 17.89 | 38.12 | 46.80 | 8.91 | 24.16 | 34.78 | 27.31 | 72.02 | 96.65 | |
AlignNet (b) | ✓ | ✗ | center | 42.96 | 73.97 | 81.20 | 12.56 | 32.47 | 43.07 | 58.29 | 96.90 | 99.67 |
✓ | ✗ | lon - 1 | 40.71 | 69.54 | 76.08 | 5.30 | 15.08 | 23.15 | 58.13 | 92.95 | 97.71 | |
✓ | ✓ | center | 50.13 | 78.61 | 84.01 | 7.12 | 20.21 | 31.66 | 49.70 | 78.71 | 87.58 | |
✗ | ✓ | center | 52.82 | 80.52 | 85.11 | 10.30 | 28.48 | 42.35 | 44.84 | 75.44 | 85.93 | |
✗ | ✓ | lon - 1 | 58.82 | 81.08 | 84.46 | 14.88 | 36.77 | 49.18 | 49.87 | 77.26 | 86.38 |
Name | Cross-Area | ||||||||
---|---|---|---|---|---|---|---|---|---|
Lateral | Longitudinal | Azimuth | |||||||
d = 1 | d = 3 | d = 5 | d = 1 | d = 3 | d = 5 | = 1 | = 3 | = 5 | |
LM [13] | 27.82 | 59.79 | 72.89 | 5.75 | 16.36 | 26.48 | 18.42 | 49.72 | 71.00 |
+train 21 epochs | 40.81 | 75.38 | 83.53 | 5.42 | 17.14 | 28.08 | 25.40 | 61.03 | 78.87 |
+bigger U-Net | 41.08 | 71.69 | 79.65 | 11.58 | 30.05 | 44.59 | 64.27 | 98.46 | 99.91 |
+OSM | 50.13 | 78.61 | 84.01 | 7.12 | 20.21 | 31.66 | 49.70 | 78.71 | 87.58 |
+start at lon - 1 | 56.62 | 78.32 | 81.62 | 22.42 | 47.31 | 57.61 | 57.57 | 92.54 | 98.97 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Emmaneel, R.; Oswald, M.R.; de Haan, S.; Datcu, D. Cross-View Outdoor Localization in Augmented Reality by Fusing Map and Satellite Data. Appl. Sci. 2023, 13, 11215. https://doi.org/10.3390/app132011215
Emmaneel R, Oswald MR, de Haan S, Datcu D. Cross-View Outdoor Localization in Augmented Reality by Fusing Map and Satellite Data. Applied Sciences. 2023; 13(20):11215. https://doi.org/10.3390/app132011215
Chicago/Turabian StyleEmmaneel, René, Martin R. Oswald, Sjoerd de Haan, and Dragos Datcu. 2023. "Cross-View Outdoor Localization in Augmented Reality by Fusing Map and Satellite Data" Applied Sciences 13, no. 20: 11215. https://doi.org/10.3390/app132011215
APA StyleEmmaneel, R., Oswald, M. R., de Haan, S., & Datcu, D. (2023). Cross-View Outdoor Localization in Augmented Reality by Fusing Map and Satellite Data. Applied Sciences, 13(20), 11215. https://doi.org/10.3390/app132011215