Off-Line Evaluation of Mobile-Centric Indoor Positioning Systems: The Experiences from the 2017 IPIN Competition
, Antonio R. Jiménez 2,†,*
, Adriano Moreira 3,†,*
, Tomás Lungenstrass 4,†,*
, Wei-Chung Lu 5,†, Stefan Knauth 6,†,*
, Germán Martín Mendoza-Silva 1, Fernando Seco 2, Antoni Pérez-Navarro 7,8, Maria João Nicolau 3, António Costa 3, Filipe Meneses 9, Joaquín Farina 4, Juan Pablo Morales 4, Wen-Chen Lu 5, Ho-Ti Cheng 5, Shi-Shen Yang 10, Shih-Hau Fang 5,*
, Ying-Ren Chien 10,*
and Yu Tsao 11
Abstract
The development of indoor positioning solutions using smartphones is a growing activity with an enormous potential for everyday life and professional applications. The research activities on this topic concentrate on the development of new positioning solutions that are tested in specific environments under their own evaluation metrics. To explore the real positioning quality of smartphone-based solutions and their capabilities for seamlessly adapting to different scenarios, it is needed to find fair evaluation frameworks. The design of competitions using extensive pre-recorded datasets is a valid way to generate open data for comparing the different solutions created by research teams. In this paper, we discuss the details of the 2017 IPIN indoor localization competition, the different datasets created, the teams participating in the event, and the results they obtained. We compare these results with other competition-based approaches (Microsoft and Perf-loc) and on-line evaluation web sites. The lessons learned by organising these competitions and the benefits for the community are addressed along the paper. Our analysis paves the way for future developments on the standardization of evaluations and for creating a widely-adopted benchmark strategy for researchers and companies in the field. View Full-Text
Figure 1
Supplementary materials
- Supplementary File 1:
Supplementary 1 (ZIP, 96983 KB)
- Supplementary File 2:
Supplementary 2 (ZIP, 41908 KB)
- Supplementary File 3:
Supplementary 3 (ZIP, 25940 KB)
Share & Cite This Article
Torres-Sospedra, J.; Jiménez, A.R.; Moreira, A.; Lungenstrass, T.; Lu, W.-C.; Knauth, S.; Mendoza-Silva, G.M.; Seco, F.; Pérez-Navarro, A.; Nicolau, M.J.; Costa, A.; Meneses, F.; Farina, J.; Morales, J.P.; Lu, W.-C.; Cheng, H.-T.; Yang, S.-S.; Fang, S.-H.; Chien, Y.-R.; Tsao, Y. Off-Line Evaluation of Mobile-Centric Indoor Positioning Systems: The Experiences from the 2017 IPIN Competition. Sensors 2018, 18, 487.
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.