Next Article in Journal
Control Design and Digital Implementation of a Fast 2-Degree-of-Freedom Translational Optical Image Stabilizer for Image Sensors in Mobile Camera Phones
Next Article in Special Issue
An Improved Indoor Positioning System Using RGB-D Cameras and Wireless Networks for Use in Complex Environments
Previous Article in Journal
Etched Polymer Fibre Bragg Gratings and Their Biomedical Sensing Applications
Previous Article in Special Issue
On Efficient Deployment of Wireless Sensors for Coverage and Connectivity in Constrained 3D Space
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Comparing the Performance of Indoor Localization Systems through the EvAAL Framework

1
ISTI Institute of CNR, Pisa 56124, Italy
2
Electronics and Telecommunications Research Institute (ETRI), Daejeon 34129, Korea
3
Centre for Automation and Robotics, CSIC-UPM, Arganda del Rey 28500, Spain
4
Institute of New Imaging Technologies, Universitat Jaume I, Castellón de la Plana 12071, Spain
5
University of Alcalá, Department of Electronics, Alcalá de Henares 28871, Spain
*
Author to whom correspondence should be addressed.
Sensors 2017, 17(10), 2327; https://doi.org/10.3390/s17102327
Received: 31 August 2017 / Revised: 27 September 2017 / Accepted: 10 October 2017 / Published: 13 October 2017

Abstract

:
In recent years, indoor localization systems have been the object of significant research activity and of growing interest for their great expected social impact and their impressive business potential. Application areas include tracking and navigation, activity monitoring, personalized advertising, Active and Assisted Living (AAL), traceability, Internet of Things (IoT) networks, and Home-land Security. In spite of the numerous research advances and the great industrial interest, no canned solutions have yet been defined. The diversity and heterogeneity of applications, scenarios, sensor and user requirements, make it difficult to create uniform solutions. From that diverse reality, a main problem is derived that consists in the lack of a consensus both in terms of the metrics and the procedures used to measure the performance of the different indoor localization and navigation proposals. This paper introduces the general lines of the EvAAL benchmarking framework, which is aimed at a fair comparison of indoor positioning systems through a challenging competition under complex, realistic conditions. To evaluate the framework capabilities, we show how it was used in the 2016 Indoor Positioning and Indoor Navigation (IPIN) Competition. The 2016 IPIN competition considered three different scenario dimensions, with a variety of use cases: (1) pedestrian versus robotic navigation, (2) smartphones versus custom hardware usage and (3) real-time positioning versus off-line post-processing. A total of four competition tracks were evaluated under the same EvAAL benchmark framework in order to validate its potential to become a standard for evaluating indoor localization solutions. The experience gained during the competition and feedback from track organizers and competitors showed that the EvAAL framework is flexible enough to successfully fit the very different tracks and appears adequate to compare indoor positioning systems.

1. Introduction

In this new digital era, where location-based services are essential in many applications and the number of devices with Internet connectivity has increased dramatically, research on the topic of indoor positioning is flourishing. In contrast to outdoor positioning and navigation, where Global Navigation Satellite System (GNSS) is a mature technology and might be considered a de facto standard, indoor positioning is still in its infancy. The main reason is that GNSS is widespread and is able to provide robust solutions for many applications in a wide range of performance/price ratios, while indoor positioning solutions are few, each specialised for a given scenario and generally expensive.
Many different technologies are being considered for indoor positioning, including Inertial Sensors [1,2,3,4,5], Radio Frequency using RFID, BLE, WiFi or UWB ranging [6,7], Ultrasound [8,9,10], Computer Vision [11], and Light [12,13,14,15]. This diversity in technologies has a significant consequence: there is no generally accepted solution for indoor positioning and navigation. In other words, there is not an indoor equivalent to the GNSS-based portable navigation devices commonly used outdoors. Nevertheless, there are already many companies offering indoor positioning solutions and the forecasts estimate an indoor positioning market of more than USD 20 billion by 2021 [16].
Although the expected commercial impact of indoor location solutions is very high, there is still a lack of standardization in the design, application areas, or evaluation metrics for a fair comparison and assessment of the most promising proposals, both in the literature and at the commercial level. As an example, most of the existing works in the literature show the results of the experiments that were carried out in the authors’ own research facilities, typically department corridors and small research laboratories, which were selected as representative real-world scenarios. Those facilities are not commonly open to other researchers to reproduce their experiments in them. In general, comparing existing working systems is not a trivial task if they have used different technologies, have been evaluated in different scenarios or their performance metrics are not equivalent.
Despite the evaluation heterogeneity found in the literature, some surveys can be found which try to give a consistent overview in this field [17,18,19]. However, most of them show the results as provided by developers, which often do not correspond to a common metric, meaning that an exhaustive comparison cannot be performed.
The research community feels the need for accepted methods of measuring the performance of the proposed indoor positioning systems, devices and technologies [1,18,20,21,22]. In fact, one can observe rising interest in providing open-access datasets, proposing benchmarking tools and, even, organizing competitions in order to facilitate comparative studies.
The EvAAL framework [21] has already been shown to be a consistent way for measuring the performance of personal navigation-oriented positioning systems in indoor environments with known characteristics and, therefore, for rigorously comparing indoor localization systems [21,23,24,25]. These studies back its ambition of becoming a standard in this field.
Although this framework originally aimed at establishing benchmarks and evaluation metrics for comparing Active and Assisted Living solutions, positioning accuracy was an important evaluation criterion. The 2016 Indoor Positioning and Indoor Navigation (IPIN) conference integrated the EvAAL community and framework in the IPIN Competition to compare indoor positioning systems due to their solid experience in organizing competitions. Specifically, three different scenarios were considered in the IPIN 2016 Competition: positioning of people in real-time, positioning of people off-line and robotic positioning. Despite the fact that the robotic positioning might not be related to AAL, it is of interest for the indoor positioning navigation community and it was integrated into the IPIN competition. Therefore, the competition considered different dimensions in the scenario space: pedestrian versus robotic navigation; on-site versus off-site evaluation; usage of smartphones versus customised sensing platforms; infrastructure-based versus infrastructure-free environments. The IPIN community used a well-known evaluation framework in its competition, whereas the EvAAL community had the opportunity to apply their framework in a challenging competition and in other contexts (e.g, the robotic track).
The main objective of this paper is twofold: on the one hand, this paper studies the diversity and heterogeneity of location solutions (different environments, sizes, sensor data and databases, used hardware, etc.) and a review of the different approaches used to evaluate them as proposed by the authors, or by international competitions. On the other hand, this paper aims at demonstrating how the EvAAL benchmarking framework [21] can be used for multiple scenarios and data streams under the large repertory of evaluation tracks at the 2016 IPIN Competition (variety in person/robots, smartphone/custom hardware, multi-storey building, on-line/off-line processing, etc.). We want to demonstrate that a unique evaluation framework, such as EvAAL, can be fitted to many different indoor location use cases.
The remainder of this paper is organized as follows. Section 2 introduces the evaluation heterogeneity found in the literature and the related work aimed at fair benchmarking and comparisons. Section 3 generally describes how the EvAAL criteria were applied to the IPIN 2016 Competition. Section 4 describes the features of the individual competition tracks and shows the results obtained by competitors. Section 5 introduces some lessons learned from the IPIN 2016 competition and the consequent conclusions.

2. Diversity Problem Review

This section is devoted to showing the different problems that arise when trying to compare different results from a large diversity of scenarios, different sensor data streams and non-standardized evaluation methods. The next subsections will present the diversity or heterogeneity problem when presenting research location solutions, public databases for testing new algorithms, and the problem related to the diversity of competition conditions and set-ups.

2.1. Diversity in Research Systems and Metrics

Some criteria to evaluate Indoor Positioning Systems (IPSs) for personal networks were proposed in [18], for example, privacy, cost, performance and robustness. The authors of [18] observed that the two main performance parameters are the accuracy and the precision, where the former is related to the geometric error and the latter was defined as the success of the position estimations with respect to a predefined accuracy (e.g., the space-based location or the percentage of error in positioning below a threshold). In general, the evaluation metrics depend on the author.
RADAR [20], the first Wi-Fi based indoor positioning system, used the quartile values of the error in positioning (defined as the Euclidean distance between the actual and estimated positions) in order to compare the proposed method to other naïve solutions in a basic analysis. The experimental testbed was located on the second floor of a 3-storey building, with an area of 43.5 m by 22.5 m, more than 50 rooms and three Wi-Fi Access Points (APs). HORUS [26], another well-known IPS developed in 2003 by Youseff et al., provided the median error in positioning and it was tested in two environments: an area of 68 m by 26 m with 172 locations and 21 APs, and an area of 36 m by 12 m with 110 locations and six APs. The two testing approaches are quite different because the density of APs was 0.003 APs/m2 in RADAR, whereas HORUS provided two scenarios with densities of 0.012 APs/m2 and 0.14 APs/m2 respectively, i.e., the scenarios where HORUS was tested had approximately four times the AP density of the evaluation testbed of RADAR. Therefore, comparing the results provided in the original references might not be fair.
In order to provide a fair comparison of RADAR (the deterministic method developed by Microsoft) and HORUS (the probabilistic technique developed by the University of Maryland), they were both implemented and evaluated using the same testbeds [26,27]. According to the data provided in [26,27,28], the density of access points in the testbed used in [26], a university department corridor, was higher than in [27,28]. This change might be due to an improvement of the Wi-Fi network. Despite this minor change in one of the evaluation testbeds, the work done in [26,27] showed that a fair comparison requires the use of the same testbed, or testbeds, rather than reusing the results provided in the literature.
Apart from the diversity of environments, there is also diversity in the hardware elements used for localization and in the metrics to evaluate an IPS. A brief resume of a few works presented in the 2016 Indoor Positioning and Indoor Navigation conference (October 2016, Alcalá de Henares, Spain) is shown in Table 1, where the conference session, base technology, evaluation set-up and evaluation metric are also shown.
According to Table 1, each research work uses specific hardware even when they use the same base technology for positioning. A Huawei Mate smartphone was used in [37], six different devices were used in [38], and a simulation was carried out in [41]. Using multiple devices in the set-up might be more challenging than using a single device or simulated data. Therefore, comparing the results of those works might not be easy at all.
Although the vast majority of papers agree that the error in positioning is defined as the shortest distance between the estimate and current position, consensus in using a particular metric to evaluate the IPS has not been reached yet, as shown in Table 1. This table shows that many metrics have been used to evaluate a IPS, but there is no clear winner metric used in all papers. This conclusion is in line with the study where 195 papers of the first edition of the Indoor Positioning and Indoor Navigation (IPIN 2010) Conference were analysed [21].
For instance, the results of [29,33] are based on the final drift (the geometric error at the end of the track) and visual analysis of the track. The work presented in [46] reports the average error per axis and angle, and the positioning error is not computed. The work presented in [40] shows some plots with measured probability and cumulative distribution instead of providing simple evaluation numbers. Estimated versus real trajectory are the results shown in [34].
Moreover, the scenarios found in the literature are very diverse: from a small 1 m2 area or a simulated 3 m by 3 m scenario to a large university hospital or a set of tracks with about 60 accumulated kilometres. Even for the same topic and base technology, the evaluation metrics and the environment depend on the author.
This diversity of evaluation set-ups is not only attached to the IPIN conference; it is a generalized problem in the field of indoor positioning:
  • GETA Sandals [55,56] have been evaluated in a few scenarios (straight track, rectangular loop, and climbing stairs) and the results have been provided as the average drift (positioning error at the end of the track) of many users and by plotting the walking distance vs. the positioning error.
  • MoteTrack [57,58] evaluation was focused on understanding how it performed under various parameter conditions and the authors carried out different tests. The results were provided as CDFs and plots based on the error distance and a few percentile values.
  • Out of Sight [59] is a toolkit for tracking occluded human joint positions based on Kinect cameras. Some in-room test were run for evaluating different contexts (stationary, stepping, walking, presence of obstacle and oclusion). The mean error on the three axes (x, y, z) and the mean positioning error were provided.
  • A Kalman filtering-based localization and tracking for the IoT paradigm was proposed in [60], where a simulation in a 1000 m by 1000 m sensor field was performed. The results and comparisons were based on the trajectory plots, and the location and velocity errors in the x and y axes over time (track).
  • A smartphone-based tracking system using Hidden Markov Model pattern recognition was developed in [22]. They carried out their experiments in the New Library of Wuhan University using three different device models with over 50 subjects walking over an aggregate distance of over 40 km. The CDF and mean accuracy were used to report the results.
The evaluation heterogeneity present in the literature does not mean that the results shown are not valid; it just makes it more difficult to directly compare the proposed methods with just the information and experiments provided by the authors. The rest of this section will show some initiatives and frameworks for fair evaluation of IPS.

2.2. Diversity in the Results Reported in Surveys

Many surveys show comparative studies about indoor positioning technologies (e.g., [17,18,19,61,62,63]). The reported accuracy is often generic and sometimes is reported just as, e.g., ’room level accuracy’, or 50% within around 2.5 m or as a simple range (2 to 5 m). In general, surveys discuss the general accuracy of positioning technologies and the future trends rather than trying to rank the analysed solutions.
Liu et al. [17] introduced a survey about indoor positioning algorithms (angulation, lateration, scene analysis and proximity), performance metrics (accuracy, precision, complexity, robustness, scalability and cost) and technology solutions (GPS, RFID, WLAN, Bluetooth, UWB, cellular).
Gu et al. [18] gave a comprehensive survey of some commercial solutions and research prototypes, which were compared by using many different features: security and privacy, cost, performance, robustness, complexity, user preferences. Trade-offs among the features and the viewpoint of a user in a personal network were also outlined.
Ficco et al. [19] introduced a novel hybrid positioning system. They included an interesting related-work section where 29 different systems, with different base technologies and accuracy ranges, were surveyed.
Xiao et al. [61] offered a comprehensive state-of-the-art survey on wireless indoor localization technologies from the device perspective (device based vs. device-free) in terms of accuracy, cost, scalability and energy efficiency.
He and Chan [62] introduced a survey of Wi-Fi fingerprinting methods, which described advances in two major areas: advanced localization techniques and efficient system deployment. Different methods exploiting spatial and temporal signal patterns were compared according to indoor site availability, additional information for localization estimation, limitations, and reported mean accuracy. Similarly, recent approaches for motion-assisted Wi-Fi localization and typical schemes of collaborative localization were also compared.
Hassan et al. [63] performed a survey on indoor positioning using visible LED light. In a first comparison, they provided general information about other positioning technologies (Wi-Fi, BLE, GSM, among others): accuracy, robustness, complexity, cost and infrastructure reusability. Even for the surveyed LED solutions, they provided an accuracy range in most of the cases.
In general, surveys have one common feature, that is, the IPS are compared using the performance as provided by the authors themselves. Moreover, the metric used to compare accuracy might differ, e.g., the performance of RADAR is 3–5 m in [19], 4 m with 50% probability in [18], and 3–5 m (2.5 with 50% of probability and 5.9 with 90% of probability) in [17]. Results shown in survey works are useful to show the differences between different indoor location technologies, but comparing methods relying on the same technology might not be easy in some cases.

2.3. Diversity in Datasets

In the machine learning and pattern recognition community, it is common to test new algorithms over a set of well-known datasets. Researchers tend to upload their collected datasets, and the employed software in some cases, to a database platform or to their own website.
While there is no equivalent platform for indoor positioning yet, you can find numerous public data online.
The Community Resource for Archiving Wireless Data At Dartmouth (CRAWDAD) platform hosts data sets about wireless data including data sets for indoor positioning and mobility. Some illustrative data sets follow:
  • A Database related to a Wi-Fi-based positioning system on the second floor of an office building on the campus of the University of Mannheim, released in [64]. The operation area was about 57 m × 32 m but only 221 square meters were covered. The test environment had twelve APs. Seven of them were administered by the university technicians, whereas the others were installed in the nearby buildings and offices. Thirteen additional access points were added for localization purposes.
  • Mobility traces at five different sites (NCSU university campus, KAIST university campus, New York City, Disney World -Orlando- and North Carolina state fair) [65].
  • Two trace files (one for Wi-Fi and one for Bluetooth) collected by the University of Illinois Movement (UIM) framework using Google Android phones [66].
  • One data set to aid the development and evaluation of indoor location in complex indoor environments using round-trip time-of-flight (RToF) and magnetometer measurements [67]. It contains RToF and magnetometer measures taken in the 26 m × 24 m New Wing Yuan supermarket in Sunnyvale, CA, USA. The data was collected during working hours over a period of 15 days.
  • The database donated in [68] contains the RSS (Radio Signal Strength) data collected with a mobile robot in two environments: indoor (KTH) and outdoor (Dortmund). The RSS metric was used to collect the RSS data in terms of dBm. The mobile robot location was recorded using odometry (dead reckoning).
The server at the University of California Irvine is a well-known repository of databases for machine learning and pattern recognition problems, the UCI Machine Learning Repository [69], where a few datasets for indoor positioning can be found:
  • The UJIIndoorLoc database is a Wi-Fi fingerprinting database collected at three buildings of the Jaume I university (UJI) campus for indoor navigation purposes. It was collected by means of more than 20 devices and 20 people [70] and was used for the off-site track of the 2015 EvAAL-ETRI Competition [25,71].
  • The UJIIndoorLoc-Mag database was a magnetic field-based database collected at one laboratory at the Jaume I university (UJI) campus for indoor navigation purposes [72,73].
  • The Indoor User Movement Prediction from RSS Data Set represents a real-life benchmark in the area of Active and Assisted Living applications. The database introduces a binary classification task, which consists in predicting the pattern of user movements in real-world office environments from time-series generated by a Wireless Sensor Network (WSN) [74].
  • The Geo-Magnetic field and WLAN dataset for indoor localisation from wristband and smartphone data set contains Wi-Fi and magnetic filed fingerprints, together with inertial sensor data during two campaigns performed in the same environment [75].
The new trend is to publish the databases on the research group website or, even, on dedicated web platforms.
  • The Signal processing for the wireless positioning group (Tampere University of Technology, Finland), provides an open repository of source software and measurement data on their website (http://www.cs.tut.fi/tlt/pos/Software.htm). Indoor WLAN measurement data in two four-floor buildings for indoor positioning studies are provided. The data contains the collected RSS values, the Access Points ID (mapped to integer indices) and the coordinates; both the training data and several tracks for the estimation part are provided as indicated in [76,77]. Other datasets with GSM, UMTS and GNSS data are also provided.
  • PerfLoc [78] is also running a competition about developing the best possible indoor localization and tracking applications for Android smartphones. The competition participants are required to use a huge database collected by The National Institute of Standard and Technology (NIST) by means of four different Android devices.
On the one hand, using data sets is useful for the research community: the IPS can be evaluated without surveying and mapping; the experiments can be reproduced; and the results can be compared when the same metric is used for evaluation.
On the other hand, a data set has a few disadvantages: it commonly covers one or a few positioning technologies; external researchers (users) do not have control over data collection; some extended information (maps, obstacles, among other useful information from the environment) might not be present; and the data format and the training/validation/test separation depends on the creator/donor.

2.4. Diversity in Competitions

In this subsection, we present the problem related to the heterogeneity or diversity of different competition tracks proposed in the last few years for indoor positioning. We mainly review the Microsoft and IPIN competition, describing the diversity of scenarios, data streams and hardware used, and the need for a standardization framework.

2.4.1. The Microsoft Competition

The Microsoft Indoor Localization Competition at the International Conference on Information Processing in Sensor Networks (IPSN) [79] is one of the most popular indoor localization competitions. It held its first and second editions in Berlin and Seattle, in 2014 and 2015. The third edition was held in Vienna in 2016, where the evaluation site was a hall of dimensions 30 by 15 m (450 m2).
In the 2016 edition [80], two different tracks were held, called “2D” and “3D”, where position data (XY or XYZ, respectively) had to be generated by competitors at each of 15 different evaluation points. A total of 49 teams showed interest in the competition, of which 35 were registered and 28 were evaluated, 12 in the 2D track and 16 in the 3D one.
In the 2D track, competitors were not allowed to install any devices in the testing hall. Three Wi-Fi APs where available for use. Competitors could use Wi-Fi-based location systems or self-contained solutions using an inertial measurement unit or a magnetometer on the competitor’s body (e.g., a smartphone alone or with some external IMUs).
In the 3D track, competitors were required to install their own hardware, for a maximum of five anchors, in order to estimate the 3D position of the same 15 ground-truth points. Competitors mainly opted for UWB, ultrasonic and laser solutions.
All teams had a whole day prior to the competition for calibration. During the competition, some teams were excluded from the awards since some of them (those based on dead-reckoning such as IMU-based approaches) needed an initial known position. Other teams were also excluded because their system was not real-time, as they needed post-processing.
The main characteristics of the Microsoft competition were kept the same during its first three editions: lack of predefined track formats at the time of subscriptions as formats were precisely defined later, based on competitor’s interest declared during the subscription; the testing environment was about the size of a big hall, with few or no walls, on a single floor; measurements were done in sequence at a series (about a dozen) of fixed points, where the competitors bring their equipment in turn, let the system stabilise, and one organiser reads coordinate estimates on the screen of the competitor’s PC; the metric was the mean of errors, defined as the Euclidean distance between the reference points and the competitor’s estimates.

2.4.2. The IPIN Competition

The International Conference on Indoor Positioning and Indoor Navigation (IPIN) is the oldest conference in this field. IPIN decided to launch its first competition in 2014 and it adopted the EvAAL framework. IPIN formally adopted the EvAAL framework in 2015 and 2016. The main features of the past IPIN Competitions were:
  • IPIN competitions take measurements while naturally moving through a predefined trajectory unknown to competitors, instead of evaluating the competitors by standing still at evaluation points;
  • IPIN competitions are generally done in big challenging multi-floor environments, possibly on multiple buildings, with significant path lengths and duration, instead of evaluating competitors in small single-floor environments;
  • IPIN competitions generally require the competitors to interface with an independent real-time measurement application and test on an independent actor;
  • The final score metrics is the third quartile of the positioning error in IPIN, which makes the accuracy results less prone to the influence of outliers and more in line with demanded accuracy for commercial systems.
The entry barrier to IPIN is high, mostly because high stability and robustness are required of the competing systems, which must be able to work reliably for a longer time in tougher conditions. Overall, the EvAAL criteria impose rigorous and realistic test conditions which are appropriate to deal with state-of-the art working systems.

2.4.3. The Scenarios in the IPIN 2016 Competition

The IPIN competition refers to a typical agent indoor navigation scenario, according to which an agent, roaming within an unknown environment, needs to know its current position. The agent moving within the environment can be either a person or a robot. In both cases, we assume that the agent carries an indoor positioning device, e.g., a smart phone or a custom device. For human agents, we refer to a wide indoor environment, such as a mall, a hospital or a big office in which navigation is not a simple task. For robotic agents, we refer to a high precision system. Due to heterogeneity of technologies and target applications, the IPIN 2016 competition includes three different scenarios:
  • positioning of people in real time;
  • positioning of people off-line;
  • robotic positioning in real time.
The first two scenarios aim at assessing the performance of localization systems targeted to humans. The third scenario refers to systems targeted to robotic agents. Competition details are available online at http://evaal.aaloa.org/2016/competition-home.
Scenario 1: Positioning of people in real time
The goal of this scenario is to measure the performance of working systems with people moving in realistic conditions. The setting was the Polytechnic School of the University of Alcalá (EPS-UAH), which is divided into four sectors (red, blue, green and yellow) of which two contiguous ones (the red and blue sectors) were used in the competitions for positioning people in real time. Figure 1 shows the floor plans of this scenario, where the usable areas are marked with an orange pattern.
In order to cover different positioning technologies, two tracks were proposed to competitors.
  • Track 1: Smartphone-based (on-site);
  • Track 2: Pedestrian dead reckoning positioning (on-site).
The main differences between these two tracks are the positioning devices allowed.
Scenario 2: Positioning of people off-line
Like the previous one, the purpose of this scenario is to measure the performance of working systems with people moving in realistic conditions. The difference is that the competition is performed off-line. The calibration and evaluation data to set up and check the indoor positioning systems are provided in advance to competitors by organizers. Therefore, all competitors have the same data and information to create and evaluate their systems.
Since evaluation time is not a tough restriction, the setting was composed by many different tracks which were pre-recorded in four heterogeneous buildings: the green sector of the Polytechnic School of the University of Alcalá (EPS-UAH), the CAR building (CSIC Arganda, Madrid), and UB & TI buildings (Universitat Jaume I, Castellon). The setting at EPS-UAH building was different from Tracks 1 and 2, in order to avoid interfering with them. Figure 2 shows the base floor plans (only ground floor for simplification) of the four buildings used for this scenario. The databases of the off-site track are available online at http://indoorloc.uji.es/ipin2016track3/.
In this scenario, only one track was available to competitors:
  • Track 3: Smartphone-based (off-site)
Scenario 3: Robotic positioning
The purpose of the last scenario is to continuously log the trajectory followed by an industrial robot. Robots are generally programmed to accomplish specific tasks, e.g., delivering postal mails among offices or delivering medical supplies to laboratories in a hospital. In the competition scenario, the industrial robot followed a closed path, including circular and straight sections alternately arranged. The path was unknown to participants in advance, as shown in Figure 3. Competitors provided both their own custom hardware, that had to be deployed and calibrated in a constrained time, and their algorithm approaches for tracking the robot trajectory.
In this scenario, only one track was available to competitors:
  • Track 4: Indoor mobile robot positioning (on-site)

2.4.4. Other Competitions

Robotics competitions have a large tradition, such as the Robot World Cup (Robocup) [81], the Urban Search and Rescue (USAR) [82,83] or the AUVSI SUAS Competition for Unmanned Aerial Systems (UAS) [84]. These competitions play an important role in the development of artificial intelligence (AI) and robotics, but none of them is focused on indoor robot positioning.

3. EvAAL Framework Applied to IPIN Competition

In this section, we first review some of the most prominent benchmarking metrics proposed to date (EVARILOS and EvAAL), and then we show how the EvAAL framework has been integrated into the IPIN competition to evaluate diverse positioning systems under multiple scenarios (multi-floor, multi-building, multi-sensor, multi user persons/robots, etc.) in order to create a de facto standard for evaluation.

3.1. Benchmarking Metrics

3.1.1. The EVARILOS Benchmarking Framework

The EVARILOS benchmarking platform [85], with a competition held in 2014, had the objective of automating the evaluation process, although the final error metrics were calculated with a manual process from the ground truth. The EVARILOS benchmarking proposal defines a taxonomy of metrics (deployment, functionality and performance related) integrated in a weighted sum to finally generate the system scores. Although it targets radio frequency-based systems, some of their ideas could be generalized to broader scenarios.

3.1.2. The EvAAL Benchmarking Framework

The first public international localization competitions (2011–2013, Valencia and Madrid) were part of the EvAAL initiative [21,23]. The EvAAL framework is characterised by several core criteria in order to evaluate the teams:
  • Natural movement of an actor: the agent testing a localization system is an actor walking with a regular pace along the path. The actor can rest in a few points and walk again until the end of the path.
  • Realistic environment: the path the actor walks is in a realistic setting: the first EvAAL competitions were done in living labs.
  • Realistic measurement resolution: the minimum time and space error considered are relative to people’s movement. The space resolution for a person is defined by the diameter of the body projection on the ground, which we set to 50 cm. The time resolution is defined by the time a person takes to walk a distance equal to the space resolution. In an indoor environment, considering a maximum speed of 1 m/s, the time resolution is 0.5 s. These numbers are used to define the accuracy of measurements. When the actor walks, measurements are taken when he/she steps over a set of predefined points. The actor puts his/her feet on marks made on the floor when a bell chimes, once per second. As long as he/she does not make time and space errors greater than the measurement resolutions, which is easy for a trained agent, the test is considered adequate.
  • Third quartile of point Euclidean error: the accuracy score is based on the third quartile of the error, which is defined as the 2-D Euclidean distance between the measurement points and the estimated points. More discussion on this is in the next section. Using the third quartile of point errors, the system under measure provides an error below the declared one in three cases out of four, which is in line with the perceived usefulness of the experimental IPS. Reference [86] argues that using linear metrics such as the mean may lead to strange and unwanted behaviour if they are not properly checked because of, for instance, the presence of outliers. A clear example of this behaviour can be seen in [25], where a competitor of the off-site track provided severe errors in very few cases and its average error and root mean squared error were negatively highly affected, however the competition metric, the third quartile, showed that the solution proposed was not as bad as the averaged error had shown. More detailed discussion is found in [21].
The core criteria are the distinguishing features of the EvAAL framework. The first EvAAL competitions additionally adopted the following extended criteria:
  • Secret path: the final path is disclosed immediately before the test starts, and only to the competitor whose system is under test. This prevents competitors from designing systems exploiting specific features of the path.
  • Independent actor: the actor is an agent not trained to use the localization system.
  • Independent logging system: the competitor system estimates the position at a rate of twice per second, and sends the estimates on a radio network provided by the EvAAL committee. This prevents any malicious actions from the competitors. The source code of the logging system is publicly available.
  • Identical path and timing: the actor walks along the same identical path with the same identical timing for all competitors, within time and space errors within the above defined resolutions.

3.2. Applying EvAAL Criteria to IPIN 2016 Competition

During IPIN 2016, four competition tracks were run in parallel. The competitors of each track were evaluated according to the third quartile of the positioning error. This error is measured based on x y coordinates (longitude and latitude). Also, a penalty P = 15 m is added for each floor error. For example, if the x y error is 4 m and the estimated floor is 2 while it should be 0, the computed error for that estimate will be 4 + 2 P = 34 m. This is in line with real users’ point of view, because some movements in the environment are restricted by physics and the architectural elements [87].
As far as the core criteria of the EvAAL framework are concerned, the IPIN competition 2016 follows them as close as possible for a given scenario.
  • The two first core EvAAL criteria are followed closely: in Tracks 1–3 the actor moves naturally in a realistic and complex environment spanning several floors of one (for Tracks 1 and 2) or few (Track 3) big buildings; in Track 4, the robot moves at the best of its capabilities in a complex single-floor track.
  • The same holds for the third core criterion: the space-time error resolution for Tracks 1–3, where the agent is a person, are 0.5 m and 0.5 s, while space-time resolution for Track 4, where the agent is a robot, are ≈1 mm and 0.1 s. In Track 4, only the adherence to the trajectory is considered given the overwhelming importance of space accuracy with respect to time accuracy as far as robots are concerned.
  • The last core criterion of the EvAAL framework is followed as well, as the third quartile of the point error is used as the final score. The reason behind using a point error as opposed to comparing trajectories using, for example, the Fréchet distance [38,88] is that the latter is less adequate to navigation purposes, for which the real-time identification of the position is more important than the path followed.
As far as the extended criteria of the EvAAL framework are concerned, the IPIN competition also similarly follows them.
  • In tracks 1 and 2, the path is kept secret only until one hour before the competition begins, because it would be impractical to keep it hidden from the competitors after the first one in a public environment. However, competitors could not add this knowledge to their systems. In Track 3, the competitors work with blind datasets (logfiles) in the evaluation so the path can be kept secret. In Track 4, a black cover is used to avoid any visual reference of the path and other visual markers.
  • The agent is independent for all tracks apart from Track 2, where the technical difficulties of the track suggested that the actor was allowed to be one of the members of the competing team.
  • The logging system is only independent in Track 1. An exception was added in Track 2 for the logging system, which was done by the competitors themselves rather than by an independent application. In Track 3, competitors submitted the results via email before a deadline. In Track 4, the competitors had to submit the results via email within a 2-min window after finishing the evaluation track.
  • The path and timing was identical for all competitors in Track 3. The path and timing was also identical for all competitors in Track 4. The paths are slightly different in tracks 1 and 2, which involved positioning people in real-time, because the path was so long that it would have been impossible to force the actors to follow exactly the same path with the same timing many times.

4. The IPIN 2016 Competition Tracks

In order to see the potential power of the EvAAL framework for evaluating different and diverse locating scenarios and systems, we now give some detail about the three different competition scenarios used in the past 2016 IPIN competition.

4.1. Tracks 1 and 2: Positioning of People in Real Time

This subsection details the rules for Tracks 1 and 2 of the IPIN 2016 Competition since they have many commonalities. The competing systems had to be engineered or prototyped so that an external actor could use them without impairing his movements.
In Track 1, actors were not trained to use the competing system. Generally, actors are people from the conference audience or people from the organizing committee willing to support the competition. In Track 1, competitors had to use standard smartphones, with the possibility of using any sensor available on the device: GPS, accelerometer, gyroscope, compass, Wi-Fi radio and barometer. Competitors were allowed to only exploit the existing Wi-Fi access points available on the competition area. Teams in Track 1 were allowed to perform two runs each during the competition day, and consider only the best result. During each run, the actor tested one competing application by using a measurement application developed by the organizers, called StepLogger.
Competitors in Track 2 were allowed to use any IMU sensor module, whether off-the-shelf or self-developed, without any limit concerning the number of devices and the mounting position on the body. Laptops, tablets and smartphones could be used to process data gathered from the Pedestrian Dead Reckoning (PDR) system in real time. Competitors were requested to provide the organizing committee with the sensor raw data, the estimated positions of the system and the key point indexes according to a specific log format. Competitors were not requested to use an external measurement application and could generate the log themselves.
The detailed competition rules are published on the IPIN web page [89,90].

4.1.1. Surveying the Area

Competitors of both tracks received detailed geo-referenced maps of the competing area and were able to survey the area during the days before the competition day. They used the survey time to check how their system performed in the area and were able to perform algorithm tune-up.
During the survey, competitors in Track 1 were mostly interested in taking Wi-Fi measurements, both fingerprints in various locations (a long and tedious task) and checks on the position of access points in the area. Most access points were, in fact, indicated on the maps, but some were not, and up-to-date ids (MAC, SSID) were not provided to competitors. After the competition, many competitors in Track 1 informally said their system would have performed much better if they had spent more time in taking Wi-Fi measurements during the survey, because the area was so big and the hours they dedicated to the task were not enough. Most notably, the winning Track 1 team needed very little survey time. To the amazement of bystanders, including some other competitors, after slightly more than half an hour spent in a specific small area of the building, they claimed that they had collected all the data required by their system.

4.1.2. Evaluation Path

A reference path is necessary to measure the accuracy of the competing systems, and was used as the ground truth. In practice, some dozen markers (keypoints) were stuck on the floor and their coordinates were measured in advance. The actor followed the markers sequentially stepping over them. For Track 1, the synchronization between the real position of the actor and the estimated position was guaranteed by the StepLogger app, running on the same smartphone where the competing app was running. When the actor steps over a mark on the floor, he pushes a button on the smartphone’s screen, and the app records the time. Since the markers are walked in a predefined order, the time stamps are easily associated with the corresponding mark and logged to persistent memory. The log thus produced is then compared with the log of continuously estimated positions provided by the competitor.
In Track 2, the same time-stamp log can be provided by the competitor if he/she uses the StepLogger app. Since most of PDR systems are implemented on laptops, it is not easy to have a communication interface with the StepLogger app, which runs on Android. Therefore, competitors generated logs including time-stamp, sensor raw data, estimated position and marker indexes indicating whether the position value is calculated at the reference point or not.
In both Tracks 1 and 2, the organisers provided an application for producing the results from the time-stamp log and the log of competitor estimates. The applications read the space-time logs and computed the errors as x y distance plus floor penalty. The third quartile of error is then computed and presented as the final score. A dedicated application, available on the EvAAL website, produces a graphical representation of the map including the ground truth and the estimated path.
The path was defined with the goal of realistically reproducing the way people move within wide indoor environments. In fact, path complexity is a distinguishing feature of the IPIN competition. For this purpose, the following rules were considered:
  • stairs (for both Tracks 1 and 2) and a lift (Track 1 only) are used to move between floors;
  • the path traverses four floors and includes the patio, for a total of 56 key points marked on the floor, 6000 m2 indoor and 1000 m2 outdoor;
  • actors stay still for few seconds in six locations and for about 1 min in three locations; this cadence is intended to reproduce the natural behaviour of humans while moving in an indoor environment;
  • actors move at a natural pace, typically at a speed of around 1 m/s;
  • total length and duration are 600 m, 15’ ± 2’, which allows to stress the competing apps in realistic conditions.
As mentioned before, the setting was the Polytechnic School of the University of Alcalá (EPS-UAH). EPS-UAH is hosted in a square building composed of four floors connected with stairs and lifts. Floors all have a similar layout, with a side of about 140 m. The central big and open round hall gives access to four wings where medium sized rooms are located. The structure is mostly made of concrete with many pillars in the central hall. Glass walls separate the patio from the ground floor indoor areas. Wi-Fi is available inside and immediately outside the building.
The path is shown in Figure 4. It is composed of 56 key points, five of which were placed in the patio. The path starts from ground level, up to floors 1, 2 and 3 by means of stairs, then goes to a terrace about 50 m long and proceeds to the ground level, goes to the patio and indoors again. When going to the lower floors, the lift is used for Track 1, the stairs for Track 2.
Key points were labelled with a tag in the form [building ID, floor, marker ID] (Key point labels were written on the button displayed by the StepLogger app, to reduce errors on the actor’s part which would require him to restart the path from the beginning). They were placed on the floor following these criteria:
  • key points were placed in easily accessible places where people usually step over;
  • distance between key points ranged from about 3 to 35 m, with a median of 8 m;
  • each key point was visible from the previous one, to ease the movement of the actor and reduce random paths between two consecutive key points.

4.1.3. Track 1 Results

In the first track, six teams were admitted [91,92,93,94,95,96]: NavIndoor, Samsung, WiMag, WMLoc, XMUH and XPosition. Each competitor had two chances to run the path, with only its best performance considered.
NavIndoor and Samsung teams obtained the best results, respectively with a third quartile score of 5.4 m and 8.2 m, while the remaining teams had scores higher than 15 m, as shown in Figure 5. Note that the localization errors are in the order of a few metres, which is acceptable for navigating an indoor environment and consistent with the EvAAL criteria. Again, we stress that we tested real working systems in a realistic environment with a realistic usage pattern and rigorous criteria: no simulations, no small or controlled environments, and no simplifying assumptions.

4.1.4. Track 2 Results

In Track 2, a total of eight teams were admitted but finally six of them competed [97,98,99,100,101,102,103,104]: NESL, SYSNAV, PARMA, DEUSTO, XIAMEN and TIANJN. Two trials were given for each team, and the best result was selected for comparison.
The results of Track 2 show a high deviation of positioning errors among the six competing teams. NESL produced the best result with a third quartile score of 1.5 m. The second ranked team, SYSNAV, showed very good performance during the initial phase, but the final score was 26.2 m. Such negative performance is due to the sunlight, an unexpected source of error, encountered when the competitor walked along a terrace on the fourth floor. All the other teams produced higher scores than 40 m, as shown in Figure 6, the main cause for this being inaccurate initial heading evaluation. Considering PDR competitors were not allowed to exploit any information gathered from the competition area, such as the Wi-Fi networks, handling of initial heading was a key factor for obtaining reasonable performance.

4.2. Track 3: Smartphone-Based (Off-Line)

In contrast with the two previous tracks, this track was entirely run off-line.
The competitors had to use databases provided by organizers to train and calibrate their systems. Those databases contained data from any sensor available on the smartphone used to log the track performed by an external actor (competition organizers and students). The competitors were not allowed to record any additional measurements on-site. Similarly, data for evaluation was also provided by organizers.
The detailed competition rules are published on the IPIN and competition web pages [105,106].

4.2.1. Surveying the Area

The main objective is to evaluate all competitors under exactly the same conditions. Therefore, they were not allowed to survey the scenarios by themselves to collect additional measurements.
Participants received a set of calibration log files with data from smartphone sensors which were recorded on different paths in a total of four different buildings. Different scenarios (four buildings), smartphones (four device models) and actors (eight people) were involved at this stage in order to maximize the heterogeneity of data. A total of 10 routes, considering the four buildings, were selected according to the same realistic-based goals established for the first and second track (see Section 4.1.2), which included 1025 calibration points (key points).
Competitors were asked to design and implement their localization systems exploiting a total of 17 calibration log files. Logs were publicly available since the very beginning of the call to guarantee that all competitors had the same time to set up their systems.

Data Format

Competitors used the log files generated by the GetSensorData application (http://lopsi.weebly.com/downloads.html) in which a user walks along a predefined path. Logs are text files where each row contains data received from a different sensor available on the the phone at a given time. Each row begins with the sensor identifier, as four capital letters, that determines the kind of sensor data (Wi-Fi, Accelerometer, Magnetometer, etc.). Then, the timestamps and sensor readings follow, separated by a semicolon. The reference data to train/generate the IPSs was geo-referenced. So, the position of well-known landmarks trough the path were provided as special entries in the log file. For the evaluation log files, explicit location of landmarks was not provided. A simple Matlab-based application was provided to competitors in order to parse and re-arrange the log files (http://indoorloc.uji.es/ipin2016track3/).

4.2.2. The Evaluation Path

Nine evaluation trajectories, or paths, were defined with the goal of realistically reproducing the way the people move within wide indoor environments, as was done for the evaluation in Tracks 1 and 2. As mentioned before, the high path complexity is a distinguishing feature of the IPIN competition. For this purpose, the following rules were considered:
  • the stairs and the lifts could be used to move between floors;
  • the paths traverse four floors in the UAH building, one floor in the CAR building, six floors in the UJIUB building and four floors in the UJITI building;
  • paths in the CAR building also include an external patio;
  • the nine paths cover a total of 578 key points;
  • total duration is 2 h and 24 min, which allows to stress the competing applications in realistic conditions;
  • actors may stay still for few seconds in a few locations; this rhythm is intended to reproduce the natural behaviour of humans while moving in an indoor environment;
  • actors move at a natural pace, typically at a speed of around 1 m/s;
  • phoning and lateral movements were allowed occasionally to reproduce a real situation;
  • competitors have all the same data for calibrating and competing.
As already mentioned, the setting was composed of four different buildings in different cities. The UAH building is in a university environment with corridors, classrooms, laboratories, among other typical facilities. The UJITI building has similar facilities but the Wi-Fi network infrastructures are different, since UAH and UJI have different strategies to deploy Wi-Fi. Unlike the two previous buildings, CSIC is a research institution and the UJIUB building has many offices hosting technological companies. In general, each of the selected buildings has unique features (occupancy, construction materials, architectural elements, Wi-Fi network policies, among many others).
Competitors tested their localisation algorithms with nine evaluation log files and they sent us the estimated locations in a predefined format. The evaluation log files were not geo-referenced so competitors had to exploit the information collected by all sensors in order to provide an accurate position estimation. It is important to mention that once the evaluation log files were released, all competing teams had strict timing to provide their results.
Since the competitors had just one chance to submit results without feedback to readjust their systems on-the-fly, up to three different results from each participant were accepted in this track, but only the best result was considered for the final ranking. In Tracks 1 and 2, the competitors had two chances, but they had feedback about their accuracy after the first run. The evaluation was performed off-line before the conference.

4.2.3. Results

In Track 3, a total of six competing teams were admitted but only five of them formally registered [91,107,108,109,110]: HFTS, UMinho, m BlockDox, NavIndoor (FHWS in Track 3 to avoid ambiguities with respect to Track 1) and Marauder.
The HFTS and UMinho teams obtained the best results, with 5.9 m and 7.3 m third quartile. BlockDox and FHWS provided slightly higher scores, at 7.8 m and 8.8 m respectively. The remaining team provided a score higher than 40 m. Results of Track 3 are shown in Figure 7.
It is worth noticing that the localisation errors we experienced are also in the order of meters: this is consistent with Track 1 results.

4.3. Track 4: Robotic Positioning

Track 4 is significantly different from other tracks due to three main facts:
  • The tracked element is an industrial robot;
  • The task not only needs discrete and usually well separated key positions to be estimated, but a detailed tracking of the actual precise and unknown robot trajectory;
  • Competitors could put sensors on board as well as locate them on given poles around the navigation area.
Detailed competition rules are published on the IPIN web page [111].

4.3.1. Surveying the Area

Participants decided both the nature and the number of sensors, and had up to 45 min to assemble, mount and calibrate their systems. Three reference points in the scenario were given to allow sensor calibration; one of them was the start/stop point for the mobile unit. Competitors had to report the consecutive points of the robot trajectory at least every 100 ms (minimum frame rate). The required information was the x , y coordinates (in millimetres) with respect to the starting point, and the corresponding time stamp.

Robot

The chosen mobile unit was a Standard Easybot from the ASTI international robotic company [112] (see Figure 8 bottom-right). The Easybot is an automatically guided vehicle implementing a magnetic guidance and its nominal velocity is 0.6 m/s, tracking the designated magnetic tape thanks to the feedback provided by a magnetic detector and the corresponding closed-loop control.

4.3.2. Evaluation Path

The magnetic path was a closed route including circular and straight sections alternately arranged (see Figure 8 top). The magnetic tape was placed on a trajectory that was previously defined according to a mathematical description; this way, we assured a known ground-truth, identical for every experiment (see Figure 8 bottom-left). The path length was 32.84 m; the trip time was around 3 min, including programmed linear velocities changes, which were equal for all participants. A black cover avoided any visual reference of the magnetic path (see Figure 8 bottom-right). A mark placed on the robot defined the reference point to be tracked by competitors.

4.3.3. Results

Track 4 received six requests for participation from teams using different technologies (UWB, visible light, acoustics and laser), but only five of them formally registered [113,114,115,116,117]. Two of them were local teams from the hosting university (University of Alcalá) and were considered as off-track competitors, three of the remaining groups formally registered, but only two teams competed and were evaluated: the ATLAS [115] team and the TopoRTLS company [114].
Figure 9 depicts the CDF of both participants, where the errors at the third quartile were 106 mm and 311 mm for the ATLAS and TPM teams, respectively.

4.4. Lessons Learned from IPIN Competition Tracks

The 2016 IPIN competition has been a successful and challenging experience both for the organizers and the competing teams. We can derive some lessons we learned during the preparation of the tracks and also during the competition days.
In smartphone-based Track 1, we observed that tests done on a realistic and long path stress the competing applications, which require a good degree of maturity to run regularly for long time periods. The independent actor may behave differently to how the competitors had anticipated. We think that these are key features for assessing the performance in realistic conditions.
As far as performance is concerned, we noted that accuracy, speed and scope of site surveying, especially for Wi-Fi measurements, was the key to success for all competitors.
In PDR Track 2, we observed that a correct initial heading is important for obtaining good performance: for three teams, a bad initial heading produced a path rotated by some degrees with respect to the ground truth, resulting in bad performance even in the presence of a correctly-shaped path. Therefore, it could be considered to provide competitors not only with the starting point coordinates, but additionally with initial heading information for Track 2. As a second note, quality of sensors turned out to be an important factor. Three teams used commercially available sensors; the others used self-developed sensor modules: the lack of engineering in the latter was sometimes crucial. In one case, a sensor module misbehaved, so the team missed the first trial; in another case, the Bluetooth communication between the sensor module and the user mobile device was unreliable, resulting in incomplete log files.
In the off-line Track 3, we observed that the best results are similar to Track 1, which is a compelling argument for the usefulness of off-line tests. On the other side, the NavIndoor team from FHWS participated in both tracks with the same system, and it reported lower accuracy off-line than in real time. It can be argued that the presence of both real-time and off-line tracks with similar characteristics has enriched the IPIN competition. Furthermore, the data sets are open-access and they can be reused by teams that did not participate in the competition. Therefore, the research community can make new comparisons of state-of-art systems.
In robotics Track 4, the resulting localization accuracy is better than in the other tracks due to the fact that the navigation area is much smaller, the ground truth path has a much higher measurement accuracy and the teams had the possibility of installing their specific equipment on-board and off-board the robot. A consistent and sufficiently precise ground-truth can be achieved by using a mathematically modelled trajectory and considering time sequence information. Similar to the smartphone-based Track 1, time necessary for equipment placement and calibration, which depends on system complexity, has been demonstrated to be an important performance-limiting factor. On the other hand, the competition set-up has to consider enough space for an industrial robot performing a diverse and complex trajectory and competition rules must provide enough description about physical elements available for participants to deploy their systems, together with size, weight and safety constraints.

5. Conclusions

We have shown how diverse current indoor location solutions are and how the different environments, different test areas, different sensor technology, etc., have a significant impact on how location results are processed and evaluated, which makes it difficult to directly compare performance.
We gave a review of the ways to approach this problem in the literature, either by using the performance measure indicated by the authors themselves, or by means of a competition. One additional approach to compare algorithms in controlled conditions has been to set up measurement databases; however, differences in formats, recording procedure and range of sensor used again makes this approach not straightforward.
As a possible solution, we propose the use of the EvAAL benchmarking framework, which we claim has the potential to become a standard way to compare systems in different application areas. It has been used since 2011 and lately in the 2016 IPIN competition, where four different tracks were focused on different use cases with significant variation along several dimensions: person vs. robot, smartphone vs. custom hardware, single vs. multi-storey building, single vs. multi-building environments, on-line vs. off-line processing. Our claim is that applying the EvAAL rigorous criteria makes it possible to directly compare the performance of heterogeneous systems in a more significant way than with other existing methods. We back this claim by presenting the details of the different tracks in the IPIN competition and showing how, in their diversity, they are still comparable.
Experience gained during the IPIN 2016 competition and personal feedback from track organizers and competitors encourages the idea that the EvAAL framework is flexible and robust enough to successfully fit very different use cases when comparing indoor positioning systems.
Finally, one of the main challenges of evaluating indoor positioning systems is considering the diversity of contexts and scenarios during the evaluation. In the 2016 IPIN competition, as it has been already mentioned, a few different contexts and scenarios were considered through the four tracks with the assumption of the normal actor’s movement. The spirit of EvAAL is to be open and integrate new tracks whenever it is possible. For instance, one difference with respect to the previous 2015 EvAAL-ETRI competition was the inclusion of robot-based Track 4. Databases to cover many multi-tier diverse buildings were introduced in the 2015 edition. As future work, the EvAAL community is open to discussing the inclusion of new tracks where the actor’s movement might be erratic (e.g., building evacuation in the case of a fire alarm), non-constant (e.g., sport tracking) or presenting non-walking patterns (e.g., people with mobility impairments) which are realistic movements in some contexts.

Acknowledgments

This work was supported by the ICT R&D program of MSIP/IITP, South Korea (2017-0-00543, Development of Precise Positioning Technology for the Enhancement of Pedestrian’s Position/Spatial Cognition and Sports Competition Analysis). Parts of this work were funded by the Spanish Government through the projects TIN2013-47630-C2-1-R, TIN2015-70202-P, TIN2016-75982-C2-1-R, TEC2015-64835-C3-2-R, DPI2013-47347-C2-1-R and TARSIUS TIN2015-71564-C4-2-R, the network REPNIN TEC2015-71426-REDT, and the José Castillejo mobility programme CAS16/00072.
The organizers of the EvAAL competition and all track chairs gratefully thank the invaluable support from the IPIN conference chairs, Jesus Ureña, Juan Jesús García and Álvaro Hernández, and funding support from the competition sponsors, The Electronics and Telecommunications Research Institute (ETRI), The Korean Institute of Communications and Information Sciences (KICS), Tecnalia Corporación Tecnológica and ASTI.

Author Contributions

Francesco Potortì, Sangjoon Park, Antonio Ramón Jiménez Ruiz were the main competition chairs. Track 1 was chaired by Paolo Barsocchi and Michele Girolami. Francesco Potortì and Antonino Crivello also participated in Track 1 organization. Track 2 was chaired by So Yeon Lee and Jae Hyun Lim. Sangjoon Park also participated in Track 2 organization. Track 3 was chaired by Joaquín Torres-Sospedra and Fernando Seco. Raul Montoliu, Germán Martin Mendoza-Silva and Antonio Ramón Jiménez Ruiz also participated in Track 3 organization and database collection. Track 4 was chaired by Maria Carmen Pérez Rubio and Cristina Losada-Gutiérrez. Felipe Espinosa and Javier Macias-Guarasa also participated in Track 4 organization. The main competition chairs and the track chairs oversaw the paper writing with the collaboration of all the co-authors.

Conflicts of Interest

The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:
AALActive and Assisted Living
APAccess Point
BTBluetooth
BLEBluetooth Low Energy
CDFCumulative Distribution Function
EvAALEvaluating Active and Assisted Living
GNSSGlobal Navigation Satellite System
IMUInertial Measurement Unit
IoTInternet of Things
IPINIndoor Positioning and Indoor Navigation
IPSIndoor Positioning System
PDFProbability Density Function
PDRPedestrian Dead Reckoning
RFIDRadio Frequency Identification
RMSERoot Mean Square Error
UWBUltra Wide Band

References

  1. Park, K.; Shin, H.; Cha, H. Smartphone-based pedestrian tracking in indoor corridor environments. Pers. Ubiquit. Comput. 2013, 17, 359–370. [Google Scholar] [CrossRef]
  2. Hardegger, M.; Roggen, D.; Tröster, G. 3D ActionSLAM: Wearable person tracking in multi-floor environments. Pers. Ubiquit. Comput. 2015, 19, 123–141. [Google Scholar] [CrossRef]
  3. Zampella, F.; Jiménez, A.; Seco, F. Indoor positioning using efficient map matching , RSS measurements , and an improved motion model. IEEE Trans. Veh. Technol. 2015, 64, 1304–1317. [Google Scholar] [CrossRef]
  4. Jiménez, A.R.; Seco, F.; Prieto, J.C.; Guevara, J. Indoor pedestrian navigation using an INS/EKF framework for Yaw drift reduction and a foot-mounted IMU. In Proceedings of the 2010 7th Workshop on Positioning Navigation and Communication (WPNC), Dresden, Germany, 11–12 March 2010. [Google Scholar]
  5. Jiménez, A.R.; Seco, F.; Zampella, F.; Prieto, J.C.; Guevara, J. Improved Heuristic Drift Elimination ( iHDE ) for pedestrian navigation in complex buildings. In Proceedings of the 2011 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Guimaraes, Portugal, 21–23 September 2011. [Google Scholar]
  6. Vera, R.; Ochoa, S.F.; Aldunate, R.G. EDIPS: An Easy to Deploy Indoor Positioning System to support loosely coupled mobile work. Pers. Ubiquit. Comput. 2011, 15, 365–376. [Google Scholar] [CrossRef]
  7. Kim, S.C.; Jeong, Y.S.; Park, S.O. RFID-based indoor location tracking to ensure the safety of the elderly in smart home environments. Pers. Ubiquit. Comput. 2013, 17, 1699–1707. [Google Scholar] [CrossRef]
  8. Piontek, H.; Seyffer, M.; Kaiser, J. Improving the accuracy of ultrasound-based localisation systems. Pers. Ubiquit. Comput. 2007, 11, 439–449. [Google Scholar] [CrossRef]
  9. Prieto, J.C.; Jiménez, A.R.; Guevara, J.; Ealo, J.L.; Seco, F.; Roa, J.O.; Ramos, F. Performance evaluation of 3D-LOCUS advanced acoustic LPS. IEEE Trans. Instrum. Meas. 2009, 58, 2385–2395. [Google Scholar] [CrossRef]
  10. Seco, F.; Prieto, J.C.; Jimenez, A.R.; Guevara, J. Compensation of multiple access interference effects in CDMA-based acoustic positioning systems. IEEE Trans. Instrum. Meas. 2014, 63, 2368–2378. [Google Scholar] [CrossRef]
  11. Bruns, E.; Bimber, O. Adaptive training of video sets for image recognition on mobile phones. Pers. Ubiquit. Comput. 2009, 13, 165–178. [Google Scholar] [CrossRef]
  12. Randall, J.; Amft, O.; Bohn, J.; Burri, M. LuxTrace: Indoor positioning using building illumination. Pers. Ubiquit. Comput. 2007, 11, 417–428. [Google Scholar] [CrossRef]
  13. Yang, S.H.; Jeong, E.M.; Kim, D.R.; Kim, H.S.; Son, Y.H.; Han, S.K. Indoor three-dimensional location estimation based on LED visible light communication. Electron. Lett. 2013, 49, 54–56. [Google Scholar] [CrossRef]
  14. Kuo, Y.S.; Pannuto, P.; Hsiao, K.J.; Dutta, P. Luxapose: Indoor positioning with mobile phones and visible light. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking, Maui, HI, USA, 7–11 September 2014. [Google Scholar]
  15. Jimenez, A.R.; Zampella, F.; Seco, F. Light-matching: A new signal of opportunity for pedestrian indoor navigation. In Proceedings of the 4th International Conference on Indoor Positioning and Indoor Navigation (IPIN), Montbeliard-Belfort, France, 28–31 October 2013. [Google Scholar]
  16. Markets and Markets. Indoor Location Market by Component (Technology, Software Tools, and Services), Application, End User (Transportation, Hospitality, Entertainment, Shopping, and Public Buildings), and Region - Global Forecast to 2021. 2016. Available online: http://www.marketsandmarkets.com/Market-Reports/indoor-positioning-navigation-ipin-market-989.html (accessed on 12 October 2017).
  17. Liu, H.; Darabi, H.; Banerjee, P.; Liu, J. Survey of wireless indoor positioning techniques and systems. IEEE Trans. Syst. Man Cybern. C (Appl. Rev.) 2007, 37, 1067–1080. [Google Scholar] [CrossRef]
  18. Gu, Y.; Lo, A.; Niemegeers, I. A survey of indoor positioning systems for wireless personal networks. IEEE Commun. Surv. Tutor. 2009, 11, 13–32. [Google Scholar] [CrossRef]
  19. Ficco, M.; Palmieri, F.; Castiglione, A. Hybrid indoor and outdoor location services for new generation mobile terminals. Pers. Ubiquit. Comput. 2014, 18, 271–285. [Google Scholar] [CrossRef]
  20. Bahl, V.; Padmanabhan, V. RADAR: An In-Building RF-based User Location and Tracking System; Institute of Electrical and Electronics Engineers, Inc.: Piscataway, NJ, USA, 2000. [Google Scholar]
  21. Barsocchi, P.; Chessa, S.; Furfari, F.; Potortì, F. Evaluating AAL solutions through competitive benchmarking: The localization competition. IEEE Perv. Comput. Mag. 2013, 12, 72–79. [Google Scholar] [CrossRef]
  22. Niu, X.; Li, M.; Cui, X.; Liu, J.; Liu, S.; Chowdhury, K.R. WTrack: HMM-based walk pattern recognition and indoor pedestrian tracking using phone inertial sensors. Pers. Ubiquit. Comput. 2014, 18, 1901–1915. [Google Scholar] [CrossRef]
  23. Salvi, D.; Barsocchi, P.; Arredondo, M.T.; Ramos, J.P.L. EvAAL, Evaluating AAL Systems through Competitive Benchmarking, the Experience of the 1st Competition; Springer: Berlin/Heidelberg, Germany, 2011; pp. 14–25. [Google Scholar]
  24. Chessa, S.; Furfari, F.; Potortì, F.; Lázaro, J.P.; Salvi, D. EvAAL: Evaluating AAL systems through competitive benchmarking. ERCIM News 2011, 84, 46. [Google Scholar]
  25. Potortì, F.; Barsocchi, P.; Girolami, M.; Torres-Sospedra, J.; Montoliu, R. Evaluating indoor localization solutions in large environments through competitive benchmarking: The EvAAL-ETRI competition. In Proceedings of the 2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Calgary, AB, Canada, 13–16 October 2015. [Google Scholar]
  26. Youssef, M.; Agrawala, A.K. The Horus WLAN location determination system. In Proceedings of the 3rd International Conference on Mobile Systems, Applications, and Services, Seattle, WA, USA, 6–8 June 2005. [Google Scholar]
  27. Youssef, M.; Agrawala, A.K. On the Optimally of WLAN Location Determination Systems; Technical Report UMIACS-TR 2003-29 and UMIACS-TR 4459; University of Maryland: College Park, MD, USA, 2003. [Google Scholar]
  28. Youssef, M.; Agrawala, A.K. Handling samples correlation in the horus system. In Proceedings of the 23rd Annual Joint Conference of the IEEE Computer and Communications Societies, Hong Kong, China, 7–11 March 2004. [Google Scholar]
  29. Gu, Y.; Song, Q.; Ma, M.; Li, Y.; Zhou, Z. Using iBeacons for trajectory initialization and calibration in foot-mounted inertial pedestrian positioning systems. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  30. Fetzer, T.; Ebner, F.; Deinzer, F.; Köping, L.; Grzegorzek, M. On Monte Carlo smoothing in multi sensor indoor localisation. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  31. Ahmed, D.B.; Diaz, E.M.; Kaiser, S. Performance comparison of foot- and pocket-mounted inertial navigation systems. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  32. Song, Q.; Gu, Y.; Ma, M.; Li, Y.; Zhou, Z. An anchor based framework for trajectory calibration in inertial pedestrian positioning systems. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  33. Caruso, D.; Sanfourche, M.; Besnerais, G.L.; Vissière, D. Infrastructureless indoor navigation with an hybrid magneto-inertial and depth sensor system. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  34. Ehrlich, C.R.; Blankenbach, J.; Sieprath, A. Towards a robust smartphone-based 2,5D pedestrian localization. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  35. Galčík, F.; Opiela, M. Grid-based indoor localization using smartphones. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  36. Zhang, W.; Hua, X.; Yu, K.; Qiu, W.; Zhang, S. Domain clustering based WiFi indoor positioning algorithm. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  37. Li, W.; Wei, D.; Yuan, H.; Ouyang, G. A novel method of WiFi fingerprint positioning using spatial multi-points matching. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  38. Mathisen, A.; Sørensen, S.K.; Stisen, A.; Blunck, H.; Grønbæk, K. A comparative analysis of Indoor WiFi Positioning at a large building complex. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  39. Berkvens, R.; Weyn, M.; Peremans, H. Position error and entropy of probabilistic Wi-Fi fingerprinting in the UJIIndoorLoc dataset. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  40. Liu, W.; Fu, X.; Deng, Z.; Xu, L.; Jiao, J. Smallest enclosing circle-based fingerprint clustering and modified-WKNN matching algorithm for indoor positioning. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  41. Mizmizi, M.; Reggiani, L. Design of RSSI based fingerprinting with reduced quantization measures. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  42. Guimarães, V.; Castro, L.; Carneiro, S.; Monteiro, M.; Rocha, T.; Barandas, M.; Machado, J.; Vasconcelos, M.; Gamboa, H.; Elias, D. A motion tracking solution for indoor localization using smartphones. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  43. Ma, M.; Song, Q.; Li, Y.H.; Gu, Y.; Zhou, Z.M. A heading error estimation approach based on improved Quasi-static magnetic Field detection. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  44. Skog, I.; Nilsson, J.O.; Händel, P. Pedestrian tracking using an IMU array. In Proceedings of the 2014 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), Bangalore, India, 6–7 January 2014. [Google Scholar]
  45. Du, Y.; Arslan, T.; Juri, A. Camera-aided region-based magnetic field indoor positioning. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  46. Laurijssen, D.; Truijen, S.; Saeys, W.; Daems, W.; Steckel, J. A flexible embedded hardware platform supporting low-cost human pose estimation. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  47. Stancovici, A.; Micea, M.V.; Cretu, V. Cooperative positioning system for indoor surveillance applications. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  48. Cioarga, R.D.; Micea, M.V.; Ciubotaru, B.; Chiuciudean, D.; Stanescu, D. CORE-TX: Collective Robotic environment—The timisoara experiment. In Proceedings of the Third Romanian-Hungarian Joint Symposium on Applied Computational Intelligence, SACI 2006, Timisoara, Romania, 25–26 May 2006. [Google Scholar]
  49. Ogiso, S.; Kawagishi, T.; Mizutani, K.; Wakatsuki, N.; Zempo, K. Effect of parameters of phase-modulated M-sequence signal on direction-of-arrival and localization error. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  50. Pelka, M.; Hellbrück, H. Introduction, discussion and evaluation of recursive Bayesian filters for linear and nonlinear filtering problems in indoor localization. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  51. Jiménez, A.R.; Seco, F. Comparing Decawave and Bespoon UWB location systems: Indoor/outdoor performance analysis. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  52. Karciarz, J.; Swiatek, J.; Wilk, P. Using path matching filter for lightweight indoor location determination. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  53. Dombois, M.; Döweling, S. A pipeline architecture for indoor location tracking. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  54. Ettlinger, A.; Retscher, G. Positioning using ambient magnetic fields in combination with Wi-Fi and RFID. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  55. Okuda, K.; Yeh, S.Y.; Wu, C.I.; Chang, K.H.; Chu, H.H. The GETA Sandals: A Footprint Location Tracking System; Strang, T., Linnhoff-Popien, C., Eds.; Springer: Berlin/Heidelberg, Germany, 2005; pp. 120–131. [Google Scholar]
  56. Yeh, S.Y.; Chang, K.H.; Wu, C.I.; Chu, H.H.; Hsu, J.Y.J. GETA sandals: A footstep location tracking system. Pers. Ubiquit. Comput. 2007, 11, 451–463. [Google Scholar] [CrossRef]
  57. Lorincz, K.; Welsh, M. MoteTrack: A Robust, Decentralized Approach to RF-Based Location Tracking; Strang, T., Linnhoff-Popien, C., Eds.; Springer: Berlin/Heidelberg, Germany, 2005; pp. 63–82. [Google Scholar]
  58. Lorincz, K.; Welsh, M. MoteTrack: A robust, decentralized approach to RF-based location tracking. Pers. Ubiquit. Comput. 2007, 11, 489–503. [Google Scholar] [CrossRef]
  59. Wu, C.J.; Quigley, A.; Harris-Birtill, D. Out of sight: A toolkit for tracking occluded human joint positions. Pers. Ubiquit. Comput. 2017, 21, 125–135. [Google Scholar] [CrossRef]
  60. Guo, J.; Zhang, H.; Sun, Y.; Bie, R. Square-root unscented Kalman filtering-based localization and tracking in the Internet of Things. Pers. Ubiquit. Comput. 2014, 18, 987–996. [Google Scholar] [CrossRef]
  61. Xiao, J.; Zhou, Z.; Yi, Y.; Ni, L.M. A survey on wireless indoor localization from the device perspective. ACM Comput. Surv. 2016, 49, 1–31. [Google Scholar] [CrossRef]
  62. He, S.; Chan, S.H.G. Wi-Fi fingerprint-based indoor positioning: Recent advances and comparisons. IEEE Commun. Surv. Tutor. 2016, 18, 466–490. [Google Scholar] [CrossRef]
  63. Hassan, N.U.; Naeem, A.; Pasha, M.A.; Jadoon, T.; Yuen, C. Indoor positioning using visible LED lights: A survey. ACM Comput. Surv. 2015, 48, 1–32. [Google Scholar] [CrossRef]
  64. King, T.; Kopf, S.; Haenselmann, T.; Lubberger, C.; Effelsberg, W. CRAWDAD Dataset Mannheim/compass (v. 2008-04-11). Traceset: Fingerprint. 2008. Available online: http://crawdad.org/mannheim/compass/20080411/fingerprint (accessed on 12 October 2017).
  65. Rhee, I.; Shin, M.; Hong, S.; Lee, K.; Kim, S.; Chong, S. CRAWDAD Dataset Ncsu/Mobilitymodels (v. 2009-07-23). 2009. Available online: http://crawdad.org/ncsu/mobilitymodels/20090723 (accessed on 12 October 2017).
  66. Nahrstedt, K.; Vu, L. CRAWDAD dAtaset Uiuc/Uim (v. 2012-01-24). 2012. Available online: http://crawdad.org/uiuc/uim/20120124 (accessed on 12 October 2017).
  67. Purohit, A.; Pan, S.; Chen, K.; Sun, Z.; Zhang, P. CRAWDAD Dataset Cmu/Supermarket (v. 2014-05-27). 2014. Available online: http://crawdad.org/cmu/supermarket/20140527 (accessed on 12 October 2017).
  68. Parasuraman, R.; Caccamo, S.; Baberg, F.; Ogren, P. CRAWDAD Dataset kth/rss (v. 2016-01-05). 2016. Available online: http://crawdad.org/kth/rss/20160105 (accessed on 12 October 2017).
  69. Lichman, M. UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science. Available online: http://archive.ics.uci.edu/ml (accessed on 12 October 2017).
  70. Torres-Sospedra, J.; Montoliu, R.; Usó, A.M.; Avariento, J.P.; Arnau, T.J.; Benedito-Bordonau, M.; Huerta, J. UJIIndoorLoc: A new multi-building and multi-floor database for WLAN fingerprint-based indoor localization problems. In Proceedings of the 2014 International Conference on Indoor Positioning and Indoor Navigation, IPIN 2014, Busan, Korea, 27–30 October 2014. [Google Scholar]
  71. Torres-Sospedra, J.; Moreira, A.J.C.; Knauth, S.; Berkvens, R.; Montoliu, R.; Belmonte, O.; Trilles, S.; Nicolau, M.J.; Meneses, F.; Costa, A.; et al. A realistic evaluation of indoor positioning systems based on Wi-Fi fingerprinting: The 2015 EvAAL-ETRI competition. JAISE 2017, 9, 263–279. [Google Scholar] [CrossRef]
  72. Torres-Sospedra, J.; Rambla, D.; Montoliu, R.; Belmonte, O.; Huerta, J. UJIIndoorLoc-Mag: A new database for magnetic field-based localization problems. In Proceedings of the 2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Banff, AB, Canada, 13–16 October 2015. [Google Scholar]
  73. Torres-Sospedra, J.; Montoliu, R.; Mendoza-Silva, G.M.; Belmonte, O.; Rambla, D.; Huerta, J. Providing databases for different indoor positioning technologies: Pros and cons of magnetic field and Wi-Fi based positioning. Mob. Inform. Syst. 2016, 2016, 1–22. [Google Scholar] [CrossRef]
  74. Bacciu, D.; Barsocchi, P.; Chessa, S.; Gallicchio, C.; Micheli, A. An experimental characterization of reservoir computing in ambient assisted living applications. Neural Comput. Appl. 2014, 24, 1451–1464. [Google Scholar] [CrossRef]
  75. Barsocchi, P.; Crivello, A.; Rosa, D.L.; Palumbo, F. A multisource and multivariate dataset for indoor localization methods based on WLAN and geo-magnetic field fingerprinting. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  76. Shrestha, S.; Talvitie, J.; Lohan, E.S. Deconvolution-based indoor localization with WLAN signals and unknown access point locations. In Proceedings of the 2013 International Conference on Localization and GNSS (ICL-GNSS), Turin, Italy, 25–27 June 2013. [Google Scholar]
  77. Talvitie, J.; Lohan, E.S.; Renfors, M. The effect of coverage gaps and measurement inaccuracies in fingerprinting based indoor localization. In Proceedings of the International Conference on Localization and GNSS 2014 (ICL-GNSS 2014), Helsinki, Finland, 24–26 June 2014. [Google Scholar]
  78. Moayeri, N.; Ergin, M.O.; Lemic, F.; Handziski, V.; Wolisz, A. PerfLoc (Part 1): An extensive data repository for development of smartphone indoor localization apps. In Proceedings of the 27th IEEE Annual International Symposium on Personal, Indoor, and Mobile Radio Communications, PIMRC 2016, Valencia, Spain, 4–8 September 2016. [Google Scholar]
  79. Lymberopoulos, D.; Liu, J.; Yang, X.; Choudhury, R.R.; Handziski, V.; Sen, S. A realistic evaluation and comparison of indoor location technologies: Experiences and lessons learned. In Proceedings of the 14th International Conference on Information Processing in Sensor Networks, Seattle, WA, USA, 14–16 April 2015. [Google Scholar]
  80. 2016 Microsoft Indoor Localization Competition. Available online: https://www.microsoft.com/en-us/research/event/microsoft-indoor-localization-competition-ipsn-2016/ (accessed on 6 July 2017).
  81. Steinbauer, G.; Ferrein, A. 20 years of RoboCup. KI Künstliche Intell. 2016, 30, 221–224. [Google Scholar] [CrossRef]
  82. Osuka, K.; Murphy, R.; Schultz, A.C. USAR competitions for physically situated robots. IEEE Robot. Autom. Mag. 2002, 9, 26–33. [Google Scholar] [CrossRef]
  83. Davids, A. Urban search and rescue robots: From tragedy to technology. IEEE Intell. Syst. 2002, 17, 81–83. [Google Scholar]
  84. AUVSI SUAS Competition. Available online: http://www.auvsi-suas.org/competitions/2016/ (accessed on 4 November 2016).
  85. Van Haute, T.; De Poorter, E.; Lemic, F.; Handziski, V.; Wirström, N.; Voigt, T.; Wolisz, A.; Moerman, I. Platform for benchmarking of RF-based indoor localization solutions. IEEE Commun. Mag. 2015, 53, 126–133. [Google Scholar] [CrossRef]
  86. Pulkkinen, T.; Verwijnen, J. Evaluating indoor positioning errors. In Proceedings of the 2015 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Korea, 28–30 October 2015. [Google Scholar]
  87. Campos, R.S.; Lovisolo, L.; de Campos, M.L.R. Wi-Fi multi-floor indoor positioning considering architectural aspects and controlled computational complexity. Expert Syst. Appl. 2014, 41, 6211–6223. [Google Scholar] [CrossRef]
  88. Schauer, L.; Marcus, P.; Linnhoff-Popien, C. Towards feasible Wi-Fi based indoor tracking systems using probabilistic methods. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  89. Barsocchi, P.; Girolami, M.; Potortì, F. Track 1: Smartphone Based (on-Site). 2016. Available online: http://www3.uah.es/ipin2016/docs/Track1_IPIN2016Competition.pdf (accessed on 12 October 2017).
  90. Lee, S.; Lim, J. Track 2: Pedestrian Dead Reckoning Positioning (on-Site). Available online: http://www3.uah.es/ipin2016/docs/Track2_IPIN2016Competition.pdf (accessed on 12 October 2017).
  91. Fetzer, T.; Ebner, F.; Deinzer, F.; Grzegorzek, M. Multi sensor indoor localisation (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  92. Bai, Y.B.; Gu, T.; Hu, A. Integrating Wi-Fi and magnetic field for fingerprinting based indoor positioning system. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  93. Herrera, J.C.A.; Ramos, A. Navix: Smartphone Based hybrid indoor positioning (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  94. Guo, X.; Shao, W.; Zhao, F.; Wang, Q.; Li, D.; Luo, H. WiMag: Multimode fusion localization system based on Magnetic/WiFi/PDR. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  95. Carrera, J.L.; Zhao, Z.; Braun, T.; Li, Z. A real-time indoor tracking system by fusing inertial sensor, radio signal and floor plan. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  96. Zheng, L.; Wang, Y.; Peng, A.; Wu, Z.; Wen, S.; Lu, H.; Shi, H.; Zheng, H. A hand-held indoor positioning system based on smartphone (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  97. Chesneau, C.I.; Hillion, M.; Prieur, C. Motion estimation of a rigid body with an EKF using magneto-inertial measurements. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  98. Ju, H.; Park, S.Y.; Park, C.G. Foot mounted inertial navigation system using estimated velocity during the contact phase (work in progress). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  99. Peng, A.; Zheng, L.; Zhou, W.; Yan, C.; Wang, Y.; Ruan, X.; Tang, B.; Shi, H.; Lu, H.; Zheng, H. Foot-mounted indoor pedestrian positioning system based on low-cost inertial sensors (work in progress). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  100. Díez, L.E.; Bahillo, A. Pedestrian positioning from wrist-worn wearable devices (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  101. Strozzi, N.; Parisi, F.; Ferrari, G. A multifloor hybrid inertial/barometric navigation system. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  102. Xiao, Z.; Yang, Y.; Mu, R.; Liang, J. Accurate dual-foot micro-inertial navigation system with inter-agent ranging (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  103. Liu, Y.; Guo, K.; Li, S. D-ZUPT based pedestrian dead-reckoning system using low cost IMU (work in progress). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  104. Yang, J.; Lee, H.; Moessner, K. 3D pedestrian dead reckoning based on kinematic models in human activity by using multiple low-cost IMUs (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  105. Torres-Sospedra, J.; Jiménez, A.R. Track 3: Smartphone Based (off-Site). 2016. Available online: http://www3.uah.es/ipin2016/docs/Track3_IPIN2016Competition.pdf (accessed on 12 October 2017).
  106. Torres-Sospedra, J.; Jiménez, A.R. Track 3: Detailed Description of LogFiles and sUpplementary Materials. 2016. Available online: http://indoorloc.uji.es/ipin2016track3/files/Track3_LogfileDescription_and_SupplementaryMaterial.pdf (accessed on 12 October 2017).
  107. Beer, Y. WiFi fingerprinting using bayesian and hierarchical supervised machine learning assisted by GPS (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  108. Ta, V.C.; Vaufreydaz, D.; Dao, T.K.; Castelli, E. Smartphone-based user location tracking in indoor environment. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  109. Moreira, A.; Nicolau, M.J.; Costa, A.; Meneses, F. Indoor tracking from multidimensional sensor data. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  110. Knauth, S.; Koukofikis, A. Smartphone positioning in large environments by sensor data fusion, particle filter and FCWC. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  111. Rubio, M.C.P.; Losada-Gutiérrez, C.; Espinosa, F.; Macias-Guarasa, J. Track 4: Indoor Mobile Robot Positioning Competition Rules. 2016. Available online: http://www3.uah.es/ipin2016/docs/Track4_IPIN2016Competition.pdf (accessed on 12 October 2017).
  112. ASTi Automated Conveyance Web. 2016. Available online: http://asti.es/en/soluciones/transporte-automatico (accessed on 4 November 2016).
  113. Tiemann, J.; Eckermann, F.; Wietfeld, C. ATLAS—An open-source TDOA-based Ultra-wideband localization system. In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  114. Wang, Z.; Zhang, L.; Lin, F.; Huang, D. Aidloc: A high acoustic smartphone indoor localization system with NLOS identification and mitigation (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  115. Ho, S.W. Indoor localization using visible Light (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  116. Katkov, M.; Huba, S. TOPOrtls TPM UWB wireless system technical description (only abstract). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
  117. Ureña, J.; Villadangos, J.M.; Gualda, D.; Pérez, M.C.; Álvaro Hernández; Arango, J.F.; García, J.J.; García, J.C.; Jiménez, A.; Díaz, E. Technical description of Locate-US: An ultrasonic local positioning system based on encoded beacons. (Out of Competition) (work in progress). In Proceedings of the 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Madrid, Spain, 4–7 October 2016. [Google Scholar]
Figure 1. Floormaps of the IPIN 2016 competition: Positioning of people in real time.
Figure 1. Floormaps of the IPIN 2016 competition: Positioning of people in real time.
Sensors 17 02327 g001
Figure 2. Floor maps of the IPIN 2016 competition: Positioning of people off-line.
Figure 2. Floor maps of the IPIN 2016 competition: Positioning of people off-line.
Sensors 17 02327 g002
Figure 3. Robotic positioning area at IPIN 2016 Track4 test. The ground-truth track is hidden by a black cover.
Figure 3. Robotic positioning area at IPIN 2016 Track4 test. The ground-truth track is hidden by a black cover.
Sensors 17 02327 g003
Figure 4. The IPIN competition path for Track 1 and Track 2.
Figure 4. The IPIN competition path for Track 1 and Track 2.
Sensors 17 02327 g004
Figure 5. Cumulative distributions of localization error in metres for Track 1.
Figure 5. Cumulative distributions of localization error in metres for Track 1.
Sensors 17 02327 g005
Figure 6. Cumulative distributions of localization error in metres for Track 2.
Figure 6. Cumulative distributions of localization error in metres for Track 2.
Sensors 17 02327 g006
Figure 7. Cumulative distributions of localization error in metres for Track 3.
Figure 7. Cumulative distributions of localization error in metres for Track 3.
Sensors 17 02327 g007
Figure 8. Evaluation path and environment for Track 4.
Figure 8. Evaluation path and environment for Track 4.
Sensors 17 02327 g008
Figure 9. Cumulative distributions of localization error in metres for Track 4.
Figure 9. Cumulative distributions of localization error in metres for Track 4.
Sensors 17 02327 g009
Table 1. Evaluation set-up of papers presented during the 2016 Indoor Positioning and Indoor Navigation conference. PDF is Probability Density Function, CDF is Cumulative Density Function.
Table 1. Evaluation set-up of papers presented during the 2016 Indoor Positioning and Indoor Navigation conference. PDF is Probability Density Function, CDF is Cumulative Density Function.
Ref.SessionBase TechEvaluation ScenarioEvaluation Metrics
[29]Hybrid IMUifoot-mounted IMU
iBeacons
Smartphone
1 walking track of 5.4 kmFinal error
Trajectory
[30]Hybrid IMUSmartphone sensor fusion4-storey building (77 × 55 m)
4 walking tracks
Average Error
[31]Hybrid IMUXsens MTw inertial sensormulti-storey office buildingAverage Error
Root Mean Square Error
[32]Hybrid IMUData acquisition platform with IMU, UWB and BT2 rooms and 2 corridors
1 walk
Position error
Heading error
Trajectory
[33]Hybrid IMUvisual-magneto-inertial systemmultiple experiments: 1 m2 area; staircase; and motion capture roomfinal drift
Trajectory
[34]Hybrid IMUGoogle Nexus 5
Sensor fusion
1 walking trackTrajectory
[35]Hybrid IMUlow-end smartphone
low-end tablet
different tests F 1
Final Disc. Ratio
Average Error
Hints
[36]RSSno device info2 roomsAverage Error
Root Mean Square Error
Max. Error
CDF
[37]RSSHuawei MateGarageAverage Error
[38]RSS6 different android devicesLarge university hospital
>160.000 m2 (3 floors)
Average Error
Error variants
échet distance
[39]RSSCompetition Database
Wi-Fi fingerprinting
3 multi-storey buildingsAverage Error
Median Error
95th percentile
building & floor rate
[40]RSSDevice not definedChina National Grand Theatre
210 × 140 m
PDF
CDF
[41]RSSSimulation (RSS)8 × 8 mComplex Scatter plot
[42]RSS7 smartphone models
(50 subjects)
Set of routes
≈50 km + ≈7 km
Average error
Histogram
[43]MagneticMIMU Platform [44]2 walks: Office and mallAverage error
Trajectory and ROC curve
[45]MagneticMagnetic and camera: Project Tango and Google Nexus 5XNoreen and Kenneth Murray Library
2 different floors with strong and weak disturbances
Average error
Visual results
Matching rate
[46]UltrasoundsSenscomp 7000r and proposed HW platformNot describedAverage error by axis and angle
[47]UltrasoundsCORE-TX [48]Indoor Surveillance
Small office with 6 rooms
Abs. Error
[49]UltrasoundsAcoustic Beaconssmall area 3 × 3Average error
Trajectory
[50]UWBIMU; UWB; and Combination20 × 20 mAverage error
Trajectory
[51]UWBBeSopon and Decawave EVK100012.4 × 9.6 mAverage Error
Median Error
90th percentile
CDF
Trajectory
Histograms
[52]S.C.SensorSamsung smartphones and proprietary podometer3 BuildingsAverage Error
[53]Hybrid Syst.Sony Xperia Z3 Compact
Samsung Galaxy S5
Office space following an open
space concept (2600 m2 approx)
Average Error
Percentile < 5 m
[54]RFIDlow cost IMU (Xsens MTi)
Samsung Galaxy S2
1 hall (10 × 7.5 m)
1 corridor (50 m)
Average Error
Avg. lateral deviation

Share and Cite

MDPI and ACS Style

Potortì, F.; Park, S.; Jiménez Ruiz, A.R.; Barsocchi, P.; Girolami, M.; Crivello, A.; Lee, S.Y.; Lim, J.H.; Torres-Sospedra, J.; Seco, F.; Montoliu, R.; Mendoza-Silva, G.M.; Pérez Rubio, M.D.C.; Losada-Gutiérrez, C.; Espinosa, F.; Macias-Guarasa, J. Comparing the Performance of Indoor Localization Systems through the EvAAL Framework. Sensors 2017, 17, 2327. https://doi.org/10.3390/s17102327

AMA Style

Potortì F, Park S, Jiménez Ruiz AR, Barsocchi P, Girolami M, Crivello A, Lee SY, Lim JH, Torres-Sospedra J, Seco F, Montoliu R, Mendoza-Silva GM, Pérez Rubio MDC, Losada-Gutiérrez C, Espinosa F, Macias-Guarasa J. Comparing the Performance of Indoor Localization Systems through the EvAAL Framework. Sensors. 2017; 17(10):2327. https://doi.org/10.3390/s17102327

Chicago/Turabian Style

Potortì, Francesco, Sangjoon Park, Antonio Ramón Jiménez Ruiz, Paolo Barsocchi, Michele Girolami, Antonino Crivello, So Yeon Lee, Jae Hyun Lim, Joaquín Torres-Sospedra, Fernando Seco, Raul Montoliu, Germán Martin Mendoza-Silva, Maria Del Carmen Pérez Rubio, Cristina Losada-Gutiérrez, Felipe Espinosa, and Javier Macias-Guarasa. 2017. "Comparing the Performance of Indoor Localization Systems through the EvAAL Framework" Sensors 17, no. 10: 2327. https://doi.org/10.3390/s17102327

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop