This section summarizes the performance results for several aspects of the LocSpeck positioning system. It highlights the performance of the different components of the system: the timing of the ranging frames, the efficiency of the medium access protocol, and finally, the positioning accuracy of the LocSpeck system using several realistic test scenarios.
4.2. Positioning and Localization Performance
This section describes the positioning results of the LocSpeck system and compares the results with those obtained by the Pozyx system. The experiment took place on the second floor of the engineering block E (ENE) building at the University of Calgary. The surface area of the testing region is 360 m
2, the length of the testing region is 48 m, and the average width is 7.5 m.
Figure 13 shows the floorplan of the testing area, the locations of the Pozyx fixed anchors, and the locations of the reference points fixed on the floor of the testing area.
The experiments were carried out using four dynamic nodes—one of them is the main node, while the other three nodes are the supporting nodes. Each node consists of a smartphone and a ranging device—both were held by a human participant. The participants were moving within the test area randomly. The test area was open to the public; however, there was light traffic during the experiments. The different smartphones used in the experiments were equipped with different sets of sensors. When inertial sensors were available, they were used to provide a PDR solution using the standalone filter. The rest of this section will evaluate the positioning performance of the main node, which was connected to the Pozyx system to collect the ground-truth trajectory.
The complete experiment consists of three separate trajectories. Each trajectory starts with the four nodes at rest. Once the experiment starts, the nodes move in random trajectories inside the testing area. The nodes occasionally stop on one of the reference position markers on the floor. The reference trajectory of the main node is captured using the Pozyx reference system. The data logging application runs on each of the smartphones and collects readings from the available sensors, the Wi-Fi received signal strength indicator along with information about the corresponding access points, and the UWB range measurement along with the address of the collaborating node.
The raw data of the three trajectories are processed using different scenarios. The first scenario evaluates the standalone positioning performance of the main node. In this scenario, all the sensors available to the main node are used by the positioning filter. The Wi-Fi fingerprint map used in this scenario is the reference map, which was created previously using a dedicated run. The objective of this scenario is to establish a performance baseline to which the performance of the collaborative approach is compared.
The next two scenarios are collaborative positioning scenarios. In the first collaborative scenario, the main node is not using any of the available sensors, except for the UWB ranging device. At the same time, the supporting nodes are estimating their positions using all the sensors available to them, along with the Wi-Fi reference map. The main node uses only the relative range measurements to estimate its position. The objective of this scenario is to evaluate the effect of collaboration in the case of node asymmetry. The main node, in this case, is in a disadvantageous position where it could not estimate its location without external aid from the collaborating nodes.
In the second collaborative scenario, the main node and the supporting nodes use the complete set of available sensors. The objective of this test is to assess the effect of collaboration when the active node already has a good estimate of its position using only measurements local to the device, without any external sources.
In the final collaborative scenario, the positioning filter is providing position estimates based on a random-walk model only. This final scenario, though seems trivial, is used to establish the lower bound of the positioning performance.
The rest of this section is divided into three subsections.
Section 4.2.1 describes the process of generating the ground-truth trajectory of each of the test trajectories. It will also elaborate on the process of generating the Wi-Fi RSSI fingerprint maps.
Section 4.2.2 is dedicated to the standalone performance using all the sensors available to the dynamic node, and without using any collaboration or relative range measurements for the positioning. Finally,
Section 4.2.3 discusses the performance of the collaborative positioning approach. In this subsection, different collaboration scenarios are evaluated.
4.2.1. Reference Trajectories and Fingerprints Maps
The positioning performance of the LocSpeck system is evaluated using three different trajectories covering the same test area.
Figure 14 shows the reference solution for the test trajectories. This reference is created using the Pozyx UWB-based system. The locations of the Pozyx anchors are highlighted in
Figure 13. The position error is evaluated at the pre-surveyed reference points.
The reference solution for each trajectory is compared to the pre-surveyed reference points on the ground. The performance of the Pozyx solution is summarized in
Table 7. For all the tested scenarios, the position error is evaluated when the main node reaches and stops over one of the reference points. This event is captured from the Pozyx reference trajectory in addition to the stop detection algorithm applied to the accelerometer data from the node of interest. The Pozyx trajectory is not used directly to evaluate the performance. It is used to indicate the location of the nearest reference point on the floor, which location is known precisely, and this reference point is used to evaluate the error in the position estimate. The small positioning error of the Pozyx system is vital to be able to distinguish between the densely placed reference points.
The reference radio map is created using the Pozyx reference trajectory in a separate run. The fingerprint map is built by observing the signal strength indicator at the reference points, then fit a Gaussian process model for each visible access point, using the position and signal strength pairs. During the positioning scenarios, the resulting Gaussian process models are used by the different dynamic nodes to aid the positioning filter.
4.2.2. Standalone Positioning Results
The standalone scenario results comprise three trajectories for the main node. Due to the stochastic nature of the particle filter, each trajectory is run through the positioning filter 20 times to produce more robust statistics of the filter performance. The standalone positioning error statistics for the three trajectories are summarized in
Table 8. For this scenario, the main node is using all the sensors available onboard the smartphone, i.e., gyroscope, accelerometer, and Wi-Fi information. The root-mean-square (RMS) positioning error across the three trajectories ranges from 4.28 to 6.65 m, while the overall RMS positioning error, in this case, is 5.92 m, as shown in
Table 8. Since these results depend mainly on Wi-Fi fingerprinting, the performance might be affected by the presence of high human mobility in the test area [
56]. Additionally, the overall performance of the standalone positioning scenario can be improved by augmenting the solution with other techniques, such as the geomagnetic field anomalies or visual scene recognition [
57,
58,
59,
60]. However, the main objective of the standalone filter in this work is to form the performance baseline, to which the effect of collaboration between nodes is to be measured, as discussed later in
Section 4.2.3.
Table 9 shows the results of the IPIN competition winners from 2015 to 2018 [
61,
62]. These results are shown for comparison with the achievable performance of the standalone mode of the LocSpeck framework. The 75% percentile of the position error is not far from the top indoor positioning system available, although the winner of the 2018 off-site track can achieve 1.1 m accuracy.
It is worth noting that the inclusion of the results in
Table 9 does not imply that the different systems can be compared directly since the performance of any positioning system will vary according to the operating conditions. The sole purpose of showing these results is to give a sense of the performance of the current state-of-the-art systems. The performance of the standalone solution acts as a baseline to which the collaborative positioning approach is evaluated.
4.2.3. Collaborative Positioning Results
In this section, two collaboration scenarios are considered. The first scenario consists of four nodes, the main node, and three supporting nodes. The main node will not use any of its onboard sensors. However, the main node will only use the UWB device to measure the relative ranges between itself and the other collaborating nodes. The other supporting nodes will use all the sensors available to them, along with the range measurement device. The objective of this scenario is to evaluate the achievable performance using relative range measurements to dynamic nodes. The second scenario is similar to the first one with one change: the main node will be using all the available sensors, in addition to the range measurement device. The objective of this scenario is to assess the effect of the collaboration on the participating nodes. In addition to these two cases, the results of the positioning using the random-walk model only is showed as well.
• Positioning using relative range measurements
Table 10 shows the performance summary for the collaborative positioning approach, using the relative range measurement only. As expected, the performance, in this case, is worse than the performance of the standalone case. However, in this scenario, the mobile node is using only the range measurements, without any of the onboard sensors. In this case, the use of the collaborative positioning framework improves the positioning error for the main node by 50%, above the performance of the random-walk model only.
Another factor that can affect the performance of the main node in the collaborative setting is the availability of the supporting nodes. The availability of the nodes is illustrated in
Figure 15, where each horizontal line represents the activity of the corresponding node. The gaps in the lines indicate that the node is not active. Although there are three supporting nodes, only two of them are active most of the time, and the third is fluctuating between the active and inactive states. The effect of the node availability is evident in the second trajectory, which has the most significant errors among the three trajectories.
• Positioning using relative range measurements and all sensors
This scenario evaluates the effect of the collaboration on the main node while using the full set of sensors available on board. When using all sensors, the main node should achieve a performance level similar to the performance of the standalone solution.
Table 11 shows a summary of the positioning performance of the collaborative positioning, while the main node is using all its sensors. The collaboration negatively affected the performance of the main node, when it uses all the sensors. The mean error has increased by 27%, the RMS error by 16.6%, and the 75% percentile error by 32.2%.
• Positioning using random-walk model only
Before proceeding to evaluate the performance of the collaborative approach using relative range measurements, it would be useful to consider the error in the absence of the collaboration between the main node and the other nodes.
Table 12 shows the positioning error statistics in this case. Without collaboration, the mean of the position error is 18.46 m, while the RMS of the position error is 21.60 m. The 75% percentile of the error is 26.57 m. The objective of this scenario is to establish a lower bound on the positioning performance.