3. Materials and Methods
Eight research methods were used in this study: a structured questionnaire with an integrated OSIVQ spatial abilities test [64
], eye tracking, creation of navigation instructions, mental maps, estimation of direction, estimation of route length, landmark identification in photographs, and additional questions. As not all of the results were in the scope of this paper’s study, only the methods whose results are referred to in this paper are introduced.
Mobile SMI eye tracking glasses recording at 60 Hz were used to measure a participant’s point of attention. The data describing participants’ eye movements were processed and analyzed using BeGaze 3.5.101, which is distributed with the SMI glasses. Special attention was paid to eye movements at decision points. Records of each participant’s route were processed manually using the Semantic Gaze Mapping method. Areas of Interest (AOIs) were sketched in advance around objects considered landmarks according to their semantic significance. Each fixation was then marked manually from each participant’s eye tracking record.
Quantitative content analysis was used to analyze navigation instructions by observing the occurrence and descriptive characteristics of landmarks created in these instructions. Special code categories were created for functional landmarks (doors, stairs) and the green evacuation signs indicating the direction of the designated evacuation route. The remaining landmarks were not differentiated into categories. Several categories related to the spatial layout of the designated route were also created: direction change, spatial relationships, distances and destination. All the resulting codes were categorized according to the spatial knowledge developed by Siegel and White [6
] (Figure 1
). The quantitative content analysis was undertaken using ATLAS.ti (v. 7.5.7).
The accuracy of navigation instructions was evaluated by separating the entire route into smaller sections. Each section always began and ended with a decision point, in other words, a turning point where the instruction had to be correctly described. If a participant gave incorrect directions or did not state a direction at all, an error was counted. An error was also counted if a participant described extra segments (e.g., extra staircases) or failed to mentioned a segment. This method was inspired by Agrawala’s [83
] dissertation research, which emphasized the importance of navigation instructions at decision points at the expense of local and contextual information and also corresponds to the findings of Kim and Hirtle [84
], who suggested that knowledge of routes is represented as a sequence of intersection-based choice points where procedural decisions must be made.
Estimation of direction and route distance was also employed as a method. Direction estimations were measured with a compass in degrees and distance estimations were obtained in meters. In the analysis, variations of estimations from real values were processed.
Identification of landmarks from photographs was used to test the participants’ visual memories. Photos were taken of different building interior scenes with landmarks, but only some of them were located on the evacuation route. Participants were then asked to separate these 17 photos of landmarks into three categories depending on whether they had seen the same landmarks while navigating the route (“Yes”, “No”) or whether they could not decide with certainty having seen the landmark (“Not sure”). For each correct categorization, participants were awarded 1 point, for incorrect answers, −0.25 points, and for “Not sure”, 0 points.
3.1. Experimental Design
In following the call to improve consistency and detail in reporting experimental design and to support the transparency, transferability and reproducibility of research studies [69
], a detailed description of our experimental design is provided below. A brief overview of the experiment’s structure and stages is shown in Figure 2
. The research methods whose results are evaluated in this paper are highlighted with a black outline (Figure 2
3.1.1. Personal Profile
A few weeks before the experiment, participants completed a web questionnaire at home. Besides basic personal information, it contained an integrated OSIVQ questionnaire [64
] to measure the participant’s cognitive style. Based on the data collected from the questionnaire, participants were categorized into two groups (two learning stimuli) to obtain a balanced distribution of cognitive styles, gender and experience with maps.
3.1.2. Intro Stage
The experiment was conducted at the Headquarters of Masaryk University. Participants were not familiar with the selected building before testing. They were brought by alternative route to the meeting room, which was the starting point of the designated evacuation route and where the experiment began.
After welcoming the participant, each provided informed consent to participate in the research study. The study was conducted in accordance with the Declaration of Helsinki, and the research protocol was approved by the Ethics Committee of Masaryk University. The eye tracking device was then calibrated, and the experiment began with an introductory stage during which the participant was briefly acquainted with the basic structure of the experiment. The participants also received instructions for working with a given visualization according to their grouping. The virtual tour group obtained introductory information along with instructions and references to the virtual tour in the form of an offline website. The floorplan group was presented with a PowerPoint presentation.
3.1.3. Learning Stage
Participants from the virtual tour group learned the evacuation route from the virtual tour available at: http://ofm.ukb.muni.cz/vt/nav/rektorat/
. An example from the virtual tour conditions is shown in Figure 3
. Participants from the floor plan group learned the evacuation route from the second part of the PowerPoint presentation, which showed the schematic plan of the individual floors of the building with the designated evacuation path (Figure 4
). The complete experimental stimuli given to the participants are available in the Supplementary Materials
. During the learning stage of the experiment, participant’s eye movements were monitored using the mobile eye tracking device. Participants in both visualization groups were given no time limit for the learning stage and could proceed backwards through the visualization.
3.1.4. Before Navigation Stage
In the first interview, participants were instructed to draw the route they had learned and to create navigation instructions. They were asked to imagine instructing a visitor who had never been inside the building. They were also asked to point towards the route’s destination (i.e., the direction) and estimate the route’s distance.
After the interview, the mobile eye tracker was calibrated for a different focal distance and participants were sent out of the room to navigate along the evacuation route. Each participant navigated the 86-meter route individually while one of the research team members followed behind at a reasonable distance to ensure the participant’s safety.
3.1.6. After Navigation Stage
When the participant arrived in the main lobby of the building, which was the designated route’s destination, the mobile eye tracking device was removed. A second interview was conducted during which the participant could modify their route drawing and was again asked to create navigation instructions for the route they had just traveled. They were asked to indicate the direction towards the route’s starting point, estimate the route’s length and answer some additional questions. The final task was identifying landmarks from photographs.
3.2. Determination of Experimental Hypotheses
Our experimental hypotheses were proposed on the basis of the results of previously conducted studies. According to the theory of Naïve Realism [44
], more realistic visual stimuli lead to the creation of vague mental representations, which, if the task requires accurate judgment, are not sufficient for identifying correct solutions to the task. In our experiment, the navigation task did not require accurate judgment. A higher level of realism in virtual tour stimuli generates increased cognitive load, whereas the virtual tour’s navigational merits should contribute to the development of procedural knowledge of the evacuation route. We therefore expected both experimental groups to perform similarly on this task.
Users who learn the route from the schematic floor plan navigate along the designated evacuation route as efficiently and effectively as users who learn the route from the virtual tour.
However, we also wanted to investigate the influence of the selected visual stimuli on participants’ mental representations of the environment. These representations were developed by participants after learning the route, and we were interested in how these mental representations changed after they had navigated the route in the real environment. Overall, we expected that users who learned the route from the virtual tour would develop a more detailed mental representation of the environment than users from floor plan group. Based on previous research and the research methods used, we hypothesized the following:
Virtual tour users will concentrate more on landmarks and their visual characteristics while navigating the route than schematic floor plan users.
Hypothesis 2.2: Virtual tour users will create more detailed navigation instructions and include more landmarks and visual characteristics in these instructions than schematic floor plan users
Based on the Self and Golledge [59
] findings, we hypothesized that females would perform better in the task of identifying landmarks from photographs whereas males would perform better in estimating directions and route length.
Males will more accurately estimate the direction and length of the route than females. Females will identify more landmarks correctly in photographs than males.
In total, 36 participants participated in the experiment: 17 in the floorplan group and 19 in the virtual tour group. All participants were volunteers and could quit the experiment at any time. The ratio of women to men was reasonably balanced in both groups: seven women (41%) to ten men in the floorplan group and nine women (47%) to ten men in the virtual tour group. More than 80% of the participants were 18 to 26 years old. No participants were aged over 40 years. Participants were of Czech and Slovak nationalities and mostly university graduates or students and had different work backgrounds.
In this chapter, results are reported in the context of the hypotheses. All data collected in the experiment was checked for normality using the Shapiro–Wilk normality test [88
]. The differences between tested groups (between-subject design) were examined using Welch’s two sample t-test (for normally distributed data) [89
] and the Mann–Whitney–Wilcoxon non-parametric test [90
]. The differences between experimental stages (within-subject design) were examined using a Wilcoxon signed rank test [91
]. The resulting p
-values < 0.05 are reported as statistically significant. We also report effect sizes using Cohen’s d
values, and following the guidelines [92
], interpret them as 0.2, 0.5, 0.8 (Cohen’s d
) and 0.1, 0.3, 0.5 (r
) as small, medium and large, respectively. All boxplots use a 1.5xIQR (interquartile range) rule and Tukey’s fences [93
] for whiskers and identifying outliers. Asterisk notation is used to visualize statistical significance (ns
-value > 0.05, *: p
-value ≤ 0.05, **: p
-value ≤ 0.01, ***: p
-value ≤0.001). The statistical analysis of results was conducted in RStudio (v.1.0.153). Descriptive statistics for all measurements for both experimental groups are summarized in Table 1
H1: Users who learn the route from the schematic floor plan navigate along the designated evacuation route as efficiently and effectively as users who learn the route from the virtual tour.
The efficiency and effectiveness of route navigation was measured by the total time spent navigating the route and the number of wrong turns taken. Data for the total navigation time were extracted from eye tracking measurements. The floor plan group showed a higher variability in time spent navigating the route, which was caused by those participants making navigation errors (Figure 5
). The difference in time spent navigating was tested using a two-tailed Mann–Whitney–Wilcoxon non-parametric test. The results were not statistically significant between the groups for all participants (α
= 0.05; W = 143.00; p
-value = 0.5728, r
= 0.0306) nor if participants who made navigation errors are excluded (α
= 0.05; W
= 46.00; p
-value = 0.0609; r
Considering the navigation effectiveness, all of the participants could find the designated route’s destination. While navigating the route, 21% of participants from the virtual tour group made a mistake during navigation, and one participant even failed twice. From the floor plan group, 35% of participants took a wrong turn while navigating, but none of them repeatedly. The difference between navigation effectiveness in the two groups was tested with a two-tailed Mann–Whitney–Wilcoxon non-parametric test. The results between the groups were not statistically significant (α = 0.05; W = 178.50; p-value = 0.5041; r = 0.0017).
Route deviations for each decision point are illustrated in Figure 6
. Photographs of the decision points are available as Supplementary Materials
. Only participants from the floor plan group (three of them) deviated from the designated route at the final decision point (5. DP), where there was an alternative exit from the building through a white glass door. Two participants from the virtual tour group deviated from the designated route at the very start (cyan and purple color) and proceeded straight ahead instead of turning right. Two participants (virtual tour group and floor plan group) passed by the staircase instead of using it (pink and brown color). Two participants from the floor plan group turned left instead of right after the first staircase (red color). One participant from the virtual tour group (dark blue color) turned left at the start of the designated route and wanted to ascend two floors on the first staircase, but then changed his mind and proceeded correctly.
As both the effectiveness and efficiency of navigation did not significantly differ between the groups of participants, hypothesis H1 can be confirmed (Users who learn the route from the schematic floor plan navigate along the designated evacuation route as efficiently and effectively as users who learn the route from the virtual tour.).
H2.1: Virtual tour users will concentrate more on landmarks and their visual characteristics while navigating the route than schematic floor plan users.
The degree of attention participants paid to landmarks was monitored using eye tracking. Route intersections (six decision points) where landmarks provided the greatest advantage to support decision-making during navigation were especially monitored. To quantify the performance of both participant groups at decision points, fixation and saccade count metrics were used. The differences in mean fixation and saccade counts at decision points were examined with a two-tailed Welch’s two sample t-test. The results between the groups were statistically significant (fixation count: α
= 0.05; t
= −3.26; df
= 20.21; p
-value = 0.0039; saccade count: α
= 0.05; t
= −3.41; df
= 20.13; p
-value = 0.0028, see Figure 7
). A strong effect of the learning stimuli types on both eye fixation count (dCohen
= 1.35) and saccade count (dCohen
= 1.41) was observed.
The degree of attention paid to specific landmarks at individual decision points (DP) was also investigated. The route had six decision points where participants had to decide which direction they would proceed. AOIs were created at these decision points for each object considered a landmark according to its semantic significance (see Supplementary Materials
Using the Semantic Gaze Mapping method, eye tracking records were processed for each participant. Fixations outside AOIs were not analyzed. Figure 8
shows the sequence of fixations at AOIs for each DP and participant. At the first DP, no difference was observed between the experimental groups. The fire extinguisher was as attractive to the floor plan group as the virtual tour group. A bigger difference can be seen at the second DP, where participants from the floor plan group paid more attention to the evacuation sign than the virtual tour group. At the third DP, participants from the floor plan group focused mainly on doors and windows and less on flowers, floor level signs and radiators. Participants from the virtual tour group focused much more on floor level signs and radiators. The degree of attention paid to doors was almost the same. The fourth DP had only two AOIs: stairs and a green evacuation sign, which attracted only one participant’s attention from the virtual tour group. More fixations on stairs were observed in the virtual tour group. At the fifth DP, participants from the virtual tour group paid more attention to flowers than participants from the floor plan group. Both groups showed almost the same number of fixations on radiators and windows. A high number of fixations on the white door was observed in some individuals from both groups. The final DP showed no difference between the groups.
Since the degree of attention paid to landmarks appeared to be dependent on the landmark type, hypothesis H2.1 cannot be confirmed (Virtual tour users will concentrate more on landmarks and their visual characteristics while navigating the route than schematic floor plan users.).
H2.2: Virtual tour users will create more detailed navigation instructions and include more landmarks and visual characteristics in these instructions than schematic floor plan users.
The navigation instructions created by the participants were analyzed in terms of their accuracy and informational content. The accuracy of navigation instructions was evaluated using the method described in Section 3
. Using the content analysis, the frequency of code categories detected in the navigation instructions was counted and compared. The richness of the navigation instructions was measured by the sum of all the code categories in a particular statement. Figure 9
and Figure 10
illustrate the differences in accuracy and richness of the instructions between groups and experiment stages. The differences were tested using a Mann–Whitney–Wilcoxon non-parametric test (between-group comparison) and Wilcoxon signed rank test (between-stages comparison). The results are shown in Table 2
and Table 3
. Statistically significant differences (α
= 0.05) are highlighted in bold cursive
The occurrences of individual code categories in navigation instructions were also analyzed. Because a strong effect of learning stimuli on the richness of the created navigation instructions was observed (richness was significantly lower in both experiment stages in the floor plan group), the relative number of occurrences was calculated for a between-group comparison. The frequency of each code category was divided by the sum of occurrences for each participant. Figure 11
illustrates the differences in relative occurrences of individual code categories in navigation instructions between groups and experimental stages. The differences were tested using a Mann–Whitney–Wilcoxon non-parametric test (between-group comparison) and Wilcoxon signed rank test (between-stages comparison). The results are shown in Table 2
and Table 3
. Statistically significant differences (α
= 0.05) are highlighted in bold cursive
Participants from the virtual tour group created significantly richer navigation instructions. In the navigation instructions created before navigation, they also included significantly more landmarks (except in the “stairs” code category), but significantly fewer visual characteristics than participants in the floor plan group. After navigation, the number of mentioned landmarks and their visual characteristics was more balanced. Based on these results, hypothesis H2.2 can be partially (except in the “stairs” category) confirmed (Virtual tour users will create more detailed navigation instructions and include more landmarks and visual characteristics in these instructions than schematic floor plan users.).
H3: Males will more accurately estimate the direction and length of the route than females. Females will identify more landmarks correctly in photographs than males.
Gender differences were also studied in the data analysis. Descriptive statistics for selected metrics for both genders are summarized in Table 4
and Figure 12
. The differences were tested using a Mann–Whitney–Wilcoxon non-parametric test (between-group comparison) and Wilcoxon signed rank test (between-stages comparison). The results of the differences are shown in Table 4
and Table 5
. Statistically significant differences (α
= 0.05) are highlighted in bold cursive
Before navigating, males estimated route length and direction significantly better than females. After navigating, males performed better only in route length estimation. An interesting trend was observed in male participants, their overall direction estimation deviation being significantly higher (α = 0.05; V= 42.50; p-value = 0.0356; r = 0.2928) after navigating than before navigating. Females performed significantly better in the landmark identification task than males. Since females estimated directions equally as males after navigating, hypothesis H3 can be only partially confirmed (Males will more accurately estimate the direction and length of the route than females. Females will identify more landmarks correctly in photographs than males.).
To summarize, participants from both groups were able to successfully navigate the designated evacuation route. The total time spent on the route was not statistically different between groups. Nevertheless, the comparison of descriptive statistics indicated possible group differences. The most significant difference may be the variation in time spent on the task (higher in the FP group), which differs between the FP and VT groups if all participants are included (Table 1
). It possibly indicates the different nature of navigation information derived from the stimuli. When the participants from FP group made navigation errors, it took them more time to get back to the intended route. This could possibly be explained by the absence of general but visually attractive landmarks (flowers, radiators, etc.) on the floor plan stimuli. This finding needs to be verified in future studies.
In a closer analysis of the route travelled by participants who made navigation errors, similarities in both groups were observed (Figure 6
). Participants from the floor plan group made errors at the fourth (3. DP) and the final DP (5. DP), turning in the wrong direction. Participants in this group perhaps only had a limited mental image of decision points, as opposed to the virtual group, who had seen detailed representations of the actual DPs in the virtual tour. At the fourth DP (3. DP), participants in the virtual group could decide according to the 2nd floor level sign, which they had previously seen in the learning stage. Analyzed eye tracking data confirmed a greater degree of attention given to this sign by participants in the virtual tour group. We argue that in both cases, the participants from the virtual tour group benefited from the additional visual information they acquired in the learning stage and therefore made correct decisions as opposed to participants from the floor plan group. However, only participants from the virtual tour group committed navigation errors at the start of navigation (0.DP). Two participants proceeded straight ahead instead of turning right at the second DP (1. DP). These results are in accordance with Dalton’s conclusions [11
] and also with the initial segment strategy [12
] that navigators are literally “following their noses” and prefer routes that have an initial straight segment. This strategy could explain participants’ decisions when they were not sure how to proceed along the route. Furthermore, acquiring the same orientation as in the learning stimuli was not possible, and initial orientation therefore required a different degree of mental translation. Floor plan stimuli also provide information about the surrounding environment and therefore better represent the spatial context of the designated route. The “turning effect” used in the virtual tour could also have affected the initial choice, as more participants stated in the interviews that they needed time to understand it.
It is important to emphasize that the two visualizations in the experiment provided different levels of detail about the evacuation route and that the virtual tour more closely represented the real indoor navigational experience. In the main task, both groups performed similarly, yet the two groups developed different mental spatial representations. This finding is consistent with studies conducted by Evans and Pezdek [18
] and Throndyke and Hayes-Roth [19
]. Considering the results presented in Section 4
, it could be said that the participants who learned the route from the virtual tour developed a more detailed mental spatial representation. This was the visualization with a higher level of realism. Participants learning from the virtual tour created richer navigation instructions, as expected in our hypothesis. After navigating the designated route, the richness of navigation instructions created by the virtual tour group decreased. In the floor plan group, however, the opposite effect was observed. One reason may be that in the second interviews, participants tried to mention only the relevant landmarks they had used for orientation. A similar conclusion was reached by Čížková [66
Based on the evaluation of eye tracking data for route decision points, a statistically higher number of fixations and saccades were found in participants from the virtual tour group. Overall, participants from the virtual tour group focused more on landmarks they knew from the virtual tour (flowers, radiators, door signs) than participants from the floor plan group, who had not previously seen those landmarks on the floor plan stimulus. Participants from both groups focused mostly on functional landmarks, which corresponds to the findings in studies by Ohm et al., [82
] and Viaene et al., [72
], and also to the content analysis results of the navigation instructions in which participants often mentioned the doors and stairs. They also frequently mentioned the evacuation arrows indicating the route’s direction, which were therefore very helpful orientation cues. However, these are relatively small objects and fixations cannot be included in the area of interest because of the eye tracking device’s degree of accuracy.
The results from the analysis of gender differences are consistent with the results of studies examining similar issues. Self and Golledge [59
] showed that females orient themselves more according to landmarks and their visual characteristics, which corresponds to the higher score of points accumulated in the task of identifying landmarks from photographs. The results also demonstrated that males oriented themselves more using direction and distance, which corresponds to the smaller spatial deviations in their estimations observed in our study. Interestingly, the overall estimations of direction by females became better after navigating the route. By contrast, mean deviation in the male participants doubled. This was mainly, however, caused by the estimation of one male participant, which was significantly worse after navigation. Female participants scored lower in estimating route length, but even the mean deviation scored by male participants was as high as the actual route length. During interviews, most of the participants reported that it was more difficult to estimate distance in indoor environments and that they tried to compare this distance, for example, to an everyday distance such as walking to a bus stop or around a running track.
Boumenir et al., [57
] conducted an experiment that was in many ways similar to this study; however, they found that virtual tour users performed more poorly. The lower effectiveness of the virtual tour may have been the result of a different design. Directions were represented using arrows that always pointed straight ahead on the monitor. Users therefore relied on a linear mental representation of space. The virtual tour in our study was designed so that when a participant clicked on another decision point, the scene automatically and slowly turned in the direction along which the route continued. We did not observe this effect providing any advantage or disadvantage in creating non-linear mental spatial representations (e.g., survey knowledge). However, how a virtual tour is designed and how a user can interact with it specifically influences their performance in solving tasks. Our results are also likely to differ since our experiment was conducted inside a building as opposed to a forest park outside.
Most of the studies mentioned above differed in some way from our study in terms of experimental conditions, visual stimuli, number of participants, and so on. This study is one of the first examining the use of a virtual tour for evacuation from a building. It is important to note that the results cannot be easily generalized because they relate to the interior of a particular building and designated route, a limited number of participants and the specific equipment used in the experiment. All the factors mentioned could possibly influence the documented results.
6. Conclusions and Future Work
Visualizations used in navigation provide the basis for developing the mental spatial representations that are crucial to effective navigation. In our study, we simulated a simple evacuation scenario of a person, who was trying to find the way to the evacuation assembly area after learning the route from presented stimuli. The degree of realism of geographic visualization is known to affect user performance in navigation tasks. The main aim of the study was to review the role of different levels of realism in graphic stimuli (2D floor plan and virtual tour) on the accuracy and efficiency of indoor navigation.
Two variants of cartographic visualizations were created for the study—a schematic plan and a virtual tour—that depicted the same evacuation route established for the purpose of the experiment. The main aim of the study was to review the role of different levels of realism in graphic stimuli (2D floor plan and virtual tour) on the accuracy and efficiency of indoor navigation. Specifically, successful completion and the level of detail of mental spatial representations developed by different user groups were examined.
The results of the experiment showed that the type of cartographic visualization did not influence whether participants completed the navigation task successfully, but the participants who learned the route from the virtual tour developed more detailed mental spatial representations of the building’s interior. A strong effect of learning stimuli on the overall richness of created navigation instructions was observed. Regarding specific landmark code categories, a greater difference was observed before the navigation stage. Navigation experience seemed to balance the observed differences, but not entirely. The type of the learning stimulus had a strong effect on the navigation process, influencing eye movement activity. Participants from the virtual group demonstrated significantly more fixations and saccades, which could imply that different cognitive processes were involved in solving the task. This matter needs to be examined more closely, however. Since the presented study was designed with high ecological validity and considered a real-life building and experience, it provides valuable insight on how the level of realism of a cartographic visualization influences the user performing the evacuation task. The results, however, relate highly to the relatively small number of participants and the building’s specific characteristics and indoor spatial metrics.
Future work could involve analyzing individual differences, especially more detailed studies of the eye tracking data collected in the learning stage from participants who made navigation errors. For more generally conclusive results, additional experiments in different buildings involving more participants etc. would be required. Bao et al., [94
] argued that it would be better to propose categories of buildings based on their navigability rather than test individual buildings. Real-time monitoring of human behavior complemented by space syntax describing specific environments could provide general results for predicting navigation success.