Methodology for Conceptual Navigational 3D Chart Assessment Based on Eye Tracking Measures

Abstract

The aim of this study was to develop a comprehensive methodology for assessing the effectiveness of conceptual navigation maps. To achieve this, a set of indicators was developed to simplify and standardise the evaluation process. A key challenge in analysing the effectiveness of maps is the variety of ways to assess their effectiveness (eye-tracking measures, visual interpretation, questionnaires), which often leads to ambiguous interpretations. In the initial phase, three separate indicators were established: readability of the map, correctness of map symbol identification, and the time required to identify these objects. For the readability indicator, a correlation analysis with eye-tracking measurements and a heatmap decomposition was carried out, which partly reduced the complexity of the data. This led to the development of a single universal indicator, representing the overall effectiveness of the map in numerical form and allowing a simple comparison of maps. Based on the methodology developed, we were able to conclude that the designed 3D concept maps are more effective than their 2D counterparts. The methodology and universal indicator presented in this study can be applied in practice to evaluate entire series of concept maps and provide designers—including those outside academia—with indicator-based tools to evaluate the effectiveness of maps.

1. Introduction

Maps unquestionably stand as one of the most important human inventions. For millennia, maps have allowed people to describe the Earth, travel the world, and return to places they have already been. During the 19th century, nations with access to water areas and with a developed merchant navy established their own Hydrographic Offices, whose aim was to produce nautical charts and nautical publications [1]. Another progression, linked to the development of electronics and the digitalisation of paper resources, made it possible to develop electronic navigational charts (ENCs). Nowadays, the map content and the format of the navigation map itself are being standardised by the IHO (International Hydrographic Organisation). Its primary purpose is to be used with an Electronic Chart Display and Information System (ECDIS) to meet the International Maritime Organization (IMO) in the International Convention for Safety of Life at Sea [2] chart carriage requirements. The product specification is detailed in S-101 [3], provided and developed by the IHO.

From paper charts, people have switched quite easily to electronic charts presented in the ECDIS systems, where it is possible not only to display the chart content but also to carry out simple analyses, such as route planning, observing the movement of other vessels in the area or determining the safety contour (isobath) [4]. Two important points about nautical charts, in particular, should be noted: firstly, nautical charts can only be understood with full knowledge and familiarity with navigational techniques; and secondly, nautical charts have always been primarily a navigational instrument rather than intended to represent the Earth [5].

1.1. Technological Challenges

Given technological developments, the visualisation of cartographic information is also changing. Nowadays, this information can take different forms, going beyond the predefined IHO standards, as can be observed in various Marine GIS systems [6,7,8,9,10,11]. Other forms and techniques for presenting nautical and maritime information may include the marine environment [12], conception of spatial presentation of ENC [13], web ENC [14], GIS systems with their own cartographic signs representing features in the area [15,16], 3D visualisation [17], marine cadastre [18], underwater topography [19], dynamic ENC map presentation [20], 3D data fusion [21], use of 3D data in simulators [22], land-based piloting [23], and real-time navigation monitoring [24].

It should also be borne in mind that new forms of visualisation are often created by the type of device. In the case of navigation maps, one can mention the increasingly widespread use of mobile phones or smartwatches. Reducing the visualisation area of quite complex navigation information can be a considerable challenge in optimising map effectiveness, especially when considering the small display sizes of smartwatches in the order of 30–35 mm and a resolution of 240–280 pixels [25].

New trends go even further towards augmented reality [26]. An example of this technology in application to inland navigation can be found in [27] and for increased efficiency of ship manoeuvring in [28]. Ready-made systems using AR are offered by Raymarine [29] and Furuno [30]. At present, however, these are not standardised on ships, but nevertheless, due to their usefulness, they should soon become one of the navigational aids, enabling the necessary navigational information to be analysed quickly and efficiently. Given the changing possibilities of visualising nautical information, the development of effective forms of visualisation is a significant challenge, especially considering the navigational safety factor.

1.2. Application of Neuroscience in Effectiveness of Map Studies

As can be seen, the dynamic development of technology implies new forms of visualisation of nautical charts, which in turn generates the need for research into their correct editing. In this context, it is also important to emphasise the complexity of the presentation of cartographic information, which depends on the graphic variables used (e.g., colour, shape, size), map symbols, map composition, the functionality of the device presenting the cartographic content, or the technical possibilities of visualisation (display size).

One of the most effective techniques for studying maps is the use of cognitive neuroscience methods—an interdisciplinary field that combines psychology and neuroscience to study how the brain enables cognitive functions such as attention, memory, perception, and decision-making [31]. This approach provides a better understanding of the mechanisms responsible for these processes. Various techniques can be used in cognitive neuroscience, such as EEG, fMRI, eye-tracking, or GSR [32,33,34]. In the present study, the eye-tracking technique was used, which involves tracking the eye movements of map users to analyse their performance.

In the case of maps, cognitive neuroscience provides a better understanding of how users interpret, analyse, or use spatial information [35]. Maps themselves should be seen as a complex form of graphic communication that requires the involvement of different cognitive functions, such as visual perception or decision-making processes. The results of the research can provide valuable information on the effectiveness of the map design, including the map symbols and selection of graphic variables, the positioning of different map elements, the assessment of map readability, or the perception of different map data, such as distance or direction.

Research demonstrating neuroscience techniques also applies to various aspects of map evaluation like its usability and user performance using eye-tracking techniques [36,37]. In cartographic and geovisualisation research, these techniques have been applied to study the impact of graphical alternatives [38], topographical height visualisation processing [39], identification of critical points along cartographic linear features [40], analysis of thematic maps [41,42], evaluation of interactive map interfaces [43], and assessment of map reading skills [44]. There is also research comparing 2D and 3D maps. Popelka and Brychtová [45] conducted a study focusing on the comparison of perception between 2D and 3D terrain contour lines. Their findings showed no significant user preference, but different scanpath strategies were identified between 2D and 3D views. Popelka and Doležalová [46] examined the effect of virtual globes versus 2D maps on symbol interpretation. The study found that the 3D virtual globe led to less accurate point symbol recognition compared to 2D maps. Liu, Dong, and Meng [47] explored the effectiveness of visual variables, such as colour and shape, in 3D maps. Their findings indicated that hue and shape were the most influential visual variables in directing attention in 3D environments compared to symbol size. Liao, Dong, Peng, and Liu [48] investigated visual attention differences during navigation with 2D versus 3D maps. The results showed that 3D maps improved navigation performance at complex decision points but increased cognitive load compared to 2D maps. Dong and Liao [49] explored the impacts of photorealistic 3D representations on pedestrian navigation performance. They concluded that photorealistic 3D maps increased self-localisation and orientation at the complex decision points but led to slower navigation performance compared to 2D maps. Lei, Wu, Chao, and Lee [50] evaluated differences in spatial visual attention in wayfinding strategy when using 2D and 3D electronic maps. Their study found that 2D maps allowed faster browsing and more efficient overall information acquisition, while 3D maps resulted in more focused but longer viewing time.

However, the above studies did not address ENC navigation maps, which are characterised by different compositions and uses [5]. By analysing previous literature, we believe that the eye-tracking technique can lead to the development of new methods which could support the elaboration of 3D marine navigational charts.

1.3. Background and Aim of the Study

The genesis of the research undertaken in this thesis was the need to test conceptual 3D navigation maps. This was due to the complexity of their construction and the testing process. The complexity of testing nautical charts is related to the necessity of testing the map more extensively in a complete operational environment. This is related to the fact that the chart is a component of the ECDIS system [51,52], which integrates various ship data in addition to the chart. In this system, the navigational chart also has an interface, and the ECDIS system itself is one component of the entire navigation bridge.

The navigator, at an operational level, uses de facto not the map itself but the entire system on which the various elements of the ship’s control are situated. In addition, the navigation process is shared with the visual observation and information provided by the various elements of the system. Another aspect is the evolution of navigational charts due to the development of the new S-100 data model [53], which will create further challenges for the visualisation of different navigational information.

Therefore, we can distinguish four levels of modes of ENC map visualisation (Figure 1). The first is the basic cartographic development, resulting from the accepted editorial principles of maps. For ENC maps, these principles are included in the IHO standards. The first level of visualisation includes the essential content of the map, that is, the topographic and cartographic model of the data. The second level stems from the functionality of ENC maps as part of the ECDIS system. This level additionally includes the interface and functionalities related to the use of the map at the operational level (e.g., visualisation of safe depths, route planning, display of various ship data or integration of data coming from other navigation devices, such as AIS or radar). The third level includes the visualisation of the ENC map as part of the navigation bridge. Information at this stage is shared with other navigation devices. Perceptually, it may be partially disturbed by additional information or posited by other navigation devices (e.g., radar). The final stage of visualisation includes the full operational environment, in which the navigator uses all available sources of information, additionally performs visual observation, and makes a real-time assessment of the navigation situation. This stage is the most complex, as it is based on the analysis of available navigational data and is related to making decisions based on them (e.g., making a manoeuvre, changing course to correct the ship’s position). Exemplary research analysing the behaviour of watch officers on the bridge and taking into account an ECDIS device is presented in [54]. We can also find research in other types of navigation, like aviation, where eye tracking was used for projecting the pilot’s cockpit [55].

Currently, there is a lack of formal methods to evaluate new ENC products, including 3D studies. This does not mean that such maps do not exist. In the case of non-standardised products such as Electronic Chart Systems, there are solutions on the market with 3D visualisation of navigation information [56,57]. In this case, it is difficult to say how much they improved the navigation process. If we consider the analysed literature on 2D and 3D map visualisation research, it is not necessarily the latter that can benefit. If we only take into account the increased cognitive load of performing tasks using a 3D map, this can negatively affect the navigation process, in which the time for analysing navigation information should be as short as possible.

Therefore, the motivation of the research process was to develop a methodology for assessing the effectiveness of conceptual 3D maps that could form a solid basis for further research—both in terms of integration into the interface and as part of the navigation bridge. They should be at least better than standardised 2D charts. Only reliable products can ensure navigational safety. The reference data for determining the effectiveness of the new product in our research was a standardised 2D map. The focus was on the content of the navigation map and the selection of its most effective form without the need for testing in a fully operational environment. This approach enables different map concepts to be tested and the most effective one to be selected for further stages of research, i.e., Levels 2, 3, and 4. Such map testing was based on previous research with the ENC 3D concept map [58]. The main problem with 3D navigational chart testing is the large amount of work at the IT level that needs to be completed to prepare the test environment for an operational-like one. It is related to the development of symbols, interfaces, and the appropriate implementation of graphics libraries, such as Open GL or programming. An example of such a testing environment from previous research is presented in Figure 2. The proposed approach in the current work involves staging the testing, in which the first stage of testing may select a concept that has implementation potential, which justifies the later workload of developing test environments at subsequent levels (illustrated in Figure 1). In the adopted research option, our map testing concerns Level 1 aspects of ENC map visualisation using a GIS-based testing environment.

The key question posed in the study was: how do we select the most effective map by testing different concepts? Based on this, the hypothesis was formulated that it is possible to develop a methodology for navigational map assessment—both standard 2D and conceptual 3D—to enable their comparison and selection for further research. In order to conduct this methodology, three evaluation criteria for navigational maps were chosen: the readability of the map, the correctness of object identification, and the time needed to identify map symbols.

The main objective of this study was to develop a methodology based on universal indicators for assessing the effectiveness of maps using eye-tracking measurements. Traditional measures, such as the number of fixations, dwell time, or time to first fixation, are often analysed separately and are strongly dependent on specific tasks or experiment conditions, which can hinder their comparability across different map designs. The proposed indicator addresses this limitation by aggregating multiple eye-tracking parameters into a unified measure, thus enabling a more consistent evaluation of the concept map at an early design stage.

Specifically, we integrated standard quantitative metrics, including the mean number of saccades, mean saccade duration, mean number of fixations and mean fixation duration, into the heatmap analysis. The aim was to explore the correlations among these measures and identify those most strongly related to the distribution of users’ visual attention. This approach aimed to simplify the overall evaluation process by reducing the number of variables needed to assess the effectiveness of the map.

Importantly, the research also involved developing a method for testing a series of navigation scenarios. Because navigational charts are used in a dynamically changing environment while the ship is in motion, assessing their effectiveness introduces additional complexity that is rarely considered in traditional mapping studies. This dynamic context of use necessitated the development of sequential map scenarios that better reflect actual navigation processes, in our case, at Level 1 ENC visualisation mode.

To support the development and evaluation of the new indicator, the study was structured around three specific research objectives. Firstly, we conducted a visual analysis of eye-tracking indices obtained from user interaction with a set of test map scenarios. Secondly, we analysed responses from a user survey that included both traditional 2D electronic navigation maps (ENCs) and proposed 3D visualisations. Thirdly, we proposed and refined a set of performance indicators, aiming for a universal indicator. In each study option, we tested 20 scenarios for 2D and 20 scenarios for 3D charts.

Based on the result of our study, we developed an effective map assessment method by combining partial research achievements.

2. Materials and Methods

2.1. Procedure

The study employed a multi-stage approach, enabling a comprehensive analysis of the effectiveness of 3D conceptual maps. The research process consisted of the following steps: initially, the study area was selected in the form of a cell from a 2D navigational map. The selection of a specific part of the map allowed for the uniform analysis and subsequent comparison of different types of maps. Based on the defined area, a 3D conceptual navigational map was developed. Subsequently, preliminary questionnaires and test scenarios were created to assess the readability and functionality of the developed maps. Before the main experiment, trial tests were conducted to evaluate the quality and usability of the research materials. The results of these tests were used to refine the scenarios and questionnaires. Based on the trial test results, corrections were made to the questionnaires and scenarios, leading to the development of their final versions. These finalised research materials were then used in the main experiment, during which participants were divided into groups testing 2D and 3D maps. The collected data were subjected to quantitative and qualitative analysis, which included indicators based on the generated heatmaps, eye-tracking measures such as fixations and saccades, and questionnaire results. The analysis of the results allowed for the formulation of general conclusions regarding the readability and functionality of navigational maps in three-dimensional space. Figure 3 provides a schematic representation of the stages of the research process.

The study was conducted with the approval of the Bioethics Committee of the District Medical Chamber. All participants were thoroughly informed about the purpose and procedure of the study and subsequently provided informed consent to participate in the experiment. The research procedure lasted approximately 10 min on average and included the preparation and configuration of measurement equipment. In the initial stage, the eye tracker was calibrated, after which participants read the test instructions displayed on the computer screen. They then proceeded to the experimental task, which consisted of 20 scenarios. Depending on the assigned study group, the test was conducted using either 2D or 3D maps.

Each scenario followed a specific format: the first slide displayed the task description along with the symbols they were expected to focus on. The next slide repeated the task description and displayed a map excerpt with a legend. The tester’s task was to identify and click on the map symbol. After completing the test, testers filled out a questionnaire regarding the test scenarios. Figure 4 presents an example of a test question. The command is “Find the north cardinal buoy on the next slide and click on it”. Below the command is a legend of the map symbols of the searched object: on the left side, there is a symbol of a buoy, and on the right side, there is a symbol of a cardinal mark north.

In Figure 5, there is a map scenario for the command from Figure 4. As with Figure 4, the top of the slide shows the content of the task. To the right of the map is a legend of the map symbols of the searched object–buoy and cardinal mark north.

2.2. Stimuli

At the start, the research area was defined, specifically, a cell of the 2D navigational map. The selected cell encompasses the range of the harbour in Świnoujście, Westpomeranian Voivodeship, Poland (Figure 6).

The 2D and 3D Chart Scenarios

Based on the selected cell of the navigational map, a conceptual 3D navigational map was developed using ArcGIS Pro software (version 3.4.0.). The same area allowed for subsequent comparisons between maps. The assumption for the transformation from 2D to 3D was to minimally adjust the standardised map into three-dimensional space. Existing map symbols were preserved but presented in 3D space. Additionally, surface objects, such as buildings and forested areas, were represented at Level of Detail 1 (LoD1) according to the CityGML standard [59]. The conceptual map also included three-dimensional models of lights.

The conception was based on keeping the relation with standardised 2D maps; we did not aim at this stage of research to develop a ready-to-use product but only test its effectiveness. Such conception can be treated as one option among many. Maps for test scenarios were selected, along with waterways, to cover various types of symbols. Figure 7 illustrates the locations with borders of the 2D and 3D maps for the test series. The spatial range of the 2D and 3D maps were similar, which allowed us to compare the same cartographic content. The reason for choosing these cells was the possibility to relate research results to previous research by using computer-based 3D navigation charts [58]. Finally, 20 maps in each series were prepared for the 2D and 3D charts. The maps in the series include various cases with simulated content similar to real conditions, which continuously change the map contents.

The preparation of twenty test scenarios was intended to simulate one stage of the navigation process—the assessment of the navigation situation using an electronic map. In a real-world setting, this task involves the watch officer analysing the content of the map, visually interpreting the types and positioning of objects, and assessing the safety of the ship’s conducting on this basis.

2.3. Participants

A total of 30 participants took part in the experiment. Participants were randomly assigned to one of two experimental conditions (2D or 3D) using simple randomisation. No stratification or matching based on prior experience, demographic characteristics, or other variables was applied. This approach was chosen to ensure an equal distribution of participants across both conditions and to maintain internal validity by minimising allocation bias. Randomisation was performed without prior knowledge of participant profiles, thereby reducing the potential for systematic differences between groups. Detailed characteristics of the testers are shown in

Table 1. Characteristics of the testers who took part in the experiment.

Category	Count	Percent [%]
Gender
Female	11	37
Male	19	63
Sum	30	100
Experience
Experienced	7	23
Inexperienced	23	77
Sum	30	100
Map
2D:	15	50
Female	4	27
Male	11	73
Experienced	4	27
Inexperienced	11	73
3D:	15	50
Female	7	47
Male	8	53
Experienced	3	20
Inexperienced	12	80
Sum	30	100

.

The average age of testers in this experiment was 25 years. Experienced testers were defined as individuals who had prior exposure to navigational maps (e.g., students of Navigation or Hydrography, lecturers teaching in these fields, and professionals working in the maritime sector). Inexperienced testers, on the other hand, were those who had experience with non-navigational maps or no prior exposure to maps at all.

2.4. Data Collection and Processing

The eye-tracking experiments were conducted in a controlled laboratory environment using a Tobii Pro X3-120 eye-tracker, a commercial device with a 120 Hz sampling rate, manufactured by Tobii AB (Stockholm, Sweden). The tracker was positioned at the bottom of a 15-inch monitor with a Full HD resolution of 1920 × 1080 pixels, where both stimuli and questionnaires were presented. Participants were seated approximately 60 cm from the screen. A standard 9-point calibration procedure was performed before each session to ensure accurate gaze tracking. All participants viewed the map scenarios under the same lighting and display conditions. Data processing and visualisation, including real-time generation of heatmaps and the application of Areas of Interest (AOIs), were carried out using iMotion software (version 10.1). Fixation and saccade data were recorded throughout the tasks and processed using the I-VT (Identification by Velocity Threshold) filter [60]. Heatmaps were generated separately for the 2D and 3D groups by averaging fixation distributions across all 15 participants in each group. These heatmaps were based on cumulative dwell time and fixation density per grid cell and visualised using a quartile-based colour scale ranging from green (low attention) to red (high attention) [61]. Heatmaps were exported in PNG format.

AOIs were manually defined as polygons covering key regions of the maps, including Pytanie (instructional text in the form of question), Legenda (simply map legend), Mapa (main area of cartographic content), and Obiekt (target object), as shown in Figure 8. The Obiekt AOI was specifically used to detect gaze directed at the correct object to be found in the task. All mouse clicks were registered during the experiment, and using screen coordinates, each click was verified to determine whether it fell within the Obiekt AOI. Identification time was calculated as the duration from stimulus onset to the participant’s first correct click on the object of interest. All collected data—fixation events, saccade events, timestamps, and click coordinates—were exported as CSV files to allow for further calculations of indicators.

2.4.1. Preparation of Scenarios for Visual Assessment

Maps were analysed based on overlaid heatmaps in 2D and 3D composition. The analysis was conducted in several variants: experienced to inexperienced and female to male testers were compared. These comparisons were carried out simultaneously using comparative video recordings. Video was prepared using Python (version 3.11.) code from tested maps with overlaid heatmaps. The researcher could watch and compare fragments of 2D and 3D maps at the same time. The analysis examined how heatmaps differed between these scenarios in specific groups. Figure 9 presents a frame from the video, where the 2D scenario heatmap is compared with the 3D scenario heatmap during analysis.

2.4.2. Preparation of Questionnaires

Additionally, as part of an assessment of the level of interest in 3D charting solutions for marine navigation, a questionnaire of survey participants was conducted. The survey consisted of 10 questions, partly general and partly specific, and was divided between 2D map testers and 3D map testers. The survey was mostly constructed of open-ended questions exploring the level of overall satisfaction (2 questions) and the assessment of the interpretive level of the content (5 questions), including object density and styling and personal views (3 questions) on the need for three-dimensional mapping visualisations. The main objective was to explore perceptions about the intuitiveness of the interpretation of the visualisations, the level of distinctness of the cartographical content representation, styling, and views presented on the test, and private opinions about the need for 3D solutions in this regard. The questions were arranged in such a way as to obtain a general view of the demand for 3D solutions in navigation maps and the general level of interpretability of the proposed images. The main difficulty in constructing the questionnaire was the fact that the entire survey concerned a series of scenarios, not a single map. For example, the tester could have had a distorted individual view of the maps when answering the questions. Hence, the construction of questions about specific images was pointless and would not have brought the expected results. The second major limitation of the evaluation method is that the results of the questionnaires may not be objective. On the one hand, the questionnaire gave the authors information on the desirability of this type of visualisation; on the other hand, it may have been biased and generated by the testers’ personal feelings rather than a proper environmental need. However, it was only the analysis of the eye-tracking data that showed whether the various proposed symbols reflected the difficulty or simplicity in perceiving and interpreting the visualisation.

2.4.3. Preparation of Heatmaps for Indicator Elaboration

In order to conduct the study, the decomposition of heatmaps was realised. To isolate individual colours of the heatmaps, heatmap segmentation was used. Separated colours represent different levels of attention (no colour indicates no attention to the object, green represents low attention, yellow indicates significant attention, and red represents very high attention). The segmentation was performed using the thresholding method. Thresholding is a popular method for segmenting an image into its components [62]. Typically, the components of thresholding are divided into two classes: the foreground, which constitutes the significant part of the image, and the background, which represents the less significant part [63]. Figure 10 schematically illustrates the process of heatmap decomposition based on image segmentation using the thresholding method.

To perform decomposition, the image was first converted to the HSV colour model. Then, the colour ranges for green, yellow, and red were set. Lower and upper bounds were defined for each range, spanning from the lightest to the darkest shade of the given colour. The colour thresholds were chosen based on a visual assessment. These ranges were used to segment the heatmap into zones of varying attention intensity in the eye-tracking data. Specifically, three HSV colour intervals were applied:

• Light green (low attention): [30, 30, 30] to [60, 255, 255];
• Yellow–green (medium attention): [20, 100, 100] to [50, 255, 255];
• Red (high attention): [10, 100, 100] to [25, 255, 255].

Based on the established thresholds, each pixel in the image was analysed and matched to one of the predefined colour ranges. Pixels meeting the criteria were assigned to the corresponding segmentation masks. The result of the applied segmentation is presented in Figure 11.

2.5. Indicator Elaboration

The development of indicators was conducted in stages and was experimental in nature. Attention was first focused on developing a map readability indicator, which was based on eye-tracking areal measures, which were heatmaps. The development of this indicator was related to the decomposition of heatmaps and the analysis of their components on the basis of mutual correlation with various eye-tracking measures. In the next step, measures were selected from the analyses to develop indicators for the correctness of map symbol identification and timing. While the correctness of map symbol identification is an obvious factor in the effectiveness of a map, identification time takes on particular importance in the navigation process. Given the movement of the vessel, the shorter time allows the navigator to make a quicker decision, which should potentially have a positive impact on the efficiency of the entire navigation process.

2.5.1. Measures

Indicator elaboration was based on heatmaps and eye-tracking measures like fixations, saccades, and time [64]. Saccades are rapid, ballistic movements that occur when the eyes move from one location to another. Fixations, on the other hand, are moments when the eyes are relatively stationary, during which perceptual encoding takes place [65]. Dwell time indicates how long participants spend looking at a specific area, while the number of revisits shows whether users return to specific elements in the image.

Both the average number and the average duration of saccades and fixations were considered. Figure 12 shows the visualisation of saccades and fixations on one test scenario.

During indicator elaboration, the Spearman correlation method was used to assess which parts of the heatmap were correlated. The correlation method was used to choose the most representative part of the heatmap to create the map readability indicator. Our intention was to reduce the number of coulometric measures in the final elaboration of the indicator; we tried to keep the rule that less means more. This was also to simplify the index, especially due to its future applications by various types of users, including non-scientific users.

Based on the generated heatmaps and eye-tracking measures, other indicators were developed. The research assumed that the heatmap-based indicator would be related to the assessment of map readability (visualisation of attention focus), the correct identification indicator would be related to the correctness of map symbol identification, and the time indicator would enable the assessment of the time needed to identify a symbol. The indicators and their descriptions are summarised in

Table 2. Map effectiveness indicators used in the research.

Indicator Type	Indicator Name	The Nature Associated with Map Efficiency
W [%]	Readability Indicator	Map readability
ID	Correct Identification Indicator	Correctness in identifying map symbols
T [s]	Time Identification Indicator	Identification time for map symbols

.

The above indicators were the basis for calculating the differential indicators and the final indicator—the universal one.

2.5.2. Readability Indicator

The map readability index expresses the degree to which map symbols can be recognised, as well as the ease with which a map can be read, interpreted, and understood. This definition is consistent with previous work on map legibility [66]. The degree of readability can also be examined using eye-tracking heatmaps, which provide insight into how users visually interact with maps. As Fairbairn and Hepburn [36] note, while heatmaps reveal where, how often, and for how long users look at different areas of a map, they do not explain why these areas attract attention or how they are cognitively processed. To develop this index, in the last step, we applied heatmap differentiation on 2D and 3D scenarios, through which we could obtain relative information about the readability of the map. We assumed that a smaller area for similar map scenarios corresponds to greater readability, which translates into a smaller area for searching to identify map symbols. In the final step, we analysed the correlation of decomposed heatmap segments with saccades and fixation measures, which gave us more information about the most correlated parts. Saccades and fixations are fundamental metrics in eye-tracking research, which help us interpret how a map is being read and understood and, in an indirect way, are taken into account in the map readability indicator.

In the current work, the readability indicator was based on heatmap decomposition and processing. The testers’ attention clusters presented by heatmaps are marked in green, yellow, and red, with red indicating the highest concentration of attention. The highest concentration was found to be on a given symbol (object) on the map. Figure 13 presents the experimental result of the task, “Find the lighthouse on the slide, showing white and red light and click on it”. The legend on the right side presents the map symbols of the searched object.

After segmentation, it was possible to calculate the pixel segment areas for each segmented region. The surface areas for green, yellow, and red regions were then summed. Using the processed heatmaps and the calculated surface areas of the segmented regions, preliminary indicators for the conceptual navigational maps were developed: W₁, W₂, W₃, and W₄. W₁ identifies the highest level of user attention focused on specific map objects. This indicator highlights the objects that caused the most attention or required significant cognitive effort from testers. W₂ quantifies the percentage of the map’s surface that was visually explored by the testers. A higher value suggests a broader search pattern, whereas a lower value indicates focused attention on specific locations. W₃ identifies the range of most intensively observed areas of the map. It helps identify which parts of the map testers found most relevant or useful. W₄ evaluates how scattered the tester’s attention was across the map. A higher value means that testers visually explored a large portion of the map, while a lower value indicates that attention was concentrated within a more confined area. The formula for calculating indicators W₁, W₂, and W₃ based on the single-scenario heatmaps for N testers is given below:

$${\mathrm{W}}_{\mathrm{j},s}^{\left(\mathrm{m}\right)}={\displaystyle \frac{1}{{\mathrm{A}}_s^{\left(\mathrm{m}\right)}}}\sum _{i=1}^{\mathrm{R}}{\mathrm{a}}_{i,s}^{\left(\mathrm{j},\mathrm{m}\right)}·100\%$$

(1)

where:

• j—index of the heatmap-based indicator, i.e., W₁, W₂, W₃;
• s—scenario number;
• m—map type (2D or 3D);
• i—index of the surface area for a given region;
• R—number of regions in scenario s;
• ${\mathrm{a}}_i^{\left(1,\mathrm{m}\right)}$—surface area in pixels for red regions (maximum attention on the target object);
• ${\mathrm{a}}_i^{\left(2,\mathrm{m}\right)}$—surface area for light green regions, including the remaining ones (maximum defined area of the heatmaps);
• ${\mathrm{a}}_i^{\left(3,\mathrm{m}\right)}$—surface area for green–yellow and red regions (significant areas of attention);
• A—total number of pixels in the entire image (scenario);
• N—numbers of testers per 2D and 3D scenarios (for each, N = 15).

Formula for calculating the W₄ index based on isolated focus fields:

$${\mathrm{W}}_4=\mathrm{k},$$

(2)

where:

• k—number of isolated focus fields (number of green-yellow and red areas).

Figure 14 visually shows the coefficients a₁, a₂, and a₃ used to calculate the indicators.

2.5.3. Correct Identification Indicator and Identification Time Indicator

These two last indicators were also based on registered eye-tracking data. The first one was based on the correctness of map symbol identification. For this aim, calculations were performed if the tester properly clicked on the map feature. The output value was 1 when the tester identified the map object or 0 in the other cases. The Correct Identification Indicator for testers in scenario s of map type m is defined as:

$${\mathrm{I}\mathrm{D}}_s^{\left(\mathrm{m}\right)}={\sum }_{k=1}^{\mathrm{N}}{\mathrm{b}}_{k,s}$$

(3)

where s denotes a single test scenario, k denotes a single participant, and b_k,s is a binary variable representing whether a participant correctly identified the target in scenario s. Variable b_k,s = 1 if the symbol was correctly identified (clicked); otherwise, b_k,s = 0.

The second indicator was based on the identification time, and a average of identification time value was assigned to it. The indicator for scenario s is defined as:

$${\mathrm{T}}_s^{\left(\mathrm{m}\right)}={\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{k=1}^{\mathrm{N}}({\mathrm{t}}_{k,s}^{\mathrm{e}\mathrm{n}\mathrm{d}}-{\mathrm{t}}_{k,s}^{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}})$$

(4)

where ${\mathrm{t}}_{k,s}^{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}}$ is the timestamp at which participant k started scenario s, and ${\mathrm{t}}_{k,s}^{\mathrm{e}\mathrm{n}\mathrm{d}}$ is the timestamp at which the identification was made (decision by clicking the next scenario).

2.5.4. Differential Indicators

To compare the 3D concept map with the standardised product, differential indices were calculated. The differential indicators were used to compare the effectiveness of the concept map with standardised 2D maps. They were calculated according to the formulas for each testing case in 2D and 3D scenarios:

$${\mathsf{\Delta }\mathrm{W}}_{\mathrm{j},s}={\mathrm{W}}_{\mathrm{j},s}^{\left(2\mathrm{D}\right)}-{\mathrm{W}}_{\mathrm{j},s}^{\left(3\mathrm{D}\right)}[\%],$$

(5)

$$∆{\mathrm{I}\mathrm{D}}_s={\mathrm{I}\mathrm{D}}_s^{\left(3\mathrm{D}\right)}-{\mathrm{I}\mathrm{D}}_s^{\left(2\mathrm{D}\right)},$$

(6)

$$∆{\mathrm{T}}_s={\mathrm{T}}_s^{\left(2\mathrm{D}\right)}-{\mathrm{T}}_s^{\left(3\mathrm{D}\right)}\left[\mathrm{s}\right],$$

(7)

where s = 1 to 20 and indicate the tested map in the 2D and 3D scenarios.

In their final form, they were presented as means for the twenty scenarios for the 2D and 3D maps:

$${\overline{∆\mathrm{W}}}_{\mathrm{j}}={\displaystyle \frac{{\sum }_{s=1}^{\mathrm{S}}∆{\mathrm{W}}_{\mathrm{j},s}}{\mathrm{S}}},$$

(8)

$$\overline{∆\mathrm{I}\mathrm{D}}={\displaystyle \frac{{\sum }_{s=1}^{\mathrm{S}}∆{\mathrm{I}\mathrm{D}}_s}{\mathrm{S}}}$$

(9)

$$\overline{∆\mathrm{T}}={\displaystyle \frac{{\sum }_{s=1}^{\mathrm{S}}∆{\mathrm{T}}_s}{\mathrm{S}}}$$

(10)

where S denotes the number of scenarios per map type (S = 20).

Additionally, we analysed these indicators, i.e., Equations (8)–(10), in the form of box plots, including median, outliers, and quartiles.

2.5.5. Universal Indicator

The motivation for further simplification of the indicator was also due to interpretative complexity. The W, ID, and T indicators are presented in the form of box plots, taking such values as mean, median, outliers, or quartiles. Such an amount of data burdens the unambiguity of interpretation in the aspect of choosing one map option. This is due to the fact that some indicators will speak in favour of the 2D map, some may be neutral, and some will prevail in favour of the 3D map. The situation may change depending on the type of map concept. The problem of interpretation can be demonstrated using the mean and median as examples. The mean will be loaded with extreme values, while the median will not include them. Even simple statistics can introduce interpretive ambiguity.

By using the final indicators to analyse the efficiency of the conceptual maps, a universal indicator was developed that combined the above metrics. In developing the universal indicator, we aimed to simplify things as much as possible, with the end result being a single value representing the effectiveness potential of the map. The main design consideration was that the final indicator should be simple, transparent, and unambiguously interpretable. This is particularly important to a non-scientific person, which may be map designers, for whom the interpretation of complex eye-tracking metrics—such as saccade dynamics or fixation patterns—may be difficult or ambiguous. Adding too many parameters to the final indicator could undermine its practical usefulness.

The creation of this indicator was based on linking more specific indicators (W, ID, and T) during its creation. The choice of the W2 indicator in Formulas (11) and (12) was based on a correlation analysis of all the measures studied. For a detailed description of the selection of W₂, see Section 3. The universal indicator was calculated according to the equations presented below:

$${\mathrm{X}}_{2\mathrm{D}}^{\left(1\right)}=\sum _{s=1}^{\mathrm{S}}\left\{\begin{array}{c}1,\mathrm{if}∆{\mathrm{W}}_{2,s}<0\\ 0,\mathrm{if}∆{\mathrm{W}}_{2,s}>0\end{array}\right.{\mathrm{X}}_{2\mathrm{D}}^{\left(2\right)}=\sum _{s=1}^{\mathrm{S}}\left\{\begin{array}{c}1,\mathrm{if}{∆\mathrm{I}\mathrm{D}}_s<0\\ 0,\mathrm{if}{∆\mathrm{I}\mathrm{D}}_s>0\end{array}\right.{\mathrm{X}}_{2\mathrm{D}}^{\left(3\right)}=\sum _{s=1}^{\mathrm{S}}\left\{\begin{array}{c}1,\mathrm{if}{∆\mathrm{T}}_s<0\\ 0,\mathrm{if}{∆\mathrm{T}}_s>0\end{array}\right.$$

(11)

$${\mathrm{X}}_{3\mathrm{D}}^{\left(1\right)}=\sum _{s=1}^{\mathrm{S}}\left\{\begin{array}{c}1,\mathrm{if}∆{\mathrm{W}}_{2,s}>0\\ 0,\mathrm{if}∆{\mathrm{W}}_{2,s}<0\end{array}\right.{\mathrm{X}}_{3\mathrm{D}}^{\left(2\right)}=\sum _{s=1}^{\mathrm{S}}\left\{\begin{array}{c}1,\mathrm{if}{∆\mathrm{I}\mathrm{D}}_s>0\\ 0,\mathrm{if}{∆\mathrm{I}\mathrm{D}}_s<0\end{array}\right.{\mathrm{X}}_{3\mathrm{D}}^{\left(3\right)}=\sum _{s=1}^{\mathrm{S}}\left\{\begin{array}{c}1,\mathrm{if}{∆\mathrm{T}}_s>0\\ 0,\mathrm{if}{∆\mathrm{T}}_s<0\end{array}\right.$$

(12)

Finally, the formulas for 2D and 3D universal maps indicators (U) are as follows:

$${\mathrm{U}}_{2\mathrm{D}}={\sum }_{d=1}^3{\mathrm{X}}_{2\mathrm{D}}^{\left(d\right)}$$

(13)

$${\mathrm{U}}_{3\mathrm{D}}={\sum }_{d=1}^3{\mathrm{X}}_{3\mathrm{D}}^{\left(d\right)}$$

(14)

3. Results

3.1. Visual Assesment

In both the 2D and 3D scenarios, testers focused their attention on similar locations, while in the 2D maps, the areas of focus were more dispersed. On 2D maps, testers focused their attention much more on the legend than they did with the 3D maps. When comparing inexperienced testers with experienced testers, it was noted that the heatmaps for inexperienced testers were more spread out, meaning that these testers performed tasks more chaotically; inexperienced testers sometimes focused on unusual objects that were not part of the task. Furthermore, the experienced testers looked at the legend less often, leading to the conclusion that they relied mainly on their knowledge when performing the test. A comparison between men and women, on the other hand, showed that men focused on fewer objects than women. The two genders also analysed the legend differently, with women focusing more on descriptions and men on symbols of objects. As a result, women were more likely to correctly identify objects than men. Some test scenarios proved to be very problematic for the testers (this was evidenced by the heatmap generated over almost the entire test scenario, indicating high uncertainty), leading to the conclusion that analysing only heatmaps is insufficient.

Table 3. Summary of observed differences in gaze behaviour between experienced and inexperienced testers.

Criterion	Experienced	Inexperienced
Attention distribution (heatmaps)	Focused	Dispersed, chaotic
Focus on irrelevant elements	Rare	Frequent (e.g., off-task objects)
Frequency of looking at the legend	Rare	Frequent
Accuracy in object identification	Higher	Lower
Chaotic task performance	Lower	Higher

provides a summary of differences in visual attention patterns between experienced and inexperienced testers, while

Table 4. Summary of observed differences in gaze behaviour between females and males.

Criterion	Females	Males
Attention distribution (heatmaps)	More dispersed	Focused
Focus on irrelevant elements	More frequent	Less frequent
Frequency of looking at the legend	Frequent	Moderate
Accuracy in object identification	Higher	Lower
Chaotic task performance	Higher	Lower

includes differences between male and female testers.

We noted that the visual assessment method was problematic for drawing final conclusions about whether the 3D map was more effective than the 2D map. We found this method ineffective for the unambiguous choice of an effective map. This method could be more useful in the assessment of one-map scenarios but is less useful during the testing of multiple maps. The strengths of this method are its identification of problematic cases and the collection of preliminary insights related to the interpretation of map content or the behaviour of tester groups. This method could also be subjective because it depends on the experience of the assessor and is time-consuming.

3.2. Surveys

In reporting the selected results, it is worth noting that 3D visualisation was considered by testers to be an effective navigational aid—17 responses were Yes out of 30 respondents, and it was rated as very important (14 respondents) and a visualisation valuable to have (11 respondents) to support navigation procedures. Overall, over half of all testers, both 2D and 3D maps showed interest in having 3D imaging. When analysing individual questions relating to purely cartographic aspects, testers mainly pointed to an increase in colour contrast at the background–symbol plane, a higher diversity of the shapes of object symbols, the use of object scaling due to the large number of symbols per map unit, which generated a cartographic cluttering effect, and changes in the size of specific object symbols. The summary of survey results is presented in

Table 5. Summary of survey results.

Precision of Results	Interpretation of Results	Objectivity	Advantages and Disadvantages of the Method
Moderately accurate based on subjective assessments	Easy but often hard to interpret answers	Low level of objectivity; answers are generalised and depend on respondents’ opinions	Advantages: easy to use on a large sample Disadvantage: subjectivity of responses; difficult to use in map series analysis
Level of interpretability of the map symbols	Identified cause for distorting the interpretability	Most problematic map symbols	Indicated reason for the difficulty in identifying the map symbol
75% of testers marked the answer that the map symbols were easy to find and interpret	High intensity of the symbols, giving a sense of clutter and confusion	Contours Wrecks Marine cable tracks Ferry routes	Colour scheme
Assessment of whether the 3D map correctly represented the navigational situation	Evaluation of a 3D map visualisation for its effectiveness as a navigational aid	Personal perception of the need for this type of visualisation	Suggestions for improving the map symbols or map content
63% of testers (16 persons) marked YES 26% of testers (8 persons) did not have absolute certainty 11% of testers require further work	56% of testers evaluated a 3D ENC map as an effective visualisation	14 testers marked as very important 11 testers marked as valuable to have 3 testers marked as not useful	More distinctive colour scheme Increased symbol-to-background contrast Map symbol scaling to avoid cluttering the image Improved 3D presentation of map symbols Map symbol design more in line with the IHO standard

.

It was difficult to find general rules for map content interpretation. This method was more helpful in problematic scenario identification and the formulation of basic assumptions. The reason was evidently the series of maps, which made it difficult to summarise the results. This kind of survey does not give us unambiguous answers about whether concept 3D maps are more effective than 2D. Summarising this method, it can be said that potential user preferences can be established, but this is not very precise and is more subjective in nature.

3.3. Map Indicators

Taking into account the first conclusions drawn from the visual and survey assessments, further research focused on the development of an analytical method based on eye-tracking indicators.

Due to the relatively large number of obtained indicators, we decided to examine the correlation among indicators W₁, W₂, W₃, and W₄ and, based on the results, reduce their number. Figure 15 presents the average quantity of correlations for the 2D and 3D maps. The average number of correlations refers to the mean count of correlation coefficients equal to or greater than 0.6 (considered a meaningful level of similarity, i.e., strong correlation), calculated among all pairs of cases within a given category. Correlations were computed separately for 2D and 3D map conditions, and the final value represents the average of these two results. Figure 16, on the other hand, shows the average correlation value for the 2D and 3D maps. The average correlation refers to the value of the correlation coefficients ≥ 0.6, again calculated separately for 2D and 3D maps. Only correlations meeting or exceeding the 0.6 threshold were included in the calculation. The final result is the average of the mean correlation values for both map types.

Figure 17 presents the results in the form of a matrix. Figure 17a shows the correlations of the readability indicators for the 2D map, while Figure 17b illustrates the correlations for the 3D map.

For both the 2D and 3D cases, the indicators W₂ and W₃ were significantly or strongly correlated with the other studied indicators (t-test for correlation significance was performed, where p < 0.05). W₂ had the highest average correlation > 0.6 and the highest correlation with the average number of saccades, average number of fixations, and time. Additionally, it was also the only W indicator of such a condition, together with Sm, Fm, T, and ID, between the 2D and 3D values (maps). Based on the obtained results, W₂ was chosen for further analysis. W₂ represents the light green regions of the heatmap, including the remaining ones (maximum defined area of the heatmaps). With this indicator, we were able to simplify, leaving behind more complicated analysis measures based on saccades and fixations.

In addition to W₂, the T indicator and the ID indicator were also presented as differential mean values for Formulas (8)–(10). The results obtained are shown in a box plot for the whole map series (Figure 18).

In the above boxplot, negative values favour the 2D map, while positive values favour the 3D map. Analysing this chart reveals that for the W₂ indicator, the 2D map decisively dominates, indicating that testers found 2D maps to be more readable. For the ID indicator, there is also a preference for the 2D map, though the difference is not as pronounced as for W₂. The results for this indicator suggest that testers interpreted symbols better on 2D maps than on 3D maps. The identification of features was hindered by wreck symbols, whose light colouring reduced their visibility. An additional challenge was the high density of depth contours, which significantly complicated the interpretation of the map. Testers also mentioned that the 3D chart displays, in that case, showed too many symbols at once, which negatively impacted user focus. In contrast, features that were easily identifiable included beacons, marinas, search and rescue stations, reporting points, pilot boarding places, turning areas, submarine cables, and ferry routes. The T indicator was more favourable for the 3D map, which implies that testers identified objects, on average, 3.9 s faster on 3D maps.

The ambiguity in interpretation was due to the fact that some of these indicators spoke in favour of the 2D map and some in favour of the 3D map. More problematic interpretation arises when analysing box charts in more detail. In the case of W₂, the mean and median were negative, indicating the advantage of the 2D map. The wide spread of results indicates a high variability between users in the aspect of map readability, and the lower quartile indicates that some participants achieved much better results in 2D. In terms of identification, we have a fairly balanced mix of positive and negative results. The median and mean indicate a small advantage of the 2D map, with many outliers (a significant advantage of object identification) observed in favour of the 3D map. The interquartile interval is small, which supports the consistency of the results. In the case of identification time, positive values prevail, indicating faster identification of objects in the 3D map. One outlier informs about the fast identification of an object with the 3D map. The interquartile range indicates that the results are fairly consistent. What are the final conclusions? Which map is more effective—2D or 3D? There can be only one answer: the final assessment of the maps’ effectiveness is troublesome. This stage of the study did not meet our objectives, as we were unable to uniquely assess the effectiveness of the map.

Universal indicators were calculated for the 2D and 3D maps, and their values were, respectively, U_2D = 26 and U_3D = 32. There were no changes in the two cases. The calculations indicate that in 26 cases, 2D maps were more effective than 3D maps, while in 32 cases, 3D maps were more effective than 2D maps. In two cases, no significant difference in efficiency between the maps was noted. The proposed universal indicator, due to its quantitative nature, provides general information about the effectiveness of conceptual maps in a binary manner (yes/no). The development of a universal indicator was the final step needed to develop a methodology for evaluating the effectiveness of the map.

4. Discussion

On the basis of the research carried out and the results obtained, it can be concluded that it is possible to develop a universal indicator with which to assess the effectiveness of a navigation map. It should be noted here that the universal indicator is based on a decision-making method of calculating a numerical advantage after summing the values obtained for the scenarios tested, which is quantitative in nature. Therefore, differential indicators should be additionally analysed in order to assess the partial advantage of the concept map in terms of readability, correct identification, and time.

On the basis of previous research on conceptual 3D maps, it can be stated that the proposed method offers clear advantages over purely survey-based approaches. In the work of [58], admittedly, the surveys made it possible to demonstrate that this type of visualisation is useful (at the level of ‘very important’ or ‘worth having’), but it is difficult to clearly state whether such declarations of map effectiveness are sufficient to justify further work on new forms of presentation, interface changes, or improvements to the visualisation system. The use of the proposed universal indicator allows for a quantitative analysis of the effectiveness of the maps based on quantitative data, which allows for a more structured evaluation. The final methodology, which includes a universal indicator, also allows for a more detailed assessment using W, ID, and T indicators, among others, which can be analysed according to the objectives of the study.

Compared to other eye-tracking measures, the improvement in performance evaluation lies in the reduction of multiple measures, such as the number of saccades, fixations, or heatmap analysis, customarily used in research [48,49,50], to a single indicator strongly correlated with them. The advantage of this method compared to other approaches using single or composite analytical indicators [66] is the direct inclusion of information about the distribution of attention in the form of a heatmap.

The limitation of a universal indicator is certainly its quantitative nature. It does not take qualitative aspects into account; nevertheless, at this stage of the research, this can be seen as a compromise to enable a quick comparison of the effectiveness of the maps. The next limitation concerns the perception of the navigation map in terms of 100 per cent efficiency. A universal indicator only reveals a quantitative advantage but does not capture important cases, such as the difficulty in identifying an object or the increased time it takes to find it on the map. Therefore, qualitative evaluation methods based on a case-by-case analysis should be used to achieve this goal. However, this limitation can be overcome by using a more elaborate methodology, as described later in this section. Another limitation relates to the need to use spatially similar test scenarios, which is, however, necessary for meaningful comparison. The difficulty in this case lies in the selection of area-similar scenarios for the 2D and 3D maps. The adopted method limits the use of any perspective views possible in 3D scenery, but we, nevertheless, believe that a full-scale test of the 3D map should be carried out once its fundamental advantage in efficiency has been established.

Visual assessment methods often lead to interpretative ambiguities. Although some scenarios indicated that the 2D map was most effective, some indicated that the 3D map was more effective, and some indicated that there were no differences. Given the problematic nature of using a visual and survey method for a series of maps, the proposed approach offers a clear advantage in the decision-making process, as it is based on concrete numerical values.

Post-test questionnaires are common tools for evaluating the solution under test. In the case of the proposed 3D navigational map visualisation solution, the authors, through a survey, were generally concerned with obtaining feedback on the demand for this type of solution and assessing the quality of the proposed cartographic layout. It was not possible to design a questionnaire with only close-ended questions, which would have been the ideal option for drawing out the very fundamental conclusions; this was mainly due to the testing of 20 different scenarios. For this reason, questions of a general nature were proposed for the most part, with which it was possible to obtain a general idea of the need for this type of solution and possible areas for improvement in terms of interpretation and the quality of the visualisation. The post-task questionnaires, despite their usefulness in assessing the subjective aspects of user interaction with 2D and 3D maps, are not without methodological limitations. There is a real risk that respondents’ answers will be biased by the tendency to present themselves in a favourable light [67], which can distort actual assessments of map readability, symbol identification, and navigation clarity. Additionally, difficulties with introspection and potential memory errors can affect data reliability, and the very wording of questions can subtly suggest desired answers [68]. Furthermore, sample representativeness and response rates pose a challenge [69], affecting the generalisability of results. The analysis of questionnaire data—particularly open-ended responses—requires caution, as standard statistical methods may not fully capture the complexity of the phenomena under study. In the context of evaluating different map types across a series of scenarios, there is a risk of habituation or contrast effects, which may affect the consistency and reliability of participant assessments. Our experience suggests that while surveys can be helpful in capturing general perceptions (e.g., whether a map is considered “worth having”), they are rather inefficient tools for drawing detailed or conclusive insights. It is difficult to obtain concrete answers when testing multiple concept map variants using only subjective assessments. To address these limitations, future research could benefit from incorporating post-task interviews, which allow for a deeper exploration of response context, and comparative ranking methods, which facilitate more direct and interpretable comparisons between map variants in terms of effectiveness. We also compare visual, survey, and indicator methods. The main conclusions are below.

Visual analysis, based on heatmaps, has a very high level of precision, allowing detailed tracking of the user’s attention. However, it is a method that is complex to interpret, requiring experience and specialised knowledge. Although the data obtained from a single scenario may be objective, the final result depends on the performance of the algorithms and the subjective judgment of the analyst. An additional limitation is the difficulty of applying this method to large sample surveys. Nevertheless, it is effective in identifying single problems and is qualitative in nature.

The survey method, on the other hand, relies on the subjective evaluations of respondents, which affects the moderate precision and low objectivity of the results. Its main advantage is the ease of application in large-scale surveys. However, responses are often general and difficult to interpret unambiguously, which limits the usefulness of this method in analysing larger data sets, such as a series of maps.

The most effective methods in quantitative analysis are indicators based on measurable parameters. They allow easy and quick interpretation of data and automation of calculations, which makes them ideal for comparisons in many test scenarios. Their limitation, however, is their quantitative nature—they do not allow the identification of specific, qualitative problems in a project. More detailed assessed methods are provided in

Table 6. Comparison of map assessment methods.

Type of Method	Precision of Results	Interpretation of Results	Objectivity	Advantages and Disadvantages of the Method
Visual (heatmap analysis)	Very accurate—allows detailed analysis of tester’s attention	Complex—requires experience in heatmap interpretation; heatmap analysis is difficult due to large colour and spatial variations	Objective for a single scenario, but interpretation depends on the algorithms processing the data and the subjective judgement of the analyst	Difficult to apply to a large sample; requires experienced professionals to interpret results; identifies individual problems; qualitative in nature
Survey (survey questions)	Moderately accurate—based on subjective assessments	Easy, but often hard to interpret answers	Low level of objectivity; answers are generalised and depend on respondents’ opinions	Easy to use on a large sample; main disadvantage is subjectivity of responses; difficult to use in map series analysis
Indicative (quantitative indicators)	Very accurate—based on measurable parameters	Easy—based on the analysis of indicators; covers all cases examined	Objective in quantitative analysis, but does not take qualitative aspects into account	Ability to automatise calculations; easy to apply in analysis of multiple test scenarios; disadvantage is quantitative nature; does not allow identification of specific problems

.

Taking into account the results of our research, we have developed a methodology for evaluating the effectiveness of conceptual 3D navigation maps. Sticking to our assumptions related to the maximum simplification of indicators, the evaluation map effectiveness is based first on simplified indicators, including the universal one. However, given its quantitative nature, the methodology for evaluating map effectiveness should also consider qualitative aspects, which may be related to the evaluation of results at a higher level of detail. Hence, the analytical process should include a generalised assessment based on quantitative measures and a more detailed one based on qualitative measures. The proposed methodology is based on an analysis from the general to the specific, whereby the specificity of the analysis depends on the specific research case (conceptual map). The methodology for evaluating the map’s effectiveness is illustrated in Figure 19. We can distinguish four levels of analysis. The first is a generalised level of analysis based on a universal indicator. The first results show whether the developed concept has an advantage in efficiency over 2D maps. The next stage is based on a more detailed analysis, taking into account mean differential values. Based on this stage, we can deduce whether the developed concept has an advantage and to what degree. This applies to sub-evaluations in terms of the readability of the map, the correctness of the identification of symbols, or the time it takes to identify them. However, given the number of statistical measures at this stage, interpretation can be problematic or ambiguous. These statistical measures can provide a partial advantage in concept map effectiveness. Especially if there are outliers that descend to the third level of analysis, where cases can be analysed at the single scenario level (2D and 3D map pair). At the lowest level, the most detailed cases can be made, even at the level of a single participant. This level can have a database structure with which to search for single cases. As we can note, the lower level of analysis gives a more quantitative assessment, especially in the case of visual assessment of the results. For example, on the last analytical level, single cases can be identified, such as those with problematic map symbol identification.

Taking into account the above consideration, Figure 20 presents a chart analysing Level 4, the symbol identification for each test scenario. Analysing this data, it is clearly visible that the symbol identification times are shorter for 3D maps compared to 2D maps. However, a detailed analysis of the chart reveals scenarios where the identification time is shorter for 2D maps than for 3D maps (e.g., scenarios 14 and 17). Additionally, by examining the chart, one can estimate the differences in identification times for individual test scenarios. This demonstrates that the method of analysing the effectiveness of conceptual maps presented in this article can also follow a “general-to-specific” methodology.

The universal indicator can play an important role in the design process, acting as a practical tool to support design decisions. It makes it possible to assess the quality of different map concepts, which avoids interpretive ambiguities in the aspect of assessing map effectiveness. Above all, it is based on concrete numerical data. As a result, in the design process, it is possible to compare different variants of solutions and make a preliminary selection of those that show the greatest development potential. Given the large number of standardised map objects [70,71], testing the cartographic content with conceptual 3D symbols at a basic level will allow the process of creating new map products to be more efficient. In addition, its simplicity and transparency mean that it can also be understood by team members without specialised knowledge, which facilitates joint project decisions.

The quantitative nature of the indicator also allows automation of the analysis, which is particularly valuable for large projects or testing multiple scenarios simultaneously. While the indicator is not a substitute for in-depth qualitative analysis, it provides a solid starting point for further research. Thus, it serves both a diagnostic and supportive function in the process of creating more optimised and functional map products.

The versatility of the indicators should be understood primarily at the basic design level as an element supporting the decision-making process. At this stage, its construction can be regarded as a pioneering approach since it is difficult to completely ignore the qualitative dimension of assessing the effectiveness of maps. Despite the simplification inherent in the universal indicator, its structure makes it possible to refer to more detailed analyses at any time.

In this context, our proposal is to use the methodology with all the elaborated indicators discussed in the paper: the map readability indicator, the correct identification indicator, and the time identification indicator. Thanks to their interrelationship and the development used in the study, it is possible to build a kind of information tree that allows multi-level analysis—even based on a single test scenario.

The results obtained may also imply further research related to the refinement of indicator methods to help assess the effectiveness of concept maps. This may involve testing other types of maps or modifying proposed solutions. Taking into account the human factor of performance evaluation, it is certainly worth focusing our attention on AI methods that can show better interpretative properties and can support the process of evaluating map effectiveness.

5. Conclusions

This paper develops a methodology for evaluating the effectiveness of 3D navigation maps. Based on the results obtained, the methodology presents a multi-level analytical process based on quantitative and qualitative measures. One of the goals of the work was to simplify the developed indicators using eye-tracking data. The philosophy of such an approach has a practical dimension since designers are not necessarily scientists; hence, quantitative indicators will certainly facilitate the process of evaluating the effectiveness of concept maps. The key indicator in the methodology is the universal indicator.

This research shows that oculographic measures better enable conceptual navigation map assessment. The universal indicator developed, despite its quantitative nature, enables concept maps to be assessed in an unambiguous way. In our case, the reference map was a standardised map product in the form of a 2D navigational map, which gives a clear answer to the question of whether the proposed concept is good enough to replace such a map in the future.

Undoubtedly, an important advantage of the proposed methodology is that several concept maps can be compared, and the most effective one is selected. This provides a solid basis for initiating further testing related to the map interface or its suitability in operational conditions on the navigation bridge. Unlike studies in the space of real 3D scenes [72], the cartographic approach in 3D mapmaking is a big challenge, especially when models containing dozens of map objects have to be elaborated.

In terms of future research, this work can form the basis for further work related to the development of more advanced decision-making methods, enabling the selection of the most effective concept map. The results of the research can form the basis for testing other forms of visualisation, such as AR (augmented reality) maps or VR (virtual reality), which is of great importance in the context of navigation safety, AI applications in such processes, and the development of decision-making methods in concept map evaluation.