Round 1

Reviewer 1 Report (Previous Reviewer 1)

Thank you for your revisions.

Author Response

Dear Reviewer!

Thank you for taking the time to read our resubmitted and revised manuscript and accepting it as it was! We have made further improvements to our text in response to comments from other reviewers, but we have kept the changes made in response to your earlier comments.

All the best,

The Authors

Reviewer 2 Report (Previous Reviewer 2)

Dear Authors,

Thank you for the opportunity to read the revised version of the paper. I have been able to identify significant changes compared to the first version of the manuscript. However, I must point out the issues raised in previous reviews.

You will find the detailed comments attached.

I wish you all the best for your future endeavours.

Comments for author File: Comments.zip

In some parts the usage of the vernacular, common or vague language is a problem:

lines 89-90: “have also provided a basis for what we can expect”

line 99: “Basically, …”

line 119: “The language in which the map is used”

line 163: “At the end, the results of their performance…” (maybe “At the end, the results on the performance”)

line 166: “The test was available online and stored the data in a MariaDB database…” (proposed change “The test was available online and the data were stored in a MariaDB database …”

Author Response

Dear Reviewer!

Thank you for again taking the time to read our resubmitted and revised manuscript! Other reviewers have already accepted our manuscript in the previous round, and the current changes are largely based on your comments. The changes have largely affected the results and discussion sections. The changes have made the former more precise and the latter more transparent. We hope that you will also find this to be the case!

#R2 comment 1:

“… but experience had the greatest impact on the accuracy of answers, confirming previous findings.”

In the manuscript, it is correctly described as an association, not an influence. Further on, I will refer to the general statement on which variable was most strongly related.

Response to R2 comment 1: This statement was removed from the abstract, as it was rephrased due to a later remark.

#R2 comment 2: Related studies

References number 14 and 15 relate to studies on map projections, which is not the case with the study material in the manuscript. I suggest providing the reader with this crucial information.

The section on Related studies would benefit from a few sentences of summary.

Response to R2 comment 2: You are right, this part does need more explanation! In the case in point, the relevant difference is the scale of the maps examined by those references, rather than the fact that projections were in focus. We have supplemented the relevant part with this information.

#R2 comment 3: The introduction to the “Results” section should be supplemented with information on which statistical tools, tests, or techniques were used and whether, importantly, the data met the specific assumptions (Chi-Square Test, ANOVA, Pearson correlation coefficient). This information could also be included in the following paragraphs concerning particular analysis.

Response to R2 comment 3: Thank you for the remark! We gave an overview of the methods in the introductory section, and we added the information on the assumptions to section 3.1, 3.2 and 3.3, to the paragraphs where we introduce each of the tests.

#R2 comment 4: In the abstract, there is a statement that the “experience had the greatest impact on the accuracy of answers”. Despite the incorrect usage of the word “impact” (instead of association), the Authors do not provide information about the overall association of the dependent and independent variables, i.e., the overall results of the test in comparison to gender (man and woman).

Response to R2 comment 4: We've rephrased the abstract indicating the number of significant associations of the independent variables, and also keeping the length requirement in mind.

#R2 comment 5: Therefore, the association between map reading skills and the accuracy of the answers was significant in the case of four questions, but the association was weak or moderate. Based on the results, it could be stated that the association was significant in four of eight cases, not that it was the most significant difference. The comment is related to all of the results (e.g. when the Authors state that “The participants’ gender had the least association with performance”, they should rather write that “The participants’ gender was moderately associated with performance in case of one question”. The same issue of misinterpretation appears in the Discussion (e.g. lines 479-481). In the current version of the manuscript, the reader could be aware only of the results for particular questions (comment is related to sections 3.1. and 3.2.).

Response to R2 comment 5: Thank you for the remark, we made the requested changes in the results (section 3.1, paragraph 5) and the discussion (paragraph 1, paragraph 2)!

#R2 comment 6: only statistically significant results should be described in the Results and in the following section. Please find attached the pdf with crossed-out sentences to be deleted (direct changes only for 3.1. The results from a general aspect).

Response to R2 comment 6: We deleted the marked sentences from the revised manuscript.

#R2 comment 7: In section 3.2 the results for the Q2 are not described.

Response to R2 comment 7: Thank you, for your remark! We added the missing description of these results.

#R2 comment 8: In Tables 2 and 3 (and sections 3.1. and 3.2.), the Authors provide information that, for example, there was a significant association between answers correctness and “Qualification” when comparing groups A to B and C (Q4), and A to C (Q6). Yet, in the text one can find only the results of chi2 for the overall comparison of the three groups, and there is no information about the detailed results of pairwise comparisons.

Response to R2 comment 8: Thank you for the remark, we added the p values of the pairwise comparisons to the related results (age groups, education groups, languages)!

#R2 comment 9: The information about the smaller test sample in the analysis of screen size influence from lines 422-424 should be included in section 2.2.

Response to R2 comment 9: We included it in the section you requested!

#R2 comment 10: Section 3.3. – Why the “I don’t know” option was not taken into consideration?

Response to R2 comment 10: In the section we wanted to focus on incorrect answers only, and the "I don't know" option was not considered as a mistake answer. Thus, using the complete case analysis method, we excluded them when assessing the incorrect answers. We mention this in the caption of figure 6.

#R2 comment 11: I must stress the necessity of statistical reasoning for section 3.3. One of the options could be a Chi-square test of equal frequencies (see e.g. https://www.ibm.com/docs/en/cognos-analytics/11.1.0?topic=tests-chi-square-test-equal-frequencies).

Response to R2 comment 11: Thank you for the advice, we applied the binominal test for this analysis, please see Results 3.3 paragraph 4!

#R2 comment 12: Lines 484-485 “This is important to highlight as the gender gap is thought to be much more significant in terms of the number of studies on this issue” - Citations needed. In the next sentence, the Authors use the phrase “At the same time, as mentioned earlier…” which implies contradiction yet presents confirmation.

Response to R2 comment 12: This contradiction has been present in studies on the subject from the very beginning, as pointed out in paragraph 3 of the "Related studies" section. Our study supports the minor role of the gender gap, which can be paralleled with the results of early studies. In contrast, many other studies have shown a much larger gender gap, but some have not, and there has also been some controversy over whether men or women perform better. To make this clearer, we have reworded the relevant section.

#R2 comment 13: The discussion is not structured in relation to the research questions. There are substantial discrepancies in the length of parts dedicated to particular RQ. The third R3 is barely mentioned. More discussion of the results of studies on language and cultural differences is needed.

Response to R2 comment 13: We have reorganised the discussion according to the research questions. The research questions are not of equal importance: more attention has been paid to the analysis of the incorrect answers, as emphasised in the title. With regard to RQ3, although we managed to answer it, we did not have much room for improvement, as there are relatively few studies of this kind. These have been mentioned in the revised manuscript.

#R2 comment 14: Conclusions should not describe statistically insignificant results as general statements (“When expert map readers make errors, it’s most likely to be related to the referencing of geographic names” which I presume relates to Q6 in Table 2).

Response to R2 comment 14: Thank you for bringing this to our attention, we have based this claim on Figure 7 and now support it with the binomial test in Section 3.3!

#R2 comment 15: In the Conclusions, Authors again misinterpret the association with influence and effect (“The present study also confirmed that map comprehension is mostly influenced by…”, “On the other hand, educational background seems to have the smallest effect on map reading.”).

Response to R2 comment 15: The phrases were changed to relation/association. We are sorry for not noticing it in the previous round!

#R2 comment 16: English: lines 89-90: “have also provided a basis for what 89 we can expect”

Response to R2 comment 16: The sentence is rephrased in the corrected manuscript.

#R2 comment 17: English: line 99: “Basically, …”

Response to R2 comment 17: We removed the concerning phrase.

#R2 comment 18: English: line 119: “The language in which the map is used”

Response to R2 comment 18: The sentence is rephrased in the corrected manuscript.

#R2 comment 19: English: line 163: “At the end, the results of their performance…” (maybe “At the end, the results on the performance”)

Response to R2 comment 19: The sentence is rephrased in the corrected manuscript.

#R2 comment 20: English: line 166: “The test was available online and stored the data in a MariaDB database…” (proposed change “The test was available online and the data were stored in a MariaDB database …”

Response to R2 comment 20: The sentence is rephrased in the corrected manuscript.

On the basis of the detailed review, we believe that the manuscript's scientific clarity has been significantly improved. This has taken both you and us quite some time, so we hope you are satisfied with the result. Thank you again for your time!  

with regards,

The Authors

Reviewer 3 Report (New Reviewer)

It is an original research article. The aim of the paper is to investigate factors influencing correct map reading tasks through an empirical research. Authors conducted a study with a large number of participants who completed an online questionnaire testing their mapping skills. The questions were related to scale usage, mental rotation and map categories recognition (referring to the relief and geographic names interpretation, as well as, recognition of linear and point symbols). Significant differences were found based on age, gender and education, and the participants' experience on map using, found to have the greatest impact on the accuracy of answers.

Authors are putting and discussing all aspects of their research clearly and the literature review they provide on the related works is sufficiently stated. They are comparing their findings with the findings of relevant existing studies and the results are promising and sufficiently summarized in the conclusions.

According to reviewer’s opinion there is one weak point. Authors are relating their research with car navigation systems several times (in abstract, in introduction and in conclusions), but the maps used in the questionnaire:

- were black-white and not colored,

- areas were symbolized by their outline and not by hue of a color or by a pattern, 

- there was a task on elevation interpretation from contour lines but relief is not represented in car navigation systems maps.

And finally, authors did not touch the issue of audio guides factor, which is an important one for car navigation systems.

It is suggested to authors to consider the following editing remarks:

1.     In lines 148-149 the segment: “Research Group on Experimental Cartography operated at the ELTE university (ktk.elte.hu)” may be replaced by the initials: “RGEC”.

2.     In lines 305-308: The procedural justification for age groups limits should be replaced by a scientific one.

3.     In line 350 the segment: “(Kim, 2018)” a literature reference?

4.     In lines 415-416 the segment: “determined by client-side JavaScript and the browser's user-agent strings” it is a repetition (see lines 323-324).

Author Response

Dear Reviewer!

Thank you for taking the time to read our manuscript carefully and for your valuable comments! Below is our response to your questions and comments, in bullet points. We hope we have understood all your questions correctly and answered them properly!

#R3 comment 1: Authors are relating their research with car navigation systems several times (in abstract, in introduction and in conclusions), but the maps used in the questionnaire:

- were black-white and not colored,

- areas were symbolized by their outline and not by hue of a color or by a pattern,

- there was a tAnd finally, authors did not touch the issue of audio guides factor, which is an important one for car navigation systems.

Task on elevation interpretation from contour lines but relief is not represented in car navigation systems maps.

And finally, authors did not touch the issue of audio guides factor, which is an important one for car navigation systems.

Response to R3 comment 1: The study investigates two scenarios, navigation and wayfinding, with test questions, as both are performed on large scale maps and typically do not have a legend. Navigation is highlighted in the paper because it has a higher risk of making a mistake in this scenario. However, we do not emphasise this in the title and in the discussion chapter we also discuss our views on the wayfinding scenario (i.e., mainly for hiking or walking, where contour lines do play roles). In this way, although the issue of navigation is emphasised in the article, we do not, in our opinion, push the topic too far towards car navigation.

The aim of the study was therefore not to investigate car navigation situations, and this has been explained in the manuscript (along with the simple, monochrome map figures). However, as our results are for large scale maps and investigate issues that arise in navigation situations, we believe that they may be useful in this respect.

Regarding the use of audio guide, we have completed the discussion in the manuscript.

#R3 comment 2: In lines 148-149 the segment: "Research Group on Experimental Cartography operated at the ELTE university (ktk.elte.hu)" may be replaced by the initials: "RGEC”.

Response to R3 comment 2: We replaced it in the corrected manuscript.

#R3 comment 3: In lines 305-308: The procedural justification for age groups limits should be replaced by a scientific one.

Response to R3 comment 3: These age groups were defined in line with our previous study in 2016 to keep the results comparable. In the previous study, the groups were defined by quartile. If we had followed this principle now, we would not have been able to track the changes. In the present study, for example, we were also interested to see whether the new completions change some statistical results over time, but we found only a few notable results in this respect (these are reported in the manuscript).

#R3 comment 4: In line 350 the segment: "(Kim, 2018)" a literature reference?

Response to R3 comment 4: Thank you for the remark, we added the missing reference!

#R3 comment 5: In lines 415-416 the segment: "determined by client-side JavaScript and the browser's user-agent strings" it is a repetition (see lines 323-324).

Response to R3 comment 5: Thank you, we removed the duplication!

All the best,

The Authors

Reviewer 4 Report (New Reviewer)

The paper is devoted to testing elementary mechanisms of wayfinding and map reading. The boundary between map reading and map interpretation is not definable, but as the authors set it out, I consider it a contribution to the discussion. The questions tested strive for simplicity, which is difficult when working with maps; even in the case of this study, some questions are visually trivial, and others have an implicit complex context. But within reason, the authors have dealt with this well. The evaluation is more descriptive than hypothesis evaluation. From the nature of the test, I would have expected using a t-test rather than a chi-square. Overall, the paper is a useful contribution to discussing the issue.

Author Response

Dear Reviewer!

Thank you for taking the time to read our manuscript carefully and for your positive overall assessment! Since no specific question has been asked, our response is limited to clarify why we used the chi2 test in our analysis.

We used the chi2 test because we wanted to know the relation between nominal type variables. We added details on this topic in the results (especially in section 3.1.). We hope you find the explanation given there satisfactory!

All the best,

The Authors

Round 2

Reviewer 2 Report (Previous Reviewer 2)

Dear Authors,

Thank you for the opportunity to review the revised version of the manuscript "Understanding Map Misinterpretation: Factors Influencing Correct Map Reading and Common Errors". The changes in the paper are substantial. Thank you for the opportunity to follow the development of the article.

To summarise, I recommend accepting the manuscript.

I wish you all the best for your future projects.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Thank you for your submission. More research on maps, map reading, map perceptions and so forth is always appreciated. I recommend that your article be revised and I look forward to that resubmission.

Recommendations for review are organized by section below.

Your initial sentence in the introduction needs a citation. I might believe the cell phone part of it, but documentation of that statement is necessary to bolster your overall argument. The last sentence of the first introduction paragraph should be cited as well. This is curious as the remainder of the section is well cited and organized, and it is satisfactory. 

The methods section is very robust and interesting, with two deficiencies to address. One - can you add an example of the map key that accompanied these maps (or, if the editors deem this too many figures - at least one key). Seeing the key is important to understand the full set of information that participants had when reading these maps. (This is especially true for question 7 - a mess without the full question, key, and materials). Second - how did you recruit these participants? Section 2.2 mentions that it was part of a survey - where and how and who and why? I also do not see a human subjects approval/process mentioned. That must be added for publication.

In the results - I think they are well laid out, though I'd be curious about a total overall score breakdown by the demographics. I am assuming you used binary gender but you never mention this - please be clear about how you collect this information. You need to standardize your presentation of the questions - questions with numbers in the first few figures become labeled text questions in the later ones. Be consistent. This will help interpretation of Figure 6 and the surrounding text which is currently dense.

Overall the discussion is reasonable but could be cited more throughout. For example, there has been extensive work done on gender and map/geographic ability, which would be useful to connect with.

I do struggle at the end of this in understanding how this exactly connects to car navigation. Reasonably, only the last map (with the rail/road network crossings) might reasonably be encountered by someone using a car navigation system. This does not invalidate the work - but it does pose questions of what purpose this serves and how it can be applied. Please consider.

Author Response

Responses to reviewer#1’s comments

Dear Reviewer,

Thank you for taking the time to read our manuscript carefully and for your valuable comments! Below is our response to your questions and comments, in bullet points. We hope we have understood all your questions correctly and answered them properly!

#1 Your initial sentence in the introduction needs a citation. I might believe the cell phone part of it, but documentation of that statement is necessary to bolster your overall argument. The last sentence of the first introduction paragraph should be cited as well. This is curious as the remainder of the section is well cited and organized, and it is satisfactory.

Response to #1: References to the two sentences mentioned have been inserted in the corrected manuscript.

#2 The methods section is very robust and interesting, with two deficiencies to address. One - can you add an example of the map key that accompanied these maps (or, if the editors deem this too many figures - at least one key). Seeing the key is important to understand the full set of information that participants had when reading these maps. (This is especially true for question 7 - a mess without the full question, key, and materials). Second - how did you recruit these participants? Section 2.2 mentions that it was part of a survey - where and how and who and why? I also do not see a human subjects approval/process mentioned. That must be added for publication.

Response to #2: Regarding the map key for question 7, we provided a key for the relevant linear features to aid the readers.

As for the participants, the 2015 survey mainly targeted university students, for which active recruitment was carried out. Later on, others also completed the questionnaire, and in the present study we were able to survey a wider population. These are also mentioned in the revised manuscript. The survey was conducted online, as mentioned in the original manuscript. Participants were informed about the conditions and the handling of the data on the test website (now mentioned in the revised manuscript). The respondents were initially mainly university students, as the test was distributed through international collegiate contacts. In the first study, the main objective was to investigate cultural differences among young people, which could still provide interesting results, but as indicated in the present manuscript, the objectives now were focusing on the mistakes. To make all these circumstances clearer, we completed the revised manuscript with the above-mentioned information at several places.

#3 In the results - I think they are well laid out, though I'd be curious about a total overall score breakdown by the demographics. I am assuming you used binary gender but you never mention this - please be clear about how you collect this information. You need to standardize your presentation of the questions - questions with numbers in the first few figures become labeled text questions in the later ones. Be consistent. This will help interpretation of Figure 6 and the surrounding text which is currently dense.

Response to #3: We appended the overall score to Table 2.

The questionnaire did indeed allow for binary gender responses. This was not specifically emphasised in the article, just as it is not emphasised in the numerous examples in the literature, but is clearly shown in the supplementary table (not included in the earlier uploaded manuscript, but included in the revised version) in the list of specific response options. Nevertheless, thank you for bringing this to our attention, as we will take it into account in our future research!

Regarding the figures, we have unified the labelling of questions, now we refer to numbers everywhere.

#4 Overall the discussion is reasonable but could be cited more throughout. For example, there has been extensive work done on gender and map/geographic ability, which would be useful to connect with.

Response to #4: We agree with you, there have indeed been many studies on this subject! For this very reason it is difficult to discuss this issue without omitting anyone who has written about it, so in the first version of the manuscript we have cited the most recent summary article (Havelkova & Hanus 2019), which does the listing for us. However, we have included some of the larger sample size studies in the revised version of the manuscript as a subchapter "Related studies", and also in the discussion. This almost doubled the number of references. We hope that we have managed to select the literature with due thoroughness!

#5 I do struggle at the end of this in understanding how this exactly connects to car navigation. Reasonably, only the last map (with the rail/road network crossings) might reasonably be encountered by someone using a car navigation system. This does not invalidate the work – but it does pose questions of what purpose this serves and how it can be applied. Please consider.

Response to #5: The study investigates two scenarios, navigation and wayfinding, with test questions, as both are performed on large scale maps and typically do not have a legend. Navigation is highlighted in the paper because it has a higher risk of making a mistake in this scenario. However, we do not emphasise this in the title and in the discussion chapter we also discuss our views on the wayfinding scenario (i.e. mainly for hiking or walking). In this way, although the issue of navigation is emphasised in the article, we do not, in our opinion, push the topic too far towards car navigation. On how the results can be used, we discuss in the discussion section (this was partly included in the original manuscript, but has been expanded in the new version).

Reviewer 2 Report

Dear Authors,
As you mention in the Results section, the study has the character of an ‘exploratory analysis’. Nevertheless, in your paper, you make some general statements based on statistically invalid results (see detailed comments below). If you decide to revise the paper, you should point out that the results are specific to your study and are not nomothetic in nature. In addition, your study lacks clearly stated research questions and the ‘Related Studies’ section, which I would expect to be quite extensive in an exploratory paper. The section on statistical analysis also should be improved (detailed comments below). But my main concern is that the submitted paper is too similar to your 2016 publication, as the number of participants is not significantly different (489 vs. 511 participants). 

Wish you all the best in your future endeavors.

Please find detailed comments on your paper below:

- Line 14 “511 participants from six countries”

The language is not an indication of nationality. If participants did not answer the question on their nationality, one cannot state only what languages they speak, not where they come from.

- Lines 20-22 “The results of the research can be used in the design of large-scale maps (e.g. car navigation), as they allow to reduce typical map reading errors by careful selection of symbol types and placements.”

The results of the study are not statistically valid, and thus cannot serve as a base for generalization.

 - Introduction

Please provide a separate section on ‘Related studies’. I strongly advise reading papers by i.a. Janet Speake or Mary Hegarty's research group on maps in navigation and on the impact of cultural background on map usage. Perhaps, the International Cartographic Association (ICA) Commission on the User Experience (UX) bibliography could be useful: https://use.icaci.org/bibliography/.

- Lines 27-28 “The maps that people use most often nowadays are most likely to appear on the screen of a cell phone or car navigation device.”

Reference needed.

- Line 40 “The accuracy of map reading”

It is not clear if the phrase is related to the map reader of the map itself.

- Line 42 “These skills are considered as statistical variables”

In some studies, these skills are considered statistical variables, but one cannot pose general a statement on that matter. The sentence needs rephrasing for clarity.

- Lines 52-54 and line 60 “These can be considered as partly learned errors that evolved during map use through inaccurate interpretation of map-related concepts [10,11]”; “maps we can only interpret what is intuitive or what is the same as today [18].”

References 10, 11, and 18 are not relevant to the topic of the study. 

- Lines 65-68 “but they do not have the same 65 meaning for the readers, and in addition, mental transformations (e.g. travel time estimation from scale) and rotation are the most important factors that influence the interpretation of the map.”

Reference needed.

- Please provide information on the ethical background of your study. Where there any Information clauses for the participants?

- Lines 78-80 “One of them was an open-ended question, the rest were multi-choice questions with 4+1 answers (4 possibilities and an “I don’t know” option)”

Why multi-choice questions if only one answer was proper? What is more, please add “the “I don’t know” option to the list of questions below (pp. 4-7). Please elaborate, on how you depicted multiple wrong answers in Figure 5.

- Section "Materials and Methods"

It is not clear in the text if you have used exactly the same research tool as in the 2016 paper. Please clarify that section.

- Provide a list of all questions used in the study (both questions on maps and questions on demographic data) in the Supplementary Materials.

- Add a figure explaining the study procedure.

 - Table 1

Add information on maps used for the question (number/code of the map).

The last cell (bottom-right) in the table seems to be missing information.

 - Lines 94-95 and Map 3

Provide examples of maps with the dollar symbol (“$”). Throughout my professional career and extensive usage of maps in my free time, I have never encountered such symbols on a map. Thus, the usage of this symbol might be confusing for participants of the study. The symbol is not mentioned on the maps discussed by Kent, A. J., & Vujakovic, P. (2009).

- Lines 201-202 “The test subjects were categorized into 4 age groups: below 20 (15%), 21 to 25 (37%), 26 to 30 (16%), above 31 (32%).”

Add information what was the highest age of the person from the “above 31” age group. Was the group consistent?

- Line 209 “we also grouped the participants by how often they used maps”

Provide the question and answers. Information about the rate of each subgroup (beginners, experts) is needed.

- Section “Results”

At this point, the statistical analysis presented is not acceptable.

Add information on what software you use to conduct analysis.

Provide detailed statistical results for all calculations (e.g. Male-Female X2 (1, N = 1238) = 11.483, p < 0.001, Cramér’s V = 0.096, p < 0.001 (with better results for female participants);

Add information on % of analyzed subgroups to Table 2 (e.g. Female 56%, Male 44%).

Please conduct the chi2 test on the distribution of the groups you want to compare. One cannot compare uneven groups. The remark particularly concerns the comparisons made within the Education, Age Category, and Language. I cannot refer to the "Map reading skills", as you do not give the information on the rate of beginners and experts.

The chi2 test allows in-pair comparisons in terms of dependence for tables 2x2 or 2x3. But it does not present the differences! I also suggest adding calculations of V-Cramer to see the degree of dependence.

Please present and discuss only statistically valid results.

The idea of analyzing the errors is interesting, but the process of data analysis was not conducted according to the state of the art. Did you perform any statistical analysis of the errors and background of the participants? As there are no statistical results given, it is not clear what is the basis of your elaboration on that topic.

- Section "Discussion"

More discussion with the findings of other researchers is needed. You provide reference to other studies only in two parts of the whole section “Discussion”. Please revise this section and the "Related Studies" section

- Lines 359-363 “Responses are basically discussed in terms of the primary map use scenario in which they might occur. In particular, incorrect answers to questions on competencies used in navigation are primarily evaluated, as they may occur in high-risk conditions. For this reason, we discuss the results below in terms of the weight of errors rather than the order of the questions.”

You have stressed the usage of study results in-car navigation. Please provide information if there was any time pressure when participants were answering the questions (which is the case while driving the car).

- Lines 367-368 “high number of participants mistook the overpass 367 symbols for a railway level crossing, despite the fact that the question explained that it was 368 a level crossing.”, and lines 433-437 “The results suggest that users probably did not have any specific misconceptions during the task of understanding hypsography while doing mental rotation, since the distribution of wrong answers was even for these tasks. On the other hand, during the interpretation of map symbols a large percentage of wrong answers assumed “rocky” is the correct answer, which might be caused by the symbol style resembled a round boulder.”

The maps provided in the manuscript do not contain a legend. Please elaborate on how it could influence the results.

- Line 407-409 “Participants who made errors failed to notice a smaller height difference of two points, and in most cases estimated the two points marked in the question to be the same height”

There are no points marked directly in the question. Points are mentioned in the possible answers. 

Comments on the quality of English concern also the clarity of the research in question, thus all the remarks are included in the "Comments and Suggestions for Authors".

Author Response

Responses to reviewer#2’s comments

Dear Reviewer,

Thank you for taking the time to read our manuscript carefully and for your valuable comments! Below is our response to your questions and comments, in bullet points. We respond first to your general comments, in which we would like to highlight the difference between our previous and current study.

Original general comment: As you mention in the Results section, the study has the character of an ‘exploratory analysis’. Nevertheless, in your paper, you make some general statements based on statistically invalid results (see detailed comments below). If you decide to revise the paper, you should point out that the results are specific to your study and are not nomothetic in nature. In addition, your study lacks clearly stated research questions and the ‘Related Studies’ section, which I would expect to be quite extensive in an exploratory paper. The section on statistical analysis also should be improved (detailed comments below). But my main concern is that the submitted paper is too similar to your 2016 publication, as the number of participants is not significantly different (489 vs. 511 participants).

Reply to the general comments: While the number of participants may look similar in the two studies, please note that the number cited in the manuscript is the number of filtered participants. The total number of participants was 611 and 721 in the two studies, respectively, an 18% increase. Furthermore, in our current work we used a more stringent filtering method than in the previous study: there we only filtered tests with a completion time of less than 2 minutes, whereas here we filtered tests between 2.5 minutes and 15 minutes based on natural breaks in the data set. It should also be stressed that the research questions in our previous publication were also different, and thus similarities are only present in that we examined the same competences because of the same questions. The positive side of this is that we can better compare the two results. In the present research, for example, we were also curious to see if there had been any changes in certain statistical results over time due to the new completions, but we found only a few noteworthy results in this respect (these are mentioned in the revised version).

Regarding the exploratory nature of the paper, we believe that we have emphasised this character in the manuscript to a sufficient degree, so that anyone reading it can give due weight to what you consider to be general statements. However, in the revised manuscript we have tried to improve the wording of our observations. The original manuscript already stated that the study is not considered representative of any population, so it is not clear to us what your basis is for believing that the findings are considered universally valid. Nevertheless, based on your opinion, we emphasize more in the revised version that the study is looking at a specific group.

We have included an explicit formulation of the research questions and a related research chapter (see our detailed answers).

Comment #1: Line 14 “511 participants from six countries”

The language is not an indication of nationality. If participants did not answer the question on their nationality, one cannot state only what languages they speak, not where they come from.

Reply to #1: Relevant statements were fixed accordingly.

Comment #2: Lines 20-22 “The results of the research can be used in the design of large-scale maps (e.g. car navigation), as they allow to reduce typical map reading errors by careful selection of symbol types and placements.”

The results of the study are not statistically valid, and thus cannot serve as a base for generalization.

Reply to #2: Since we have no information about the structure of the population that uses large-scale digital maps for orientation or navigation, our sample cannot be validated with it. So, your observation is true in this respect. However, this is not only true for our study, but for virtually all the studies in this field. The statistical methods used have been explained more precisely in the revised version, based on your comment (see our later reply). They have been carried out in the same way as in similar studies. We are not sure if we understand your comment correctly, but we would not consider it realistic to expect that the development of digital maps on portable devices would suddenly stop because the results of exploratory surveys would be discarded. However, in the revised manuscript we have tried to improve the wording of the concerning sentence.

Comment #3: Introduction

Please provide a separate section on ‘Related studies’. I strongly advise reading papers by i.a. Janet Speake or Mary Hegarty's research group on maps in navigation and on the impact of cultural background on map usage. Perhaps, the International Cartographic Association (ICA) Commission on the User Experience (UX) bibliography could be useful: https://use.icaci.org/bibliography/.

Reply to #3: Thank you for your suggestion! The collection on the ICA's UX group page is very useful, and although several of the authors listed there were already cited in the first version of the manuscript, even more are cited in the revised version. However, in the earlier version we did not cite any of the authors you explicitly mentioned. This has been corrected in the revised manuscript!

In the revised version, the studies related to our specific research questions have been summarised as a new subsection at the end of the Introduction chapter.

Comment #4: Line 40 “The accuracy of map reading”

It is not clear if the phrase is related to the map reader of the map itself.

Reply to #4: Both the map and the map reader affect the accuracy of the map reading. This statement refers to the accuracy of the information transfer (i.e. map reading) and not to the medium (map) or the receiver (map reader) similarly, as in Lobben 2004. This work is cited in the sentence in question.

Comment #5: Line 42 “These skills are considered as statistical variables”

In some studies, these skills are considered statistical variables, but one cannot pose general a statement on that matter. The sentence needs rephrasing for clarity.

Reply to #5: We rephrased the sentence in question.

Comment #6: Lines 52-54 and line 60 “These can be considered as partly learned errors that evolved during map use through inaccurate interpretation of map-related concepts [10,11]”; “maps we can only interpret what is intuitive or what is the same as today [18].”

References 10, 11, and 18 are not relevant to the topic of the study.

Reply to #6: We do not agree with this suggestion, as reference 10-11 in the earlier manuscript gives an example of one possible cause of the errors detected (the "learned errors") and reference 18 gives another example of the risk of relying on intuitive recognition of "familiar" signs. To make this more precise, we have clarified the wording in the revised manuscript!

Comment #7: Lines 65-68 “but they do not have the same 65 meaning for the readers, and in addition, mental transformations (e.g. travel time estimation from scale) and rotation are the most important factors that influence the interpretation of the map.”

Reference needed.

Reply to #7: Fixed!

Comment #8: Please provide information on the ethical background of your study. Where there any Information clauses for the participants?

Reply to #8: Thank you for your comment, we wrote about this in the corrected version of the manuscript! Before starting the test, the participants were noted that they are taking part of a research survey measuring the map reading skill and how long the test takes approximately. They were also informed that the test is anonymous, does not collect personal data and that the data collected is used for research purposes only.

Comment #9: Lines 78-80 “One of them was an open-ended question, the rest were multi-choice questions with 4+1 answers (4 possibilities and an “I don’t know” option)”

Why multi-choice questions if only one answer was proper? What is more, please add “the “I don’t know” option to the list of questions below (pp. 4-7). Please elaborate, on how you depicted multiple wrong answers in Figure 5.

Reply to #9: A multi-choice question means that the user can only choose from a list of answers. This is an accepted terminology in the literature to our knowledge and has been used by other authors (e.g. Chang & Antes, 1987 and Liebenberg, 1998, cited by us). In our case, only one answer could be selected by the respondent, which means that there was no possibility to select multiple wrong answers. To clarify this, the relevant part of section 2.1 has been completed. We have also added the option "I don't know" to the list of possible answers.

Comment #10: Section "Materials and Methods"

It is not clear in the text if you have used exactly the same research tool as in the 2016 paper. Please clarify that section.

Reply to #10: We don’t understand why the first sentence of that chapter did not make this clear. We cannot phrase it any more clearly than “we utilized a preexisting map reading test of the Research Group on Experimental Cartography”. However, in the new version we have indicated elsewhere in several places that it was the same test.

Comment #11: Provide a list of all questions used in the study (both questions on maps and questions on demographic data) in the Supplementary Materials.

Reply to #11: We completed the manuscript with a supplementary material listing the demographic questions. The rest of questions used in the study is available at fig. 1-4.  It will also be included in the supplementary materials along with the test results.

Comment #12: Add a figure explaining the study procedure.

Reply to #12: A figure on this topic was added to the manuscript!

Comment #13: Table 1

Add information on maps used for the question (number/code of the map).

The last cell (bottom-right) in the table seems to be missing information.

Reply to #13: Thank you for noticing the missing cell information! This and the requested column were added to the table!

Comment #14: Lines 94-95 and Map 3

Provide examples of maps with the dollar symbol (“$”). Throughout my professional career and extensive usage of maps in my free time, I have never encountered such symbols on a map. Thus, the usage of this symbol might be confusing for participants of the study. The symbol is not mentioned on the maps discussed by Kent, A. J., & Vujakovic, P. (2009).

Reply to #14: The symbol is found on large scale topographic maps of the countries influenced by the former Soviet Union's mapping system (e.g., Bulgaria, Romania, and Hungary) and depicts vineyards. An English-language reference to the map key of the Soviet topographic maps has been added. We note, however, that a similar 'vine climbing on a stick' symbol is used to indicate vineyards on some western (e.g., French) topographic maps.

Comment #15: Lines 201-202 “The test subjects were categorized into 4 age groups: below 20 (15%), 21 to 25 (37%), 26 to 30 (16%), above 31 (32%).”

Add information what was the highest age of the person from the “above 31” age group. Was the group consistent?

Reply to #15: We gave additional information on the age groups. The groups were consistent, only a few years were missing above 60.

Comment #16: Line 209 “we also grouped the participants by how often they used maps”

Provide the question and answers. Information about the rate of each subgroup (beginners, experts) is needed.

Reply to #16: The questionnaire is included as supplementary material, as requested.

The rate of each subgroup is added to Table 2 and 3.

Comment #17: Section “Results”

At this point, the statistical analysis presented is not acceptable.

Add information on what software you use to conduct analysis.

Provide detailed statistical results for all calculations (e.g. Male-Female X2 (1, N = 1238) = 11.483, p < 0.001, Cramér’s V = 0.096, p < 0.001 (with better results for female participants);

Add information on % of analyzed subgroups to Table 2 (e.g. Female 56%, Male 44%).

Please conduct the chi2 test on the distribution of the groups you want to compare. One cannot compare uneven groups. The remark particularly concerns the comparisons made within the Education, Age Category, and Language. I cannot refer to the "Map reading skills", as you do not give the information on the rate of beginners and experts.

The chi2 test allows in-pair comparisons in terms of dependence for tables 2x2 or 2x3. But it does not present the differences! I also suggest adding calculations of V-Cramer to see the degree of dependence.

Please present and discuss only statistically valid results.

The idea of analyzing the errors is interesting, but the process of data analysis was not conducted according to the state of the art. Did you perform any statistical analysis of the errors and background of the participants? As there are no statistical results given, it is not clear what is the basis of your elaboration on that topic. Section “Results”

At this point, the statistical analysis presented is not acceptable.

Add information on what software you use to conduct analysis.

Provide detailed statistical results for all calculations (e.g. Male-Female X2 (1, N = 1238) = 11.483, p < 0.001, Cramér’s V = 0.096, p < 0.001 (with better results for female participants);

Add information on % of analyzed subgroups to Table 2 (e.g. Female 56%, Male 44%).

Please conduct the chi2 test on the distribution of the groups you want to compare. One cannot compare uneven groups. The remark particularly concerns the comparisons made within the Education, Age Category, and Language. I cannot refer to the "Map reading skills", as you do not give the information on the rate of beginners and experts.

The chi2 test allows in-pair comparisons in terms of dependence for tables 2x2 or 2x3. But it does not present the differences! I also suggest adding calculations of V-Cramer to see the degree of dependence.

Please present and discuss only statistically valid results.

The idea of analyzing the errors is interesting, but the process of data analysis was not conducted according to the state of the art. Did you perform any statistical analysis of the errors and background of the participants? As there are no statistical results given, it is not clear what is the basis of your elaboration on that topic.

Reply to #17: We added the software name and version to section ‘Results’.

The statistical results of both Chi2 and Cramer's V were added to the text.

The percentages of the groups were also added to table 3, and 4.

Chi2 test was originally run during the evaluation, but as mentioned in our previous answer above, the statistical results of both Chi2 and Cramer's V are now added to the text.

During the error analysis we investigated the relative frequency distribution of the data (see: https://openstax.org/books/statistics/pages/1-3-frequency-frequency-tables-and-levels-of-measurement). This is a method used in recent papers, see for example Fazzolari, Michela, et al. "Experience: improving opinion spam detection by cumulative relative frequency distribution." Journal of Data and Information Quality (JDIQ) 13.1 (2021): 1-16. Based on this we believe that that this method should be considered state of the art.

In our analysis, as we hope is now clearer from our amendments, we discuss statistically valid and significant results.

We did not perform any analysis based on the errors and the participant background. The reason behind this is that in some of the groups (e.g., elementary school graduates) the population number is so low that we didn't want to further divide them by the type of mistake they had made. Because of this we only investigated the whole population for these types of analyses. Hopefully in the future we will have more participant for this test, and we can do the analyses for the groups too.

Comment #18: Section "Discussion"

More discussion with the findings of other researchers is needed. You provide reference to other studies only in two parts of the whole section “Discussion”. Please revise this section and the "Related Studies" section

Reply to #18: In the revised version, a chapter on related studies has been added and the discussion has been completed in the light of that.

Comment #19: Lines 359-363 “Responses are basically discussed in terms of the primary map use scenario in which they might occur. In particular, incorrect answers to questions on competencies used in navigation are primarily evaluated, as they may occur in high-risk conditions. For this reason, we discuss the results below in terms of the weight of errors rather than the order of the questions.”

You have stressed the usage of study results in-car navigation. Please provide information if there was any time pressure when participants were answering the questions (which is the case while driving the car).

Reply to #19: Thank you for the idea, we did not apply time pressure on the users (although we did measure the answering time), but we added this as potential future research to the Conclusions section.

Comment #20: Lines 367-368 “high number of participants mistook the overpass 367 symbols for a railway level crossing, despite the fact that the question explained that it was 368 a level crossing.”, and lines 433-437 “The results suggest that users probably did not have any specific misconceptions during the task of understanding hypsography while doing mental rotation, since the distribution of wrong answers was even for these tasks. On the other hand, during the interpretation of map symbols a large percentage of wrong answers assumed “rocky” is the correct answer, which might be caused by the symbol style resembled a round boulder.”

The maps provided in the manuscript do not contain a legend. Please elaborate on how it could influence the results.

Reply to #20: This was discussed in line 377-380 of the first version of the manuscript. In the revised manuscript, we added that the use of map legends may improve performance in these cases in a map-use test. We also added that popular digital mapping tools (e.g., Google Maps) and other studies with navigation-oriented maps (e.g., Dillemuth, 2009; Pugliesi, 2009; Ramos et al. 2016) do not include map legends, so we did not provide any.

Comment #21: Line 407-409 “Participants who made errors failed to notice a smaller height difference of two points, and in most cases estimated the two points marked in the question to be the same height”

 There are no points marked directly in the question. Points are mentioned in the possible answers.

Reply to #21: Thank you, we clarified this statement!

Reviewer 3 Report

I think it is a well thought out and interesting article, even more so at the present time when the use of maps has spread in a way that would have been unthinkable years ago.

The paper is well organised and presented. However, I think that a key element for any sample study is missing. This is the description and justification of the sample. What type of sampling has been used? Random, stratified random, etc.

You say a total number of respondents, but you do not break down the number of respondents represented in each of the categories analysed. It is NOT known, for example, how many men or how many did it in English (is language related to nationality?).

It is also necessary to reflect on and justify these decisions. What has led to the choice of these age groups? What is the reason for choosing these languages/countries and not others?

Similarly, there is no explanation of how the sample was carried out: was there a specific time frame, and over what period of time?

I think that all these elements would help to understand the significance of the results and whether they are comparable because of the homogeneity of at least the number of respondents.

Thanks

Author Response

Responses to reviewer#3’s comments

Dear Reviewer,

Thank you for taking the time to read our manuscript carefully and for your valuable comments! Below is our response to your questions and comments, in bullet points. We hope we have understood all your questions correctly and answered them properly!

#1 The paper is well organised and presented. However, I think that a key element for any sample study is missing. This is the description and justification of the sample. What type of sampling has been used? Random, stratified random, etc.

Response to #1: The data collected in the survey were obtained either on a voluntary or on a controlled basis and can therefore be considered non-probability sampling (despite the fact that it may be representative for the whole population in terms of sex ratio, e.g. for Hungarian respondents, but we did not validate the sample on this). This was pointed out in the manuscript (also in the first version) at the end of the discussion, with the comment that the research was exploratory. In the revised manuscript, we have tried to further clarify the nature of the sample in Section 2.2.

#2 You say a total number of respondents, but you do not break down the number of respondents represented in each of the categories analysed. It is NOT known, for example, how many men or how many did it in English (is language related to nationality?).

Response to #2: The distribution of participants within each demographic category are now added to Table 2 and 3 as percentages, from which the readers can deduce the number of participants in each category.

Regarding the issue with language and nationality, we have clarified it in the reviewed text that they do not necessarily mean the same.

#3 It is also necessary to reflect on and justify these decisions. What has led to the choice of these age groups? What is the reason for choosing these languages/countries and not others?

Response to #3: These age groups were defined in line with our previous study in 2016 to keep the results comparable. In the present research, for example, we were also interested to see if the new completions would change some statistical results over time, but we found only a few noteworthy results in this respect (these are mentioned in the revised version). The languages were chosen based on the research group's contacts with universities abroad to facilitate the collection of data from foreign participants. Clarifications have been made in section 2.2 "Test sample" to explain these circumstances.

#4 Similarly, there is no explanation of how the sample was carried out: was there a specific time frame, and over what period of time?

Response to #4: We are not sure if we understood your comment correctly, but if you are referring to the length of the data collection period and the completion time, these were mentioned in section 2.2 "Test sample" of the original manuscript, although we have now elaborated on them to make it clearer. If you are referring to the limitation on completion time, our answer is that we did not put any time pressure on users (although we did measure response time), but we have added this to the Conclusions section as potential future research.

#5 I think that all these elements would help to understand the significance of the results and whether they are comparable because of the homogeneity of at least the number of respondents.

Response to #5: We agree with you that the points made in your comments provide important information about the study, and thank you again!

Round 2

Reviewer 1 Report

Thank you for your responses. I am generally satisfied, though concerned that a map key was not provided to students for some of the questions. Maps should not be assumed as intuitive enough to understand without symbolic explanation as different countries/entities have their own standards. However that is what it is, as it were, so I will not withhold publishing on that regard.

However, despite the statement of how this was distributed (necessary and appreciated!) no statement on approval by an Institutional Review Board (IRB), Human Subjects Projection Panel (HSP/HSPP), independent ethics committee (IEC), ethical review board (ERB), or research ethics board (REB) is present. There needs to be a statement clearly saying 'This research was approved by INSTITUTION's BOARD NAME as APPROVAL ID'. Given this was funded research, certainly one of these exists.

Until completed this paper should not be published.

Some minor edits especially of new content would help with the overall flow.

Reviewer 2 Report

Comments and suggestions are provided in the pdf file.

Comments for author File: Comments.pdf

Please verify the manuscript regarding the words used in colloquial speech.