Mapping with ChatGPT

Round 1

Reviewer 1 Report

The research investigates two main methodologies of map creation using ChatGPT, namely designing thematic maps with predetermined or public geospatial data and forming mental maps from textual descriptions of geographic areas. The paper finds that ChatGPT brings significant advantages to the field of cartography, including easing the learning process for professional map designs, enhancing efficiency in large-scale map production, and promoting spatial thinking and understanding. Despite these benefits, the study acknowledges the current limitations of ChatGPT in map-making, including its unequal advantages for different user demographics and the necessity of user intervention for quality control. 

This is an interesting and timely study that explore the potential of large language models, specifically ChatGPT, in innovating map-making procedures, which will be of a great interest to a broad range of readers in GIScience and other relevant fields. My concerns and suggestions are listed below: 

1.     The authors mentioned two versions of GPT are available, but it’s not clear which version the current study used.

2.     For Figure 3, the authors may want to double check the direction of the north arrow is correctly oriented in the map based on the specific map project used here.

3.     With regard to the first limitation, it might be beneficial to re-evaluate whether this truly constitutes a limitation within the GIS mapping context. As GIS fundamentally involves the use of spatial data and coding (with tools also based on programming code), utilizing ChatGPT to generate codes for map creation seems to maintain this foundation rather than disrupt it.

4.     Some relevant discussions regarding limitations 2 and 3 (and some potential solutions) can be found in the Automatous GIS paper cited this study, although that paper does not specifically focus on mapping. It would be beneficial to the readers if the authors could further discuss/incorporate the findings from the previous study.  This would also provide an opportunity to highlight unique challenges specifically tied to GIS mapping when using ChatGPT, which would be informative for readers.

5. It would be great to share the codes generated by ChatGPT for each or some selected presentative prompts included in this study.

6.     The resolution of all the figures in the paper is insufficient for clear understanding.

The language is ok and the paper is easy to follow. 

Author Response

The research investigates two main methodologies of map creation using ChatGPT, namely designing thematic maps with predetermined or public geospatial data and forming mental maps from textual descriptions of geographic areas. The paper finds that ChatGPT brings significant advantages to the field of cartography, including easing the learning process for professional map designs, enhancing efficiency in large-scale map production, and promoting spatial thinking and understanding. Despite these benefits, the study acknowledges the current limitations of ChatGPT in map-making, including its unequal advantages for different user demographics and the necessity of user intervention for quality control. 

This is an interesting and timely study that explore the potential of large language models, specifically ChatGPT, in innovating map-making procedures, which will be of a great interest to a broad range of readers in GIScience and other relevant fields. My concerns and suggestions are listed below: 

1. The authors mentioned two versions of GPT are available, but it’s not clear which version the current study used.

Re: Thank you for the comment. We actually stated in the 3^rd paragraph of our original manuscript that “In this study, we explore making professional maps with ChatGPT (GPT-4)”

2. For Figure 3, the authors may want to double check the direction of the north arrow is correctly oriented in the map based on the specific map project used here.

Re: Thank you for the comment. Since there are multiple reviewers shared concerns about the original Figure 3, we’ve re-designed the map by using graticule to specify spatial extent, instead of using scale bar and north arrow.

3. With regard to the first limitation, it might be beneficial to re-evaluate whether this truly constitutes a limitation within the GIS mapping context. As GIS fundamentally involves the use of spatial data and coding (with tools also based on programming code), utilizing ChatGPT to generate codes for map creation seems to maintain this foundation rather than disrupt it.

Re: thank you for the valuable opinion. We agree that “GIS fundamentally involves the use of spatial data and coding (with tools also based on programming code)”. However, whether mapping with ChatGPT maintains or disrupts the way of map creation is debatable. For GIS professionals, it maintains and enhanced the map creation process given they will most likely use spatial data and coding for mapping anyway. For non-professionals, being able to skip the hardest technical part, namely writing codes, is a huge advantage of mapping with ChatGPT. One of the authors recently introduced this approach to his mapping course, of which the majority of students were not in GIS/geography major, the feedback was certainly encouraging. Although mapping with ChatGPT still relies on coding at its core, it is still a disruptive tool if the coding part stays as a “black box” to the users.

We rephrased the relevant texts to make it clear that the “disruption” or “innovation” depends on the level of users.

4. Some relevant discussions regarding limitations 2 and 3 (and some potential solutions) can be found in the Automatous GIS paper cited this study, although that paper does not specifically focus on mapping. It would be beneficial to the readers if the authors could further discuss/incorporate the findings from the previous study. This would also provide an opportunity to highlight unique challenges specifically tied to GIS mapping when using ChatGPT, which would be informative for readers.

Re: thank you for this suggestion. In the conclusion, we have added a future direction is to implement an intelligent and autonomous mapping agent. The agent should contain a decision-making core to overcome the current limitations, such as dependence on external conditions and unsatisfactory initial maps (Li and Ning 2023).

5. It would be great to share the codes generated by ChatGPT for each or some selected presentative prompts included in this study.

Re: we’ve uploaded all of our codes and results in a dedicated GitHub page: https://github.com/jinwenxu/ChatGPT4Mapping and shared that URL in the Data Availability Statement in the end of the paper.

6. The resolution of all the figures in the paper is insufficient for clear understanding.

Re: we’ve revised all the maps in the paper with improved resolution.

Reviewer 2 Report

The authors know very well what ChatGPT can and cannot do. According to the precisely written requirements, ChatGPT can write program code to create a map, and the authors must then use some software to draw the map. They point out that an experienced user will know how to write requests better than an inexperienced one. The first map is rarely satisfactory (Fig. 2) and should be improved with additional requirements (Fig. 3).  In the end, the authors describe well the pros and cons of ChatGPT in map making and suggest that ChatGPT should be coupled with some AI mapping tool in the future. In the text, the authors refer to Tobler (2020), which is not correct. In the list of references Li, Z. is not in the right place.

Author Response

Re: thank you for the comment. We have corrected the citations. We have also improved the readability of all our maps.

Reviewer 3 Report

It is an interesting work on the use of the ChatGPT tool for the creation of thematic maps. The proposal is also innovative, since ChatGPT is an emerging technology.

But it is more of a job of description, not of experimentation. That is, it does not have the structure: Introduction-hypothesis-methodology-data collection-results-discussion-conclusions. Therefore, conclusions based on data processing (results) cannot be obtained after applying a methodology.

On the other hand, as it is an emerging technology that has only just begun, the methodology for creating a map using this tool seems very innovative and interesting, as well as the weaknesses and strengths described by the authors. Therefore, I propose a complete rewrite of the article in which the introduction makes clear the non-experimental approach of the paper, that is, that the work has a more descriptive approach, and that it does not provide data on the usability or the possible impact of its use in the teaching field, or its impact on the development of geospatial thought... etc. which could be indicated as future work.

That being said, there are some specific recommendations:

Abstract

I don't know if “language models” is the correct term to refer to VahtGPT. Consider review. Likewise, the phrase “has ushered in a new era of innovation and opportunity across various industries and disciplines” is hardly appropriate for a scientific article. I recommend checking this out. In general, the abstract does not describe any concrete results, nor any conclusion supported by data: “…We argue that ChatGPT can be a handy tool for map …”, “…it has a huge potential to revolutionize industry, education, and research in the mapping world”. I recommend a complete rewriting of the abstract considering what is exposed and structured into: research object, methodology, summary of results and conclusions (for an experimental approach) or else making it clear that it is a more descriptive than experimental article.

1. Introduction

The phrase “we believe ChatGPT has a great potential to revolutionize the way we design and produce maps.” It is not backed by data or methodology or results that support it. I recommend removing it.

2. Thematic map

I congratulate the authors for the detailed description of the process of creating a map using the Chat Gpt interface, as well as the abundance of figures illustrating the process.

3. Mind map

I like the double approach adopted: Skech map and mental map via coding.

4. Discussion and conclusions.

This section usually talks about the results obtained after applying a methodology, but the authors limit themselves to giving "our opinion", which is not correct. It is necessary to make clear the descriptive focus of the paper.

Author Response

It is an interesting work on the use of the ChatGPT tool for the creation of thematic maps. The proposal is also innovative, since ChatGPT is an emerging technology.

But it is more of a job of description, not of experimentation. That is, it does not have the structure: Introduction-hypothesis-methodology-data collection-results-discussion-conclusions. Therefore, conclusions based on data processing (results) cannot be obtained after applying a methodology.

On the other hand, as it is an emerging technology that has only just begun, the methodology for creating a map using this tool seems very innovative and interesting, as well as the weaknesses and strengths described by the authors. Therefore, I propose a complete rewrite of the article in which the introduction makes clear the non-experimental approach of the paper, that is, that the work has a more descriptive approach, and that it does not provide data on the usability or the possible impact of its use in the teaching field, or its impact on the development of geospatial thought... etc. which could be indicated as future work.

Re: thank you for the comment. This paper does not follow a traditional approach based on controlled experiments. What we have done include a series of exploratory tests, to e. We have reorganized the manuscript to better fit the logic and make the paper more readable. We also moved the possible impacts on relevant fields, e.g., teaching, to future works as we have not validated them in this study.

That being said, there are some specific recommendations:

Abstract

I don't know if “language models” is the correct term to refer to ChatGPT. Consider review. Likewise, the phrase “has ushered in a new era of innovation and opportunity across various industries and disciplines” is hardly appropriate for a scientific article. I recommend checking this out. In general, the abstract does not describe any concrete results, nor any conclusion supported by data: “…We argue that ChatGPT can be a handy tool for map …”, “…it has a huge potential to revolutionize industry, education, and research in the mapping world”. I recommend a complete rewriting of the abstract considering what is exposed and structured into: research object, methodology, summary of results and conclusions (for an experimental approach) or else making it clear that it is a more descriptive than experimental article.

Re: thank you for the comment. We have completely rewritten the abstract to address the points mentioned above.

1. Introduction

The phrase “we believe ChatGPT has a great potential to revolutionize the way we design and produce maps.” It is not backed by data or methodology or results that support it. I recommend removing it.

Re: thank you for the comment. We have rephrased it in the revision.

2. Thematic map

I congratulate the authors for the detailed description of the process of creating a map using the ChatGPT interface, as well as the abundance of figures illustrating the process.

Re: thank you for the encouragement.

3. Mind map

I like the double approach adopted: Skech map and mental map via coding.

Re: thank you for the encouragement.

4. Discussion and conclusions.

This section usually talks about the results obtained after applying a methodology, but the authors limit themselves to giving "our opinion", which is not correct. It is necessary to make clear the descriptive focus of the paper.

Re: thank you for the comment. We rephrased this part to make the discussions more appropriate based on our test results.

Reviewer 4 Report

My initial suggestion is to recharacterize the utility of ChatGPT for map making. I don’t think “lowering the learning curve of professional map design” is really what ChatGPT does here. “Lowering the barrier to producing maps” might be more accurate. I would not classify the maps that ChatGPT produces in this paper as professional - as in the quality of someone who is an expert cartographer or visualization specialist paid to make maps. Additionally, this is not really lowering the learning curve or helping people truly understand the process of making these maps (i.e., learning Python, Matplotlib, or GIS), it is just giving them the final product.

I would also suggest you avoid the use of non-precise adjectives such as “handy” or “huge” (see abstract).

The introduction overall is very brief, and I think would benefit from exploring some established tools/platforms that exist for making it easier to produce maps. While these are not based on ChatGPT - and few use any AI/ML, while some are also paid platforms - they are important considerations when it comes to practical ways to make creating maps easier. A widely used example is ArcGIS online, which requires no coding and is increasingly oriented towards users without technical/programming/cartography skills, and has been widely used in academic settings for teaching, projects, etc.

In addition, the introduction provides little specific insight into relevant features/capabilities of ChatGPT such as writing code, interpreting data sources, producing outputs, etc. Given the lack of substantive background on ChatGPT relative to this paper, it is a bit of a stretch to quickly jump to “we believe ChatGPT has a great potential to revolutionize the way we design and produce maps”. 

One of my broader concerns is the distinction between ChatGPT being able to produce code to achieve a relatively straightforward and narrowly defined task (which is well supported by existing Python packages/examples and data), vs actually performing meaningful spatial thinking and identifying a novel function of ChatGPT.  While I personally find making static/web maps a really fun use of ChatGPT, it is already very well established that ChatGPT can write code fairly well, particularly when leveraging well established packages such as Matplotlib, and this paper does not really provide novel insight into that core ability of ChatGPT. 

Some of the language describing the prompts would also seem to suggest ChatGPT has a deeper understanding of maps rather than just changing a code variable. E.g. “Besides choropleth maps, there are other map types available to choose from…” makes it sound like these map features are built into ChatGPT. For readers less familiar with Matplotlib, GIS and associated Python packages this could be confusing/misleading. 

I appreciate the emphasis on needing to run code in a separate environment and that initial maps are often not ideal, but I think additional details on the specific steps required to iterate on maps is needed. One common issue associated with using code from ChatGPT is that it usually requires debugging. Even if the code from ChatGPT did not require debugging in your examples, it is a critical consideration given the intended use of lowering the learning curve / barrier for making maps for people who presumably do not know how to write code to produce maps. You touch on this topic at the end of the Prompt 1 section, stating it may be more convenient to edit code for the last mile, but that may be much easier said than done for your intended users.

The section on mental maps is where you really explore the spatial thinking ability of ChatGPT and is the more novel portion of your work. This opens the door to a lot of potential questions about how ChatGPT can understand space and incorporate information. However in its current form this seems like it is more of a thought-provoking example rather than research. I’d personally suggest expanding your work on mental maps to address a clearly defined research question.

The last items I wanted to address are the descriptions of ChatGPT in the discussion. While ChatGPT does make user interaction more human and intuitive in some ways, it has many flaws and is certainly not foolproof. ChatGPT is incredibly dependent on the specific language and approaches to generating prompts, and simply engaging in a conversation with ChatGPT as you would a human will not always result in ideal outcomes. This may not have been a notable concern for the example prompts/interactions you provided, but related limitations and idiosyncrasies of ChatGPT and other LLMs in responding to prompts are well established.

There were a handful of minor typos and issues with sentence structure, I suggest a thorough proof reading

Author Response

My initial suggestion is to recharacterize the utility of ChatGPT for map making. I don’t think “lowering the learning curve of professional map design” is really what ChatGPT does here. “Lowering the barrier to producing maps” might be more accurate. I would not classify the maps that ChatGPT produces in this paper as professional - as in the quality of someone who is an expert cartographer or visualization specialist paid to make maps. Additionally, this is not really lowering the learning curve or helping people truly understand the process of making these maps (i.e., learning Python, Matplotlib, or GIS), it is just giving them the final product.

Re: we agree that “lowering the barrier to producing maps” is a more accurate description than “lower the learning curve”. We have revised the manuscript accordingly.

I would also suggest you avoid the use of non-precise adjectives such as “handy” or “huge” (see abstract).

Re: we have rephrased the relevant sentences.

The introduction overall is very brief, and I think would benefit from exploring some established tools/platforms that exist for making it easier to produce maps. While these are not based on ChatGPT - and few use any AI/ML, while some are also paid platforms - they are important considerations when it comes to practical ways to make creating maps easier. A widely used example is ArcGIS online, which requires no coding and is increasingly oriented towards users without technical/programming/cartography skills, and has been widely used in academic settings for teaching, projects, etc. 

Re: thank you for this comment. In this revision, we have added a summary of existing mapping tools in the introduction. We have also briefly discussed why mapping with ChatGPT can potentially offer some unique advantages, which is one of the motivations of this study.  

In addition, the introduction provides little specific insight into relevant features/capabilities of ChatGPT such as writing code, interpreting data sources, producing outputs, etc. Given the lack of substantive background on ChatGPT relative to this paper, it is a bit of a stretch to quickly jump to “we believe ChatGPT has a great potential to revolutionize the way we design and produce maps”. 

Re: thank you for this comment. In the introduction, we added a summary of the relevant features or capabilities of ChatGPT, including writing codes, search for data sources, and comprehend users’ requests through conversations. These characteristics of ChatGPT motivates us to explore its potential for producing maps.

One of my broader concerns is the distinction between ChatGPT being able to produce code to achieve a relatively straightforward and narrowly defined task (which is well supported by existing Python packages/examples and data), vs actually performing meaningful spatial thinking and identifying a novel function of ChatGPT.  While I personally find making static/web maps a really fun use of ChatGPT, it is already very well established that ChatGPT can write code fairly well, particularly when leveraging well established packages such as Matplotlib, and this paper does not really provide novel insight into that core ability of ChatGPT. 

Some of the language describing the prompts would also seem to suggest ChatGPT has a deeper understanding of maps rather than just changing a code variable. E.g. “Besides choropleth maps, there are other map types available to choose from…” makes it sound like these map features are built into ChatGPT. For readers less familiar with Matplotlib, GIS and associated Python packages this could be confusing/misleading. 

Re: thank you for the valuable comment. In contrast with making mental maps, making static/web maps with ChatGPT does not fully exploit its thinking capability, which truly separates it from existing tools. However, we think it is still meaningful to exploit how well ChatGPT can respond to users’ mapping requests and find the right mapping functions from associated Python packages to use. Users do not need to speed time on gaining a panoramic view of the Python packages to know what map features/styles are available. Instead, ChatGPT serves as an AI-based assistant to choose the right mapping function to fulfill users’ needs. We have revised the relevant texts to describe ChatGPT’s role more accurately when creating static/web maps.

I appreciate the emphasis on needing to run code in a separate environment and that initial maps are often not ideal, but I think additional details on the specific steps required to iterate on maps is needed. One common issue associated with using code from ChatGPT is that it usually requires debugging. Even if the code from ChatGPT did not require debugging in your examples, it is a critical consideration given the intended use of lowering the learning curve / barrier for making maps for people who presumably do not know how to write code to produce maps. You touch on this topic at the end of the Prompt 1 section, stating it may be more convenient to edit code for the last mile, but that may be much easier said than done for your intended users.

Re: thank you for the comment. We acknowledge this limitation of mapping with ChatGPT. In some of our tests, the whole process can be free of debugging. However, it is quite uncertain and is beyond our control. Only after several attempts to improve the map with follow-up prompts, we shall know if debugging is needed for the last mile. In this discussion, we added sentences to clearly state this limitation.

The section on mental maps is where you really explore the spatial thinking ability of ChatGPT and is the more novel portion of your work. This opens the door to a lot of potential questions about how ChatGPT can understand space and incorporate information. However in its current form this seems like it is more of a thought-provoking example rather than research. I’d personally suggest expanding your work on mental maps to address a clearly defined research question.

Re: thank you for the comment. We also think making mental maps better showcases ChatGPT’s spatial thinking capability that separates it from other mapping tools. In this revision, we rephrased the relevant texts in that section to formalize our research design. We also made it clear in the future work that we plan to comprehensively evaluate ChatGPT’s spatial thinking capabilities in a separate study, of which creating mental maps is one crucial aspect.

The last items I wanted to address are the descriptions of ChatGPT in the discussion. While ChatGPT does make user interaction more human and intuitive in some ways, it has many flaws and is certainly not foolproof. ChatGPT is incredibly dependent on the specific language and approaches to generating prompts, and simply engaging in a conversation with ChatGPT as you would a human will not always result in ideal outcomes. This may not have been a notable concern for the example prompts/interactions you provided, but related limitations and idiosyncrasies of ChatGPT and other LLMs in responding to prompts are well established.

Re: thank you for the comment. We agree that interacting with ChatGPT is intuitive but not necessarily foolproof. In the discussion, we added sentences to objectively describe the pros and cons of user interaction. Our conclusion has also been modified to state mapping with ChatGPT is still in an early stage. It is a promising yet immature approach to creating maps. It brings us a new solution, but it will not necessarily replace the existing popular mapping solutions.

Reviewer 5 Report

In an attempt to generate maps using ChatGPT, the authors created thematic maps utilizing geospatial data, both provided and publicly accessible. Additionally, they sought to generate mental maps based on descriptions of geographic space. This endeavor demonstrates an exploration of ChatGPT's potential applications in GIS research. Nevertheless, there are several limitations to consider in this study.

Firstly, the authors assert that they have generated "professional" maps using ChatGPT. However, many of the maps presented in this manuscript do not meet professional standards. Issues such as inappropriate north arrows and legends, even when modified with follow-up prompts, as well as potential uncertainties in map attributes, compromise their professionalism. ChatGPT is more like a playful tool rather than a professional cartographer. Therefore, caution must be exercised when recommending its use for mapping purposes, as it can lead to erroneous results that users may not be aware of.

Secondly, the authors could provide additional information regarding the capabilities of ChatGPT. The conclusions of this study should enlighten users on how to effectively employ ChatGPT for mapping tasks, delineating its potential functionalities, limitations, and the potential risks associated with its usage.

Lastly, the figures presented in the manuscript are unclear.

Author Response

In an attempt to generate maps using ChatGPT, the authors created thematic maps utilizing geospatial data, both provided and publicly accessible. Additionally, they sought to generate mental maps based on descriptions of geographic space. This endeavor demonstrates an exploration of ChatGPT's potential applications in GIS research. Nevertheless, there are several limitations to consider in this study.

Firstly, the authors assert that they have generated "professional" maps using ChatGPT. However, many of the maps presented in this manuscript do not meet professional standards. Issues such as inappropriate north arrows and legends, even when modified with follow-up prompts, as well as potential uncertainties in map attributes, compromise their professionalism. ChatGPT is more like a playful tool rather than a professional cartographer. Therefore, caution must be exercised when recommending its use for mapping purposes, as it can lead to erroneous results that users may not be aware of.

Re: thank you for the comment. We agree that some of our maps created with ChatGPT are imperfect. In this revision, we first revised our maps in terms of map design and readability, e.g., we replaced the awkward north arrow and scale bar with graticule in Figure 3, and we improved the legend in all maps. Second, we rephrased sentences to emphasize that ChatGPT is more of a tool for beginner users to create satisfactory maps rather than for expert users to create highly professional maps.  

Secondly, the authors could provide additional information regarding the capabilities of ChatGPT. The conclusions of this study should enlighten users on how to effectively employ ChatGPT for mapping tasks, delineating its potential functionalities, limitations, and the potential risks associated with its usage.

Re: thank you for the comment. We have revised the conclusion to state how to effectively employ ChatGPT for mapping, and who should use it for different situations. The potential functionalities, limitations, and risks are also discussed.

Lastly, the figures presented in the manuscript are unclear.

Re: we’ve revised all the maps in the paper with improved resolution.

Round 2

Reviewer 1 Report

My concerns are addressed. 

Author Response

We very much appreciate your assessment of our work. Thank you.

Reviewer 3 Report

The authors have made the changes that I proposed in my previous review. I understand that the article is interesting and publishable.

Author Response

We very much appreciate your objective assessment of our work. Thank you.

Reviewer 4 Report

Thank you for addressing my comments. I think the edits and framing changes put this article in a much better spot. I did come across some additional articles (albeit preprints) that might be relevant to include.

https://arxiv.org/abs/2306.00020

https://eartharxiv.org/repository/view/5544/

There are handful of minor typos and sentence issues that could be improved.

Author Response

Thank you for the suggested additional literature. We have included them in the latest revision. We have also gone through the manuscript again and corrected several typos or grammatical errors.