Seeing Climate Change: A Framework for Understanding Visualizations for Climate Adaptation
Round 1
Reviewer 1 Report
This is a review paper, which is very comprehensive and informative. I think that it is a good addition to the geovisualization community.
The overall flow is good and everything is appropriately organized. The only constructive criticism I can think of is to add a nice summary of the paper in the conclusion section.
I will support this work for publication.
Author Response
The authors would like to thank this reviewer for their time in providing the review. We are happy to hear that the reviewer finds our manuscript acceptable for publication.
Reviewer 2 Report
Comments
A very welcome topic, there are many unexamined geovisualization tools and methods for communicating complex data on climate change (and other themes) to choose between. Any systematic way to help users to select geovisualization for the right purpose and audience is a step forward.
Overall
Although the topic is relevant and that some parts, especially the results section, are good and comprehensive, there are several parts of the paper that need to be developed and revised:
- The empirical approach is quite weak: a literature review is not enough to finalize a CVAP framework; I suggest that the framework should be tested also on practitioners, decision makers and researchers. The framework is for them, right?
- A follow-up comment is that the presented framework (and its result with classification of different geovisualization tools) is too complex and non-transparent for practitioners and to some extent also for researchers. I believe that the framework's accessibility, content and transparency can be increased by testing it on different groups.
- Accordingly, the development of the new framework based on previous and existing research, including used concepts and dimensions, needs to be presented in a more transparent way - there are too many questions and ambiguities in how different geovisualization tools/approaches are classified and placed within the presented model (even if the scale is binary). How are geovisualization tools assessed based on the dimensions (themes) of risk, realism and interactivity? There is an invisible basis for assessment grounded on a high degree of subjectivity (difficult to avoid in research - but the transparency can be improved). This is probably also related to the fact that the different dimensions are both qualitative and quantitative, concrete and abstract in their forms – the assessment process becomes unlear.
- There are ambiguities in the method chapter of how the review and development of the framework took place, see also specific comments below. One problem is that several methodological descriptions and clarifications appear in the result and discussion sections. I would suggest a restructuring of the paper with a clearer balance between method and result sections, e.g. some methodological explanations, such as the Map Use Cube etc., should be moved from result section to the method section. Another example that you state that the approach is qualitative in the discussion at the end of the paper? This should be stated and explored already in the method section.
- It is the result of the framework that is useful to users, e.g. the CVAP populated with geovisualization products to choose between (figure 3), or the CVAP itself – the process of classifying geovisualization tools and methods into themes? This is confusing?
Specific
Row
74 No source for “sea ice monitoring”
108 Structured literature review, in what way?
116 “several iterations” How many and in what way?
117 “included parameters” Which parameters? Comes later but needs to be clarified already here.
122 In what way is the review comprehensive?
124 Topic = Criteria? What do you mean by criteria? In what way? Not clear!
124 Structure, in what way?
126 “The criteria together capture the benefits and the inherent trade-offs…” How?
133 “The CVAP cube is a visual framework developed for understanding different types and applications of geovisualizations for climate adaptation” First time the cube is mentioned? Mention the source if the CVAP cube is reused from previous research, explain the meaning of the cube more clearly if it is you who develops it. How did you develop it from The Map Use Cube created by Maceachren and Kraak?
138 Motivate why these keywords were chosen.
140 “The included axes were derived from recurrent and salient themes such as amount of interactivity, realism, and level of risk associated with the type of impact resulting from the decision-making process” in which way, refer to scientific sources!
143 “The parameters were selected because they effectively and comprehensively describe the main differences in climatic visualization products in a clear and concise manner.” Not convincing, this needs to be developed and clarified in more detail.
151 “Each geovisualization was assigned a value of “Lower” or “Higher” amount of interactivity, risk, and realism respectively, relative to the other geovisualizations examined with CVA” How were the levels of “Lower” or “Higher” assigned? Which criteria were used?
166 I don't see the connection here? What is the risk-value related to? Geovisualizations tools or implications? Is risk the most appropriate concept for assessing the suitability of different geovisualization tools to be a support in decision making?
170 How do you rank interactivity on a scale (even if the scale is binary)? How to determine the position on the ranking scale?
178 “however, the ideas presented within the older model can still provide a foundation for building new concepts” In what way?
246 “Products were ranked relative to one another…” How?
455 Not clear how the themes are deployed in the cube? How have the lines been placed?
455 Geovisualization instead of visualization?
456 Why do you state that the approach is qualitative in the discussion? Should be clarified already in the method section.
472 “The CVAP framework was created with the intentions of formulating a comprehensive understanding about both the current state and the (relatively recent) history of geovisualization use for climate change adaptation.” I do not think you reach that goal with the paper. I think the history of geovisualization use for climate adaptation is a separate paper.
Author Response
Reviewers comments appear in bold for clarity.
Although the topic is relevant and that some parts, especially the results section, are good and comprehensive, there are several parts of the paper that need to be developed and revised:
1. The empirical approach is quite weak: a literature review is not enough to finalize a CVAP framework; I suggest that the framework should be tested also on practitioners, decision makers and researchers. The framework is for them, right?
Our project consists of two phases, the first is to produce the framework, and the second is to test the framework with practitioners. The second phase has unfortunately been delayed because of COVID-19. In the meantime, we feel that there is a sufficient contribution from this paper (phase 1) that, if published, will nicely set up phase 2. To address this concern of the reviewer and to clarify our intent, we have added the following text at the very end of the conclusion section:
"As part of an ongoing research project, the CVAP framework will be tested in focus groups and workshops in which practitioners and decision makers can evaluate the how well the cube is representative of their experiences with understanding climate change information."
2. A follow-up comment is that the presented framework (and its result with classification of different geovisualization tools) is too complex and non-transparent for practitioners and to some extent also for researchers. I believe that the framework's accessibility, content and transparency can be increased by testing it on different groups.
Please see the response to comment 1 above.
3. Accordingly, the development of the new framework based on previous and existing research, including used concepts and dimensions, needs to be presented in a more transparent way - there are too many questions and ambiguities in how different geovisualization tools/approaches are classified and placed within the presented model (even if the scale is binary). How are geovisualization tools assessed based on the dimensions (themes) of risk, realism and interactivity? There is an invisible basis for assessment grounded on a high degree of subjectivity (difficult to avoid in research - but the transparency can be improved). This is probably also related to the fact that the different dimensions are both qualitative and quantitative, concrete and abstract in their forms – the assessment process becomes unclear.
The authors have revised the text to make more clear the description of how cube was formulated. We have added this text in the last paragraph of the methods section with the following text:
"The geovisualizations that were included in CVAP are represented as spheres inside the dimensions of the cube. Each geovisualization product was analyzed, with special focus directed towards the amount of interactivity, risk, and realism associated with the geovisualization. They were then compared amongst themselves, and during this stage each visualization product was assigned a relative value of “Lower” or “Higher” based on the amount of interactivity, risk, and realism associated with the geovisualization. This was a relative comparison performed amongst only the geovisualizations included in CVAP."
4. There are ambiguities in the method chapter of how the review and development of the framework took place, see also specific comments below. One problem is that several methodological descriptions and clarifications appear in the result and discussion sections. I would suggest a restructuring of the paper with a clearer balance between method and result sections, e.g. some methodological explanations, such as the Map Use Cube etc., should be moved from result section to the method section. Another example that you state that the approach is qualitative in the discussion at the end of the paper? This should be stated and explored already in the method section.
The authors agree with this comment, and a similar sentiment was also shared by one of the other reviewers. After multiple attempts at addressing this comment, the authors decided that it was best to revise the methods section to be more clear about how the importance of the literature review in evaluating previous frameworks and in developing the CVAP framework. Please see section 2 and section 2.1 for these changes.
5. It is the result of the framework that is useful to users, e.g. the CVAP populated with geovisualization products to choose between (figure 3), or the CVAP itself – the process of classifying geovisualization tools and methods into themes? This is confusing?
The authors acknowledge that we did not make this clear. To address this comment, we have added text in both the abstract and in the introduction that reads:
"The overall goal of the framework is for it to be used as a tool for researchers and practitioners to use as a decision support system to discern an appropriate type of geovisualization product to implement within a specific use case or audience."
Specific
Row
74 No source for “sea ice monitoring”
Sea ice monitoring was removed.
108 Structured literature review, in what way?
This is clarified with the revised text in the methods
116 “several iterations” How many and in what way?
This is now clarified in the following text directly above Figure 1:
"CVAP was refined after several iterations of this process by creating the cube and evaluating its effectiveness at representing the various considerations that go into selecting and using geovisualization tools."
117 “included parameters” Which parameters? Comes later but needs to be clarified already here.
This sentence was deleted as it was confusing.
122 In what way is the review comprehensive?
The word "comprehensive" was replaced with "structured"
124 Topic = Criteria? What do you mean by criteria? In what way? Not clear!
Yes - you are correct. To address this confusion, we simply changed the word "criteria" to "topic".
124 Structure, in what way?
The revised text in the methods section provides a better description of how the literature review was structured.
126 “The criteria together capture the benefits and the inherent trade-offs…” How?
This sentence was changed to:
"These topics together are consistently identified in the academic literature when examining research focused on managing solutions for mitigating climate change risks."
133 “The CVAP cube is a visual framework developed for understanding different types and applications of geovisualizations for climate adaptation” First time the cube is mentioned? Mention the source if the CVAP cube is reused from previous research, explain the meaning of the cube more clearly if it is you who develops it. How did you develop it from The Map Use Cube created by Maceachren and Kraak?
The following sentence was added to the first paragraph in the methods section:
"This translated into a 3D cube that provides a means for organizing visualizations along axes in terms of types, application, and intended audience for the tools, similar to manner to MacEachren and Kraak (1997) and Bohman et al. (2015)."
138 Motivate why these keywords were chosen.
In response to this comment, the following sentence has been added to the end of section 2.1:
"These were selected based on their relevance to the aspect of visually communicating or presenting geospatial data in an appropriate manner or format to a user."
140 “The included axes were derived from recurrent and salient themes such as amount of interactivity, realism, and level of risk associated with the type of impact resulting from the decision-making process” in which way, refer to scientific sources!
Scientific sources have now been added and the paragraph has been revised to read:
"The included axes were derived from recurrent and salient themes such as amount of interactivity (Bohman et al., 2015; Lovett et al., 2015; Neset et al., 2016; Newell et al., 2017b), realism (Newell & Canessa, 2015; Newell et al., 2017b; Pettit, Raymond, Bryan, & Lewis, 2011), and level of risk (Grainger et al., 2016; Lovett et al., 2015; Morseletto, 2017; Neset et al., 2016; Newell et al., 2017b; Vandenbroucke et al., 2013) associated with the type of impact resulting from the decision-making process."
143 “The parameters were selected because they effectively and comprehensively describe the main differences in climatic visualization products in a clear and concise manner.” Not convincing, this needs to be developed and clarified in more detail.
This sentence is now deleted as we have included multiple citations to explain how we derived the parameters (please see response to previous comment).
151 “Each geovisualization was assigned a value of “Lower” or “Higher” amount of interactivity, risk, and realism respectively, relative to the other geovisualizations examined with CVA” How were the levels of “Lower” or “Higher” assigned? Which criteria were used?
Text has been added to this paragraph in response to the reviewer's comment. The text reads:
"The geovisualizations that were included in CVAP are represented as spheres inside the dimensions of the cube. Each geovisualization product was analyzed, with special focus directed towards the amount of interactivity, risk, and realism associated with the geovisualization. They were then compared amongst themselves, and during this stage each visualization product was assigned a relative value of “Lower” or “Higher” based on the amount of interactivity, risk, and realism associated with the geovisualization. This was a relative comparison performed amongst only the geovisualizations included in CVAP."
166 I don't see the connection here? What is the risk-value related to? Geovisualizations tools or implications? Is risk the most appropriate concept for assessing the suitability of different geovisualization tools to be a support in decision making?
The authors acknowledge that this is not clear. We have revised the text to describe that risk refers to the scenario for which the geovizualization in being applied. The text now reads:
"The results of the literature review of that contributed to the development of CVAP are summarized in Table 1, organized by the review of the important and shared characteristics between geovisualization, climate adaptation planning, and public and stakeholder engagement. For the purpose of this study, risk refers to the type of decision being made for which a geovizualization is employed. A high rating of risk refers to a scenario in which geovisualization tools are used to support decision-making associated with significant implications towards the well-being of communities (and human life), environmental systems, or public infrastructure. A lower risk rating implies a scenario that the tool is used for decisions which have a less significant impact on property loss or do not pose a great hazard to someone’s safety."
170 How do you rank interactivity on a scale (even if the scale is binary)? How to determine the position on the ranking scale?
It is relative interactivity. We have revised this sentence to explain this. The sentence now reads:
"This was a relative comparison performed amongst only the geovisualizations included in CVAP."
178 “however, the ideas presented within the older model can still provide a foundation for building new concepts” In what way?
This sentence was removed as it could not be supported.
246 “Products were ranked relative to one another…” How?
The following sentence has been added to the paragraph:
In ranking them relative to one another, decisions were made as to where the authors concluded, based on their prior expertise, the products aligned along the axes.
455 Not clear how the themes are deployed in the cube? How have the lines been placed?
This is now described in the methods section in response to a previous comment.
455 Geovisualization instead of visualization?
This has been change to geovisualization.
456 Why do you state that the approach is qualitative in the discussion? Should be clarified already in the method section.
This is now explained in the methods section with the following sentence:
"Second, the literature review was then used to inform the qualitative development of a framework for classifying geovisualizations, referred to here as the Climate Visualizations for Adaptation Products (CVAP) framework."
472 “The CVAP framework was created with the intentions of formulating a comprehensive understanding about both the current state and the (relatively recent) history of geovisualization use for climate change adaptation.” I do not think you reach that goal with the paper. I think the history of geovisualization use for climate adaptation is a separate paper.
The authors agreed and have thus removed the sentence.
Reviewer 3 Report
Climate change is a hot topic of research and essential to human beings. Although tremendous research results have been obtained, these results have little impact on stakeholders. The reason is a lack of tools severing as a bridge between research results and stakeholders. This study provided such a tool and, therefore, is of particular use.
The type of this manuscript is shown as a “Structured Literature Review.” This type is not appropriate and suitable. I suggest changing this term to “technical note,” which is an acceptable type to IJGI.
The organization and writing of this manuscript can be improved substantially. Section 3 should be a significant contribution of this study, but it seems to be a literature review. This is terrible in terms of scientific writing. This manuscript can be accepted after an improvement in writing.
Author Response
The authors thank this reviewer for their comments on our submission. We agree that the presentation of the paper needed improvement, especially regarding the presentation of the literature review. Here is how we addressed the reviewers' comments:
Comment: The type of this manuscript is shown as a “Structured Literature Review.” This type is not appropriate and suitable. I suggest changing this term to “technical note,” which is an acceptable type to IJGI.
Response: The authors did conduct a formal literature review, but unfortunately it was not emphasized accurately in the manuscript. To address this shortcoming, we have revised the methods section (specifically section 2.1) to clearly describe the importance of the literature review, and how it was conducted (e.g. date it was conducted, keywords used). We have also changed all mentions of "review" to "literature review", as we noticed that this may have caused some confusion.
Comment: The organization and writing of this manuscript can be improved substantially. Section 3 should be a significant contribution of this study, but it seems to be a literature review. This is terrible in terms of scientific writing. This manuscript can be accepted after an improvement in writing.
Response: As mentioned above, the authors are in agreement that improvement to the manuscript was needed. As such, the authors revised the entire manuscript to correct for any grammatical and structural problems, and are now confident that the paper reads much better.
Reviewer 4 Report
The paper presents a framework for geovisualization tools in the domain of climate change adaption called the Climate Visualizations for Adaption Priducts (CVAP), it focuses on visualisation features, audiences and intended outcome or purpose of visualisation medium.
The researchers start with a literature review to build their theoretical framework and dimensions. For reproducability it would be good if the authers would cite their full list of keywords used for the search. From the current text it is difficult to understand what the literature review brought to light and how it ended up in the new framework. What papers were the result of the literature review? Only the ones mentioned in Tabel 1, then this does not feel like a comprehensive review. Are the resulting papers the ones mentioned in 3.2? It would be good if the results of the literature review were presented in a structured way. Other papers that present literature reviews provide tables with summaries, keyword analysis, etc.
The core of CVAP is a cube based space for classifying geovisualizations. The cubes purpose is to allow researchers, decision makers, etc. to choose the right visualisation tool for a certain task and audience. While the authors explain the final cube in quite some detail, the cube does not live up to its requirements. The target audiences are not included in the cube and are in general left aside for most of the discussion. In several regards it feels, like the original map use cube, is in this regard even better equiped to provide these insights.
In an evaluation of the cube, the authors use a selection of papers and map them onto the cube. The authors take a very simplified mapping approach by only placing the projects in the extremes of each dimension. Looking at other cube-like frameworks, authors are using the whole range of dimensions not just the extremes. The authors then overlay a 2-dimensional column-layout on top of the 3-dimensional cube to map themes of the purpose of each system. From a simple model abstraction point of view i find it highly doubtable that in a realistic setting mixing a 3-d space with a 2-d space will work out, especially when using the full range of dimensions. I think the purpose part of the framework (2d themes) needs some further work, as this is kind of one of the most import aspects of the framework, as well as including target audiences in the framework. Maybe the cube is not the right form for this?
I still belive that the researchers but thorough research into this and the idea would present a good contribution for the climate adaption community. If the literature review is explained in more detail, especially how it led up to the model and the model space is refined, particularly the inclusion of target audiences, this paper should be accepted.
Author Response
Reviewer's comments are in bold.
The paper presents a framework for geovisualization tools in the domain of climate change adaption called the Climate Visualizations for Adaption Priducts (CVAP), it focuses on visualisation features, audiences and intended outcome or purpose of visualisation medium.
The researchers start with a literature review to build their theoretical framework and dimensions. For reproducability it would be good if the authers would cite their full list of keywords used for the search. From the current text it is difficult to understand what the literature review brought to light and how it ended up in the new framework. What papers were the result of the literature review? Only the ones mentioned in Tabel 1, then this does not feel like a comprehensive review. Are the resulting papers the ones mentioned in 3.2? It would be good if the results of the literature review were presented in a structured way. Other papers that present literature reviews provide tables with summaries, keyword analysis, etc.
The authors have revised the methods section describing the literature review so that it is more clear how the review took place. We did not however include a table of all studies that were returned from the keyword search as this resulted in over 100 papers. We therefore only included those that were utilized in this study. The new text in the methods reads as follows:
"A structured literature review was performed between May to August, 2019 of frameworks pertaining to the topics of geovisualization, climate adaptation planning, and/or public and stakeholder engagement. These three topics were used to structure the literature review, as they all relate to the necessity of developing effective planning tools for climate adaptation. These topics together are consistently identified in the academic literature when examining research focused on managing solutions for mitigating climate change risks. This is a challenge as stakeholders, researchers, and members of the general public often have different perceptions of the most appropriate measures to implement in order to prepare for the potential impacts of climate change. The literature review was framed by searching multiple university search engines (e.g. Web of Science) and Google using the following keywords: geovisualization, visualization, climate framework, climate change, climate adaptation, public engagement, landscape planning, and risk mitigation. geovisualization, visualization, GIS, climate adaptation, landscape adaptation, landscape visualization, climate change, spatial decision support system, and risk mitigation. These were selected based on their relevance to the aspect of visually communicating or presenting geospatial data in an appropriate manner or format to a user."
The core of CVAP is a cube based space for classifying geovisualizations. The cubes purpose is to allow researchers, decision makers, etc. to choose the right visualisation tool for a certain task and audience. While the authors explain the final cube in quite some detail, the cube does not live up to its requirements. The target audiences are not included in the cube and are in general left aside for most of the discussion. In several regards it feels, like the original map use cube, is in this regard even better equiped to provide these insights.
We agree that this is a current challenge of the paper, and it is something that we intended to address with a second stage of our research. Unfortunately, our focus groups and workshops were cancelled due to COVID-19; we were intending to have several practitioners utilize the cube to determine where they felt they were located along the axes. Instead, we have added text at the end of the paper to describe our future intentions. This text reads:
"As part of an ongoing research project, the CVAP framework will be subsequently tested in focus groups and workshops in which practitioners and decision makers can evaluate the effectiveness of the cube in representing the tradeoffs with various products."
In an evaluation of the cube, the authors use a selection of papers and map them onto the cube. The authors take a very simplified mapping approach by only placing the projects in the extremes of each dimension. Looking at other cube-like frameworks, authors are using the whole range of dimensions not just the extremes. The authors then overlay a 2-dimensional column-layout on top of the 3-dimensional cube to map themes of the purpose of each system. From a simple model abstraction point of view i find it highly doubtable that in a realistic setting mixing a 3-d space with a 2-d space will work out, especially when using the full range of dimensions. I think the purpose part of the framework (2d themes) needs some further work, as this is kind of one of the most import aspects of the framework, as well as including target audiences in the framework. Maybe the cube is not the right form for this?
As with the previous comment, our intention was to have stakeholders help us determine the relative positioning of the geovisualization products in more of a continuum along the various axes, but instead we resorted to having to cluster them in the corners of the cube. This problem is address at the end of the paper where we describe the need for stakeholders to assist with this process. This added text reads:
"Focus groups and workshops can provide such stakeholders with the opportunity to see where they feel specific geovisualization tools should be positioned with the cube, which would improve our current presentation in which we have all products clustered into the cube’s corners. Furthermore, stakeholders could also determine where they feel their level of comfort with using geovisualization tools resides along the cube’s axes, thus providing new knowledge on how well various tools are aligned with stakeholder experiences and expectations."
I still belive that the researchers but thorough research into this and the idea would present a good contribution for the climate adaption community. If the literature review is explained in more detail, especially how it led up to the model and the model space is refined, particularly the inclusion of target audiences, this paper should be accepted.
Thank you. We feel that we have address these issues.
