Mind the Gap: Towards a Typology of Climate Service Usability Gaps

Abstract

Literature on climate services presents a large diversity of different services and uses. Many climate services have ‘usability gaps’: the information provided, or the way it is visualized, may be unsuitable for end users to inform decision-making processes in relation to adaptation against climate change impacts or for the development of policies to this end. The aim of this article is to contribute to more informed and efficient decision-making processes in climate adaptation by developing a typology of usability gaps for climate services. To do so, we first present and demonstrate a so-called ‘climate information design’ (CID) template with which to study and potentially improve the visual communicative qualities of climate services. Then, two climates services are selected for a further, qualitative explorative case study of two cases in the north and south of the Netherlands. A combination of focus group sessions and semi-structured interviews are used to collect data from Dutch governmental stakeholders as well as private stakeholders and NGOs. This data is then coded to discover what usability gaps are present. We then present twelve different types of usability gaps that were encountered as a typology. This typology could be used to improve and redesign climate services.

1. Introduction

Europe is increasingly confronted with the effects of climate change. In some regions climate change can lead to drought, while in other areas heat stress or increasing precipitation can become a major problem [1,2]. In addition to climate change mitigation, there is a need for climate adaptation to guarantee that contemporary and future societies can endure these challenges. This means that decision-makers and other stakeholders need to understand their responsibilities and obligations concerning climate change impacts and adaptation [3]. In adaptation processes, so-called climate services are used to communicate climate data to adaptation professionals and other stakeholders to facilitate well-informed climate adaptive decision-making.

Climate services (CS) offer information about climate change, its potential impacts, and possible adaptation measures [3,4,5,6]. The exact definition of CS is still subject to debate. As a point of departure, we look to the European Commission’s CS initiative, where it is recognized that the term ‘climate services’ has a broad meaning [7]. Based on several definitions in a variety of literature [3,4,5,6,7,8,9,10,11] we posit that CS: (1) provide useful information and knowledge related to climate change or climate change impacts; (2) are used for (climate) informed decision-making by decision-makers, including non-governmental/private individuals and organizations; (3) act as guidance and counselling in their use; (4) entail the transformation of climate data into customized products (5) encompass a variety of different tools, such as maps, projections, scenarios, and assessments; (6) support climate change adaptation, mitigation, and disaster risk management; (7) are produced as a result of a specific demand (user-driven) [3,12]. The goal of CS is to support decision-making and can be directed towards various end users: politicians, managers, private enterprises, inhabitants, etc. As such, CS also play a role in education and awareness-raising [13]. In short, we define CS as information services that provide information about climate change, climate impacts, climate adaptation, and mitigation measures for decision-makers and other stakeholders to create understanding, raise awareness, and make decisions.

Hamaker et al. distinguish between ‘climate data services’, ‘adaptation services’, ‘mitigation services’, and ‘disaster risk management’ (Figure 1). According to Hamaker et al., CS are based on observational data (satellite data, measurements). This data is used as input for climate data services (climate models, forecasting, maps) to assess useful information, often related to climate change impacts [14]. Plans for adaptation, mitigation, and disaster risk reduction are developed based on these insights. These plans can then lead to the actual action: the implementation of the measures.

According to this conceptual model (Figure 1), a CS can be interpreted as the sum of climate information (often modeling output), the actual service, and the response (a plan and climate adaptive action). Currently, we observe that there is not a wide variety of CS available that take into account this conception as a whole. A review of the CS that is available via the European climate adaptation platform indicates that the majority of CS provide information with regard to the planning phase, while only a marginal number of tools focus on the acting phase [15]. The current use of climate data and scientific knowledge within CS is focused on providing access to climate information, rather than communicating knowledge with a particular goal and audience in mind. The problem facing end-users of CS seems not a lack of knowledge per se, but rather knowing which knowledge to use and when, as well as knowing how to deal with risks and uncertainties related to different kinds of climate knowledge.

These observations imply the existence of ‘usability-gaps’ [16] at various levels (e.g., strategic and operational) and in different phases of adaptation processes. For example, policy-makers could have access to a CS that shows various climate risks in their area, but lack the information about stakeholder responsibility needed to invite the appropriate partners for a risk dialogue. This gap is also addressed by Weaver et al., who argue that usability gaps result from a mismatch between a CS and its audience [17]. The delivered CS, which are often projected on a large spatial scale and require technical expertise to understand, do often not contain the required information for the local end users to—in some planning systems—develop policies, allocate budgets, or implement measures [16,17,18]. As a result, planners and policy makers perceive the information provided by these CS as being relatively uncertain, too geographically and temporally distant, and to be addressed by someone else in the decision-making hierarchy rather than at their policy level. This perception is reinforced by the CS’ lack of information about the precise impacts of climate effects at the local scale [3,19]. Weaver et al. point out that these problems with CS’ usability gaps can lead to a delay in decision-making by end users, and consequently to a delay of taking climate adaptation and mitigation measures [17].

In our research, we put forward the view that the development process of CS can be less linear by better involving end users in the design of the CS. However, a clear classification and understanding of the variety of possible usability gaps is needed to both target specific communicational deficiencies in existing CS and to develop new, innovative CS. The main goal of this article is therefore to contribute to better informed and more efficient decision-making processes in climate adaptation by developing a typology of usability gaps for CS. This typology is based on a climate information design (CID) template, which provides an analytical framework for studying the communicational qualities of CS.

The goal of this paper translates into the following main research question: What usability gaps in current climate services should be addressed to enable well-informed climate adaptive decision-making? To answer this question, we first present the aforementioned CID template with which to study and potentially improve the communicative qualities of CS. Second, we operationalize this template into an analytical framework with which we evaluate two climate services by analyzing the communication needs of stakeholders in two cases in the Netherlands using an explorative complementary case study analysis. We then illustrate the different types of usability gaps that we encountered, after which we arrive at a topology of usability gaps. In the discussion, we recommend a living lab-type of approach for experimenting with different climate information designs to further refine the CID template.

2. A Template of Climate Information Design

2.1. Climate Change Communication

Within planning practice, climate change adaptation processes of urban areas proceed through different phases, e.g., from the indication of a potential problem all the way to the implementation of interventions, and through various cycles Specifically, Moser and Ekstrom distinguish the understanding, planning, and management phases [20]. The policy model used by Moser and Ekstrom is based on the more commonly used policy-making cycle model [21,22], which provides a framework for the steps through which stakeholders develop and implement policies. Generally, the first steps consist of information gathering and agenda setting, followed by policy making. The policy will be implemented and monitored on its ability to solve the original problem. If needed, this cycle repeats with a new round of information gathering for additional or replacing policies.

The communication needs of stakeholders can change during the different phases of such circular adaptation processes. For example, during the initial understanding phase the process is focused on establishing a sense of the scale at which climate change could have impacts in an area [23], or on creating awareness about this impact [24]. In the planning phase, these communication needs shift towards informing decision-makers in regard to where potential measures should be implemented and how well these planned measures would perform [25]. In the management phase, climate information can be used as a baseline for evaluating the final implemented measures [26]. The evaluation outcomes may serve as validation for engaging in a potential new round of the planning process. In the Netherlands, communication needs change during the different phases of adaptation as well. A recent study concluded that climate services play different roles within the different phases of the decision-making process of governmental actors, as distinguished by Moser and Ekstrom [27].

Climate services function as communication tools within the context of planning processes for climate change adaptation. Considering the importance of the communicative qualities of CS, we first need a more clear understanding of climate change communication to develop a useful climate information design template. Moser states a number of challenges related to undertaking climate change communication [28]. For example, the idea that the object of communication consists of invisible causes with distant impacts that play out in a natural environment of which modern man has largely been insulated [29]. This makes communicating potential benefits of taking action against climate change difficult, since positive effects might only be perceivable on the long-term, if ever. More fundamental issues lie at the heart of the public’s awareness of both climate change processes and the anthropogenic influence on these processes [30]. The cause for this lack of awareness is understandably rooted in the high levels of complexity and uncertainty that surround climate change. Appropriate signals for the need to take action are therefore lacking and are easily trumped by the rhetoric employed by climate-sceptics and powerful institutions that benefit from maintaining the status quo [28].

To meet these challenges, key elements of the communication process should be designed in a coherent, often tailored way. A first element, as stated by Moser, consists of the purpose and scope of communication, whereby she makes a distinction between: (1) informing and educating individuals about climate change, (2) achieving some type and level of social engagement and action, and (3) bringing about changes in social norms and cultural values [28]. This distinction connects to the understanding, planning, and management phases of the policy cycle of climate adaptation processes, as proposed by Moser and Ekstrom.

The second element refers to the type of audience that needs to be addressed since ‘different audiences require distinct frames, goals, messages, and messengers’ [28] (p. 39). Frames are mental constructs that enable audiences to perceive and value climate change problems and solutions in their own way, highlighting specific aspects and downplaying others [31]. Means of (visual) communication have the discursive power to frame their content in a certain way, and thus re-frame the perception of that content by its audience [32]. Attuning the purpose of climate change communication to specific audiences requires the communicator to formulate their message—the third element—with a particular frame in mind. The appropriateness of the message, according to Moser, is therefore heavily dependent on context and, consequentially, is hard to define. However, to increase the chances of success, one should always consider the internal consistency of the message, know the audience’s mental frames or interpretive habits [33], consider the emotional impact of a message’s accompanying imagery, and ensure a proper engagement by keeping the audience’s attention. Furthermore, the authority of the messenger that delivers the message is a determinant for the extent to which an audience ‘trusts’ the provided information. Such authority relates to that of individuals, but also of institutions and organizations.

Finally, the modes and channels of communication, e.g., written, verbal and non-verbal ways of communication, determine the possibilities for interacting with the audience. The effectiveness with which an audience is reached is influenced by the extent to which the modes, channels, and the social settings wherein they are used facilitate one-way or two-way communication. Climate services, with specific modes and channels of communication, often function independent of their original messengers and context of creation. This means that CS contain rather complex messages to be interpreted at a distance by a potentially segmented audience. Supposedly, the self-explanatory and objective qualities of CS are derived from the visual modes they employ, e.g., maps, graphs, or infographics. However, the use of these visual modes and channels adds further levels of complexity to climate change communication. Images are prone to unchecked distribution when they are not accompanied by their original descriptions to provide a frame for interpretation [32]. Instead, audiences are left to interpret climate services based on their own mental frameworks. The communicative success of a CS therefore depends on the way a message and its meaning is anchored in its mode and channel of communication. As ‘immutable mobiles’ [34] that contain detailed ‘inscriptions’, climate services should possess the agency for consistently mobilizing a variety of stakeholders towards taking climate adaptive actions that are independent of the spatial and temporal contexts within which the CS are used.

To ensure a proper ‘inscription’ the visual communication process of CS should be analyzed and facilitated with great care. The field of information design can aid in structuring this communication process, and potentially serve as a bridge between science and policy. The relevance of this notion of information design for visual communication becomes clear when we consider the idea that CS cater to specific heterogenic audiences [35]. The form of (raw) climate research data is often inadequate for the communication of findings to policy-makers. In other words: “to make data valuable, it has to be structured, transformed, and presented in a meaningful way” [3] (p. 5). This may be, for example, in the form of a report or a map, as well as many other ways of communicating information depending entirely on the message and the audience of this information. For example, an extensive technical report may be of value to someone working in a governmental department who has an extensive background in his or her field. To an untrained citizen, that same report could also appear as a report full of difficult, technical language. Instead, showing photographs that depict the impact of urban flooding or property damage might communicate the same information much more effectively.

In short, the study of the communicational qualities and potential usability gaps of CS can benefit from the use of an analytical framework that is based on an information design for climate change communication: a ‘climate information design’ (CID) template (Figure 2). This framework is based on a critical semiotic perspective on visual communication in participatory climate adaptation processes, assumes that climate service visualizations have an implicit or explicit goal, and assumes that the quality of visualizations as communication tools depends on the extent to which that goal is achieved [36]. Climate data visualizations, such as CS, that do not lead to the appropriate climate adaptive action taken by its targeted audience can therefore be considered unsuccessful. Such miscommunication can occur due to an inconsistency in any of the visual communication components: (1) the desired interpretation (and resulting action) by its intended audience, (2) the framing of a message for a specific audience with a specific purpose in mind, (3) the appropriate information, (4) the readability of the choice of visual expression and appropriate medium of presentation.

2.2. Analytical Framework

According to Raaphorst et al., three additional communicative qualities of the audience, purpose, information, and format are distinguished within the CID template: validity, readability, and interactivity [36]. Operationalized for the study of CS, validity addresses whether a CS reaches and affects the appropriate audience, the purpose with which a CS is created fits the policy cycle, the type of climate information presented is correct, and its visual format is coherent with the type of audience, purpose, and information. Readability, in this regard, entails whether the CS is attuned to the visual language of its intended audience, the purpose of the CS is transparent, the information depicted is clearly understandable, and whether it is clear how the visual format should be read. Finally, interactivity relates to adhering to the viewing literacy of the audience, the possibilities for re-purposing the CS and adding or adapting the information that is presented, and modifying the visual mode in terms of scale or color.

Usability, conceptualized according to the CID template (Figure 2), entails the extent to which a climate service enables stakeholders of climate adaptation processes to access, understand, and use climate data, which can be expressed in relative levels of validity, readability and interactivity. As such, the usability gap, as mentioned by reference [16] can be visualized through this template by comparing the needs of the end user with the purpose, information, and format presented in the CS. In theory, the notions of validity, readability, and interactivity each imply a usability gap at each of the four levels in the CID template (stakeholder, purpose, information, visual format). Subsequently, twelve potential usability gaps are defined in a preliminary analytical framework (

Table 1. Analytical framework that describes twelve usability gaps for climate services (CS).

	Validity	Readability	Interactivity
Stakeholder	Is the desired action the responsibility of the targeted audience?	Does the visual language, and its possible connotations, match the interpretive frames of the audience?	Is the visual literacy required for interpreting the CS suitable for the target audience?
Purpose	Is the purpose (understand, feel, act) suitable for the phase in the policy cycle?	Is the purpose of the CS clear? (otherwise people act before understanding)	Can the CS be repurposed by the user?
Information	Is the information shown correct/trustworthy?	Is it clear what information is presented in a CS?	Can the information be modified?
Visual format	Does the visual mode enable an accurate representation of the climate phenomenon?	Is the type of mode, and its way of reading, clear? (a story map requires a different viewing than a standard GIS map)	Can aspects of the mode (zoom level, color scheme, etc.) be modified?

). This analytical framework forms the basis of the coding scheme of the case study analysis.

3. Materials and Methods

The potential usability gaps of climate services are studied using an explorative complementary case study approach [37]. The research scope of this approach consists of two parts: (1) a qualitative study of the communication needs (purpose, information, and visual format) of stakeholders in climate adaptation processes for two case studies in the Netherlands; (2) the demonstration of the CID template by illustrating the twelve usability gaps using a content analysis.

3.1. Case Study Policy Context

The case study discusses two Dutch cases. A brief introduction to the policy environment is provided in order to better understand the position of the cases within the wider range of policies that are driving climate adaptation in the country. At the moment, the annual Delta Program (Dutch: Deltaprogramma), and especially the section ‘Delta Plan Spatial Adaptation’ (Dutch: Deltaplan Ruimtelijke Adaptatie; DPRA), is the leading policy document in the Netherlands in regard to climate adaptation. This document dictates the necessary processes that regional and local governmental stakeholders (municipalities, provinces and water boards) go through to become climate resilient [36]. The DPRA section outlines the necessary steps to reach this goal as well, as the moments in time when these steps should have been completed. Two notable goals can be distinguished within this document: (1) by 2020, at the latest, all regional and local governmental stakeholders need to have had climate adaptation incorporated within their implementation agenda, which may contain both policies and spatial interventions; and (2) in 2050, at the latest, spatial measures and policies must be implemented.

By 2020, three steps need to have been taken by all regional and local governmental stakeholders: (1) in 2019, at the latest, all regional and local governmental stakeholders should have conducted a ‘stress-test’ that provides insight in the vulnerable locations in their region for climate change related impacts; (2) all regional and local governmental stakeholders should have conducted risk dialogues with regional and local societal partners (e.g., citizens, agricultural, businesses, nature organizations) that are at risk of experiencing these impacts. During this dialogue, it is established which risks are acceptable and who will take the responsibility for taking measures to reduce the risks and vulnerabilities that are deemed unacceptable; (3) based on these risk dialogues, and outcomes of the stress-tests, a strategy and implementation agenda are established on what (spatial) measures and policies should be implemented to achieve the aforementioned 2050 goal. In turn, this strategy is formed into an implementation agenda comprised of a budget and a schedule of planned (spatial) measures and policies that need to be executed by regional and local governmental stakeholders.

Additionally, governmental stakeholders should capitalize on linkage opportunities with other ongoing and/or planned spatial developments to speed up the implementation of measures, to reduce nuisance for local inhabitants during the implementation and to yield financial benefits. Additionally, the Ministry of Infrastructure and Water Management has allocated funding for the promotion and facilitation of the climate adaptation process (e.g., through exemplary projects, financial incentives, and knowledge sharing initiatives). Finally, contingency plans need to address the impacts of unforeseen weather conditions.

Also in 2016, the National Climate Adaptation Strategy (Dutch: Nationale Klimaatadaptatie Strategie; NAS) was introduced in the Netherlands [38]. This strategy is the Dutch answer to the EU call to its member-states to develop national climate adaptation strategies [39]. Like the Delta Program, this strategy covers the Dutch ambition to becoming climate resilient. Within this strategy, the focus is on improving the connection and collaboration between stakeholders. Furthermore, it builds on the Delta Program and attempts to accelerate climate adaptation where possible [39]. On paper, the NAS is complementary to the Delta Program [38]. However, in practice the Delta Program and the DPRA are leading in the Netherlands. The DPRA, with its clearly defined adaptation trajectory, is supported by a larger budget and experiences more exposure to climate adaptation professionals compared to the NAS.

This tangle of climate adaptation policies in the Netherlands is difficult to navigate for governmental and non-governmental stakeholders who, often for the first time, need to implement climate adaptive measures at these levels of spatial and temporal scale. Climate services are pushed to the fore as facilitating media during project meetings, workshops, and public outreach events, but it is unclear to what extent these services meet user demands. The following section describes the two Dutch case studies: Northeast Brabant and the FLUVIUS region, as well as the climate services that are primarily used within these cases; a regional adapted version of the NAS-Adaptation Tool and the Climate Stresstest, respectively.

3.2. Case Study Locations and Climate Services

3.2.1. Northeast of the Province of North Brabant, The Netherlands

Climate adaptation in the Netherlands is primarily associated with rising sea levels and flood hazards [40,41]. For example, a recent case involves the summer of 2016, when heavy rainfall and hailstorms caused over more than 500 million euros in damage to agricultural activities in the region [42]. However, at the regional scale water scarcity and drought can also have significant impacts, especially concerning freshwater supply for agriculture and nature. Extreme rainfall and flooding in certain periods are expected to go hand in hand with drought in other periods, which can affect both rural and urban areas. Finally, also heat stress in urban areas due to extreme temperatures is gaining recognition from regional and local stakeholders [43].

To face the challenges of climate change, the province of North Brabant is cooperating with regional governmental stakeholders in a joint adaptation agenda for the work region of Northeast Brabant (Figure 3). This work region consists of 550.000 inhabitants, twelve municipalities, and two water boards. In support of this cooperation, the province is currently building a climate knowledge portal. This portal will be the main toolkit for providing instruments, knowledge, and best practices on climate change adaptation for all parties involved in the region. Important questions for the province relate to keeping climate adaptation on the long-term (political) agenda, connecting short-term actions to long-term challenges and investments, learning from efforts/mistakes when it comes to climate change, going beyond the pilot, making sure upscaling and further implementation takes place, and communicating about climate change and adaption with the local and regional stakeholders.

The main climate service promoted during the adaptation process in Northeast Brabant is the NAS Adaptation Tool. This tool distinguishes the four main trends in climate change: heat, precipitation, drought, and sea level rise. The tool ‘maps’ the primary direct and indirect consequences of these trends in a diagrammatic way, specified to specific spatial sectors, e.g., nature, agriculture, health, energy, etc. The tool is presented as a visual aid for clarifying risk and opportunities, and for seeking out relevant knowledge partners for cooperation.

3.2.2. FLUVIUS Work Region, The Netherlands

The Fluvius work region is located in the IJssel-Vechtdelta (Figure 3), in the North-Eastern part of The Netherlands where the rivers IJssel and Vecht flow into the IJssel lake. This region consists of six municipalities with in total 210.000 inhabitants. The abundant presence of water has driven the social and economic development in this region. The regional and local governments collaborate for a water robust and climate proof region with the aim to, by 2100, provide a water safe area where life, work, recreation and entrepreneurship can prosper. These collaborating governments have launched a program called ‘Living with Water in the IJssel-Vechtdelta’ in 2014. One of the aims of the program is to improve the awareness of the communities within the region of the effects of climate change. By communicating the risks of climate change during participatory development sessions of coping strategies, communities in different sectors (e.g., nature, agriculture, etc.) may learn to become more resilient to floods or heat. Some specific issues in the project area relate to reaching out and connecting to different groups within communities (e.g., elderly people, youngsters, or immigrants) and, secondly, to organizing dialogues with these groups.

The main climate service that is being used for conducting the stress test within this case study is the Climate Effect Atlas. This atlas shows the potential vulnerabilities within particular areas of the FLUVIUS region in relation to the aforementioned main trends in climate change. Through spatial modeling, heat, drought, water nuisance, and flood scenarios are visualized using interactive maps, accompanied by story map components, which offer additional background information in layman’s terms. The main purpose of the stress tests is to identify and prioritize vulnerable and vital spatial functions in the selection of areas for climate adaptive action.

3.3. Data Collection: Documentation, Semi-structured Interviews, and Focus Group Meetings

An inventory of the communication needs of the stakeholders in both case studies was made using semi-structured interviews and focus group meetings. The CID template was used to structure the interview protocol as key questions that were addressed during the interviews and focus groups relate the needs of stakeholders to the available climate services. For example, questions that relate to the stakeholder category were: With whom does your organization need to cooperate to reach climate adaptation goals? Are the required stakeholders involved in the adaptation process? What are your expectations of those stakeholders in terms of responsibilities? Other questions relate to the purpose category: What is your role within the climate adaptation process? Are you able to fulfill that role based on the climate information that is currently available? The information category was addressed by asking questions like: What climate related information have you already gathered? What information would you like to have concerning climate change impacts and adaptation, is there something missing? For the visual format category, questions were: How is the information that you currently use visualized: is it a written document, a map, a model, something else? If there is information missing, what visual format would you like this information to have?

In total, 11 interviews were conducted with local governmental and private stakeholders, as well as NGO’s (see

Table 2. Organization that each interview respondent belonged to.

FLUVIUS Region	Date of Interview	Northeast Brabant	Date of Interview
Nature interest organization; NGO	2018-10-30	Agricultural interest organization; NGO	2019-10-14
Agricultural interest organization; NGO	2018-11-29	Local health organization; public	2019-10-14
Drinking water company; semi-public	2018-12-04	Nature interest organization; NGO	2019-10-16
Safety region; public	2018-12-13	Housing corporation: private	2019-11-05
Local health organization; public	2019-01-21	Safety region Northeast Brabant; public	2019-11-06
Housing corporation; private	2019-02-05

and

Table 3. Organization that each focus group respondent belonged to.

FLUVIUS Region	Date of Focus Group Meeting 1	Northeast Brabant	Date of Focus Group Meeting 1
Municipality Hoogeveen/De Wolden	2018-06-26	Municipality of Meijerijstad	2018-11-01
Municipality Midden-Drenthe		Municipality of Boekel
Municipality Westerveld		Municipality of Uden
Municipality Steenwijkerland		Municipality of Oss
Water board Drents Overijsselse Delta		Province of North Brabant
Province of Drenthe		Water board Aa en Maas
Province of Overijssel		Royal Haskoning DHV

¹ While there are several stakeholders mentioned for the focus group sessions, it could be that there were employees from other governmental stakeholders present as well. Not all participants’ affiliations were documented during the session, so we had to rely on our own field notes.

). Additionally, as mentioned, a focus group session with employees of at least eight local municipalities, three provinces, and two water boards was held in both regions. The respondents and focus group attendants were found using a purposive sampling strategy, looking for people that, to our judgement, were best positioned to provide us with the necessary data [44]

The data consists of summarized transcripts of 11 interviews. Focus group document materials such as researcher’s summaries of the work sessions and field notes were used for contextualization. The transcripts were anonymized using a random ID-coding system so that the respondents’ affiliation to a particular stakeholder type (e.g., municipality, water board, health organization) could still be reproduced. Field notes consisted of photographs taken from customized CID templates and flip-overs of brainstorming sessions, as well as in-situ observations made by the researchers. The climate services were reviewed online. The background documentation that accompanies the services was collected from the internet [45,46,47].

3.4. Data Analysis

The CID template of Table 1 provides the analytical framework and coding scheme for the evaluation of the two climate services, i.e., NAS Adaptation Tool Northeast Brabant and the Climate Effect Atlas FLUVIUS, as well as for the content analysis of the interview transcripts and focus group summaries. This data was coded and analyzed using the data analysis software program Atlas.TI.

First, an analytical reading was done of the two climate services using the CID template of Figure 2. This means that for both climate services, nominal variables [48] such as the intended audiences, the information provided, the intended purpose, and the types of visual format are described based on using the actual tools, but also on the background documentation that accompanies the services. Second, summarized transcripts of the interviews and summaries of the focus groups were coded using the usability gap analytical framework of Table 1. The twelve types of usability gaps were identified by reviewing and assessing the respondents’ statements. The stated cooperation with stakeholders, need for information, required purpose, and visual preferences were compared to the qualities found in the currently used climate services. In this comparison, the variables were coded using the values of “0” (no overlap found) and “1” (overlap found). The usability gap was noted for each time a variable was evaluated with the value of “0”. Finally, the amount of times that the twelve usability gaps occurred were counted and discussed in relation to each other and to the policy context to see whether internal hierarchies of relative importance could be identified.

4. Results

4.1. ‘Reading’ the Climate Services

The following section depicts the two climate services, i.e., NAS Adaptation Tool Northeast Brabant (Figure 4 and

Table 4. Climate Information Design qualities of the NAS Adaptation Tool Northeast Brabant.

	Drought	Precipitation	Heat	Sea Level Rise
Stakeholder	Local government; Regional government.	Local government; Regional government.	Local government; Regional government.	Local government; Regional government.
Purpose	Understand	Understand	Understand	Understand
Information	Effect and impact of thematic trend; specified to specific sectors; nature of consequences; temporal scale; Scale analysis: which problems occur at the local or regional scale? - Lower humidity. - Soil drought. - Quality of surface water. - Increase in frequency of extreme drought. - Decrease in river discharge capacity.	Effect and impact of thematic trend; specified to specific sectors; nature of consequences; temporal scale; Scale analysis: which problems occur at the local or regional scale? - Change in surface water quality due to runoff. - Increase runoff from higher sandy soils. - Increase river discharge, more fluctuation. - Increase of extreme weather (wind gusts, lightning strikes, hail). - Increase extreme peak discharge.	Effect and impact of thematic trend; specified to specific sectors; nature of consequences; temporal scale; Scale analysis: which problems occur at the local or regional scale? - Early start of and longer lasting growing season. - Geographic shift of climatic zones. - Increasing periods of extreme high temperatures. - Mild winters. - Warm summers. - High temperature of surface waters, bigger quality issues.	Effect and impact of thematic trend; specified to specific sectors; nature of consequences; temporal scale; Scale analysis: which problems occur at the local or regional scale? - Higher water levels.
Visual format	Infographic	Infographic	Infographic	Infographic

) and the Climate Effect Atlas FLUVIUS (Figure 5 and

Table 5. Climate Information Design qualities of the Climate Effect Atlas FLUVIUS for the most important climate change impacts.

	Drought	Precipitation	Heat
Stakeholder	Local government, Regional government, agricultural businesses, citizens.	Local government, Regional government, agricultural businesses, citizens.	Local government, Regional government, citizens, public health organization.
Purpose	Understand; Act	Understand; Act	Understand; Act
Information	Effect and impact of thematic trend, information about:—Max total potential rain shortage in millimeters, now and in 2050. - Explanations for the increase of the frequency and intensity of extreme dry weather. - Drought sensitivity of different land use types in relation to the increasing occurrence of low ground water levels in 2050. - Examples of forest fires in North Brabant as a result of drought, coupled with a fire sensitivity index of natural areas. - Water seepage and filtration.	Effect and impact of thematic trend, information about: - Amount of precipitation during an iconic rainstorm on July 28, 2014. - Explanations for the increase of the frequency and intensity of precipitation. - The damage impact of hail storms, and an explanation that the frequency and intensity of hail storms will increase. - Expected number of ‘very wet’ days (>25 mm precipitation/day) per year now, and in, and in 2050. - Percentage of built up area per neighborhood. - External links to websites regarding concrete interventions taken to limit damage due to water nuisance.	Effect and impact of thematic trend, information about: - Amount of ‘tropical’ days now and in 2050 (max temp > 30 degrees centigrade). - Amount of warm nights (min. temp. >20 degrees centigrade) in urbanized areas, now and in 2050. - Cooling effect of water and vegetation.Percentage of ‘green’ area per neighborhood. - External links to websites regarding concrete interventions taken at the individual and urban level. - ‘Heatplan’ information of the RIVM (Netherlands National Institute for Public Health and the Environment)
Visual format	Map; Story map; Text; Photographs.	Map; Story map; Infographic; Video; Text; Media news item.	Map; Story map; Infographic; WebLink; Text.

) along the categories of the CID template, i.e., stakeholder, purpose, information, and visual format.

4.2. Illustrating the Usability Gaps

In this section, we present examples of the twelve usability gaps as they are recognized during the interviews with the internal and external stakeholders of the Northeast Brabant and FLUVIUS regions.

4.2.1. Stakeholder

The first and foremost component of the CID template consists of the intended audience of the climate service, i.e., the stakeholders involved in climate adaptation processes. It is essential that all efforts of information selection, purpose formulation, and visualization are attuned to a specific target audience for the climate service to have its desired effect. This component can be divided into three forms of usability: stakeholder validity, stakeholder readability, and stakeholder interactivity.

Concerning stakeholder validity, not all information is relevant for every stakeholder, and a variety of stakeholders requires a variety of climate information. For example, as a Public Health Organization explained: “Heat is not really an issue for us, unless you can somehow connect it to temporary water buffers in urban areas. When the excess water heats up the risk of infection also increases”. This category is also affected by the aforementioned impact/effect distinction. From an academic perspective one is inclined to say everyone is effected in some way by climate change, but it is not until those effects are translated into tangible impacts that audiences understand what is being communicated.

Besides showing tangible impacts, the way those impacts are visualized should also be attuned to a spatial language familiar to the audience of a climate service, i.e., stakeholder readability. Maps, for example, might show geographical locations where a potential flood could occur, but, as one municipality notes: “When it comes to communicating with local inhabitants, using their experiences of climate change effects and impacts would help, through images, to make the map become more ‘alive’.” Besides images, such as photographs of flooding situations, one could also take visualization a step further into augmented reality and art installations. Some stakeholders note novel means for communicating climate change using different ways to engage users: “there was this project on a public square with lasers that projected the water levels to raise awareness about flood risk.” This refers to the ‘Waterlicht’ expositions by Dutch artist Daan Roosengaarde [49].

In the stakeholder interactivity category, one of the main trade-offs is the relation between data complexity and visual literacy. Novel visualization techniques can be highly accessible in terms of interactivity, as the aforementioned ‘Waterlicht’ project can be viewed by anyone who visits the exposition by just walking through it. However, the installation shows only one main attribute: water height as a result of a flood. Other technologies, such as a sophisticated GIS portal or application that contains a lot of detailed flood information, might require more technical skills for a user to operate. As one policy advisor recommended: “What could work, is that we appoint a central person with GIS expertise at the water board, who can add layers of data to prevent a proliferation of different systems.” Although such an idea would help to safeguard data consistency, it would also re-centralize the content of the CS and prevent higher degrees of interactive validity and purpose.

4.2.2. Purpose

The second main component of the CID template relates to the purpose of the climate service. This component contains, among others, the intention, framing and fitness of information. Similar to the stakeholder component, three types of usability are distinguished.

Climate services serve a particular range of purposes. Purpose validity relates to the notion that, despite the information being correct, it might not be the right type of information for the end user’s needs. Some of the information that is presented by a CS is simply deemed too abstract to be useful. For example, most of the soil maps that are available are not detailed enough for micro-scale or precision agriculture decisions, as the Agricultural Organization states: “We need information at the level of a square (centi)meter, we are already familiar with the global picture”. Sometimes, end users are helped more by a CS that enables them to make their own decisions regarding implementation. If they would like to envision a different approach to dealing with the various climate issues information that shows alternative uses might be preferred, like the Drinking Water Facilities organization states: “When it comes to dealing with groundwater levels it would be better not to focus on water discharge, but also on buffering. This would help to deal with the drought issues that are starting to occur more frequently”. A local housing corporation realizes there is a lot to be done when it comes to raising awareness amongst their tenants with regard to the added value of small-scale interventions: “We want to do more with rain water runoff, by supplying rain barrels for people that are interested, hoping other people will catch on as well”. Information on how such small interventions add up to a decrease of flood risk could provide incentives for people to get involved by conducting climate adaptation measures themselves.

Purpose readability is the extent to which users are able to understand the purpose of a climate service. Written policy documents and advisory reports can be overwhelming in their content, and underwhelming in terms of visual attraction, especially within a bureaucratic environment. For example, the Public Health Organization stated they noticed that “a lot of the reports we produce end up on a shelf, forgotten. We are now trying to make our products more visual, by using maps and infographics, to communicate the importance of our findings better.” Here, the purpose of ‘communicating importance’ is explicitly linked to using the visual language of risk maps for a technical audience, and infographics for a wider public audience.

Purpose interactivity relates to the ability of users to re-purpose a climate service. This includes the option of addressing a different audience, as the Nature Interest Organization would like: “our main purpose is to contribute knowledge to the policy-making process, but we also see a role for our organization in informing local citizens.” In relation to the usability of climate services in times of an actual crisis, the Safety Region doubted whether sophisticated flood models are responsive enough: “A model, do you always need an internet connection? How accessible is this information in times of crisis? How stand-alone are these systems, do we need external power?” This stakeholder is also hesitant in relying on climate services produced by others as they would also like to adapt them in line with the strategy of the organization: “we used to be a very reactive organization, but we want to be more proactive, yet we don’t have any cyber experts.” An interesting point made by a local municipality relates to the prescribed informative purpose of most climate services, and that they could also be used as a conversation starter: “Maps should also show the things that are unknown: put question marks on the map, and use those to start a dialogue with colleagues about options for climate adaptation.”

4.2.3. Information

The third component of the CID template consists of the information that CS contain. Within our template, the same three types of usability apply to information as well.

The Climate Effect Atlas relies on computer models for simulating the effects of prolonged precipitation. Some of the information shown does not correspond to flooding situations that have recently occurred, as one of the municipalities noted that “the model predicted water levels of up to 2 meters in our city, which is physically impossible”. Such observations occur frequently; stakeholders compare the images that models provide with their own experience for a ‘reality check’. For stakeholders, a fundamental issue of information validity occurs when such images do not reflect their own experience. Another validity issue relates to accessibility and feasibility. For example, a Public Health Organization doubts whether certain information will be available due to the high production costs: “An analysis of electricity hubs and their power radius would be relevant, but it is very expensive to produce such maps”. Others consider that the information they need is difficult to obtain due to the temporal nature of some climate effects, as the Regional Environmental Advocacy group notes: “The effects of drought on natural areas are difficult to see immediately, but rather after a couple of years. For example, whether particular flora and fauna survived the summer.” For them, short-term depictions of changes in flora and fauna are not viewed as valid until they can be placed in a longer temporal perspective.

Information readability refers to whether it is clear from the CS which type of information is shown. For example, when the Safety Region discusses the maps containing ‘heat stress’, they find it difficult to grasp precisely what kind of ‘stress’ these maps refer to. To them, such information remains intangible: “These ‘stress tests’ contain images such as maps, but how reliable is that information; are the red spots really that bad?”. Similarly, the Public Health Organization noticed that the general public often can locate the reports that they produce, but were not able to understand what their graphs and table depicted: “which is not that strange. Despite using panels of citizens for making an inventory of what people do in times of heat stress, we only recently started to document our findings in ways that are more accessible and understandable”. A CS could benefit from a clear description or manual that explains what type of information is shown, and how it should be interpreted.

The current Climate Effect Atlas is presented in a closed format with low information interactivity. The maps are publicly accessible, but the information they contain cannot be modified. This excludes possibilities for crowd-sourced input, which could be useful for making the rather abstract maps more meaningful to specific users. The Public Health Organization in the region noted the following: “It is difficult to talk about problems of greenness in our province, considering it is the greenest of the Netherlands. Yet we do have our share of urban problems, so locating the subtle qualities of our green spaces through some kind of public involvement would be beneficial”. For stakeholders such as safety organizations, CS do not only inform their strategies in advance, they can also play a vital role during a crisis situation. Possibilities for adding information to the CS also relates back to issues of validity, as the Regional Safety Organization notes: “Maintaining the ground truth is paramount in times of crisis. We should be able to map current, factual situations as they take place on the ground.”

4.2.4. Visual Format

The fourth component of the CID template constitutes the visual, tangible manifestation of the climate service. Once again, three types of usability are presented.

In the practice of climate adaptation a variety of municipalities work together in ‘Deltaplan Spatial Adaptation Regions’. Such regions are formed on a policy level, not necessarily based on shared climate effects and stakeholders. The challenge of designing CS for such regions lies in the diversity of content and stakeholders, which makes it difficult to determine a single strategy of visual format validity, as one municipality remarks: “We cooperate between 12 municipalities that all partake in risk control. How do you present a coherent story?”. The internal consistency of the CS becomes especially important with CS that contain a diverse dataset, different purposes, and cater to a differentiated audience group.

The visual format readability category relates to a relatively simple notion: is the visual component of the CS readable? As one of the Nature Interest Organizations states: “Graphs are not always clear for people”. Maps are subject to similar visual (un)clarity, as a municipality notes: “a blue spot on the map is just blue, of course it corresponds to water, but what about its impact? Is there water in people’s houses? Vital infrastructure being compromised?” This point occurred several times. It relates to the very core of climate change communication: an intangible phenomenon with very tangible impacts is visualized using the most abstract type of spatial representation: a map. This distinction between impacts and effects seems to correspond largely to a distinction of intangible and tangible, or abstract and concrete, respectively. Several stakeholders emphasize the importance of a clear legend that helps to decode the maps into clear climate impacts.

Climate services can include different types of information, but also information on various spatial and temporal scales. A static map, for example, only shows a fraction of the information potentially housed by a climate service database. Visual format interactivity could improve the options of making a wider range of data accessible in terms of spatial scale. For example, one municipality remarked: “It would be good if users could zoom in on specific climate adaptation developments that are taking place.” The interactivity of a climate service could also provide easy access to more relevant information, while maintaining the richness of climate data that is offered: “Most maps are static, which means that sometimes people cannot see the right information despite that it is embedded within the map. An interactive GIS system that allows people to switch layers on and off would solve this.”

4.3. Counting the Usability Gaps

The statements of interview and focus group respondents about their experiences and needs surrounding the use of CS were coded using the analytical framework of Table 1. The coded statements were compared to the ‘readings’ of the two climate services (Table 4 and Table 5) to determine whether and to what extent the 12 different usability gaps occur. When we count the number of times the different codes that relate to the twelve usability gaps are observed we can see an emphasis in the number of occurrences for particular gaps along the two axis (

Table 6. Count of usability gaps for FLUVIUS and Northeast Brabant.

	Validity	Readability	Interactivity
Stakeholder	16	9	6
Purpose	36	1	8
Information	27	4	4
Visual format	1	4	2

). First, the category of Validity scores high on the information, purpose, and stakeholder components of the climate information design, compared to the readability and interactivity categories. Second, the gaps related to visual format score relatively low in the Validity and Interactivity categories, as well as the Purpose component in the readability category.

The scores of Table 6 entail that 36 statements made by the interview and focus group respondents describe a ‘purpose validity’ usability gap in the NAS Adaptation Tool Northeast Brabant and the Climate Effect Atlas FLUVIUS. This means that, although the information shown in the CS is correct, it is not the right information, or not detailed enough to be deemed useful. The ‘visual format validity’ usability gap was recognized only once, which would suggest that the CS use an appropriate visualization technique for the climate information they depict. However, this difference in occurrence could also relate to the way that climate adaptation professionals think about CS, i.e., what they consider to be important components. Stakeholder readability is also a relatively important concern, as nine statements refer to the fact that the information show by the CS should be readable for the target audience. The ‘purpose interactivity’ gap, which refers to the possibility for users to re-purpose a CS, scores high in the interactivity category. This indicates a need for making CS more ‘mobile’ within an organization, or within different phases of the adaptation process. This notion is addressed further in the discussion section. Next, we will also discuss the differences in occurrence in terms of internal hierarchies in the climate information design, and link these hierarchies to the policy context, and to the recent practice of developing climate services.

5. Discussion

The methodology presented in this paper could offer a good starting point for assessing and improving the communicational qualities of existing climate services at the one hand, and for determining the experiences and needs of climate adaptation professionals surrounding climate services at the other. This means that the CID template of Figure 2 serves two purposes: the researcher fills in the template for a ‘reading’ of a CS, whilst it can also be used during interviews to guide respondents through the different categories of usability gaps that could possibly exist. An overview of usability gaps, such as that of Table 6, could then provide a means of prioritization for improving the climate service. Consequently, the same template should be used for assessing the qualities of an ‘improved’ climate service as new issues of validity, readability and interactivity could occur once new information is introduced or when a different stakeholder is addressed.

Our analysis shows that all twelve usability gaps are found in Dutch climate adaptation practices. The questions central to the CID-framework have been found to be applicable to CS outside of the Netherlands as well [50]. However, a detailed international comparison of CS using the CID framework lies outside of the scope of this paper. Such a comparison would entail a thorough understanding of local climate adaptation policies, stakeholder configurations, and access to the local end user for evaluating the qualitative perception and use of CS. Nevertheless, future research of international cases could provide complementary examples, and a potential broadening, of our typology.

The empirical examples presented in this paper are used to illustrate twelve different gaps in climate service usability. Although examples of every type of usability gap were found, not all of the gaps occurred equally often. As such, a hierarchy of usability gaps can be seen in Table 6, from roughly left to right (validity, readability, and then interactivity), and top to bottom (information, purpose, stakeholder, and then visual format). This hierarchy could reflect the manner flood adaptation professionals deal with the development of climate services in practice. Based on the interviews and focus groups, it seems that the development of climate services thus far occurs very much in a data-driven way. In current practice, main questions in the early stages of climate adaptation processes revolve around data availability and data accessibility. This means that the development process of climate services is not only structured in top-down order (starting from the data), key questions central to the CID template are also left out, e.g., (1) What audience or stakeholder should be addressed? (2) for what purpose? (3) what is the information need of the audience? (4) what modes of visual presentation are suitable to convey that type of information?

Similar to the hierarchy of the four components, an order to reasoning in climate service development processes is found in the relation between validity, readability, and interactivity. Regarding all four components, the usability gaps related to readability and interactivity are not mentioned that often. One explanation could be that climate professionals are just starting to realize that specific audiences should be addressed for different reasons within a climate adaptation process. Consequentially, practitioners did not yet have time to think about appropriate visual formats. As such, the hierarchy represents not only a communication need, but an interest and disinterest for particular aspects of climate change communication as well. Another reason could be that climate data is often presented in a predetermined type of visual presentation, e.g., flood maps, heat maps, etc. This is could be seen as a testament to some of the more dominant visual languages traditionally used in climate adaptation processes. These traditions neglect a much more diverse group of stakeholders (e.g., local citizens and businesses) that is increasingly involved in climate adaption and is not so familiar with traditional ways of presenting scientific data.

The purpose of a climate service is largely determined by what is needed to facilitate or advance the adaptation process along its different phases [27]. Even though the new DPRA climate adaption processes in the Netherlands are still in an early stage, many climate professionals are geared towards presenting concrete adaptation measures and interventions, rather than first increasing the understanding of and raising the awareness about climate change and its impact. In doing so, the sense of risk and urgency that surrounds climate change is circumvented and stakeholders are encouraged to make low-level interventions based on instrumental considerations. Such considerations often include low costs and little inconvenience, but are not based in a broader understanding of climate phenomena or awareness about more radical interventions that might be needed in the future. A key issue that arises is the extent to which such a short-term strategy helps to increase support for more radical interventions needed on the long-term.

Climate adaptation is taking a more participatory turn and requires fostering alliances between stakeholders that are new to adaptation practice. Engineering firms seem to offer concrete protocols for the process of developing climate services that meet the demands posed on these current adaptation processes. Municipalities, being new to these kinds of climate adaption processes, seem to gladly accept the services offered. However, it is uncertain to what extent the protocols that are being used are actually tested and suitable for the developing practice, or whether they offer the opportunity for co-developing climate services ‘along the way’ in the coming years. Long-term commitment of the facilitating parties to climate service development seems pertinent, as the demands posed on climate services change along the various phases of climate adaptation processes. Whether it relates to a specific information need, required purpose, or a preferred visual language, co-developing climate services in an iterative process can ensure higher chances of success.

6. Concluding Remarks

The goal of this paper was to define a typology of climate service usability gaps by exploring and demonstrating a CID template with which to study how climate services are currently being developed and used in the practice of climate adaptation. This relation to adaptation practices is paramount, and determines the justification for using a particular climate service. As services, this means that communication always serves a particular purpose: facilitating and advancing climate adaptation. By that logic, any climate service that is partially or wrongly used (or not used at all) requires serious evaluation and potentially redesign.

Although the CID template and the analytical framework presented in this paper stress the need for a good ‘internal coherence’ of all twelve types of usability, we cannot assume that a ‘coherent’ climate service will find its audience, or that its audience will actually use it. For example, one of the usability gaps, i.e., purpose interactivity, entails the possibility for re-purposing a climate service. This particular gap was mentioned a couple of times within the context of internal communication within organizations. This means that climate services should not only be used to inform external stakeholders or audiences. They are also useful for internal stakeholders since municipalities and water boards often face a lack of understanding and willingness to cooperate within their own organization. Often, successful usage requires a proponent; an actor that actively promotes the climate service within his/her organization or social group. Consequentially, thinking about how climate services will be used in practice, or be developed for the most appropriate audience between and within organizations, should be part of any development process. Finally, the framework, and especially the hierarchies within the two axis of information-purpose-format-audience and validity-readability-interactivity, encourage both climate professionals and stakeholders to continuously reflect on and adhere to the policy processes they are part of. This could guarantee a more purposeful development and refinement of climate services, potentially resulting in a more informed, critically engaged, and pro-active configuration of stakeholders within a climate adaptation process.

As a final recommendation, we suggest that a next step in the critical development of climate services would be the setting-up of an iterative living lab-type of process wherein climate services are designed, evaluated, and redesigned together with potential end users. Day-to-day practice, however, might not accommodate the amount of time and resources for developing climate services along the lines proposed. Although most climate adaptation professionals acknowledge the value of good communication, they do not always provide the means or take the time for doing so. The use of our CID template could then provoke an interesting (and perhaps much needed) discussion on how climate adaptation processes are structured, planned, and carried out. This could mean that for attaining sustainable climate adaptation on the long-term, a serious reconsideration of adaptation processes and the role of climate data within them might be warranted.