Evaluating a planning support system's use and effects in urban adaptation An exploratory case study from Germany

: Planning Support Systems (PSS) are increasingly used to support collaborative planning workshops in urban adaptation practice. Research has focused on developing such tools and evaluating their use in workshops but has not measured tools’ e ﬀ ects over time on real planning processes, on the participants involved, and on the ﬁnal outcomes. The role that tools play in adaptation planning, therefore, remains unclear. A longitudinal case study was made to evaluate a PSS, the Adaptation Support Tool (AST), in a design workshop for sustainable urban water management, in Berlin, Germany. The case study also served to test the evaluation framework and generate insights regarding systematic evaluations of tools in planning processes. The case study was carried out over eighteen months, to capture both the details of the workshop and its longer-term e ﬀ ects on the project and participants. Our results show that the AST’s most evident e ﬀ ects were (1) contributory and less tangible in nature (e.g., supporting learning), than directly causal and concrete (e.g., a ﬀ ecting planning decisions), and (2) a function of the process and context in which the workshop took place. This study demonstrates that making systematic, longitudinal evaluations are valuable for studying the role of PSS in urban adaptation planning. E ﬀ ects are described in this study by: e ﬀ ects on learning; e ﬀ ects on problem situation; e ﬀ ects on planning process; e ﬀ ects on decisions made; (un)intended e ﬀ ects. investigation, S.M.; methodology, S.M., F.H.M.v.d.V. and J.H.S.; project administration, F.H.M.v.d.V.; software, F.H.M.v.d.V. and R.B.; supervision, F.H.M.v.d.V. and J.H.S.; validation, J.H.S.; visualization, R.B.; writing—original draft, S.M.; writing—review and editing, S.M., F.H.M.v.d.V. J.H.S.


Introduction
The unique challenges of climate change for cities have been recognized for some time and have inspired the burgeoning field of research and practice in adaptation planning [1,2], a complex undertaking, involving existing structures and infrastructures, interconnected urban systems, and myriad public and private stakeholders [1,3,4]. As such, adaptation planning is inherently spatial and deeply political [4,5]. Engaging stakeholders in the planning process is widely viewed as necessary to address the complexity and multi-actor nature of urban adaptation [1,6,7]. To this end, collaborative planning workshops are often held in the preliminary phases of a project, with the intention of building relationships between stakeholders, exchanging knowledge and views, clarifying problems, and identifying mutually acceptable and technically viable solutions [6]. Whatever the aims of a specific collaborative planning workshop, ultimately, they are meant to positively affect the larger planning process of the project, the participants involved, and the final outcomes.

1.
Making evaluations specific enough to be meaningful to a particular case, yet generic enough to offer useful and usable insights for broader research and practice.

2.
Making evaluations flexible enough to capture locally relevant factors and unintended and unexpected effects yet structured enough to be recognizable and comparable to other applications. 3.
Capturing both the details of a tool used in a workshop, and the longer-term effects on the planning process, the participants, and the outcomes.
There is a recognized need to better understand the role of tools in real planning applications. Such knowledge will provide valuable insights to practice and feedback to theory [4,10]. The focus of our research is to explore how tools used in collaborative planning workshops influence these activities and their outcomes, and how such evaluations can be carried out. To this end, we undertook a single exploratory longitudinal case study of a collaborative adaptation planning workshop in Moabit West, a district of Berlin, Germany. This workshop made use of a Planning Support System (PSS), called the Adaptation Support Tool. In this article, we present our approach to evaluation, the results of our case study, and offer insights drawn from our experience with this evaluation. The conceptual framework of analysis is presented in Section 2, the research design for our study in Section 3, our research methods in Section 4, the case study results in Section 5, a discussion of our findings and our experience with the framework in Section 6, and conclusions in Section 7.

Conceptual Framework of Analysis
The use of a collaborative planning support tool can be conceptualized as nested within a workshop that occurs during a planning process, which itself takes place within a larger context [18]. Understanding the role of a tool requires examining these different layers and their inter-relations. While the literature offers useful frameworks for evaluating participatory planning processes [33,34] among others and individual workshops [31,[37][38][39][40], we are unaware of a framework that accounts for the nested nature of tool use needed to study the role of planning support tools in practice. Another requirement for our framework is that it be useful for evaluating not only PSS but planning support tools more broadly.
We took a pragmatic approach to develop a framework for our aims by borrowing and learning from the existing frameworks for evaluating activities and processes. As elaborated elsewhere [41][42][43] and described in Table 1, our framework identifies seven factors for evaluating workshops within planning processes. This framework explicitly links the input and the workshop with results and effects within the larger context. This reflects our fundamental assumption of the relations between what happens in a workshop and its outcomes in a context-rich planning process. Furthermore, our framework distinguishes between the process and content of a workshop, which is useful for evaluating a tool, and particularly a PSS, as planning support systems aim to support both process and content aspects [10]. Table 1. Evaluation factors, with the elements by which each factor was described in this study [42,43].

Context
The context in which a planning process and a specific workshop take place has important implications for what can be achieved by a planning support tool [22]. Contextual factors can include political and physical conditions, social, technical and ecological systems, and what events have come before. The elements of context that are relevant for a tool's role and effects in a specific case can vary significantly. Context is described in this study by: local project setting; institutional setting; project structure and process.

Input
The input to a workshop encompasses everything that was provided to it, such as the available data, the stakeholders related to the issue, and the objectives of the workshop. Input should not be confused with participants' contributions during the workshop, which is content. Input is described in this study by: Aim and role of workshop; organization of workshop; stakeholders and workshop participants.

Evaluation Factors
Process A workshop's process includes the procedures, communication, and ways of working during the activity. This is not to be confused with the overall planning process, in which the workshop takes place. Tools typically intend to support a workshop process through improved interactions and communication.
Process is described in this study by: workshop structure and procedures; communication; way of working.

Content
Content refers to the substantive material used during a workshop, including information, knowledge, models, maps, perspectives, and values that are shared by participants or provided by organizers. Planning support tools are typically an important source of substantive content. Content is described in this study by: quality and type of data and information used; depth and breadth of content; tool or method used.

Results
Results are the direct products of a workshop, which include artefacts, like maps, models and planning documents, and less tangible outcomes like alliances and agreements. Tools aim to improve the quality of workshop results through improved content and processes.
Results are described in this study by: workshop results; documentation of results; value and relevance of results to the planning process and stakeholders.

Use
The use of results includes the direct and indirect ways a workshop's tangible and less tangible results are used over various time frames and by different actors, for different purposes (direct use of a tangible result, for instance, would be a planner applying an idea or measure from a tool directly in the next steps of developing the plan. Indirect use of less tangible results would be a stakeholder leveraging a new alliance to influence decisions on the plan). The use of results leads to effects and also captures the value and meaning of the results for different stakeholders.
Use is described in this study by: direct use of results; indirect use of results; unused results.

Effects
Effects are the workshop's impacts on the system or actors involved. Assessing effects is complicated as they have different forms and are realized at different temporal and spatial scales. There are direct effects from a workshop, such as learning and new relationships, and indirect effects through the use of results, such as influencing later decisions. Effects may be intended or unintended, and a workshop may clearly be the cause of an effect or only contribute to it. In adaptation and collaborative planning, less tangible effects, such as creating shared meaning, are as important as traditional, more concrete ones [3,36]. Effects are described in this study by: effects on learning; effects on problem situation; effects on planning process; effects on decisions made; (un)intended effects.

Case Study Description
The case study focuses on a workshop for designing sustainable urban water management measures in Moabit West, in central Berlin, Germany ( Figure 1). Moabit West faces pressures typical for Berlin and other European cities, namely the challenge of providing affordable housing to a growing population, supporting a dynamic economy, addressing climate change [44], and adapting an already densely built environment [45].
The design workshop studied in this research formed part of a larger project (planning process), Smart Sustainable District Moabit West (SSD-Moabit West) that was funded by Climate KIC and organized by CHORA Conscious City, at Berlin University of Technology (CCC-TUB). The SSD-Moabit West project focused on three themes: sustainable urban water management, energy efficiency and low-carbon mobility. The working groups for each theme included local public and private partners, European knowledge partners from Climate KIC's SSD consortium, and the project managers at CCC-TUB. During the design workshop, each working groups focused on finding pilot projects that could be implemented in Moabit West. A designated integration manager from CCC-TUB looked for opportunities to integrate the themes.
The year-long project was initiated and designed by CCC-TUB and involved two workshops between the local and European partners. The first workshop, in March 2016, addressed agenda setting, building relationships between the partners, and deciding on the role of the European partners. The second workshop, in September 2016, focused on designing pilot projects for each theme and exploring opportunities for integration. The event comprised:

•
A short plenary introduction for all project partners and invited participants; • Parallel half-day design sessions for each working group to design pilot projects; and • A plenary integration session to identify opportunities for collaboration between pilots from the different working groups.
The design session for the sustainable urban water management group (here forth: water group) forms the focus of this study and employed a PSS called the Adaptation Support Tool. A schedule of the day, activities, materials, and outcomes is available in Appendix A.
Sustainability 2019, 11, x FOR PEER REVIEW 5 of 26 • A plenary integration session to identify opportunities for collaboration between pilots from the different working groups.
The design session for the sustainable urban water management group (here forth: water group) forms the focus of this study and employed a PSS called the Adaptation Support Tool. A schedule of the day, activities, materials, and outcomes is available in Appendix A.

Adaptation Support Tool
The Adaptation Support Tool (AST) is a web-based PSS for planning nature-based and traditional infrastructure spatial adaptation measures in the urban environment ( Figure 2). The AST is designed for use in facilitated workshops, where small groups of stakeholders co-create spatial adaptation plans at the neighbourhood to city scale. More information on the AST can be found in van de Ven et al. [6].
The touch-table screen of the AST consists of three panels. The left panel is used for input, the middle panel contains a map interface that is used for developing an adaptation plan, and the right panel shows output parameters in real-time. Each panel is summarized below.
The input panel consists of a Setup tab to specify properties of the project area, for example, soil type, land use, and scale of interest. Based on the entered properties, a ranked list of adaptation measures is generated from a library. This list is shown in a second tab Measures [47]. For each measure in the library-currently 72-information and pictures are provided by selecting Info.
The map window contains base layers, like street and satellite view maps or aerial photographs, which are used for spatial referencing. On top of these base layers, semi-transparent thematic maps can be displayed as overlays, like elevation, critical objects, and flood or heat maps. The layers help users understand the climate challenges in an area, and to choose effective locations for interventions. Adaptation measures can be selected from the list in the input panel and applied in the map interface. The estimated effectiveness and cost of the measure are then calculated, based on its dimensions, and the local properties and climate conditions. For more details on the models, see van de Ven et al. and Voskamp and van de Ven [6,47].
The output panel contains a legend of the measures that have been applied in the map and a list of key performance indicators: storage capacity, flood return period, heat stress reduction, drought reduction, water quality, and a first estimate of construction and maintenance costs. At the start of a workshop, targets are entered for each performance indicator, and bar graphs are used to show the cumulative percentage of each target achieved. A second tab, Details, provides the quantified contribution of each applied measure.
Plans developed using the AST are intended as the input for more detailed design efforts by water managers, urban planners, and landscape architects. The intended added value of the AST lies in collaboratively developing ideas of possible measures and their locations, based on the dialogue of stakeholders and informed by real-time, evidence-based feedback from the tool. AST sessions are

Adaptation Support Tool
The Adaptation Support Tool (AST) is a web-based PSS for planning nature-based and traditional infrastructure spatial adaptation measures in the urban environment ( Figure 2). The AST is designed for use in facilitated workshops, where small groups of stakeholders co-create spatial adaptation plans at the neighbourhood to city scale. More information on the AST can be found in van de Ven et al. [6].  In the interest of transparency, we clarify the relation of the authors to the project, the workshop and the tool evaluated. The local organizers in Berlin designed the workshop and selected the AST to support their aims. The first author was invited to make an independent evaluation and played no role in the project itself, nor in planning or facilitating the workshop, nor in the development of the The touch-table screen of the AST consists of three panels. The left panel is used for input, the middle panel contains a map interface that is used for developing an adaptation plan, and the right panel shows output parameters in real-time. Each panel is summarized below.
The input panel consists of a Setup tab to specify properties of the project area, for example, soil type, land use, and scale of interest. Based on the entered properties, a ranked list of adaptation measures is generated from a library. This list is shown in a second tab Measures [47]. For each measure in the library-currently 72-information and pictures are provided by selecting Info.
The map window contains base layers, like street and satellite view maps or aerial photographs, which are used for spatial referencing. On top of these base layers, semi-transparent thematic maps can be displayed as overlays, like elevation, critical objects, and flood or heat maps. The layers help users understand the climate challenges in an area, and to choose effective locations for interventions. Adaptation measures can be selected from the list in the input panel and applied in the map interface. The estimated effectiveness and cost of the measure are then calculated, based on its dimensions, and the local properties and climate conditions. For more details on the models, see van de Ven et al. and Voskamp and van de Ven [6,47].
The output panel contains a legend of the measures that have been applied in the map and a list of key performance indicators: storage capacity, flood return period, heat stress reduction, drought reduction, water quality, and a first estimate of construction and maintenance costs. At the start of a workshop, targets are entered for each performance indicator, and bar graphs are used to show the cumulative percentage of each target achieved. A second tab, Details, provides the quantified contribution of each applied measure.
Plans developed using the AST are intended as the input for more detailed design efforts by water managers, urban planners, and landscape architects. The intended added value of the AST lies in collaboratively developing ideas of possible measures and their locations, based on the dialogue of stakeholders and informed by real-time, evidence-based feedback from the tool. AST sessions are meant to capture locally specific factors like acceptability, constraints, and opportunities early in a planning process, and to create shared learning through dialogue and interaction with the tool [6].

Relation of the Authors to the Project, the Workshop, and the Tool Evaluated
In the interest of transparency, we clarify the relation of the authors to the project, the workshop and the tool evaluated. The local organizers in Berlin designed the workshop and selected the AST to support their aims. The first author was invited to make an independent evaluation and played no role in the project itself, nor in planning or facilitating the workshop, nor in the development of the AST. These conditions provided the opportunity to evaluate a PSS used within a real workshop, as well as its effects on the remainder of the planning process. The second and third authors facilitated the AST session during the workshop and formed part of the tool's development team but were not involved in the collection or analysis of data. The fourth author provided an external check on the research design, data analysis, and results and has no association with the workshop, project, or tool.

Research Methods
A case study approach provided an appropriate method for examining the role of a tool within a planning process, where a holistic analysis promises the most useful insights [48]. Accordingly, a single, longitudinal case study was used to evaluate the role of the AST in the SSD-Moabit West sustainable urban water management design workshop and its longer-term effects. Our methods are elaborated below.

Evaluation Factors
The evaluation factors are each described by several elements (Table 1). These elements derive their use in other applications from a literature review [41,42] and are further refined in this case via the deductive and inductive data analyses (Section 4.3). While a comprehensive description of each evaluation factor is unrealistic, in our selection of the elements in Table 1, we required that the elements be (1) general enough to be meaningful in most applications and to our exploratory study, (2) comprehensive enough to capture all elements germane to understanding the role of a tool in practice, and (3) concise enough to support a pragmatic evaluation. Finally, our evaluation factors focus on the elements needed to understand the role of a tool within a workshop and planning process, as opposed to describing characteristics or functionality of a tool.

Data Collection and Types of Data
In longitudinal case studies, data is collected at more than one point in time to track changes in relevant factors [48]. In this case study, the analysis was based on data collected over 18 months, in five phases from before the workshop, during and after it, at the end of the project (planning process) and one year after the end of the project.
A table detailing the data collected in each phase is provided in Appendix B. A summary of the different types of data collected and how they were used is provided below: • Interviews. Seventeen semi-structured interviews of one to three hours were carried out with the project management team and workshop participants. Audio recordings and written notes were transcribed for analysis.

•
Discussions. In addition to the formal interviews, informal on-the-record discussions were used for confirming impressions and information, as well as asking for the views of a wider range of informants. Audio recordings and written notes were transcribed for analysis. • Documents. A range of documents was reviewed, including planning documents, reports, websites, team emails, and work products.

Data Analysis
The data analysis comprised three steps, an inductive (thematic) analysis, a deductive analysis and a meta-analysis. Text analysis, using the software package Atlas.ti version 7.5.18 (ATLAS.ti Scientific Software Development GmbH, Berlin, Germany) formed the primary method in the inductive and deductive analyses.
• Inductive (thematic) analysis was made first to surface codes and themes that emerged from the case study. • Deductive analysis was undertaken using the pre-defined evaluation factors and a list of describing elements from literature. The list of elements (codes) was later refined to those listed in Table 1 (see Appendix C for original list). • Meta-analysis was used for two purposes. First, to compare the inductive and deductive analyses for different and common findings. From this assessment, a comprehensive list of themes and codes was created. Second, the meta-analysis was used to examine a number of elements that were not well captured in text, yet were important for the evaluation, for example, assessing the quality of work products.
The multiple data sources and three-part analysis were used to reduce bias in the evaluation and to strengthen the accuracy of the findings through triangulation [49].
All participants gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Delft University of Technology's Human Research Ethics Committee.

Results
The following sections summarize the findings for each evaluation factor and end with a short reflection on the role of the tool, based on our analysis. We report the results qualitatively and descriptively for two reasons. First, the small number of participants in our study and its inherently qualitative and exploratory nature would make quantitative reporting unrepresentative and misleading. Second, most of our results are based on a combination of data sources, for instance, questionnaires and interviews, which are not easily conveyed in graphical or quantitative formats. The results summarized below are elaborated in more detail, including interview quotes in Planning Support Tools in Urban Adaptation Practice [43].

Context
Although describing the context ( Table 2) does not tell us directly about tool use, it gives critical information for understanding the use and effects of tools. Thus, context is important to the evaluation of tools in use. Elements of context were surfaced primarily through the thematic analysis. The interviews and document reviews were particularly useful for this. Nature-based adaptation measures have clear benefits for the social and physical systems in Moabit West. They can improve aesthetic quality, create more shared green space in the community and address present and growing problems of heat stress and storm water flooding. However, the environmental permitting and property rental conditions create institutional barriers to implementing nature-based solutions on both public and privately-owned land. Furthermore, the workshop was carried out in a project with a strict schedule, budget, and performance requirements from its funder, which served as limiting boundary conditions. The AST's aim to support the design of nature-based measures to adapt to urban areas was well suited to the planning topic and local project setting. Furthermore, the fact that the AST is an off-the-shelf tool that can be used with basic, easily available input information, meant the tool could be deployed within the limited timeline and budget of the project. In these ways, the AST seems an appropriate tool for the given contextual conditions.

Input
Similar to context, assessing workshop input offers insights for understanding the role of the tool in the workshop and its effects afterwards. The input (Table 3) was evaluated mostly through the thematic and deductive analyses of interviews, documents, and observation. The AST was selected by the organizers for its ability to support the aims of the workshop: exploring and designing nature-based solutions in public and private spaces in the early phase of adaptation planning. Furthermore, the collaborative design element of the tool could foster dialogue between stakeholders to address institutional barriers to the implementation of such measures and could serve as a pilot for collaborative planning workshops in Berlin. Previous research has highlighted the importance of task-technology fit in PSS usefulness [50]. Finally, the tool is intended to be used by small groups, like that in the workshop, and the type of information presented by the tool is well suited to institutional stakeholders.

Process
Observations, interviews, and questionnaires were used to evaluate the workshop process, using the inductive, deductive, and meta analyses (Table 4).

Analysis of the Role of the Tool in the Workshop Process
Our analysis indicates that the AST supported the workshop process mostly through communication and the way of working. These were among the most valued aspects of tool use by participants.
Communication between participants and an interactive way of working were supported through the tool's map interface, real-time feedback, and library of measures. Responding to the tool's content and working around a common object created a dynamic and engaged process during  1.
Introduction to session and participants 2.
Sustainable urban water management best practices presentation and discussion 3.
Review of measures in the AST library 4.
Creation of a short-list of preferred measures 5.
Design by implementing preferred measures in the tool's map interface Our analysis indicates that the AST supported the workshop process mostly through communication and the way of working. These were among the most valued aspects of tool use by participants.
Communication between participants and an interactive way of working were supported through the tool's map interface, real-time feedback, and library of measures. Responding to the tool's content and working around a common object created a dynamic and engaged process during the workshop. This finding highlights the interconnected nature of content and process in workshops and the tool's role in supporting both. By providing content, the tool was central to supporting a productive workshop process and realizing its achievements in a time-efficient manner.

Content
The observations, interviews, questionnaires, and documents were most relevant for evaluating the content, using the inductive, deductive, and meta analyses (Table 5). Our analysis indicates that the tool was a significant source of substantive content in the workshop and also elicited the communication of content held by the local and European partners. This included subjects not covered by the tool, such as social or institutional issues. The tool's most valued elements, its library of measures and real-time feedback, are its content. Given the participants' mixed level of familiarity with adaptation measures, reviewing the tool's library created a common knowledge base for the design. Similarly, the map interface created a focal point for discussions and a shared spatial language. The interactive nature of the touch table was also valued for adding content in a more dynamic and creative way than traditional workshops.

Results
While the inductive, deductive, and meta analyses were used to evaluate the results (Table 6 and Figure 4), the meta-analysis was particularly helpful.  Measures include a water square, porous pavement, disconnecting paved surfaces, green roofs, urban agriculture, tree pits, and swales.

Analysis: Role of the Tool in the Results
The tool was central to the results achieved in the design workshop. The main work product, the plan of measures on the map, was created in the tool and based on information provided by the tool's library and indicators. Participants, however, found the less tangible results, like learning and collaboration, most valuable. It is more challenging to determine the tool's role in achieving these types of results, but evidence includes participants reporting that the tool made the workshop more effective and efficient than others they had attended, and significantly more so than local planning practice without workshops. The tool also supported dialogue and improved communication between participants, as discussed in earlier sections, and so contributed to the less tangible results.

Use
The evaluation of use (Table 7) focused first on the use of results at two time frames: during the SSD-Moabit West project and after it. The meta-analysis was especially useful for comparing planned or intended uses of results with actual uses over time. Table 7. Summary of the findings related to the use of workshop results, listed by the describing elements. Measures include a water square, porous pavement, disconnecting paved surfaces, green roofs, urban agriculture, tree pits, and swales. The tool was central to the results achieved in the design workshop. The main work product, the plan of measures on the map, was created in the tool and based on information provided by the tool's library and indicators. Participants, however, found the less tangible results, like learning and collaboration, most valuable. It is more challenging to determine the tool's role in achieving these types of results, but evidence includes participants reporting that the tool made the workshop more effective and efficient than others they had attended, and significantly more so than local planning practice without workshops. The tool also supported dialogue and improved communication between participants, as discussed in earlier sections, and so contributed to the less tangible results.

Use
The evaluation of use (Table 7) focused first on the use of results at two time frames: during the SSD-Moabit West project and after it. The meta-analysis was especially useful for comparing planned or intended uses of results with actual uses over time. Table 7. Summary of the findings related to the use of workshop results, listed by the describing elements.

Direct use of results
During the Project • Immediately following the design session, the plan was used in the integration session with other groups to select a pilot project • The plan was used by water group and project leaders in the documentation, follow up stakeholder meetings, and to elaborate on the design and analysis After the Project • The plan formed the basis of the 'tree pit' pilot project that continued in the year following the end of the project

Indirect use of results
During the Project • Strengthened relationship between two key partners used in~30 meetings to secure permits and implementation of the pilot After the Project • Experience from the workshop inspired one participant to propose it in a spinoff project Unused results

•
Plan was used to identify a suitable location for the pilot but not to select tree pit measure, which was pre-selected by group leaders Both the work products and the less tangible results from the tool were used directly and indirectly during and after the project. The plan developed in the tool was a useful artefact and the most used result for project reporting, presenting, and as the basis of further analysis and design elaboration. The plan was valued for communication purposes and for developing the pilot project. However, the plan was not used to select measures, as would be expected. This was attributed to the pre-selection of a preferred measure and to contextual factors of the project that limited openness to pursue new measures proposed in the plan. Overall, the work products, namely the map and measures developed with the tool, were used for official purposes. The less tangible results, such as strengthened commitment, were used to motivate the actions of partners in this project and to inspire future initiatives. The use of work products is interesting in this regard, as the purpose of the AST is to create designs that inform planning, but contextual factors of the project appear to have limited this use while making the designs useful in other important ways and making the less tangible outcomes useful to the planning process.

Effects
The longitudinal study was particularly useful for capturing effects (Table 8) over time, using the inductive, deductive, and meta analyses. The tool's strongest effect during the project was on learning. Here, the collaborative workshop and the tool's library of measures and interactive character appear to have contributed to learning through the provision of content and improved communication and interaction. The tool supported the intended effects of the project by providing a successful pilot of collaborative tool-based planning workshops, as evidenced by the proposed use of the tool for other projects in the city. The learning and strengthened commitment credited to the tool also played a role in achieving approval for the tree pit pilot; however, these effects are too indirect and enigmatic for strong claims.
The tool did not have a discernible effect on the rest of the planning process, on the decisions made, or on the physical problem situation. This appears to be largely due to contextual factors of the project, like its timeline being too short to consider new ideas or to engage other stakeholders.

Discussion
We conceptualized the use of planning support tools as nested within a workshop, which occurs during a planning process, which itself takes place within a larger context. In the previous section, we described our findings from applying the evaluation framework to the SSD-Moabit West design workshop for sustainable urban water management, which made use of the Adaptation Support Tool. In the discussion, we reflect on the use of the tool in the workshop, the connections to the planning process and its context, and finally, on our research methods.

Reflections on the Use of the Tool in the Workshop
Our results have shown that the design workshop was perceived positively and that participants enjoyed working with the one another and the tool in collaborative adaptation planning. Beyond outcomes that could reasonably be expected from dialogue alone, we ascribe the following attributes to the use of the tool: • Providing information about the many adaptation measures in the tool's library, which created a common knowledge base and vocabulary for the design. While some of these outcomes could arguably be achieved with paper maps and more traditional forms of a workshop, the tool provided time efficiency to achieve these results within three hours. In this way, the tool's unique contribution was to combine the analysis, design, and dialogue aspects of conceptual planning in a time efficient and informed manner. These findings are consistent with the main added values of PSS identified by Pelzer et al. [10], namely, learning, communication, collaboration, consensus, efficiency, and more informed products.
When reflecting on the value of the tool in the workshop, most participants remarked that the way of working was more efficient, creative, interactive, or inspirational than traditional planning. Previous PSS research has shown that some types of users, such as planners and community members, may find an analytical map-based tool disruptive to their manner of working [30,41,51]. In Berlin, the match between the tool's engineering frame and the participants' technical backgrounds probably contributed to their comfort with the tool. It would be valuable to evaluate tool use in a more mixed group.

Reflections on Connections with the Planning Process and Context
The most relevant effects of the tool on the planning process were contributory and less tangible in nature, such as learning. This is not surprising, given the preliminary level of design in the workshop and the focus on overcoming barriers to implementing pilots. The lack of more causal and concrete effects, however, is a function of the context. Most critically, the project structure defined, to a large extent, the role that the tool could play in the planning process. A better alignment between the workshop aims and project schedule seems like it would have provided a better opportunity for the tool to affect the planning process in a material way. Research has shown the critical influence of context on the role of a tool [22,27,28,42].
Learning plays an important role in adaptation [1,3,52,53] and forms a common aim of planning support tools, and PSS more specifically [10]. Therefore, the learning effects created by using the AST are important outcomes.

Case Study
A longitudinal case study proved a useful approach for examining the role of a tool in a workshop and planning process, particularly as this approach allowed us to evaluate the use of results and the effects over the life of the project and beyond. As expected, the complexity of real-world planning made it challenging to tease out the most contributory and less tangible effects that were realized. Here, the richness of context was important for understanding the 'how' and 'why' of what was observed.
The SSD-Moabit West design workshop was a representative case of adaptation planning. Evaluating 'real' workshops, as opposed to those designed for academic research, offers obvious benefits for the representativeness of results. However, 'real' workshops are imperfect research objects. In our particular case, there was insufficient time for using the tool to its full capacity in the workshop, which limited the extent of the design; local data was not used in the tool, limiting the value of its indicators; and some stakeholders were missing from the workshop, whose perspectives were absent from the dialogue. Evaluating a 'real' workshop, however, revealed the critical importance of context in determining the role a tool is able to play.
While case studies are meant to offer in-depth insights in place of broadly generalizable results, it is useful to compare our findings to other research on PSS use in workshops. In doing so, we see a number of consistent themes emerge. For instance, Pelzer et al. [10], found that users' valued the MapTable planning support tool more for collaboration and communication support than for outcomes. Similarly, the same tool's shared map interface was found to stimulate knowledge sharing and dynamic interactions in workshops, which contributed to learning [51]. Meanwhile, Arciniegas and Janssen [20] found that another map-based touch table tool supported improved communication and new insights into land use planning, Russo et al. similarly found map interfaces are highly valued by practitioners [27]. Mirroring our own findings, Pelzer [54] noted the need for alignment between a tool's functionality and the stage of planning in order to promote effective tool use. Finally, a number of researchers [22,27,28,42,55] have highlighted context as a determining factor in the role, and even the meaning, of a PSS and its outcomes. While many of these findings were based on simulated workshops and different tools, our corroboratory results are based on a real planning workshop using the AST. Most importantly, while prior research has focused on the workshops alone, we have evaluated the effects of PSS use over the longer-term planning process, a novel step in this field.

Evaluation Framework
We required a framework that recognized the nested nature of tool use within workshops, planning processes, and context. Our framework was useful for structuring the evaluation and for ensuring that it was systematic and comprehensive. The framework, however, is descriptive in nature; it simply structures data so that it is addressed, regardless of expectations and preconceptions. The evaluator must make sense of the data and draw causal links. The framework was used as the basis of the deductive analysis, while the inductive and meta-analyses were used to surface explanatory threads, major themes, and to check for components missing from the original framework. The three analyses combined, allowed locally specific and relevant themes to emerge while ensuring the evaluation was still systematic and produced reliable and comparable results.
Our choice for a framework that does not focus on tool characteristics, but instead focuses on the factors that help understand the role of the tool in a broader setting, provided a holistic and rich picture of what worked (and not), how it worked, and why. A challenge in using and reporting on the framework is overlap, which comes from the interconnectedness of factors. For instance, the tool's content, such as the map interface, also played an important role in the process of the workshop. In this article, we chose to report the results according to the structure of the framework to illustrate both the application of the framework, as well as the results of this particular case.

The Challenges of Evaluation Revisited
We started this article by describing several challenges in evaluating the use of tools in adaptation planning workshops and processes. Reflecting on our framework and evaluation, we find:

•
The evaluation factors and describing elements broad enough to be relevant for a variety of tools and workshops and specific enough to produce meaningful insights for our case.

•
The framework and analysis method flexible enough to reflect local conditions and capture emergent themes yet structured enough to produce systematic and comparable results.

•
The longitudinal case study approach appropriate and effective for capturing the effects of the tool on the planning process and participants, and for revealing the importance of context, while still capturing the details of tool use in the workshop itself.

•
The nested view of tools helpful for understanding the use of the tool, its results, and effects.

•
The descriptive and qualitative nature of the framework a potential weakness in its reliability, but a strength in its ability to be applied to a wide variety of cases and tools, in different contexts.
The results of our evaluation of the AST used in the design workshop for SSD-Moabit West are specific to the tool and the application studied. The use of tools, the tools themselves, and the conditions in which they are used are too varied to make claims about the generalizability or transferability of our results. Nevertheless, by using a systematic and structured approach to the evaluation, the outcomes should be comparable and useful in other cases. The reliability of the framework can only be confirmed through testing in more applications. However, the skill of the evaluator is an important and less predictable variable in the quality of results. Validity is a more interesting question and a common challenge in qualitative studies [48,49], such as ours. We found several strategies useful for limiting bias: • Using many sources and types of data for triangulation.

•
Basing the evaluation on a structured framework for ensuring a systematic review of all the data.

•
Using different approaches in the data analysis for capturing a comprehensive view of the data. • Using a longitudinal study for ensuring the consistency of our findings over time.

•
Checking our themes, hypotheses, explanations, and findings with key informants. • Using an evaluator who is independent of the tool, the workshop and the project.

•
Engaging an external reviewer to check the evaluation design, analysis and results.
Finally, we recognize that the access, time, and resources to carry out a comprehensive and longitudinal evaluation are luxuries not readily available to most projects. While our analysis was time consuming, the framework could also be used in a 'lighter' evaluation. Workshop organizers or evaluators could fill in most of the framework based on their knowledge, soliciting input and feedback from key participants, as needed. Care should be taken to think critically about the evaluator's biases and preconceptions, as they will not have the benefit of thorough data collection or analysis.

Conclusions
We set out to understand what role a PSS, like the AST, plays in collaborative planning workshops, and what effects such tools have on these activities, the participants, and the planning processes in which they are used. We also wanted to test how such evaluations could be carried out effectively. There are a number of challenges in evaluating the role of tools in a way that the results are both meaningful to a specific case, and more broadly, useful and comparable. Such evaluations must be flexible yet structured and detailed yet wide-reaching. In this article, we have presented a framework for making evaluations, along with our results and experiences in applying it to a longitudinal case study of the AST, used in a design workshop for sustainable urban water management, in Moabit-West, Berlin. Our findings showed that the tool's role was mostly contributory and less tangible in nature (e.g., supporting learning and communication) as opposed to directly causal and concrete (e.g., affecting the planning process, decisions or problem situation). Perhaps most importantly for illustrating our assertion that the effect of tools should be studied within the wider arena of the planning process and context, our results showed that the role of the tool was largely a function of contextual conditions, such as project structure and timing.
While one case study is a modest contribution to understanding the role of tools in collaborative planning workshops, the longitudinal case study approach allowed us to evaluate the effects of a tool over time, and to test a framework for evaluation. We found our framework and research approach addressed the challenges of evaluations and provided useful and usable results. Naturally, more applications are needed to test the reliability of the framework, while additional evaluations of tools in real applications are needed to continue improving the quality of PSS and of adaptation planning practice and theory. Finally, evaluations that account for the nested nature of tools within workshops, planning processes, and context can help to capture effects beyond the workshop.

Acknowledgments:
The authors thank the Climate-KIC Smart Sustainable District-Moabit West project for serving as a case study. We are especially grateful to Nadine Kuhla von Bergmann and Georg Hubmann at CHORA Conscious City, Department for Sustainable Urban Development, Faculty of Architecture, TU Berlin, and to the members of the sustainable urban water management group, who participated in this research. Livius Hausner was particularly helpful in providing information and documents. Finally, we thank the anonymous reviewers and the editor of this journal for their helpful comments on our manuscript.

Conflicts of Interest:
The authors declare no conflict of interest. The funders played no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.  Bezirksamt Mitte von Berlin District authority, tasked with approving, operating and maintaining any measures in public streets or green areas in Moabit West. The offices of streets and green spaces, and of nature conservation were invited.

Workshop Agenda and Invited Stakeholders
Senatsverwaltung für Stadtentwicklung und Umwelt City department for urban development and environment. The departments of water resources and ground water were invited.

Deltares
Dutch institute for applied research in the field of water and subsurface. Facilitated the Adaptation Support Tool design session.

TNO
Dutch institute for applied sciences. Developed the best practices document.

Appendix B
Data Collection Table A3. Data collected by phase over project duration.  Agreements, plans for next steps From interviews and discussion in sessions, the planned next steps were recorded, as well as agreed actions of different actors. These were also reported by session organizers to the project management team. During breaks and after symposium, short discussions several partners, participants, organizers and managers, to check information and ask for impressions. Written records and audio memos were used to record discussions. Table A3. Cont. Project status updates and reporting shared during interviews Table A5. Tracking codes used in the evaluation.

CONTEXT-Challenges General
To identify the role of these challenges as they reinforced/counteracted the role of the design session and tool

PROCESS-Changes
To identify if the process of the design session strayed from plans

PROCESS-Ending
To identify how the design session ended

RESULTS-Planned Actions
A type of result that is a plan to take action by a participant. Later compared to actual actions taken following design session.

USE-Planned Use
Track intended use of results for different time periods for comparison with actual use. Identified realized, unrealized and realized but unforeseen uses of results and their time frames

EFFECTS-Types
Track different types of effects over time

Needed actions
Track actions that were identified as necessary to reach certain aims, like implementation. Later checked which actions were taken and the results.
Next steps/expectations Track the plans and expectations of different actors to compare with the actual process and what transpired.

Participation Identify role of participation and views on participation
Perspectives/views Track different perspectives of actors in the project over time to identify changes, contradictions, shared views, etc. Table A6. Prompting codes used in the evaluation.

RESULTS-Consistency
Assessing the consistency of the results with the input conditions, actors, process and content of the design session and project