Are Cities Aware Enough? A Framework for Developing City Awareness to Climate Change

: Cities are growing and becoming more complex, and as they continue to do so, their capacity to deal with foreseen and unforeseen challenges derived from climate change has to adapt accordingly. In the last decade, an e ﬀ ort has been made to build city resilience and improve cities’ capacity to respond to, recover from and adapt to climate change. However, certain city stakeholders’ lack of proactive behavior has resulted in less e ﬀ ective city resilience-building strategies. In this sense, the importance of developing stakeholders’ awareness of climate change in order to ensure proactivity is documented in the literature. However, there is a lack of studies that deﬁne how, when and what should be done to develop stakeholders’ climate change awareness at a city scale. This paper presents a framework to develop stakeholders climate change awareness as a result of a systematic literature review and a co-creation process with the participation of 47 experts through a focus group and a Delphi study. The framework deﬁnes a four-step process and includes nine policies that seek to develop stakeholders’ climate change awareness. The framework concludes determining the responsibilities of each stakeholder by deﬁning the policies they should implement, and the e ﬀ ect one policy might cause on other stakeholders and among policies.


Introduction
Most of the world's population now lives in cities, and it is forecasted that 68% of the population will live in urban areas by 2050 [1]. As cities continue growing, their capacity to deal with foreseen and unforeseen challenges derived from climate change has to adapt accordingly. Climate change presents some differences to other challenges due to its uncertainty in both short and long-term climatic scenarios as well as its magnitude affecting both globally and locally sectors like economy or health [2,3]. In the last decade, the concept of city resilience has been developed and used to enhance cities' capacities in responding to, recovering from and adapting to climate change [4]. Although the literature provides several definitions for the concept of "city resilience" or "climate resilience" [5][6][7], in this study, we follow the one given by the European project Smart Mature Resilience (SMR) as it encapsulates the definitions given by United Nations Office for Disaster Risk Reduction (UNISDR) [8] and Intergovernmental Panel on Climate Change (IPCC) [9] and provides a holistic perspective on the concept. SMR defines city resilience as "the ability of a city to resist, absorb, adapt to and recover from acute shocks and chronic stresses to keep critical services functioning, and to monitor and learn from on-going processes through city and cross-regional collaboration, to increase adaptive abilities and strengthen preparedness by anticipating and appropriately responding to future challenges" [10].
Cities recognize their role as contributors to climate change and they are attempting to reduce their impact on the environment while simultaneously building resilience to the irreversible effects of climate change and transforming themselves into sustainable cities [11]. Several studies and projects that develop conceptual models and frameworks to build city resilience and become a sustainable city can be found in the literature [10][11][12][13][14][15][16][17][18][19][20]. The aim of these studies is to define the attributes and actions needed to improve the resilience level of cities. Building city resilience depends on different factors such as city's governance, social system, economic system, natural resources and infrastructures of a city [21,22]. Not considering all these factors while building climate resilience leads the developed strategies to be less effective than planned [23,24]. In this research, we focus on cities' governance, more precisely in the effect of certain city stakeholders' passive behavior. In practice, the lack of certain city stakeholders' proactive behavior while city resilience is operationalized has resulted in wasted effort or maladaptation [25,26]. City stakeholders are defined as "any individual, group or organization within a city who can affect or can be affected by the resilience-building process" [27]. Given this definition, city stakeholders can be clustered in three groups: public entities (local governments, governmental associations), private companies (the private sector, academia or professional associations) and community groups (regional and civil society organizations, donors) [28].
Hence, ensuring stakeholders' proactivity is crucial in order to effectively operationalize city resilience [25,26]. In this vein, developing city stakeholders' awareness in the context of climate change has been demonstrated to be an effective way to transform behavior from being passive to proactive [29][30][31][32][33]. Awareness has the ability to enhance stakeholders' understanding of climate change, collaboration among stakeholders, commitment to deal with the challenge and proactivity, which leads to increased city resilience operationalization and decreases the effects of climate change [30,31,33]. Understanding climate change will increase the perception of the risks derived from climate change. Both proactivity and commitment will increase stakeholders' action and efforts allocated to deal with climate change. Finally, collaboration among stakeholders will boost both understanding and commitment as participating with other stakeholders will increase the knowledge and the actions to face climate change. However, there is a lack of clear definitions with regard to how awareness is developed, what steps are involved in the awareness-development process, what actions are needed to go from one step to another, and who should lead these actions [29,34,35]. In this research, we understand awareness not only as the first step prior to developing any resilience-building process [36,37] but also as a requirement that must be met during the resilience development process because it drives transformation [38,39].
Our research seeks to answer the following research questions: (RQ1) How should the process to develop city stakeholders' climate change awareness be? (RQ2) How should the participation of city stakeholders be? (RQ3) What actions should city stakeholders implement in order to build awareness? To that end, this paper presents a framework with the objective of contributing to the lack of climate awareness development process at a city scale. In detail, the paper aims to show the methodology used to define the framework, the activities carried out during the followed co-creation process and the results and conclusions obtained.
The paper is structured as follows. First, a state of the art concerning city resilience and the development of climate change awareness is presented. Then, Section 3 describes the followed mixed methods methodology. Results are presented in Section 4, and in Sections 5 and 6, the contribution made by the results and the main conclusions of the paper are discussed.

City Resilience and Stakeholders' Awareness of Climate Change
In the last decade, an effort has been made to build climate resilience at a city scale [1,[10][11][12][13][14][15][16][17][18][19][20]40]. For example, the Sendai Framework provides cities with four priorities and seven goals to prevent and reduce known and unknown risks and gain city resilience [17]. However, the strategies for building climate resilience have often fallen short, and the reason for this to happen has been studied in the literature [21,23,[41][42][43][44][45][46][47]. Overall the studies highlight that the complexity of having to address cities as a complex system of systems hamper the proper operationalization of city resilience as all the systems composing a city (governance system, social system, economic system, natural resources and infrastructures system) need to be managed and taken into account [21,22]. In general, studies have been more focused on developing technical studies concerning critical infrastructures, the economic system or how to manage the natural resources in urban areas. Indeed, there is a lack of studies that analyze the social factors concerning certain city stakeholders being passive towards building city resilience [48].
In this context, recent studies highlight the positive effect of developing awareness to enhance stakeholders' proactivity, improve city resilience operationalization and become a sustainable city [11,[49][50][51]. For example, Olazabal et al. [51] evaluated four cities' climate resilience based on the framework they have defined and highlighted the weaknesses and strengths of each case. One of their main conclusions was that the four cities need to develop stakeholders' climate change awareness to improve their city resilience level. Table 1 summarizes the most used and referenced studies about the definition of climate resilience frameworks referenced at the beginning of this section, and how developing stakeholders' awareness is embedded within these frameworks. All in all, the frameworks highlight the importance of awareness as it enhances the understanding, participation, commitment and collaboration level of stakeholders concerning the challenge of climate change.
Different definitions can be found in the literature for climate change awareness that follow the conclusions made in Table 1. Although there are some differences depending on the context-awareness is used, most of them define awareness based on the following attributes: perception, understanding, willingness to act, commitment and collaboration (see Table 2). In order to become aware of climate change, it is important to perceive that climate change presents a problem for the sustainability of our planet. Understanding the risks and the impacts derived from climate change is key to perceive the need to deal with it [8,[52][53][54][55][56][57]. Furthermore, awareness is materialized in the willingness to act and participate in taking measurements to face the challenges posed by climate change [8,48,52,[54][55][56][57][58][59][60]. This participation depends on the commitment level, the higher the commitment level the better the quality and efficiency of the implemented efforts to deal with climate change [48,[57][58][59][60][61]. Finally, acting collaboratively helps improving the awareness level since facing the problem in a cooperative way enriches the knowledge about the problem, and the commitment to face climate change [57,59,61]. In this research, we follow the definitions given in the literature and define developing climate change awareness as the process to perceive and understand climate change as a hazard as well as to increase the willingness of taking action in a committed and collaborative way to adapt and face the challenges of climate change. Table 1. Resilience frameworks and awareness contribution.

Author
Resilience Definition Awareness Development [18] -Four dimensions to define city resilience-building process: (1) Leadership and Governance: leaders commitment; engagement of relevant city stakeholders; multi-governance collaboration; learning.
Enhances the committed participation of stakeholders. (2) Preparedness: anticipation of expected and unexpected disasters; increase the flexibility and adaptive capacity; education and training programs.
(3) Infrastructure and Resources: increase the robustness and resistance level of infrastructures; reliability of critical infrastructures. (4) Cooperation: collaboration (partnership) at the local, regional, national and international level; communication.
Enhances the collaboration and alliance creation. [17] -10 essentials to define city resilience-building process: (1) Organize for disaster resilience.
(2) Identify, understand and use current and future risk scenarios. -Four dimensions to explain city resilience: (1) Leadership and Strategy: promote leadership and management; empower stakeholders, foster long-term planning.
Enhances the knowledge that empowers stakeholders.
(2) Health and Wellbeing: meet basic needs; support livelihoods and employments; ensure public health services.
(3) Economy and Society: promote cohesive and engaged communities; ensure social stability, security and justice; foster economic prosperity.
Enhances the co-responsibility and engagement of stakeholders. (4) Infrastructure and Environment: ensure continuity of critical services; provide reliable communication and mobility; provide and enhance natural and manmade assets. [13] -Two city resilience planning stages (assessment and readiness) composed of six characteristics: (1) Attention to the current situation.
(2) Attention to trends as future threats.
Enhances the knowledge and understanding of future threats.
(3) Ability to involve the public. Enhances the collaboration and commitment of stakeholders. (4) Ability to initiate action.
Enhances the willingness to act. (5) Ability to set goals. (6) Ability to learn from previous experience.
Enhances the understanding of hazards. [14] -Four concepts to understand city resilience: (1) Vulnerability: demography analysis; informal urban spaces; uncertainty; spatial distribution. Enhances the understanding of hazards.
Enhances the increase of knowledge and understanding of hazards.  However, even if the relevance of developing climate change awareness has been demonstrated, procedures for developing awareness are still lacking, which hampers the ability to properly develop awareness [29]. In this vein, few studies can be found in the literature that define policies to develop city stakeholders' climate change awareness when building climate resilience. For instance, Lu and Stead [13] assessed the way resilience is addressed in policymaking for the specific case of Rotterdam. Their study revealed that Rotterdam defined a set of policies to develop community groups climate change awareness through educational sessions. However, the rest of the city's stakeholders are not considered, and the policies are limited to the field of education and not framed within a procedure.
On the other hand, Siriporananon and Visuthismajarn [62] carried out a study in the city of Hat Yai to study the success of the policies defined by the Asian Cities Climate Change Resilience Network. The study concludes that even if policies concerning city stakeholders' awareness-development can be found, the policies are not as effective as planned. The study reveals that policies to develop awareness should be balanced between self-interest and public interest, have private participation and not only be focused on carrying out workshops and share knowledge among stakeholders.
Moreover, Pietrapertosa et al. [11] analyzed the program presented by the European Commission in 2009 and concluded that even though the importance of developing stakeholders' climate change awareness is highlighted, the suggested policies are focused on building climate resilience and not on developing climate change awareness.
Hence, the literature addresses the need to develop city stakeholders' climate change awareness in order to contribute to the city resilience building process. However, a process that guides cities in the awareness-development process and the participation of stakeholders has not been specified. In this paper, we present a framework for developing city stakeholders' climate change awareness. In contrast to the above studies, this research is not only focused on community groups' climate change awareness, but it also takes into account the participation of private companies and public entities when developing climate change awareness. The framework defines an ideal process for developing awareness to understand and act in a committed and collaborative way to deal with climate change. Moreover, the framework specifies the policies that each stakeholder group should implement, how these policies are interrelated to each other, as well as their effect among stakeholders and their implementation order in the awareness-building process.

Research Methodology
The research methodology employed in our study to develop the framework consisted of three phases (see Figure 1). A combination of different methods with different characteristics was applied. Theoretical methods such as systematic literature review were complemented with participatory methods such as focus group and Delphi study. On the one hand, the systematic literature review helped to identify the processes to build awareness based on the existing scientific literature. On the other hand, the focus group was on-site with regional experts on the topic, and it facilitated the debate and knowledge exchange among the experts. Moreover, the Delphi study was carried out using online questionnaires to enable the involvement of international experts.
As a result, the systematic literature review defined the steps in the process of building city stakeholders' climate change awareness. In order to complement this theoretical process to build stakeholder climate awareness with the tacit knowledge in mind of the participating ten experts, the focus group resulted in a list of policies for boosting the process. Finally, to analyze the list of policies and gather the knowledge from international experts, a three-round Delphi study with 37 participants (different from the ones participating in the focus group) was carried out.
The co-creation process followed in this research involved a total of 47 experts in the field representing public entities, private companies and community groups. The aim of carrying out first a systematic literature review followed by two participatory methods like the focus group and the Delphi study was to complement the theory found in the literature with the expertise of the 47 participants.

Phase 1: Systematic Literature Review
The systematic literature review was carried out using the Scopus electronic database. Only academic papers published in scientific journals were selected as a way to ensure a more standard set quality. Specifically, only journal articles published after 2000 and with at least two citations were selected. The fields of medicine, biology, earth sciences and agriculture were not taken into account due to contributing to the challenge of climate change from a technical-scientific perspective that is out of the scope of this research. This research is focused on the socio-technical aspects of the awareness development process. Moreover, three keywords comprised the base of the searches to ensure content related to the aim of the research would be found. The keywords were: "awareness" AND "Climate Change" AND "resilience".
Once the initial list of relevant articles was obtained, all the abstracts were read to ensure that the papers contributed to the definition of the awareness-development process. Then, the full articles were thoroughly overviewed. During this process, papers were dismissed at different stages for reasons like being too technical concerning the applied methods such as mathematical models to analyze and predict the effects climate change might cause or too focused on the recovery phase of climate change related crises. Note the article focuses on the passive behavior of stakeholders when dealing with climate change and aims to contribute to the lack of stakeholders' climate awareness that is hampering the proper operationalization of the climate resilience-building strategies. Finally, duplicate papers were removed, leaving a total of 48 papers from the initial 98. Iturriza et al. [63] explain in more detail the systematic literature review carried out in this research. The resulting fourstep procedure to develop awareness of climate change is presented in Section 4.1.

Phase 2: Focus Group
A focus group is defined as "a research technique that collects data through group interaction on a topic determined by the researcher" [64]. Focus groups generally involve four to twelve participants with different characteristics and backgrounds [65].
The focus group presents several benefits to our research. On the one hand, carrying out focus

Phase 1: Systematic Literature Review
The systematic literature review was carried out using the Scopus electronic database. Only academic papers published in scientific journals were selected as a way to ensure a more standard set quality. Specifically, only journal articles published after 2000 and with at least two citations were selected. The fields of medicine, biology, earth sciences and agriculture were not taken into account due to contributing to the challenge of climate change from a technical-scientific perspective that is out of the scope of this research. This research is focused on the socio-technical aspects of the awareness development process. Moreover, three keywords comprised the base of the searches to ensure content related to the aim of the research would be found. The keywords were: "awareness" AND "Climate Change" AND "resilience".
Once the initial list of relevant articles was obtained, all the abstracts were read to ensure that the papers contributed to the definition of the awareness-development process. Then, the full articles were thoroughly overviewed. During this process, papers were dismissed at different stages for reasons like being too technical concerning the applied methods such as mathematical models to analyze and predict the effects climate change might cause or too focused on the recovery phase of climate change related crises. Note the article focuses on the passive behavior of stakeholders when dealing with climate change and aims to contribute to the lack of stakeholders' climate awareness that is hampering the proper operationalization of the climate resilience-building strategies. Finally, duplicate papers were removed, leaving a total of 48 papers from the initial 98. Iturriza et al. [63] explain in more detail the systematic literature review carried out in this research. The resulting four-step procedure to develop awareness of climate change is presented in Section 4.1.

Phase 2: Focus Group
A focus group is defined as "a research technique that collects data through group interaction on a topic determined by the researcher" [64]. Focus groups generally involve four to twelve participants with different characteristics and backgrounds [65].
The focus group presents several benefits to our research. On the one hand, carrying out focus groups enables us to capture information that will help to better manage the process of knowledge development [66]. On the other hand, the focus group methodology has the capacity to gather a broad range of viewpoints and all the possible crucial particularities that come with participants with different expertise [66,67]. In addition, focus groups help the participants to explore and clarify their own views in relation to others [68].
Consequently, we chose the focus group method as it facilitates to gather the information we needed to define the preliminary list of policies.

Participants
Our focus group had a total of ten participants, all of them from the city of San Sebastian in Spain and with a different field of expertise (see Table 3). San Sebastian is a coastal city located in the north of Spain, with a total of 186,000 inhabitants. In recent years, the city has experienced unusual heatwaves and heavy rains that have caused unexpected floods and high, violent sea tides that have flooded the streets with sea water and damaged them. Consequently, the city has been working on their climate change strategy and proactively taken part in European projects to build city resilience [10]. Hence, given their experience and current work in building city resilience, we chose the city of San Sebastian to carry out the focus group session assuming that its stakeholders have a great deal of embedded knowledge to contribute to the first phase of the co-creation process. In detail, public entities participants were selected from the city of San Sebastian in Spain due to their active participation in similar studies related to enhancing city resilience and their work towards the challenge of climate change. Private companies' experts were chosen due to the given service, having participants from consultancies that seek to apply sustainable habits or climate change research centers. Finally, community groups were chosen based on their aim as a group such as working with vulnerable population or increasing climate change-related dialogues. Hence, the stakeholders who participated in the focus group were involved in dealing with climate change so that they could contribute to the focus group by sharing their experience. The aim of the focus group was to define a preliminary list of policies to develop climate change awareness and study their effect in the awareness-development process.
The ten participants were divided into three different groups according to the type of stakeholder they represented (public entity, private company and community group). Firstly, participants were asked to brainstorm the main policies that each stakeholder might lead to developing awareness on a citywide scale. Next, from the policies listed, the three most relevant policies for developing stakeholders' awareness were selected by each group to carry out a deep analysis of the three policies.
The participants were then mixed and reconfigured as two groups, such that each group had participants from public entities, private companies and community groups. Taking into account that the aim of the second activity was to determine the effects that implementing the policies would have on city stakeholders, having a balance of stakeholder types in both groups was essential.
However, as the focus group was conducted with ten city stakeholders, the results were limited to the expertise of the participants. Further research was needed not only to define more global and transversal policies but also to study more the list of policies composing the framework in greater depth. To that end, a Delphi study was carried out with a wider group of international participants.

Phase 3: Delphi Study
The Delphi method is a systematic and iterative survey process for structuring a group communication process in order to obtain consensus about a complex problem under study [69]. In fact, the Delphi method provides access to the opinions of multiple experts from different regions at a reasonable cost [70]. There are four key elements that characterize this methodology; anonymity, iteration, controlled feedback and statistical aggregation of group response [71]. First, anonymity encourages experts to express their opinions freely without fear of disagreeing with others [72]. Second, the interactive structure divided into the different rounds allows participants to reconsider their answers based on the information they receive from other experts [73]. Third, the controlled feedback process provides the opportunity for participants to justify or change their opinion based on other participants' comments [74]. Finally, the Delphi study allows to carry out a quantitative analysis of the gathered results [72].
Due to the above-mentioned characteristics, we chose to use the Delphi method as it facilitates reaching a consensus during the validation process of the framework [73]. In fact, the Delphi study has been successfully used in the field of city resilience as a method to obtain consensus and validate preliminary results [18,75].

Participants
Selecting the participants is a critical step in the Delphi study since the quality of the answers received will condition the results of the study [76]. It is suggested that between 15 and 30 heterogeneous experts should be carefully selected to ensure the quality of the feedback received from the Delphi process [77].
A total of 70 experts were invited to participate in our online three-round Delphi study. Thirty-seven experts accepted the invitation. They included men and women from six different nationalities (Spanish, German, Danish, English, Greek and Italian) and different expertise backgrounds. However, only 29 of these initial 37 participants were able to complete the two surveys composing the Delphi study due to time constraints, giving us a final participation rate of 41.4%. Public entities had the highest participation rate (41%), followed by private companies (32%). Community group participation was more difficult to achieve (27%), as several invitees from this group stated that they were not familiar with the methodology and did not feel comfortable responding to the online questionnaire.

Delphi Study Structure
The objectives of the Delphi study were: (1) to validate the nine preliminary policies defined in the focus group; (2) to study how the implementation of one policy affected another; and (3) to specify, if possible, the ideal implementation order of the policies based on the previously defined awareness-developing process. Given the objectives, the Delphi study was designed to use two different questionnaires. Firstly, there was a first round or explanatory round to study the suitability of the nine actions in developing city stakeholders' climate change awareness as well as the existence of potential precedence relationships among the nine policies. This was followed by a second round that consisted on a feedback process in which the participants had the chance to analyze and change their answers based on other participants' responses. Finally, a third round or confirmatory round evaluated the results obtained in the first round as well as integrated the policies into the four-step procedure to develop awareness of climate change. A fourth round was not necessary as the participants got to a high level of consensus in the third round of the Delphi study (see Sections 4.2 and 4.3)

Results
In this section, the framework resulting from the co-creation process is presented. The following sub-sections describe the four-step awareness development process, the nine policies, the effects of the policies and their integration in the four-step process.

Awareness-Development Process
Based on the 48 papers selected from the systematic literature review, a four-step gradual procedure that every city stakeholder goes through when developing climate change awareness has been defined, starting with the lowest level of awareness, "Step 1-Passive", and ending with the highest level, "Step 4-Synergies" (see Table 4). Table 4. Awareness-development process defined in the framework.

AWARENESS-DEVELOPMENT PROCESS Step1-Passive
Step2-Static Step3-Proactive Step4-Synergies In the first step, city stakeholders still have doubts about the existence of the challenge of climate change. In fact, their behavior is passive when it comes to climate change. In order to advance, the first requirement is to realize that there is a need. The question that defines this step would be "do we perceive the challenge of climate change?" Experience regarding climate change hazards is key in this aspect, as having suffered a climate change event makes stakeholders realize that what has been done is not enough to face climate change, and they start to perceive the existence of the problem and the need to act [78,79].

Step 2: Static
In the second step, awareness has started to develop as stakeholders perceive and begin to understand the vulnerabilities and hazards related to climate change, yet they do not act accordingly. Therefore, during this step, it is essential to understand why city stakeholders engage in passive behavior and design incentives that make them more aware of climate change-related issues and foster a transformation in behavior. However, someone needs to lead the process of behavior transformation and incentives. As Ban Ki-Moon stated in 2009, leaders should set the example, yet these leaders can be of any kind (public, private or community group); the only must is to ensure leaders are committed, as commitment will make them lead and act in a proactive way [80]. The question in this second step would be "do we want to make a change?"

Step 3: Proactive
In the third step, willingness to act has been achieved as a consequence of the incentives and actions in step 2 but only at an individual level. Each stakeholder acts separately, not considering possible collaborations. Each stakeholder is committed but acting separately, without considering possible collaborations. The question in this step would be "what can we do to improve this situation?" The stakeholders recognize the vulnerabilities and hazards around them and act accordingly in facing them. Even if incentives are applied in this step, actions are carried out altruistically because each stakeholder perceives personal benefits and good in doing so [62,81]. The commitment in this third step is seen as an opportunity to initiate a change and effectively face the problem of climate change.

Step 4: Synergies
In the fourth step, both awareness and proactive behavior have been achieved in a multilevel way. This time, city stakeholders act collectively against climate change's short-and long-term effects. In this step, stakeholders seek both personal convenience and a universal good that might not directly result in personal benefits. As a consequence, the behavior in this last step means being in a process of continuous change as existing barriers disappear, silo-thinking is eliminated and stakeholders' communication increases [79,82]. As stakeholders have experienced the whole process, new daily habits are introduced [83], new collaboration networks are created, and new needs appear. The question in this fourth step would be "who else can be part of the group?" In this last step, the effectiveness of the plans increases as city stakeholders have not only improved communication among them but also are committed and acting jointly [35,84]. The newly created collaboration networks facilitate communication [85], and this, in turn, makes it easier to implement plans and make them more effective.

Policies
As a result of this study, nine policies for developing city stakeholders' climate change awareness were defined. Developing awareness is a transversal and collaborative process that requires the engagement of all stakeholders [26]. Because every stakeholder has a role when developing climate change awareness, this study defines a set of three policies for each stakeholder.
The policies were first defined in the focus group and then iterated in the Delphi study until the given definitions were accurate: most of the participants agreed or strongly agreed with the given policies' definitions (see Table 5). In both rounds of the Delphi study, a five-level Likert-type scale (strongly agree-5, agree-4, neutral-3, disagree-2, strongly disagree-1) was used to evaluate the definitions. Participants were also able to add comments concerning the policies. Both the average and the standard deviations of the given answers were calculated to indicate the level of consensus among participants [86]. The criterion used to determine there was a consensus on the definitions was to achieve an average value equal or higher to four to ensure a high agreement level about the definitions and a standard deviation lower than one in order to confirm the low variability on the responses. Table 5 summarizes how after making the suggested changes concerning the used vocabulary or missing concepts, in the second round of the Delphi study, the agreement level increased in all the cases, and the standard deviation decreased, indicating that consensus was achieved. In detail, in the first round, both PU1, development of norms and sustainable procurements, and PR3, development of new technologies for new business models, were not meeting the criterion having an average agreement level below four. In both cases, participants suggested adding missing concepts such as "sustainable procurements" and "new business models" that are key in the definition of the policy (see Table 6).  Table 6 concludes with the final definitions of the policies, classified according to the city stakeholder that leads each policy: Public entities lead the implementation of the policies denoted PU1, PU2 and PU3; private companies lead the implementation of the policies denoted PR1, PR2 and PR3, and community groups lead the implementation of the policies denoted CG1, CG2 and CG3.
The policies were tested in the city of Kristiansand, Norway. We considered Kristiansand to be an interesting city not only because of the city's past experiences related to climate change such as extreme unpredicted floods but also because Kristiansand has actively been working on developing awareness of climate change through European projects. In fact, Kristiansand is part of the European Covenant of Mayors (https://www.globalcovenantofmayors.org/cities/kristiansand/) and the Smart Mature Resilience (https://smr-project.eu/kristiansand/) project and has developed actions like the definition of a framework for efficient and timely communication between responders, industries and the public. Having a city that has already started to develop awareness was convenient so that they can evaluate the nine policies defined in the framework. Twelve interviews were carried out with public entities, private companies and community groups to gather the information concerning the policies and to what extend they were implemented in Kristiansand. Table 6 summarizes the evidences found for each policy and the identified barriers when implementing them in practice.
Overall, Kristiansand's case showed that even if public entities allocate efforts to build a climate change-resilient city and transform Kristiansand into a sustainable city, the allocated efforts are not effective if private entities are not engaged in adopting new habits and community groups do not participate. Actually, the interviews concluded that public entities in Kristiansand are the most aware stakeholders that seek to increase both private companies and community groups awareness. Public entities are followed by community groups that understand and perceive climate change, yet their active participation has not been achieved. Finally, private companies are the less aware stakeholder, which makes sense taking into account the industrial history of the city that is used to business-as-usual habits. Hence, it can be concluded that the policies were not only boosting the awareness development process but also that implementing one policy affects other policies' implementation. Therefore, the relationships among policies should be analyzed to better understand the climate-awareness development process.  [80,87] This policy determines the adoption of existing norms, standards and sustainable procurements but also the development of new norms and sustainable procurements in order to develop awareness and increase city resilience toward climate change. • Working on new legislation and implementing sustainable procurements to boost private companies.  [62,89] This policy consists of carrying out transparency actions such as publishing companies' emissions, sharing consumption data, communicating information about sustainable actions taken, financial movements or sharing problems and barriers encountered in the process of developing sustainable habits.
• Progress. Consultant companies can be found in the city that aim to help other companies to adopt this policy by setting incentives such as performance indicators.
• Private companies, mainly from the oil and gas industry, need to be engaged in these types of policies.  [62,80] This policy is related to the adoption of new business values such as the adoption of an inclusive and sustainable economy, low carbon emission practices or sustainable habits like recycling, in order to pursue sustainable production and resource consumption.
• Progress. Consultant companies can be found in the city that aim to guide companies in the adoption of new values.
• Not all the private companies understand the relevance of having a new business model that puts an end to the "business as usual" concept.

•
Lack of the required time and efforts that are needed to be aware.
PR3: DEVELOPMENT OF NEW TECHNOLOGIES FOR NEW BUSINESS MODELS [80,90] This policy consists of developing technologies to transform private companies into new business models that seek profit while also generating social benefits, such as social vulnerability reduction. For example, investing in sustainable technology makes companies reduce their emissions, which reduces pollution and increases social welfare.
• Progress. The process company can be found in the city that has developed a special technology to reuse the waste produced in their production line.
• There is still work to do in PR1 and PR2, which hampers proper implementation.
COMMUNITY GROUPS CG1: ORGANIZE EVENTS AND WORKSHOPS [91,92] This policy consists of organizing events and workshops to discuss the main concerns and problems derived from climate change like decreased society welfare due to the effects of climate change, vulnerabilities due to new climate scenarios or the lack of knowledge concerning what to do or how to act in a sustainable way.
• Working on holding more events and workshops to engage citizens and other stakeholders.
• High percentage of participants tend to be the same. A part of society still needs to be engaged.
CG2: ORGANIZE CAMPAIGNS [80,93] This policy consists of organizing campaigns through social networks, media or physical events with the objective to create a positive impact on the three stakeholder types' awareness but with greater emphasis on community groups.
• Progress. Working with influencers made them increase the impact of the campaign.
CG3: DEMAND A CHANGE [39,88] This policy consists of demanding public entities and private companies' change the way they face the challenge of climate change. To do so, demanding action and transparency concerning their decisions and emissions is posed in this policy in order to ensure good sustainable practices are adopted and enabled.
• Progress. Implementing CG1 enables community groups with the channels to demand a change.
• Citizens in Kristiansand happen to be passive, even though new movements are starting to happen.

Effects and Relationships between Policies
To define the relationships among policies, first, the results obtained from the focus group were used, followed by the results obtained from the two rounds of the Delphi study. Analyzing and defining the relationships among the policies was key in order to understand the participation of city stakeholders while developing climate change awareness as well as to understand the dynamics of the process of developing stakeholders' climate awareness.
In the focus group, the participants defined the main effects the policies might have on stakeholders. Participants agreed that the policies led by public entities have a direct effect on both private entities and community groups. The policies oriented towards engaging private companies increase companies' knowledge through mandatory norms and provided tools. For example, PU1: Development of norms and sustainable procurements, has an effect on private companies' habits in that it leads to the reduced emission of greenhouse gases, which is a contribution to dealing with climate change. The policies related to having an effect on community groups are focused on sensitization activities such as talks, workshops and conferences, where the effect is on drawing citizen's attention and perception towards the challenges of climate change.
In the case of the policies led by private entities, we concluded that private companies have identified policies that present climate change as an opportunity for a company to grow, become a reference in the market and differentiate from the competence. In fact, policies like PR1: Corporate responsibility actions or PR2: Develop new values for new business models, emphasize the importance of the company's reputation and public image, as well as the company's being a reference in the field. In contrast to the other proposed policies, the ones defined by private companies mainly affect their own company.
Finally, policies lead by community groups affect not only community groups but also public entities and private companies. For example, CG3: Demand a change refers to strikes similar to the one led by the climate activist Greta Thunberg. These strikes have caused several effects, such as the reaction from the UK Environment Department (public entity) that has recently highlighted the need to act, the increase in attention from some energy sector companies (private companies) that are expecting to see the changes the strikes suggest, as well as millions of people (community groups) that have got out to the street to demand change every Friday [94].
Based on the results obtained in the focus group, we studied the effects of the policies in greater depth by specifying the relationships among the policies through the Delphi study. To that end, in the first round of the Delphi study, participants were asked to mark what policies are affected when one specific policy is implemented. Only the relationships marked by at least half of the participants were selected in the first round, resulting in a total of 32 relationships. In the third round of the Delphi study, experts were asked only about the 32 relationships identified in the first round. In this case, we have followed Diamond et al. [95] in considering only relationships identified by 75% of the participants in order to end up with the final set of relevant relationships. After applying the 75% threshold, we ended up with a total of 14 relationships. Table 7 shows the data for the whole process.
The resulting 14 relationships in the Delphi study confirm the conclusions obtained in the focus group. Public entities seek to have an effect on the policies of both private companies and community groups. However, the policies led by private companies mainly affect other private companies' policies. Community group-led policies, however, are the only ones that have an effect on all three city stakeholders.

Integrating Policies in the Awareness-Development Process
Once the relationship among policies was analyzed, an effective order of implementation was defined in the third round of the Delphi study in order to ensure the relationships among the policies were considered. This section describes the integration of the four-step gradual process defined in Section 4.1 and the nine policies presented in Section 4.2.
In the third round of the Delphi study, participants were asked to specify the step in which the policies should be implemented in order to be effective. The participants concluded that the same policy could be implemented in more than one stage in order to boost the awareness-development process. Table 8 shows in which steps the policies should be implemented in order to be effective. The table specifies the percentage of experts who think the policy is significant to be implemented in that stage. For example, having 74% of participants agree that PU1 should be implemented in Step 2 means that 74% of participants stated PU1 is significant to be implemented in Step 2. Note that we considered the implementation of a policy in one step when more than 50% of the participants stated so. As Table 8 summarizes, some policies are implemented during the four steps while others are only suggested to be maintained during two steps. In this sense, the identified nine policies have been defined to have both an effect in the short and long term during the process [54]. Hence, there are specific policies like CG2: Organize events and workshops that should be implemented on a specific step of the climate change awareness development process while others such as PU3: Sensitization actions are defined in a more strategic level and need to be continuously implemented in the process. Table 8 also concludes that experts perceive community groups as being the ones who are important in starting the change. Given that these groups are the ones demanding things to be done differently, it is not surprising that their importance is so highly rated in this regard in the first steps. Moreover, participating experts indicated almost all nine policies are implemented between step 2 and step 3. They argued that a greater effort is needed to be done between those two steps to go from static to proactive behavior towards climate change. In this sense, public entities were perceived by the experts to be of great importance, as all their policies are implemented in these two middle steps and affect both community groups and private companies. Finally, experts placed the policies led by private companies in the last steps. This makes sense as the policies led by private companies only affect private companies.

Discussion and Contribution
This research presents a framework for defining the development of city stakeholders' climate change awareness resulted from a co-creation process with experts in the field (see Table 9). The followed methodology enabled us to consider not only the theory found in the systematic literature review but also the tacit knowledge embedded in the minds of the 47 experts who took part in both the focus group and the Delphi study to define specific policies that boost the climate awareness-development framework. This way, this research aims to contribute to the lack of procedures that define how climate awareness should be developed at a city scale. The presented framework answers the three research questions through the description of a four-step development process: (RQ1), the participation of stakeholders; (RQ2), the policies that boost the process; (RQ3), the relationships among policies and their implementation order. The resulting framework has been defined at a strategic level so that cities can assess their awareness level and be guided through the awareness-development process. The steps facilitate cities' evaluation of their current situation, and the policies enable decision-makers to go forward in the awareness-development process. Indeed, any city that aims to develop climate awareness could use the framework to be guided in the process. More precisely, as public entities are both leading the development process and are knowledgeable of the technical concepts used in the framework, the main user, yet not the only one, should be public entities.
Moreover, as described in Step 1 (Section 4.1.1), it is essential that city stakeholders experience real climate change threats in order to initiate the awareness-building process. Having lived real events brings about a clear change in the stakeholders' climate change awareness since they clearly perceive how badly they could be affected if nothing is done. From the second step on, stakeholders' attention towards climate change increases, and they start being more committed and taking actions to reduce climate change impact and become a more sustainable city. Moreover, and closely related to this is the fact that in step 2 and step 3, stakeholders start to get more knowledge about the issue of climate change and have more interest in learning and understanding about this challenge and how to deal with it. In the last step, once the three mechanisms are activated, it is important to ensure that the collaboration and thus the synergies among stakeholders are achieved so that awareness is maintained and reinforced among all stakeholders.
In order to operationalize this process, the policies to be implemented by each city stakeholder have been defined. The interviews carried out in the city of Kristiansand highlighted the importance of analyzing the existing interrelationships among the polices and including them in the framework. The framework defines public entities as proactive leaders when dealing with climate change. They seek to guide both community groups and private companies through the awareness-development process (see Table 9). In fact, the three policies led by public entities are related to implementing mandatory actions, making knowledge accessible and generating attention about climate change. The Delphi study corroborates this conclusion as the policies led by public entities affect both private companies and community groups and are of high importance in the awareness-development process. In addition, the policies led by public entities are key in the middle steps, during the transformation from being static to being proactive.
The framework also addresses the important role community groups play when developing city stakeholders' climate change awareness. In fact, CG3: demand a change is the first to be implemented. The results of the focus group define community groups as an active resource in society, one that is able to deal with climate change and plays the role of demanding a change and participate in the change. This conclusion is coherent with the results in the Delphi process that related the policies led by community groups with all the stakeholders, meaning community groups have a large effect on the implementation of other policies.
Finally, the need to have the participation and engagement of private companies is reflected in the framework too. As a result of the focus group, private companies can be considered as the actors who are responsible for the key products and services of society. Consequently, their active participation is essential in order to get stakeholders to transform and develop awareness in order to be climate change resilient. More specifically, in the co-creation process, the policies identified by private companies present climate change as an opportunity for a company to grow. In fact, the results of the Delphi study conclude that applying the policies led by private companies mostly affects the companies.

Conclusions
The importance of developing city stakeholders' climate change awareness to ensure more effective implementation of cities' climate change resilience-building strategies has been addressed. Developing awareness is one of the key factors while developing city resilience to climate change in order to ensure the developed strategies are as effective as planned. City stakeholders need to be proactive when it comes to building climate resilience. Adopting a proactive behavior facilitates the operationalization of existing climate-resilience strategies and consequently boost cities to shift into sustainable urban areas. As stated in Table 1, developing stakeholders' awareness not only makes them understand and perceive the challenges climate change poses but also increases the willingness to act and participate in facing these challenges. Awareness increases stakeholders' commitment level; the higher the commitment level, the better the efficiency of the implemented efforts to deal with climate change [48,[57][58][59][60][61]. Finally, the collaboration among the different stakeholders enhances the awareness level since it enriches the knowledge about the problem and the commitment towards dealing with the challenges of climate change [57,59,61]. Nevertheless, even though the positive effects of developing stakeholders' climate change awareness have been noted in the literature, studies in this field have been more focused on developing technical knowledge about climate change than on social factors such as development of awareness.
This paper presented a framework to raise stakeholders' climate change awareness. The article takes a step beyond already existing studies [11,13,62] as it defines a detailed process that guides city stakeholders in developing awareness of climate change. The framework was developed through a systematic literature review and a co-creation process composed of a focus group and Delphi study. The framework defines a process composed of four steps and a list of nine policies that seek to develop stakeholders' climate change awareness at a city scale.
This study concludes by addressing the importance of the three stakeholder groups when developing climate change awareness. In defining a framework with three policies for each of the stakeholders, one of the main conclusions was that not only public entities but also both private companies and community groups must participate when developing climate change awareness.
The policies listed might be considered as being defined at a strategic level. However, our research is a first attempt to not only define a set of policies that considers all the city stakeholders as being important in the awareness-development process but also defines each stakeholders' participation in the process. Moreover, the policies defined have a wider scope than the policies addressed in the literature, as we have defined policies that are related to sustainable procurements and technology innovation rather than being centered only on education and workshops.
Further research is needed to test the theoretical framework in a city. Specifically, the strategic policies defined should be further concretized and particularized in order to guide city stakeholders when operationalizing them. Indeed, the presented framework only shows the theories developed and specifies the participation of stakeholders, but examples could be added to guide stakeholders when operationalizing the framework.