Public Engagement in Energy Research

Public Engagement in Research is a key element in “Responsible Research and Innovation”; a cross-cutting issue in current European research funding. Public engagement can advance energy R&D, by delivering results that are more in-line with society’s views and demands; and collaboration also unlocks societal skills and knowledge. This paper structures the ways to look at engagement, and gives some pointers on how to implement it in energy R&D, with various levels of intensity. The publics to engage with can be citizens, future users, affected persons, but also organisations that represent them. We have selected methods and tools that showcase a broad range of types of engagement that have been applied in The Netherlands or the UK. The cases are grouped based on the role of the researcher in the engagement process. These roles relate to discussing with, consulting, involving, collaborating with and supporting the various publics. This study shows that there is a diversity of tools and methods already available—open to variation, combinations and further developmentthat facilitate the participation of society in energy research. Not only for democratic reasons, but also for instrumental benefits in creating innovations to help solve the Grand Societal Challenge of the energy transition.

towers or by pure market forces on their own, but need a broader knowledge base. One of these grand challenges defined by the European Commission (EC) is the transition to a secure, clean and efficient energy supply. The EC also defined challenges to achieve green transport, take climate actions and make resource use more efficient-including moving towards a bio-economy. Other challenges are found in health, food and in structuring our society [3].
For this paper, we studied how public engagement can be part of energy research and innovation. We aim to inspire those for which this is a new element to their research, and hope to trigger a discussion and more publications on innovative approaches to public engagement in energy research. In no way should the overview given in this study be taken as complete and ultimate; even more so because it is predominantly based on Dutch and UK experiences. Our study forms part of a larger European project, called Engage 2020, which gives a broader overview of current and potential practices in engaging citizens and their organizations in the research and innovation process in general [4]. In a next phase, we aim to discuss the potential for a crossover of methods used in the various disciplines or grand challenges.

Method and Focus
When studying public engagement in research and innovation, one can discern different target groups to engage with, and various phases of the research and innovation process. In our overarching project Engage 2020, we decided to distinguish between the involvement of citizens or civil society organisations (CSOs), when describing various methods for engagement, since the approaches and motivations usually differ. Both citizens and their organizations can also have specific roles, e.g., as (potentially) affected people, as future users, or as employees, which allows for different engagement approaches. An important part of all methods of engagement is the selection of participants. For some engagement methods, representivity is important, for others this is a lesser issue. Sometimes participants are called stakeholders; this can apply to both individuals and organisations. In general, the word stakeholder is used to indicate that they have something "at stake" in the issue at hand. During the process of engagement the way and degree to which stakeholders are involved can change.
In our study, we further distinguish four different levels in the research and innovation process: -The formation of research and innovation policy; - The development of research and innovation programmes; - The definition of a research and innovation project; -Actual research and innovation activities.
These four levels can also call for different methods of engagement. In our paper, we focus on engagement in the latter two levels of research and innovation: defining the projects, and the research and innovation activities themselves. Engagement in policy and programme development is usually done at government and funder's levels, and thus more remote from those involved in doing research themselves.
We obtained our information by two routes: (1) In December 2013, the Engage 2020 consortium set up an international online survey which was used to collect examples of methods or tools being used at the time to engage civil society stakeholders at all levels of the research and innovation process across all grand societal challenges. The consortium partners invited relevant parties in their networks and professional associations to contribute. The majority of these are professionally organising public engagement, such as science communicators, community-based researchers, and technology-assessors. Over 200 questionnaires were filled out over a period of four months. Based on this input and research on fields of practice in engagement by the consortium [5], 57 methods and tools were identified. For all of the methods and tools a specific factsheet was made, based on the information in the questionnaire and additional desktop research. The report on the factsheets was published online in October 2014 [6].
(2) As part of, and in addition to the work carried out in step 1, we obtained specific information on engagement in energy research; with a focus on engagement during project definition or the actual research and innovation activities. Most of this information was gathered from the Dutch language area. The main catalogues used were: -National Academic Research and Collaborations Information System (Narcis) of the Dutch Data Archiving and Networked Services (DANS); -SmartCat, which is the University of Groningen's localized version of WorldCat; -Web of Science.
The keywords used in the initial weeks to gain an overview of the field of work were: citizen science, participatory research, public participation, crowd science, participatory sensing, focus group, discourse analysis, civic science, public engagement, action research, public participation in scientific research (PPSR) and similar words in Dutch.
The final step in the research for this paper was the selection of methods and tools that show a broad range of types of engagement that have been applied in The Netherlands and the UK. Each method is illustrated with a short example of how it was applied in the Dutch or UK energy sector, with a focus on The Netherlands. Some examples were found in literature, others were found or further clarified through professional contacts, and events related to energy and participation that were visited in 2014.
These methods and tools were selected in such a way that they became typical examples for the range of intensity of engagement that the various approaches have; from single-event discussion meetings to full-scale almost collegial co-operation between citizens and professional researchers.

Theoretical Background
Public participation in research comes in many forms. When considering public participation in policy making, the ladder of Arnstein (see Figure 1) is a well-known classification in eight categories [7]. The keywords used in the initial weeks to gain an overview of the field of work were: citizen science, participatory research, public participation, crowd science, participatory sensing, focus group, discourse analysis, civic science, public engagement, action research, public participation in scientific research (PPSR) and similar words in Dutch.
The final step in the research for this paper was the selection of methods and tools that show a broad range of types of engagement that have been applied in The Netherlands and the UK. Each method is illustrated with a short example of how it was applied in the Dutch or UK energy sector, with a focus on The Netherlands. Some examples were found in literature, others were found or further clarified through professional contacts, and events related to energy and participation that were visited in 2014.
These methods and tools were selected in such a way that they became typical examples for the range of intensity of engagement that the various approaches have; from single-event discussion meetings to full-scale almost collegial co-operation between citizens and professional researchers.

Theoretical Background
Public participation in research comes in many forms. When considering public participation in policy making, the ladder of Arnstein (see Figure 1) is a well-known classification in eight categories [7]. Arnstein's ladder-though slightly implicit-suggests that the highest achievable engagement is that where the decision is in the hand of citizens. The bottom two rungs are described as non-participation. In presenting examples of participation in energy research, we avoid the categorization where the top level is the best. There is a range of approaches which may be suitable for a specific situation. There are also large differences between sectors, concerning their experience with engagement. Thus, a small step may actually make a big difference to current research and innovation processes in a specific sector. It simply takes time to develop engaged practices. However, we firmly belief that in doing research, the best results are achieved in co-operating, thus in co-constructing knowledge. Where Arnstein's ladder is appropriate for engagement in political decision making, it is less tailored to engagement in project definition and the actual research and Arnstein's ladder-though slightly implicit-suggests that the highest achievable engagement is that where the decision is in the hand of citizens. The bottom two rungs are described as non-participation. In presenting examples of participation in energy research, we avoid the categorization where the top level is the best. There is a range of approaches which may be suitable for a specific situation. There are also large differences between sectors, concerning their experience with engagement. Thus, a small step may actually make a big difference to current research and innovation processes in a specific sector. It simply takes time to develop engaged practices. However, we firmly belief that in doing research, the best results are achieved in co-operating, thus in co-constructing knowledge. Where Arnstein's ladder is appropriate for engagement in political decision making, it is less tailored to engagement in project definition and the actual research and innovation process, where in fact "partnership" would seem the best: research is co-designed, combining the "best of both worlds".
A similar approach is found in the "IAP2 public participation spectrum", published by the International Association for Public Participation (IAP2), which describes public participation in five different categories, depending on the "power balance" between the policy maker and the citizen, or as they state, the "level of public impact" [8]:

1.
Informing: To provide the public with balanced and objective information to assist them in understanding the problem, alternatives, opportunities and/or solutions; 2.
Consult: To obtain public feedback on analysis, alternatives, and/or decisions; 3.
Involve: To work directly with the public throughout the process to ensure that public issues and concerns are consistently understood and considered; 4.
Collaborate: To partner with the public in each aspect of the decision including the development of alternatives and the identification of the preferred solution; 5.
Empowerment: To place final decision-making in the hands of the public.
In the structure we developed, adapted from the sources above, we will use the "intensity of engagement" as a descriptor, without judgement on what is "best". However, we will not include traditional, top-down ways of informing people about research results, since there is no engagement there. This means that this form of communication does not deliver to instrumental motives for engagement, nor does it fully deliver to the democratic motives for engagement.
Our description of categories, as given in Figure 2, is based on the role of the research organisation and the process they are mainly facilitating; the initiative for the engagement process shifts from the researchers towards the civil society from left to right. The intensity of the engagement is highest when there is a collaborative research. In all other forms, the participation of civil society or that of the researchers is less. The selection of the labels we have used to describe certain types of engagement has been challenging and is understandably open for some debate. In Table 1 we explain these. For each of the categories listed, examples will be given of projects in The Netherlands or abroad which have used various methods of engagement. The actual engagement methods applied can be different for the project definition phase and the actual research and innovation activity (similarly, they can be different for the levels of science policy making or research programme development). innovation process, where in fact "partnership" would seem the best: research is co-designed, combining the "best of both worlds". A similar approach is found in the "IAP2 public participation spectrum", published by the International Association for Public Participation (IAP2), which describes public participation in five different categories, depending on the "power balance" between the policy maker and the citizen, or as they state, the "level of public impact" [8]: 1. Informing: To provide the public with balanced and objective information to assist them in understanding the problem, alternatives, opportunities and/or solutions; 2. Consult: To obtain public feedback on analysis, alternatives, and/or decisions; 3. Involve: To work directly with the public throughout the process to ensure that public issues and concerns are consistently understood and considered; 4. Collaborate: To partner with the public in each aspect of the decision including the development of alternatives and the identification of the preferred solution; 5. Empowerment: To place final decision-making in the hands of the public.
In the structure we developed, adapted from the sources above, we will use the "intensity of engagement" as a descriptor, without judgement on what is "best". However, we will not include traditional, top-down ways of informing people about research results, since there is no engagement there. This means that this form of communication does not deliver to instrumental motives for engagement, nor does it fully deliver to the democratic motives for engagement.
Our description of categories, as given in Figure 2, is based on the role of the research organisation and the process they are mainly facilitating; the initiative for the engagement process shifts from the researchers towards the civil society from left to right. The intensity of the engagement is highest when there is a collaborative research. In all other forms, the participation of civil society or that of the researchers is less. The selection of the labels we have used to describe certain types of engagement has been challenging and is understandably open for some debate. In Table 1 we explain these. For each of the categories listed, examples will be given of projects in The Netherlands or abroad which have used various methods of engagement. The actual engagement methods applied can be different for the project definition phase and the actual research and innovation activity (similarly, they can be different for the levels of science policy making or research programme development).   Creating opportunities for contributions to deliberations and research activities or contributing to research execution as more than a subject in the project.
Working together on research initiation and/or execution, so there is co-ownership of the project.
Societal actors are in the lead in the research initiation and execution. On their request, they are supported by researchers or institutions.

Discussing
Unorganized discussions on matters relating to research and innovation take place in the media, and there are many organized events in which there is room for more than a one-way communication on research and innovation (e.g., at science centres or at pop festivals). In this paragraph, we pay attention to one world-wide phenomenon where research can be discussed in an open and informal setting. This is the Science Café.

Science Café
A Science Café is an event organised in an informal setting (e.g., in a pub) as a place of dialogue with participants coming from all walks of life and academia. An expert presents a subject in a concise and open manner after which the floor is open for a discussion. The moderator facilitates the sharing of a wide range of views on the subject at hand.
Publishing findings is a major part of the scientific process. Sharing knowledge with the general public by researchers is encouraged by many scientific institutions. Science Cafés offer an infrastructure for interaction which goes beyond informing the audience. In the face to face interaction the experts have ample opportunity to gather responses to their message and take away new questions. Often this is a way to gather alternative views and relevant narratives especially when the events focus on controversial issues. From the perspective of the general public the science café is often seen as a place for obtaining knowledge and forming opinions. The interaction is not only with the expert, but also with the other participants in the discourse.
Across the world there are hundreds of Science Café events being held every month, well prepared and held in an informal setting. The Science Cafés are known under a number of different names and there are various different flavours. Key ingredients are the bringing together of lay people and experts outside of an academic context. There is room for a presentation by an expert but the event includes interaction and discussion. The organisers are usually not for profit organisations that regularly organise these events.
On a tutorial webpage short films have been shared for organisers of science cafes on the practical issues which are involved [10]. The events are usually less than two hours long and presentations by speakers should be short. These could be around 40 min but some cafes limit it to five min in a total session of one hour. Some facilitators prefer presentations without slides to encourage a more informal interaction. Generally there is one speaker but there are also models with multiple experts.
A key ingredient is a moderator who should also prepare the experts to ensure there are lively and useful discussions.
In The Netherlands there are a number of places where there are regular Science Cafes, for example in Groningen and Deventer. The subjects covered are very diverse and include energy topics. In 2013, a "Kenniscafé" (literally "Knowledge Café") was organized in Groningen about the controversies around shale gas [11]. Three experts from different faculties of the University of Groningen were invited to contribute from their perspectives. There was a professor of geo energy; a field that includes the aspects of exploration and exploitation of shale gas. There was a professor of environmental psychology and a professor of the regulation of energy markets. The experts were allowed to present some slides from their perspective, but the interaction with the lay people present is at the core of the event. There were also a number of professionals working in the field present in the audience. For the experts this was an opportunity to discuss face to face their ongoing research on this subject and closely related fields.
The Kenniscafé in Groningen is regularly organized by the "Studium Generale" department, in which the two Groningen universities co-operate. There is a free entry and about 80-100 people participate each time. Science Café's serve as a meeting of minds, a dialogue outside of the usual spaces.
From an engagement perspective, the discussion is very open, and one tries to remove boundaries between presenters and audience; e.g., by not putting presenters on a stage but on a bar stool. This depends on room facilities though. One can discuss any phase of the research and innovation process, from research policy to actual on-going research projects. Participants can be of any kind, though the method focuses on individual citizens.
Science Cafés are seen as a way "to build capacities for "lay" people to deliberate and engage with scientists, and to understand and critically analyse complex socioscientific issues" [12]. There have been some evaluations of science cafés that were held in relation to emerging science technologies such as synthetic biology and nanotechnology. Navid and Einsiedel conclude, based on five Science Café's in Canada on synthetic biology that Science Cafés "were viewed positively for the relaxed atmosphere, small crowd size and informality of the venue . . . " and they "are an effective upstream engagement platform for discussing emerging science technologies." [13] Similar conclusions were drawn from a study among visitors to Science Café's in The Netherlands, on the risks and benefits involved in nanotechnology. For the people interested in the subject, both speakers and visitors, it was seen as a worthwhile approach to sharing and discussing a complex topic [14]. Thus by design, a Science Café is a low-intensity, informal form of engagement, without any formal strings attached; it's value for all participants seems to be in being confronted with new viewpoints-which can have a certain impact on attitudes and choices in the future.

Consultation
In order to obtain stakeholders views on your research and innovation project, you can simply consult them. This can be done by polls, surveys or by invitation to a meeting, depending on the phase in the research and innovation process. We will discuss two methods to allow stakeholders' views to be included in the research design; user committees and Q methodology-which is a method to ensure that all stakeholders' views are represented.

User committee
A user committee involves users, and can include other stakeholders as well, in the formal monitoring and steering of the research and innovation process. The name "user committee" may be a bit too strict in that sense, but it is common language in public-private partnerships.
In The Netherlands, the Responsible Innovation Program (NWO-MVI) has been requiring "valorisation" panels since 2009. Their 2014 Call states that "apart from representatives of the private partners, the valorisation panel includes all other actual and potential users and/or user groups. Relevant societal stakeholders can also be included in the valorisation panel. Also representatives from organisations that are willing to disseminate the research results and to valorise these among the target group that they represent can be included in the valorisation panel".
The valorisation panel should be involved from the initiation and remain involved throughout the entire project duration. The valorisation panel's main task is "to contribute its knowledge and expertise, and to confront the researchers with the everyday user practice, so that the researchers can incorporate this in their choices".
Other research programs, e.g., the ones operated by the Dutch Technology Foundation STW, have a more traditional industrial user approach to the role of a user committee [15].
The faculty of Technology, Policy & Management of the Delft University of Technology has successfully completed a number of NWO-MVI projects related to energy. One of these was the project "Embedding social responsibility in the design of offshore wind energy systems". Of the seven organisations represented in the valorisation panel, six represented industry partners [16]. Most of these were directly related to offshore wind industry, with the exception of one oil and gas exploitation company-which as a user of the North Sea has competing interests with the offshore wind projects. The only committee member that did not represent industry interests was the North Sea Foundation; an environmental non-governmental organisation (NGO). In our view, public engagement could have been expanded by, e.g., including representatives of coastal communities and of employees involved in the operation and maintenance of wind farms under harsh conditions.
In general for the NWO-MVI projects across all sectors, the representativeness of the committee is an area of concern. Typically, there are many representatives of industry as a "user" and CSOs are underrepresented. Often NGOs that participate represent an industry association, which was also the case in the offshore wind project-thus, these NGOs are not CSOs (Civil Society Organizations).
There are exceptions of course. Project summaries published in 2010 for the NWO-MVI program show 16 user committees [17]. Of the total 135 member organizations, 21 can be classified as a CSO. One project had six CSOs among 10 members; another project had no CSOs among 14 members.
From an engagement perspective, a broad "user" committee allows for formal engagement in which stakeholders have a real say on the research and innovation process, from the project definition phase through the actual research and innovation process. Not only "users" can participate, but also other stakeholders (as representatives of organizations or communities; not as individual citizens). A broad participation in such a committee is not yet mainstream though.

Q Methodology
Controversial issues in the public debate involve stakeholders and experts with a wide variety of viewpoints. The Q methodology is a research tool from social sciences that can be used to gain insight into the diversity of perspectives. Furthermore, it can be used to select relevant participants for further dialogue about the issues at hand. It has been used as a tool to facilitate the start of a consultation process. This makes Q methodology especially appropriate to shape the constraints for research programs and projects based on those, and have discussions at a sectoral level.
Q methodology was developed in the 1930s by the psychologist William Stephenson to find correlations between diverse individual viewpoints. The method is applied by social scientists across a wide range of fields. In The Netherlands, a number of researchers have applied it as a tool to uncover perspectives on controversial subjects in the field of energy.
When organising a dialogue it is fundamental to facilitate the meeting of stakeholders with diverse viewpoints on the issue under discussion. Often the assumption is made that by selecting participants on the basis of their affiliation a wide range of views is represented. This is not always the case though. The Q methodology can be a tool for stakeholder selection where the emphasis lies on the representation of diverse perspectives in the dialogue. The Q methodology involves three main steps [18]: i.
Definition of the concourse When looking at a specific issue the "concourse" is the sum of all the statements about the issue. This step results in a set of statements called the Q set which defines the discourse. For example, in the Dutch biomass dialogue in 2007, there was an active public debate. So a desktop analysis of newspaper articles could be used to sketch the concourse. Sometimes this approach is not effective because of a lack of public debate. Then a different approach may be necessary. In a recent study on the acceptability of hydrogen technologies, focus groups were organised to gather data for the Q set. ii.
Interviews and perspective identification The sample of statements collected in the first phase is presented to the interviewees, who each make a Q sort. This is a ranking of the statements in the Q set according to their personal agreement or disagreement with the statements. At the end of each of the sessions the interviewer has two sets of data, the Q sort and the narrative where the interviewee explains their choices. The whole set of interviewees is called the P sample. They are selected to represent a variety of viewpoints. One purposive sampling approach is to snowball, by asking each participant for names of people with similar and different views. This can then be used to invite more participants and to monitor whether the interviewees represent groups with a wide variety of viewpoints. iii.

Analysis & Conclusions
A statistical analysis of the Q sorts finds correlations between individual viewpoints. There are various software packages, such as PQMethod, that can support this analysis. Together with the analysis of the narrative this leads to conclusions about shared ways of thinking.
In 2007 the Q methodology was used as a preparation for a national biomass dialogue in The Netherlands [19]. The purpose of the dialogue was to develop sustainable biomass chains and to identify what was needed, including new national policies, to facilitate the development of these chains. Over the period of a few months, 75 people were interviewed, selected with the snowballing principle to include a wide range of views, backgrounds and affiliations. In this case, six perspectives were found, which was somewhat more diverse than expected. This confirms that there was quite a lot of complexity and uncertainty in the issues under discussion. Based on the interviews six types of stakeholders or actor types were identified as relevant for the dialogue. These were both national and local governments, knowledge institutions, energy companies, NGOs and Small and Medium Enterprises (SMEs). To illustrate this diversity a graphical analysis was done by Cuppen, Figure 3, on the prevalence of the perspectives across the various actors [20]. This indicates the extent to which respondents agreed with the opinions underlying the six perspectives. For NGOs there was a clear preference for "hit the brakes" but in general there was a diversity of perspectives across the actor types.  The six perspectives for the 75 participants from six actor types:
Security of supply with global, certified, 2nd generation biomass; 5.
Efficiency the goal: biomass a means? 6.
Just do it, step by step.
Often participants in a dialogue are selected by affiliation and are expected to represent the perspective of a certain actor. The use of the Q methodology in selecting stakeholders and feeding the start of the dialogue had a number of advantages. Firstly by describing the variety of perspectives, in this case six, and their distribution across the various stakeholders. It was useful to expand the discussion beyond arguments that were purely for or against. These perspectives were used to feed the workshop sessions to help participants understand the diversity in perspectives. Secondly the distribution of perspectives also facilitated the selection of participants to maintain a diversity both in actors involved but also in the perspectives represented in the discussions. One of the ways this was checked in the biomass dialogue was by evaluating the extent to which participants met new actors they had not worked with in previous projects or events. Q methodology thus helped to make positions clear, find nuances and even common ground and understanding. This makes decisions on research or investments in biomass easier to take.

Involving
Worldwide, there is an enormous increase in projects in which citizens are involved in the research activities. The most common umbrella terms for this involvement are Citizen Science, or Patient-Involvement (PI) in Research in the medical domain. A traditional example of Citizen Science is bird-counting done by the public. More recent examples are citizens measuring particulate matter concentrations in the atmosphere with their smart-phones (iSpex)-a process with a reliability similar to professional measurements, because of the large number of observations possible in this way as compared to a sited single observatorium-or the Galaxy-Zoo project in which amateurs code space-photographs to discover planets-humans are better at this than computers. Thus, citizens are active in gathering or analysing data; scientists can then use these data. And obtain them cheaper and faster than without citizen support. Of course, involvement can go beyond this and be more encompassing. The boundary with the next level; of engagement, "collaboration", is not sharp.
Also in traditional product development, "users" are a category regularly involved in the innovation process. We will highlight two examples of "involvement" in this section: Participatory Design and the Hackathon.

Participatory Design
Participatory design can be done together with citizens concerned with a certain issue, e.g., the environment. The starting point is consultation with individuals and community organisations. This is followed by an interactive design process which includes field tests with the users of the developed technologies and devices.
Designing for users is common practice in the field of industrial design. The user is also seen as a customer and the future buyer of the product. Participatory design with citizens who are for instance concerned with their local and global environment has a similar approach. There are a number of interesting projects pioneering in the area of participatory design.
An example of participatory design is the Energy and Co-Designing Communities (ECDC) project, which is being organised by Departments of Sociology & Design of Goldsmiths in London, UK. They work with existing communities engaged in reducing energy demand, and aim to understand the range of perspectives and knowledge they embody [21]. The project's website claims that this approach is unique, since it draws out "the imaginative, playful, emotional and potential aspects of people's use of technologies". This goes beyond the traditional approach that maps what "people do and think in the context of 'useful' technologies" [22] The participatory design process used in the ECDC can be summarized as follows [23]: i. Workshop with communities (citizens engaged in a certain issue), academics & local government, including collaborative mapping sessions, probes and workbooks to feed the discussion; ii.
Design & adaptation during deployment.
The ECDC project uses materials drawn from field trips, workshops and probes. The thinking in the team has roots in both industrial design and Science and Technology Studies. Part of their method follows an ethnographic approach, which can for example be seen in the fieldtrips. The probes presented on the website are materials sent to the community as a data gathering tool. The project created workbooks, based on these inputs and the designers' own ideas. These workbooks are collections of illustrations of potential technological and social innovations. Some of these are a little weird and extreme, but useful to feed the co-creation sessions. From the start, the project expected to " . . . design novel devices and give them to participating communities to test in their own settings" [21]. The researchers are soon publishing some results on the Energy Babble, an innovative computer moderated communication device that was created.
In 2011, the Dutch Ministry of Economic Affairs followed a similar approach when it opened a call for Smart Grid Pilot Projects (IPIN), administered by NL Agency. These were called "proeftuinen", which could be translated as "tasting gardens", i.e., testing grounds. The projects created spaces to experiment with changes in the energy networks, and study, e.g., how users at a residential level interact with the energy system. During the execution of the twelve projects, NL Agency organized a number of workshops to exchange experiences in involving users in the innovation program. The position paper at the start of the sessions encouraged "sharing concerns" and emphasized co-creation to improve results and participation in the projects [24]. User engagement has been described extensively for Power Matching City-Phase 2, which is one of the IPIN projects [25]. There was an intervention in one street in the city of Groningen, with 18 households participating. Some of these households were members of a local energy co-operative called "Grunneger Power". Various interactive events were organized to involve participants, for example to co-design the interfaces of in-home devices, which functioned as an energy monitor. However, there seems to be little analysis of the extent to which there is co-ownership by the users in all the IPIN projects, or the extent to which their concerns are leading the innovation.
In the press release at the end of the project, the headline was "Implementation of smart grids could generate €3.5 billion for The Netherlands" [26]. There is a mention of the residents of the area involved in the project, but the message is clearly framed in terms of economic benefits to the partners involved. The "instrumental value" of the co-operation seems to benefit the innovation partners especially. The design and execution of the project paid less attention to co-ownership of the project and the benefits of outcomes for the involved households and the Dutch residents in general.

Hackathon
A hackathon is an event, usually held in a weekend, often aiming to build ICT solutions in teams. The concept was developed around the turn of the century and initially had a strong focus on software development together with volunteers. Through the years, people with a non-technical background have increasingly become involved.
The name "hackathon" comes from the combination of the words hacking & marathon. An intensive and exhausting event, with the purpose of creating novel solutions. Thus, this relates to the positive connotation of hacking as in "engineering change". The events can include a competitive element, in which teams compete for prize money, e.g., by organising pitches at the end of the event to compare the results.
The organisers and hosts of the events are often a number of organisations. There is a central theme or challenge which is put forward in the invitation to participate in the event. This is usually expanded with talks at the start of the event.
In the UK the National Health Service has organised a number of hackathons a year since 2012 to support their own staff and inspire innovations by involving people from outside their organisations. There are also examples in the UK of sessions to develop tools for life sciences, and events to improve transparency in government [27].
In October 2014, two Dutch Distribution System Operators (DSO) supported a hackathon called Apps for Energy [28]. The DSOs are currently rolling out smart meters, which are generating energy use data at a household level. This is creating a potential for new products and services to lower household energy use, or to facilitate the alignment of various local renewable energy generation options. The hackathon was organised to encourage the development of these new services [29]. The event was hosted by a business incubator. 70 volunteers participated for some 40 hours over a period of three days. Other organisations where involved in mentoring the teams. The mentors included experts from companies involved in energy data and ICT services, but also experts on open data and public data sources, such as the Dutch Central Bureau of Statistics. Three prizes were awarded at the end of the event for a total of € 6.000.
The DSOs in The Netherlands are publically owned. They are obligated to roll-out the smart meters based on national regulations, which are changing in similar ways across the EU. The acceptability of the smart meters has been under discussion in The Netherlands. Possibly, the development of relevant services in the hackathons will increase the acceptability of the smart meters. Primarily, the involvement of users taps into a new source of creativity.
Thus, the engagement that we see in both these "involvement" processes is very instrumental; it uses creativity, skills and time of the public to advance research and better align technology and society.

Collaborating
New knowledge can be created together, in a collaboration of professional researchers and those for whom research is not their daily profession. It requires a constant search for win-win situations, in which all parties gain from co-operation. Apart from this "reward", collaboration requires recognition of each partner's contribution, respect for different views, clear roles, a good relationship to be build and maintained, and results (so a clear focus on outcome and impact); the six R's of co-operation [30].
We will describe two approaches to collaborate; the Science Shop and its tools that facilitate the joint setting of research questions (definition of projects), and the Charrette, an engaging method for co-creation, developed in the context of urban planning.

Science Shop
If the engagement starts from requests by civil society, and projects are set-up together, then almost by definition the research will be well aligned with society's demands and views, and results can easily be taken up in society. This demand-driven operation is typical in so called Science Shops, which can be found at universities or as stand-alone organizations all over the world (under various names). They do research on request for civil society organizations.
We will take an illustrative example in the energy domain from the Science Shop at the University of Groningen. A residents' group from the north of the country approached the Science Shop with the issue that large wind turbines near their homes caused a lot of noise during the night. The government claimed that according to the official national model used to calculate noise there was no problem. Physicists (staff and students) at the university then looked into the mathematics of the model. With additional field experiments they could falsify the model and show it to be unsuitable for current wind turbines, which are much higher (i.e., 100 m) then the turbines that the model had been developed for (30 m, 60 m). In the PhD thesis that resulted from all this work [31], they also constructed a new model. Subsequently, the government changed the official national model accordingly [32].
Without the option of approaching the Science Shop, this new knowledge would not have been created. It is interesting in this case that it is still the prerogative of politicians to decide how much noise is acceptable. However, they now have the tools to translate this acceptable noise level into distance to homes for wind turbines.
Science Shops were established in the 1970s at the Dutch universities (with similar developments in, e.g., Canada and USA), and are now active in many countries. A Science Shop is defined as a unit that provides independent and participatory research support in response to concerns experienced by civil society [33]. For universities, it is a way to fulfil their Third Mission (outreach, or knowledge transfer, next to Research and Teaching). Through a Science Shop they can ensure that research is accessible to everyone, including civil society organisations and non-profits (complementary to curiosity driven or industry driven research). The Science Shops are united in the Living Knowledge Network [34].
Civil society driven research leads to interesting research topics for staff and students, and offers social and political learning for students, next to developing problem-solving skills. It offers good PR for the university. This is a win-win-win situation. Policy makers benefit from additional knowledge that is created, to make better informed decisions.
Because at university-based Science Shops the research is mostly done by students under staff supervision, as part of the curricula, there are only low costs involved. Mostly, bachelor or master thesis research is used to perform research for a CSO. For professors, supervising this research already counts towards their teaching hours. The learning experiences for students of this engaged research contribute to their competences as demanded by, e.g., the Dublin descriptors. These-or similar-are used at many universities to assess the final competences of graduates [35]. It is also motivating for students to do research for a societal organization [36,37]. At the same time, working with students also has limitations, especially in time planning. When additional funding is available, researchers can be hired.
Even without establishing the full infrastructure of a Science Shop, one can use elements of its work in research programs and projects definition. Or one can, e.g., invite an existing Science Shop to join a consortium.
The Science Shop, as infrastructure, offers an existing network of CSOs in the region, within which trust relations have been established. When starting from scratch or involving CSOs in a stand-alone project, a needs survey among (relevant) CSOs can be done, to see if the expressed needs match research interest/capacity within the institute or the consortium submitting a research proposal. Science Shop staff have good experience in process management of these co-operative projects.
In a start-meeting, the research objectives and time frame are agreed, expectations managed, and sources of knowledge identified. It is also very important at this stage to find out what the real question behind the initial one is, and to decide whether the research should be more encompassing or presumptions have to be investigated first.
The CSO participates in the sounding-board of the project. Results are made public. Through this co-operation, the research is both independent and participatory. Further involvement of the CSO is possible, i.e., in the actual research process, depending on the context. In university based Science Shops, the university has final responsibility to deliver a product that abides by academic quality standards. Other Science Shops are stand-alone organisations, which usually work in partnership projects with CSOs and research institutes, or perform part of the research themselves. Responsibilities are then distributed in the team.

Charrette
Charrettes are used in urban planning to facilitate input from the community in a specific geographic area. They are intensive workshops for many stakeholders to work together, including policy-makers, experts and the general public. A key element is the integration of design activities in an early phase to make implementation plans and/or research proposals. This method was developed in North America as a tool for community planning and further developed in The Netherlands within the project "Grounds for Change" and by the "Laboratorium Ruimte + Energie + Milieu". The Lab R+E+M consortium organised Charrettes with special attention to energy ("energie") in relationship to spatial developments ("ruimte") and environment ("milieu"). More than 20 Charrettes have been documented and a number of practitioners still facilitate these type of events [38]. In this section we will compare the LAB R+E+M approach [39] in The Netherlands with the North American model as it is described in "Participatory Methods Toolkit. A practitioner's manual", which was published in 2006 by two Belgian foundations [40].
In Wikipedia some background information is given on the roots of the term. The word Charrette is French for "cart" or "chariot". In the École des Beaux-Arts in Paris in the 19th century, it was not unusual for student architects to continue working furiously in teams at the end of the allotted term, up until a deadline, when a Charrette would be wheeled among the students to pick up their work for review while they, each working furiously to apply the finishing touches, were said to be working en Charrette, in the cart. [ . . . ] The term metamorphosed into the current design-related usage in conjunction with working right up until a deadline [41]." At the core of the method is an event called the Charrette workshop which aims to have the right ingredients and pressure for a creative process with the experts and stakeholders. Usually this takes place over a number of days. This needs extensive planning and preparation but also a thorough follow-up. There are many variations on this format; one of the defining elements is the number of participants.
In the North American model the main workshop is prepared by a steering committee of 10-15 people. They decide on the focus of the Charrette and will require input from stakeholders. This should both be done by involving a number of representatives of stakeholders in the committee, and tapping into the networks the members have in the community. This will at the same time feed the process of inviting a broad range of participants. The steering committee is also responsible for the gathering of information that feeds the Charrette workshop. This should be a wide range of data related to the specific area but can also include expert opinions and example projects from other areas.
In the Dutch examples, the process was generally owned by local governments. So the preparation is very much a co-operation between the facilitators and the local authorities. Often there was a 40-60 ratio in internal and external participants. Thus, almost half the participants were from various departments or agencies of the national government or from the various departments of the local government involved. A goal of these type of events also was to facilitate the interaction between various departments, even those from within the same government.
There are examples of Charrette workshops with more than 1000 participants. Generally the number of participants in the Charrette will be in the range of 50 people and the process can include an event open to the general public. In The Netherlands there were no examples of events for the general public and the focus was more on expert participation. In general there were 25 participants with some exceptions of sessions with up to 50 participants. The Charrette consists of a number of sessions and usually there are three types of sessions before the public session: i.
Getting to know each other and the core issues. It can be useful to do a site visit but the information that has been gathered beforehand to bring the participants up to speed is the key; ii.
Gathering input from the participants. In depth interviews in subgroups should be held to gain insight into the views of each stakeholder. Each participant also prioritises the issues that they raise; iii. Integration.
Finally the collected information is integrated and analysed in preparation for the public meeting. This involves brainstorming and analysis to make some first recommendations.
The purpose of the public event is to check the direction of the Charrette team effort. So they present their preliminary analysis and the overview of goals and objectives that has been gathered from the stakeholders. Based on the feedback the focus can be adjusted if necessary.
After the public event the Charrette team has a number of sessions alternating between working in small groups and in a plenary setting. The groups are divided to work on one specific issue with at least one relevant expert in the team. They generate proposals for their specific issue and present that to the whole group. The responses on their presentations then feedback into the design process in the sub-group sessions.
The Charrette generates many ideas and possible solutions. These need to be shared with the general public in a second open meeting. So this phase includes the preparation of the event and presentation, but also a final document which needs to be readable for the general public. As mentioned before the Dutch Charrettes did not include these public events.
For all the Charrettes a summary of the event was made with a description of the process and main results. These final documents were always shared with the participants and in some cases with a wider audience. In all publications from the Charrettes there is a strong graphical element which can be useful for communicating across disciplines.
The Charrettes were often related to new housing projects and included energy dimensions at various levels. Some design activities focused on new energy concepts at the household level, others more at a neighbourhood level or the way in the neighbourhood energy system relates to the bigger system. The innovation part also related to the embedding in administrative processes, which is made easier when the Charrette happens early on in a project development.
The involvement of civil society in the Charrettes in The Netherlands was not strong. There is one clear exception which was organised in 2009 in The Hague, where the participation of local inhabitants of the area was encouraged and considered crucial. The North American model has various elements which could be integrated to strengthen the public engagement. Mainly this could be done by including different parties in the event, but also by considering the integration of a public event in the process. Organisers of Charrettes see a lot of value in the new ideas generated, but also in the sense of ownership and commitment created. By focusing more on a wider inclusion, this should also lead to ownership in the local community.

Supporting
If researchers have a "supporting" role, society in some form is in the lead. One can think of communities organising their own research and requesting technical support to have scientifically sound results to support their case in local land use discussions, as described in case studies on public participation in environmental research [42]. In the Dutch energy sector, we have not found clear cut examples of this and other types of supporting interaction.
We will discuss here two areas where we found at least a weak form of "support" based on civil society taking the lead in the research. Firstly we will look at knowledge transfer mechanisms around local energy initiatives and secondly take an example of an NGO which is supported by scientists in making fact sheets on sustainable behaviour for the general public.

Knowledge Exchange
Since the 1980s there are a number of community owned wind turbines in The Netherlands. In the past ten years, there has been a significant increase in the number of local energy initiatives. These have focused on organising home investments in energy saving measures and solar power installation. There is a lot of interaction between these initiatives in supporting each other, but there is also often financial and knowledge support from local governments. A number of organisations have been created which organise events and platforms to facilitate the exchange of practical advice and best practices in all types of areas. So not only technical information, but also information relating to legal and organisational aspects.
The relationship with knowledge institutions does not seem to be well structured and we did not find current practices of effective interaction. At a conference in 2013, organised by a platform of local initiatives a number of participants in initiatives indicated that they are primarily an object of research and do not feel supported [43]. Also the results of research are hardly available and accessible to facilitate learning. Possibly the strongest link to sources of knowledge and the further development of relevant knowledge are some volunteers in the initiatives. At a very practical level it is common for participants to bring their professional experience and networks as support to the organisations. This can be energy related expertise, but also the experience of communication professionals or civil servants. In this way there are also links to research organisations by academic researchers who work as volunteers for these initiatives. For example the chairman of the Grunneger Power cooperative is a retired professor of sociology who still has strong links to ongoing research related to energy initiatives [44].

Quality Control
Friends of the Earth Netherlands, called Milieudefensie, started the "Milieutelefoon" (Environmental Telephone) in 1987. Citizens could call for practical information to enable them to live in an environmentally friendly way. The Milieutelefoon was later placed in an independent non-profit organisation, called Milieu Centraal (Environment Central), 1998. This organisation received a subsidy from the Ministry of the Environment and developed into an independent, fact-based resource for citizens (with its information now available on-line as well) [45,46].
Milieu Centraal uses a scientific board (researchers working at research centres or universities) to advise them on the content of the fact sheets. These have been made by in-home researchers, and have been commented on by stakeholders (thus, here's an engagement process as well). The fact sheets should give undisputed information; therefore, finally, the scientific board forms the gate keeper for quality control [47]. Milieu Centraal currently gives information on energy-savings, solar panels, low energy homes and equipment [48].
The distinction between "supporting" and "collaboration" is not always sharp. Also in, e.g., research through Science Shops there are NGO-partners that take more or less the lead in shaping the process. This shows that the intensities of collaboration form in fact in a continuum.

Discussion and Conclusions
Public engagement in energy research can be done in many different ways and this has been illustrated with various methods and example projects in The Netherlands and the UK. This paper highlights public engagement which is done already, even within current constraints that, e.g., academic researchers face. The increased attention and funding (requirements) for public engagement, RRI, and the focus on "outcome" instead of "output" may allow more academic staff to become active in public engagement and might well be the "game-changer" that will allow hiring, promotion and tenure criteria for academic staff to be more in favour of public engagement. In the UK, 20% of the direct funding from the Research Councils to Universities is now allocated based on "societal impact", and this will increase to 25% [49]. In The Netherlands, the "number of publications" has been removed as a separate quality indicator from the Research Assessment Protocol [50], and so-called "valorisation" indicators have been introduced [51]. These include, e.g., the number of popular presentations or publications, the number of societal organizations worked with, etc. Further research would be required to see whether the potential impact of these changes will be realized.
In categorising the role of the research organisation or researcher in public engagement projects we have used five labels: discussing, consulting, involving, collaborating and supporting. For each of these roles and categories of projects we have analysed at least one example project to illustrate the application of a specific method. In this range of categories, the distinguishing factor is the level of engagement, and the ownership of initiative in the research. The borders between the categories are not always clear cut. In the execution of similar projects there can be some differences in the level of ownership by stakeholders. On the other hand, the categories have proven useful to illustrate and discuss the diversity in engagement methods.
Drivers for public engagement in energy research can be both democratic and instrumental, for governments, public funders and researchers alike. This has been confirmed in the examples discussed. For all projects one of the elements was relevant, for many both, though usually one was dominant. When looking at democratic drivers for public engagement it is clear that for science cafes (discussing) this is the dominant driver. By discussing the research with lay people, the "tax payer" is informed of how research funding is being spent. A discussion also allows, or even forces, the researcher to become acquainted with the views, wishes and demands of society. In a more formal way, the user committee (consulting) is an example where the funding body requires the involvement of stakeholders from society as a building block for responsible innovation. This engagement thus contributes to the democratisation of research by giving society a bigger role in the agenda setting of specific projects. Another element of the democratic driver is facilitating the public discourse about research strategies. In the example of Q methodology (consulting), the engagement method ensured the inclusion of a broad range of perspectives that are relevant for the democratic discourse.
At the core of the instrumental driver is the expectation that broader involvement in research will lead to better research. Indeed, in the case of the participatory design (involving) of Smart Grids, there was a clear economic benefit to the innovators. In addition, from our study it is possible to conclude that there is a willingness within the research community and society in general to invest time and energy in public engagement projects. This can be seen clearly in the hackathons (involving) and Charrette (collaborating) which have been organised as short events over the years. It has been possible to mobilise end users and stakeholders to participate in innovative activities to develop together new solutions to energy challenges faced by communities and society in general. The demand driven research as, e.g., done through Science Shops (collaborating) clearly shows that research results can directly benefit society. The case of the wind turbine noise is an example of how research can help to make the energy transition move along with less negative effects such as noise pollution.
The transition to a system with a secure, clean and efficient energy supply is a big challenge in Europe and across the world. There is a lot of research and development being done and many funds are available from governments and public funders. In this study we have shown that there is a diversity of tools and methods already available that facilitate the participation of society in energy research. These could be further developed, combined-the level of engagement being in fact a continuum of options-and implemented. Not only for democratic reasons, but also for instrumental benefits in creating innovations to help solve the Grand Societal Challenge of the energy transition.