Toward a Socially Desirable EU Research and Innovation Agenda on Urban Waste: A Transnational EU Citizen Consultation

: Growing waste production has become a global sustainability challenge that is in need of innovative solutions. It has been argued that greater public engagement in science and technology policy could stimulate new directions for innovation and provide orientation for the development of more acceptable, sustainable, and desirable innovations that address societal needs. In 2013, the European Commission (EC) piloted a EU-wide study to engage citizens in setting the research and innovation (R&I) agenda on urban waste. This yielded the question: ‘To what extent are the proposed ideas considered innovative, feasible, and meaningful for the EC R&I agenda?’ This article addresses the outcomes of 100 focus group discussions held in 27 European Union (EU) countries in which 992 citizens were consulted. Citizens discussed a number of (self-experienced) barriers and concerns regarding waste management, and proposed and prioritized over 350 ideas to realize a (near) zero-waste society. Although citizens found it complex to propose innovative and feasible ideas, their priorities were consistent with current EU research policy areas of importance (e.g., reduce packaging, stimulate recycling), and in addition, citizens proposed various new ways to strengthen current waste management (e.g., innovations to enhance convenience in household waste management). We that citizen involvement in EC R&I agenda-setting across the EU is feasible, and leads to meaningful input. However, in making sense of this input, it is essential to take contextual differences into account. Various recommendations are given for future agenda-setting activities at the EU level.


Introduction
In 2013, the European Commission (EC) Directorate-Generals of Environment and of Research and Innovation engaged European Union (EU) citizens in research and innovation (R&I) agenda-setting on municipal solid waste (MSW) management. This initiative happened in response to two developments. First, it was recognized that sustainability challenges, such as that of MSW, require new ways of knowledge production and decision-making, where the focus is on research collaboration among scientists from different disciplines as well as between scientists and non-academic stakeholders (i.e., business, government, and civil society) [1][2][3][4]. It has been argued that in order to change established waste management practices, there is a need for public engagement in the political, institutional, and social arenas in which decisions are made [5,6]. Second, in this process, citizens play a crucial role in achieving cleaner and more sustainable societies through the ways in which they purchase products and deal with waste in and around their homes. At the same time, it is citizens whose voices are least heard and listened to in decision-making during the early stages of R&I. The EC has recognized that citizens' engagement in science and technology policy can generate new forms of social intelligence and create mutual benefits by stimulating new directions for innovation [7]. In addition, such engagement is seen as a way to promote an "active European citizenship-the recognition, development, and cultivation of a European identity among citizens" [8].
The initiative was unique in the sense that the EC had never before involved citizens directly in the process of R&I agenda-setting at a EU-wide scale. During the last decade, there have been several Europe-wide initiatives to engage stakeholders, including the wider public, in debates on science and technology policy, such as citizens' deliberations on brain science and public consultations on nanotechnologies [9,10], environmental planning, and new policies [11]. These consultations focused on identifying (societal) concerns and issues that citizens value most in relation to the topic in question. In the transnational agenda-setting initiative discussed in this article, the EC consulted citizens to think more abstractly, going beyond identifying barriers and concerns related to waste management, and propose and prioritize innovative and feasible ideas to improve it. On the basis of this input, the EC would define topics for the next Horizon 2020 work program related to 'Urban Waste and Innovation'.
Given the novelty of the endeavor, the EC faced two uncertainties. The main one was the extent to which citizens were able to produce meaningful knowledge for its R&I agenda on MSW. Glicken [12] distinguishes three types of knowledge-"cognitive, experiential, and value-based knowledge"-where cognitive knowledge stems from technical experts, experiential knowledge comes from people's personal experience, and value-based knowledge is related to social interest and social values. Previous successful research agenda-setting endeavors involving non-experts have typically engaged with participants representing a unique perspective; for example, the perspective of patients with knowledge of a specific illness or disease [13,14], activists fighting for a cause [15], local residents or community members in areas of planning/risk assessment [16], or consumers [17]. These participants had already been through a process of reflection, either due to personal experiences or by being members of an overarching body (e.g., civil society organization (CSO), patients' organization, non-government organization (NGO), community association) that can provide structure in organizing the distinctive knowledge that these participants represent and articulate. As a consequence, the emerging socially robust research agendas appealed mainly to their articulated experiential knowledge, mostly consisting of personally experienced barriers and concerns, and wishes for the future. In this case, participants were consulted as members of the general public. It was assumed that MSW is a broadly felt societal challenge and an issue that every citizen could relate to and talk about, as everyone to some extent produces and deals with waste, and thus clearly possesses some experiential knowledge on this subject. However, for a R&I agenda, innovativeness and feasibility become relevant quality criteria for which some degree of cognitive knowledge is required. How to deal with this in structuring the consultations, and to what outcomes does that lead?
A second uncertainty relates to the transnational character of the study, as this may give rise to specific methodological challenges [18]. Although previous transnational participatory studies [19] have shown that such initiatives are in principle feasible and that citizens value them, participation tends to be much more difficult to organize. First, at the transnational level, the diversity of the "target group" is both significantly larger and much more heterogeneous than at the national level, as each society has its own values and cultural norms that impinge on the structure of public debate [20]. In addition, EU countries differ in their overall waste management system [21,22] due to different (economic, structural, and political) drivers and geographical position. Can we speak of the general EU public, or should we keep in mind that we are dealing with various publics? Due to these differences, we hypothesize that the perceptions of EU citizens on current MSW management practices, as well as visions on how to achieve sustainable waste futures for Europe, differ. To what extent can a EC R&I agenda incorporate this diversity of perspectives, while still being appealing to the wider EU?
In this article, we present the results of the consultation, called VOICES [23] (Views, Opinions and Ideas of Citizens in Europe on Science), which was led by Ecsite (the European Network of Science Centers and Museums) in collaboration with academic researchers from the Athena Institute of the Vrije Universiteit Amsterdam, and assessed the value of this consultation for EC R&I agenda-setting. This transnational EU-citizen consultation aimed to gain insight into (1) their knowledge on, and behavior toward MSW management; (2) barriers and concerns related to MSW management; and (3) their ideas for achieving a zero-waste society. In addition, we investigated the quality of these citizens' inputs from the perspective of EC R&I agenda-setting, as well as how to deal with the diversity resulting from this transnational consultation process. Lessons learned are provided for future transnational EU-citizen consultations.

Approach
Since the project aimed to develop more in-depth understanding of citizens' perceptions by identifying their thoughts and ideas, using their own framing and words, a quantitative approach [24][25][26] in which perceptions are identified based on pre-defined answers was considered less suitable. Rather, a qualitative approach was selected, namely the focus group (FG) method. In FGs, participants respond to, and build on the views expressed by other participants, stimulating shared creative thinking, and providing the opportunity to gain in-depth insight into participants' ideas, values, wishes, and concerns [27][28][29][30]. Given that they can be used in a transparent, uniform, and structured way, FGs have proven to be an appropriate public consultation method at both the national [31] and international [32] level. FGs can be conducted at the national level with a group of diverse citizens who speak the same language and are able to take the specificities of the local context into account. Given that the project was transnational in scope, it was important to address how to manage the diversity and the complex logistical set-up [18]. Preparation, data collection, analysis, and reporting of the FGs took place in the period from January 2013 to February 2014.

FG Participants
In each of the 27 EU countries, citizens were selected by local recruiting agencies. Maximum variation sampling [33] was used to involve a cross-section of society in terms of demographic information, including sex, age, and education, as well as housing situation, family composition, and municipality diversity. In addition, participants should reflect the urban-rural ratio of the country. In total, 992 European citizens from 557 EU municipalities participated. There was an almost equal proportion of men (49.1%) and women (50.9%). The participants were aged from 18 to 77 years, and were equally divided among three age groups: 18-35 years, 36-50 years, and 50-77 years. Educational levels were diverse: 40.5% of participants had a high level of education, 39.5% had a medium level, and 20% had a low level. In terms of housing, each country had an almost equal distribution between participants who lived in apartments (49.8%) and houses (50.2%). Participants from both urban and rural areas who were resident in at least five different municipalities, including bigger towns and smaller villages, were represented in each FG. Table 1 provides an overview of the overall demographic composition and housing of the participants.

Location and Number of FGs
In 21 countries, the FGs were held at local science centers and museums (permanent venues for public discussions and exhibitions on science, technology, and society). In six countries (Bulgaria, Cyprus, Luxembourg, Malta, Romania, and Slovakia), this was not possible, and six other organizations (e.g., social research agencies) were sub-contracted for this purpose.
In most of the participating countries, three FGs were conducted with 10 participants each, all at the same location. However, in countries with a population of above 25 million (France, Germany, Italy, Poland, Spain, and the United Kingdom), two sets of three FGs were conducted, each set at a different location. In the Netherlands, four FGs were held instead of three, due to several no-shows during the first three FGs. In total, 100 FGs were conducted in 33 different locations: Austria (Vienna), Belgium

Design of the FGs
In line with the EC's strategies, the FG design explicitly approached the challenge of MSW from two perspectives: the waste hierarchy (see Figure 1) and the life-cycle thinking approach. The five-step waste hierarchy [34] can be used as a rule of thumb when choosing between options for waste management: prevention is the most preferred option, and disposal in landfill is a last resort. However, all of the products and services have environmental impacts at various stages of their existence. To avoid shifting the negative impact from one stage to another, the life-cycle approach [35] involves looking at all of the stages of a product's life-from the extraction of raw materials for its production to its manufacture, distribution, use, and disposal-to find out where improvements can be made to reduce the environmental impacts and use of resources. Both perspectives were incorporated in a PowerPoint presentation that was used to inform the FG participants about the EC's vision on waste management, serving as a theoretical platform (i.e., cognitive knowledge) on which they could further construct their ideas.

Implementation of the FGs
Over a period of three hours, the participants discussed (1) their knowledge about and ways of handling household waste; (2) barriers and concerns related to MSW pathways; (3) their research ideas to achieve a zero-waste society; and (4) their priorities among the proposed research ideas. The FGs ended with an evaluation in which participants gave feedback regarding their experience of the FG. The FG script was piloted twice in Amsterdam and twice in Brussels, and subsequently fine-tuned. Results of the pilots are not included.
Each FG took place in the local language, and was led by a moderator who guided the discussions and ensured that all of the participants expressed their views and all of the topics were covered. Every science center/museum and sub-contracted organization assigned local staff to serve as moderators. These moderators received a comprehensive two-and-a-half-day training in Brussels on the use of the FG script, the facilitation technique, transcribing the discussion, and writing a summary report.

Data Analysis
After each FG, the moderators wrote a summary report, which was sent to the participants for their validation and comments. All of the FG discussions were audio-recorded and transcribed verbatim. With the exception of Ireland and the United Kingdom (UK), transcripts were translated from the local language into English. Data analysis was conducted by using the qualitative data-analysis software MaxQDA (v.11, VERBI Software GmbH, Berlin, Germany) and processed anonymously by a team of 18 researchers. A systematic procedure was established to ensure both the quality and uniformity of analysis and reporting. Data were analyzed, both by structured analysis and open coding. Structured analysis used a pre-designed coding sheet based on an analysis of relevant literature and the pilot FGs. Furthermore, open coding ran parallel to the structured analysis, and allowed for unforeseen insights to emerge. As a result of the open analysis, certain codes in the coding sheet were amended, and codes were added. Team meetings were regularly organized in order to ensure consistency between researchers.
To assess the unique and distinctive perspective (experiential knowledge) of the citizen participants, an advisory board (AB) (n = 8), comprising experts in the field of public participation, R&I, urban waste/environment issues, social innovation, governance, and qualitative methodologies in social research, reviewed the outcomes of the FGs. They received the full list of ideas proposed by the citizens and five randomly selected country reports (Bulgaria, France, Latvia, Malta, and the UK), and were asked to reflect on the participants' ideas in terms of innovativeness and feasibility.

Implementation of the FGs
Over a period of three hours, the participants discussed (1) their knowledge about and ways of handling household waste; (2) barriers and concerns related to MSW pathways; (3) their research ideas to achieve a zero-waste society; and (4) their priorities among the proposed research ideas. The FGs ended with an evaluation in which participants gave feedback regarding their experience of the FG. The FG script was piloted twice in Amsterdam and twice in Brussels, and subsequently fine-tuned. Results of the pilots are not included.
Each FG took place in the local language, and was led by a moderator who guided the discussions and ensured that all of the participants expressed their views and all of the topics were covered. Every science center/museum and sub-contracted organization assigned local staff to serve as moderators. These moderators received a comprehensive two-and-a-half-day training in Brussels on the use of the FG script, the facilitation technique, transcribing the discussion, and writing a summary report.

Data Analysis
After each FG, the moderators wrote a summary report, which was sent to the participants for their validation and comments. All of the FG discussions were audio-recorded and transcribed verbatim. With the exception of Ireland and the United Kingdom (UK), transcripts were translated from the local language into English. Data analysis was conducted by using the qualitative data-analysis software MaxQDA (v.11, VERBI Software GmbH, Berlin, Germany) and processed anonymously by a team of 18 researchers. A systematic procedure was established to ensure both the quality and uniformity of analysis and reporting. Data were analyzed, both by structured analysis and open coding. Structured analysis used a pre-designed coding sheet based on an analysis of relevant literature and the pilot FGs. Furthermore, open coding ran parallel to the structured analysis, and allowed for unforeseen insights to emerge. As a result of the open analysis, certain codes in the coding sheet were amended, and codes were added. Team meetings were regularly organized in order to ensure consistency between researchers.
To assess the unique and distinctive perspective (experiential knowledge) of the citizen participants, an advisory board (AB) (n = 8), comprising experts in the field of public participation, R&I, urban waste/environment issues, social innovation, governance, and qualitative methodologies in social research, reviewed the outcomes of the FGs. They received the full list of ideas proposed by the citizens and five randomly selected country reports (Bulgaria, France, Latvia, Malta, and the UK), and were asked to reflect on the participants' ideas in terms of innovativeness and feasibility.

Ethical Issues
Each participant signed an informed consent form, which provided information on the topic and the aims of the FG. This form obtained approval for audio-recording the FGs, use of the resulting data for research purposes, including the use of quotes, and data storage for five years. Participants were informed that participation was voluntary, and that they were free to withdraw at any time, without having to give any reason. Each participant received around 50 Euros (with slight differences between countries) to cover travel and other costs.

Results
This section first describes what European citizens know and do with respect to (their) MSW management. Next, participants' barriers and concerns with respect to MSW management are outlined. Third, their ideas for a zero-waste society are presented. In the final section, we present the reflection of the advisory board (AB) on the outcomes of the FGs. Throughout this section, illustrative quotes are provided. Quotes are coded to identify country, focus group (FG), participants (P), and moderators (M), and are presented in a standard format.

Participants' Knowledge and Behavior Related to MSW
There is a large diversity in whether participants recycle and separate waste correctly, depending on both personal (e.g., knowledge, skills, and motivation) and contextual (e.g., housing situation, local waste management system, infrastructure) factors. Indeed, the local waste management system was found to have a strong influence on how European citizens deal with their waste.
It was noted that participants from certain Western EU countries (e.g., Belgium, Denmark, Germany, and Sweden) reported that they usually recycle and separate waste as directed, due to a well-organized waste management system. They expressed satisfaction about the quality and availability of facilities, and the way that the waste system was organized (e.g., schedule for waste collection published). However, some participants from these countries indicated that separating and recycling waste is complex because of the many rules and exceptions. Participants who reported that they occasionally or never recycle and separate waste were mostly citizens of Eastern European countries (e.g., Bulgaria, Latvia, Romania, and Slovakia). They indicated that in general, they did not separate and recycle waste because of the less developed waste management system (e.g., schedules for waste collection are lacking or unreliable, and fewer accessible waste collection facilities).
In general, participants had limited knowledge of waste pathways, and in fact, most had no idea what happened to their waste after being collected by local waste management companies. Participants from Western European countries had slightly more knowledge about waste pathways.

Barriers and Concerns Related to MSW
In the following section, the identified barriers and concerns are presented and clustered according to the waste 'life-cycle thinking' approach as it was introduced to the participants: (1) waste prevention and production; (2) waste management at the household level; and (3) waste disposal and pathways. Table 2 provides an overview of the most frequently mentioned barriers and concerns.

Waste Prevention and Production
The most frequently formulated barrier was excessive (plastic) packaging of products, with many products being individually wrapped, often with various types of material and multiple layers, and (cheap) packaging materials (e.g., plastic) forced upon them.
When I go to a store, it is wrapped, already complete; for example, a cream is in a good, really in a plastic or in a small box, still, there is a larger box and when I go from the store, they push me to a plastic bag, and I really don't need it. (Slovakia, FG3, P1) Citizens acknowledged that this is partly due to the EU's hygiene standards (and citizens' demands), as well as due to product marketing. It was felt that packaging adds to the price, while disposing of it also costs money. Furthermore, many participants indicated that few eco-friendly packaging alternatives are used.
In a number of FGs, it was also mentioned that nowadays, shops do not provide opportunities for people to bring their own packaging (e.g., jars), and that citizens themselves do not bring their own (eco-friendly) shopping bags and keep using plastic ones. Another frequently mentioned barrier was waste production as a result of overconsumption, for which both producers and consumers were held responsible. Consumers buy more than they need, resulting in more waste.
We're used to buying food and then throwing it away. Next, I thought about clothes on more than one occasion, how shops are packed full of clothes. We buy them without thinking. OK, maybe we give some of them away but, really, most of them are just thrown away, and the production just goes on.
On the other hand, producers often package their products in bulk, and some participants indicated that marketing strategies (e.g., targeting children) also persuade people to consume more than necessary. The limited lifespan of products was also mentioned in a number of FGs, and participants specifically addressed the "planned obsolescence" phenomenon: products are manufactured in such a way that they have a short lifespan. A few participants also mentioned that recycled products (e.g., bio-bags) are not aesthetically attractive or of good quality, and are therefore undervalued. In some FGs, participants mentioned that too many advertisement leaflets significantly contribute to waste production. Finally, some participants indicated that many citizens have limited awareness on the subject of waste prevention as a strategy for waste production, and therefore do not take measures to prevent the production of waste.

Waste Management in Households
In almost all of the FGs, participants mentioned the inconvenience of waste bins and bags in and around the house, as these take up much space, produce unpleasant smells, and attract vermin. It was noted that this issue was mentioned more often by participants living in small houses and apartments in urban areas/cities. Furthermore, most of the participants expressed the view that proper waste separation requires time, effort, and money. Many shared concerns regarding products that need to be cleaned (e.g., in some cases, glass), packaging comprising different materials that cannot be separated easily, and the financial costs that arise when buying different bags and bins for separate waste collection, all of which are perceived as burdens.
But the recycling of cans and glass, I rebel a little against, as I waste way too much water and cleaning on them before I can put them in that recycling place because they have to be cleaned. (Denmark, FG3, P3) In almost all of the FGs, participants talked about citizens' limited awareness on the subject of proper waste separation, generally because products are not labeled with proper disposal information, e.g., how to dispose of cooking oil or cat litter, and consequently, these products end up in a general waste bin.
I need to see written on packaging 'this box must be thrown into this, the inner package into that' . . . Finally, in many of the FGs, participants expressed concerns about citizens' negative attitude and behavior toward proper waste separation. They specifically mentioned that people in general are lazy or lack motivation, or they tend to mirror others' negative behavior, or they just do not feel responsible for dealing with waste properly. In a few of the FGs, citizens' negative attitude toward waste separation was seen as a common socially accepted phenomenon.

Waste Disposal and Pathways
In most of the FGs, participants mentioned that the accessibility and availability of public waste bins around the house was not optimal, as they found that these were inconveniently located (i.e., too far from the house) or because there were not enough bins/containers for sorted waste. Furthermore, participants expressed their irritation about waste collection and indicated that waste collection schedules are confusing or unavailable, and that waste is not collected frequently enough. As a consequence, people place their waste beside the bins, causing unpleasant smells and attracting vermin, or they throw their waste in the wrong bins, or dump their waste at illegal sites.
If they would collect it more frequently, you know, like two or three times a week . . . right now we take away the rubbish, after we keep it in our apartment for some time, or we put it in some other bin that is nearest, but then we fill up their bin, you know. (Slovenia, FG1, P1) Several participants also complained about the complexity of the waste management system, which often involves communal bins of different colors for different kinds of waste, which can be confusing when neighboring municipalities use different colors for their bins. Participants also experienced barriers related to the accessibility and affordability of waste collection points for larger (bulk) waste. They indicated that it is a barrier to transport bulk waste to these designated waste collection points, as these were considered to be inconveniently located, have inconvenient opening hours, and be crowded on the weekends. It was commented that when waste collection points charged a fee for waste disposal, it would encourage illegal dumping. In most of the FGs, especially in Eastern EU countries, participants see the current existence of landfills and incineration points as problematic, as these have a negative environmental impact. In a range of countries, several participants considered that the negative attitude/behavior of citizens toward proper waste disposal forms a barrier, as people are lazy and just do not make the extra effort to do this properly.
People don't like to walk an extra 50 m to go to the car park; instead we double park in front of the shops. The same with rubbish: we don't take the trouble going another 100 or 200 m, but throw it in the rubbish bin or throw it out of the window. (Portugal, FG2, P3) It was also mentioned that some citizens have insufficient awareness of proper waste disposal and pathways (what, how, and why to recycle), and therefore do not do it properly. Finally, in almost all of the FGs, concerns were raised that regardless of citizens separating waste, it might end up together for incineration or landfilling. They suspect that some of the waste management companies do not collect and treat sorted waste separately, possibly due to other interests.
[P1] But when they come [ . . . ] to pick up the rubbish, and they pick up all three containers together, in one truck . . . I wonder why we are collecting it separately . . . .
[P7] This is completely not motivating people to collect waste separately. (Bulgaria, FG3)

Citizens' Ideas on a Zero-Waste Society
The notion of the waste hierarchy framework was introduced to invite participants to think of ideas to achieve a zero-waste society. Collectively, the participants proposed over 350 different ideas for research, innovation, and policy on MSW. We identified two broad categories of ideas, each consisting of four sub-categories (see Table 3). In the following section, we give an overview of the diversity of these proposed ideas. We focus on ideas that were mentioned and/or prioritized in at least eight countries, as this emphasizes the transnational character of VOICES. Ideas often reflect the personal situation of participants, focusing on innovations at the household and community level. Table 3. Research domains of proposed ideas and number of different ideas mentioned across all of the focus groups (FGs).

Technological Ideas
First, participants proposed a broad variety of technical engineering ideas, which require the development of technologies, machines, or techniques to realize. Popular ideas in this sub-category were related the development of machines that would use waste as a resource to create energy (at the household and community level) or new products/materials. Other ideas, focusing on enhancing in-house waste management, include machines that can automatically sort, separate, and/or compress waste, and (home) buildings equipped with an integrated waste disposal system. In many countries, participants also came up with ideas to get rid of waste by dissolving or disintegrating it, dumping it in oceans or volcanoes, and by sending it into space or to the moon, the sun, or black holes. Such ideas were sometimes accompanied by a discussion among participants in which they agreed that this type of short-term solution would not contribute to reaching a zero-waste society.
[P3] No, personally a clear future vision for me would be, maybe we can achieve space transport and shoot all that crap to Mars.
[But] what will bring the energy back?
[P10] At least the crap is out of sight, but it's not gone. (Austria FG3) Second, a range of other ideas focused more on preventing waste by using fewer, less harmful resources. In almost all of the FGs, participants suggested the need for more research in developing packaging materials that are environmentally-friendly (e.g., biodegradable), and that it is crucial to substitute conventional plastics. In addition, packaging that is reusable, recyclable, and/or edible was presented as a worthwhile idea. Citizens also expressed that more research into developing recyclable and/or re-usable products with a longer lifespan would lead to less waste production.
[...] materials that could be recycled more times, that might be more durable so they last longer [...] They're less likely to break. But you can also recycle them more times, so there's not like a limit of how many times. (United Kingdom, Newcastle-upon-Tyne FG1, PX) Third, participants proposed bio(techno)logical ideas such as (micro) organisms that could aid in the process of waste treatment. Also popular across the FGs was the idea of developing food in the form of pills, as this would take up less space and prevent waste.
[P5] I don't know if that's even feasible, but there wouldn't be any packaging, actually, there would just be countless amounts of pills . . .
[P5] Exactly, and as soon as you put them inside some machine, they'd be turned into a cooked meal. (France, Paris FG1) Finally, in a number of countries, participants proposed that ICT could be used to raise awareness of proper waste management, stimulate recycling, and trigger behavioral change. As such, the development of an app that provides general information on appropriate waste separation and disposal measures, and contextualized information regarding waste collection schedules and collection points was proposed.
Since we don't know what happens to the rubbish after we throw it away, this app will tell us where it is. (Italy, Naples FG2, P3)

Non-Technological Ideas
First, a broad range of policy-related ideas were proposed. Participants across all of the FGs mentioned that incentives, in the form of vouchers, money, tax reduction, or rewards, could be used to improve recycling and trigger behavioral change. They also mentioned that waste collection centers should be free of charge. Simultaneously, they argued that disincentives, in the form of fines, should be imposed on those who do not manage waste properly. In most of the FGs, participants proposed regulations/incentives aimed specifically at producers. As such, participants thought it would be valuable to introduce legislation obliging producers to prioritize the usage of recyclable/re-usable/ecofriendly materials over other materials. In some of the FGs, it was made explicit that specific incentives to reduce the amount of harmful materials, by for example banning plastic or increasing taxes on these materials, could be adopted. In addition, they proposed that the use of 'bio-products' can be encouraged among consumers by making these more affordable while imposing higher taxes on products that are not 'bio'. Furthermore, the participants proposed standardizing and regulating waste management at the national, regional, and/or EU level, and devoting more government funding to research and projects aimed at improving waste management. Second, a variety of ideas related to MSW management and logistics were proposed. Participants suggested the idea of expanding the current deposit return system for various other types of waste, such as bottles, glass jars, aluminum, electronic appliances, cooking oil, and all types of plastic. Citizens also considered it relevant to improve recycling points (e.g., more conveniently located points, better parking space). Participants often talked about supermarkets, making it possible for consumers to bring their own packaging and buy in bulk.
[P4] We wrote down that you should encourage bulk distributors. This happens in organic stores for cereals and it could be useful for pasta, rice; it stops waste [...] [P7] For the first visit, you buy the milk with the bottle, and then afterwards you go with the same bottle. (France, Grenoble FG2) Third, a number of ideas related to communication and education, which were mostly aimed at increasing awareness and triggering behavioral change among citizens, were put forward. Many participants highlighted the importance of education specifically aimed at children and youth on waste-related issues (e.g., urgency, consequences, proper waste management) as a strategy for waste prevention and management. Furthermore, more research on how to change lifestyles and behavior regarding proper waste management was also addressed across all of the FGs. They also suggested that labels providing information on how to recycle specific products would be helpful.
[P4] Yes, we also had that idea, it should say on the package, how it should be sorted. And in large print, not in some tiny print that requires a magnifying glass to be readable.
[P10] In colors maybe, like it is on deposits for bottles.
[P4] Well, yes, some marking, package classifications or something. (Estonia, FG2) Finally, a variety of local initiatives were suggested. Local production was embraced among participants, and in several of the FGs, it was mentioned that communities could benefit from producing their own food in gardens and on rooftops. Participants also suggested that recycling and the re-use of products could be improved by setting up consumer exchange shops/centers.
[P5] In Berlin, there is now a range of 'lending shops' where people bring things they don't need and then someone else can take it if they need it.
[P3] Yes, there are already sort of internet cafés and places where people bring their stuff and then there are proper specialists who try to repair it. Toasters and televisions and if they can't fix it, then you've got to buy a new one of course. But at least the ideas come back. And I think that's great, [it's] much better than nowadays when you can't fix it yourself, or when the electrician says, no, I haven't got the part for that, you know, or it's not worth it, chuck it away, you know, and then there's still people who say they'll give it another go. (Germany, Bremen, FG1)

VOICES Advisory Board's Reflections on the Citizens' Ideas
Although not all of the proposed ideas incorporated the vision of a zero-waste society, it is clear that they were inspired by (1) one or more of the identified barriers/concerns; (2) the waste hierarchy concept; and/or (3) the life-cycle thinking approach. The question now becomes: 'How to make sense of the ideas resulting from the consultations?'. The VOICES advisory board (AB) was asked to reflect on these ideas in terms of innovativeness and feasibility based on their experience and expertise in the fields of social research or waste management. In general, they found that while many of the proposed ideas were not innovative or feasible, a considerable number seemed to be potentially valuable for shaping EC R&I on urban waste, especially in the field of social and human sciences research into policy, management, and communication. Below, we briefly present the main reflections of the AB.
The AB noted that the degree of innovativeness varied among the ideas, and that most of the ideas were not innovative compared with current waste management practices in general (e.g., provide incentives to make consumers recycle more, educate people about the importance of MSW). However, while a number of ideas might already be in place in some countries, these can still be considered novel for a specific country/region where these ideas have not yet been adopted. The AB considered these ideas to be innovative within a specific context (e.g., setting up a system with different colored bags with clear information on when these are collected, and the introduction of waste taxes). Citizens who proposed more innovative ideas (e.g., a disposal bin with a barcode card that holds information on the people living in one's household, limiting the amount of waste that can be disposed of, or an eco-friendliness card providing supermarket discounts) were mostly from countries in northwest Europe (e.g., Austria, Belgium, France, and the UK), which are characterized by more advanced MSW systems. The AB argued that citizens not only came up with ideas based on their own knowledge and creativity, but also in terms of building on current practices within their local context. Although these ideas are not innovative at the general level, it would be useful to publicize them across the EU and so also adopt these "successful" ideas in countries that have not yet done so. Interestingly, in 13 countries, citizens themselves also proposed the idea to "adopt good MSW practices from other countries".
Participants suggested a range of ideas that were considered less valuable for EC R&I on urban waste, and the AB noted that this was mostly related to the nature of the ideas. First, several ideas were considered far-fetched and less feasible as they required, for instance, significant infrastructural changes (e.g., the use of pipelines to reduce packaging, having certain foods as powders or in concentrated form, entering the home via a system similar to the water supply), or would involve consuming vast amounts of energy to run as well as the energy and materials to build the suggested machines and devices (e.g., machines that would turn waste into new materials). Also, a number of ideas were not considered socially/ethically acceptable (e.g., develop food in the form of pills, or install cameras at waste collection points to monitor who throws waste in the wrong container) as these might trigger a negative reaction among citizens, or politically acceptable (e.g., reduce consumption by reducing choice in supermarkets).
The AB also indicated that although participants collectively proposed a large number of ideas, many of which were less innovative and feasible, they did prioritize ideas that were more innovative and feasible. For example, the idea to "send waste into space, bury it under sea, dump into a volcano, earth, black hole, sun" was mentioned in 19 countries, but highly prioritized only in one. Another example is the idea to have a "system to transfer products directly from the shop/producer into the house", requiring huge infrastructural adaptations and as such hampering feasibility, which was mentioned in 10 countries, but ultimately prioritized in none. Rather, citizens highly prioritized education and awareness programs, and ideas that would make the sorting and disposal of waste more convenient within and around their household. One of the AB members mentioned: . . . it can be stated that realistic ideas obtained the highest priority in most of the cases. Thus, the prioritization leads to an assessment of more or less realistic ideas . . . this is some kind of internal quality control that helps to identify the most popular futures research areas . . . Another interesting finding from the AB's reflections is that it might seem straightforward to only focus on popular ideas-ideas mentioned in, for example, at least eight countries, and/or highly prioritized by a significant number of citizens-but in doing so, certain promising ideas, that were poorly represented overall (mentioned in only one or two countries), may remain ignored. While these ideas lacked representation across all of the FGs, and therefore tend to be invisible among the large range of ideas, the AB noted a number of ideas that they considered innovative and potentially effective. For example, only in Luxembourg did participants propose and highly prioritize the idea to "focus on marketing to increase demand for more sustainable products". The AB commented that demand can be a powerful force for social change, and if carefully done, this idea could have great impact. Also, only in Estonia and Portugal did participants mention and prioritize the idea of arranging win-win contracts between waste management companies and other large organizations (e.g., universities) on heating and/or power supply.
Finally, the AB indicated that citizens do take a life-cycle perspective on waste management, but that participants generally focused more on problems they experienced themselves when dealing with waste, yielding ideas close to their own realities. In doing so, ideas related to convenience in the household and improving the local waste management system were considered important. This should not come as a surprise, because people generally tend to think from their own reality and perspective, and indeed were invited in the FG to present that perspective. One of the members of the AB remarked: After some more thinking, it became clear that convenience in households is, from a citizen's point of view, one of the most important issues, and hence absolutely justified to talk about. To a certain extent, the increase of convenience by new waste technologies may therefore also be a good way to make people support modern waste management.

Discussion
This section reflects on the results from the FGs, and discusses their relevance for the future of EC R&I on urban waste management, and on EC policy and public engagement in general. In addition, methodological considerations are discussed.

Results from the FGs
Participants were able to reflect on their personal knowledge, and experienced barriers and concerns related to MSW management. In general, participants recognized the importance of reducing the amount of waste and appreciated the idea of recycling, but often felt constrained to act accordingly. We observed clear differences between countries, e.g., many people particularly from Eastern European countries indicated that they occasionally or never separate and recycle waste, while in most Western European countries, waste separation and recycling was common practice among participants. This is consistent with Flash Eurobarometer 316 [25], in which at least a third of the respondents from Lithuania (33%), Romania (38%), Latvia (40%), and Bulgaria (42%) did not separate waste for recycling or composting, while these numbers are much lower for Luxembourg (3%), Austria (3%), Denmark (3%), Finland (4%), the Netherlands (6%), and the UK (7%).
The articulation of barriers and concerns served as a basis for ideas to achieve a (near) zero-waste society. Although we noted that it was more complex for participants to think of concrete innovative and feasible ideas for the future, many ideas were put forward. On the one hand, citizens mentioned ideas that were already being practiced in some countries (lacking innovativeness) while, on the other hand, some ideas were quite futuristic, requiring technological breakthroughs and large infrastructural changes (lacking feasibility). Given their limited knowledge of environmental sciences and technologies, it is less straightforward for lay people to pre-assess the relative novelty and feasibility of their ideas. At the same time, it was part of our design to allow a brainstorm session that encouraged creative and out-of-the-box thinking and welcomed all ideas. However, when asked to prioritize ideas in the FGs, participants chose more realistic and feasible ideas, showing that common sense knowledge enables citizens to roughly estimate feasibility.
Citizens specifically prioritized small-scale innovations to be implemented in or near their own homes from which they would benefit directly. This is reflected in the popularity of ideas such as the self-sorting waste bin, composting facilities in the neighborhood, or homes with a built-in disposal system. In addition, participants usually gave high priority to communication and education related to waste prevention and management, indicating that they considered citizens' lack of awareness and knowledge as an important barrier to proper waste management. In almost all of the FGs, citizens were convinced that educating children at school and showing the consequences of pollution to adults is necessary for proper waste management. However, knowledge levels may differ widely among European citizens, and thus concrete knowledge gaps need to be assessed before educational campaigns can be designed. For example, De Feo and Gisi [36] undertook a quantitative study of awareness, attitudes, knowledge, and behavior in relation to urban waste prior to designing tailored educational campaigns.
With respect to the transnational character of the project, it is worth mentioning that contextual influences exist and matter. In particular, we found that the difference in the extent to which a countries' respective waste management systems can recycle waste effectively influences knowledge, perceptions of barriers and concerns, as well as ideas and priorities. EU countries can be ranked based on the percentage of MSW recycled (see Table 4) [35]. Reflecting on our results, we note that participants from countries with a higher amount of 'MSW recycled' live in a context where the waste management system is more complex and advanced, with various instruments to stimulate recycling, including taxes on landfill and incineration, bans on landfill of non-pre-treated organic waste, mandatory separate collection of non-packaging waste, and economic incentives for recycling [35]. In countries with a lower percentage of 'MSW recycled', there are fewer instruments, and more waste is incinerated and landfilled, and participants propose a variety of other ideas that could be considered novel in their context, but rather dated for a R&I agenda. This suggests that if citizens' perspectives and ideas are to be incorporated in a transnational R&I agenda, based on their relative novelty, the chances are higher that the agenda will be primarily defined by citizens from those countries with more advanced waste management systems.
Countries also differ in, for example, their waste network designs: the process of collection, separation, sorting, and reprocessing [37]. Some countries (e.g., Germany and the UK) have their own domestic material-recovery facilities, while smaller countries (e.g., the Netherlands) have most of the sorting process done in neighboring countries. Both the country's geographical location and infrastructures (such as railway and waterway), as well as the capacity and quantity of the available facilities, influence the design of the waste network. Finally, collection planning-entailing the organizing unit for collection, allocation of waste types to the curbside or drop-off, type of trucks-differs across EU member states. For example, in Spain, mixed waste is usually collected at the curbside, while paper and glass are deposited at drop-off points. All of these contextual differences might have influenced citizens' perception of the waste management system, and their ideas on how to achieve more sustainable waste systems.
In our study, participants were not only engaged on the basis of their experiential knowledge perspective-as 'users' in the waste management system-they were also asked to provide their ideas and preferences on future research directions. The proposed ideas are the output of a 'black box', which has as its input the articulated experiential knowledge (barriers and concerns), guiding cognitive knowledge (waste hierarchy, life-cycle thinking, and zero-waste society), contextual influences (local waste management practices), and cognitive skills (ability of citizens to think in abstract, creative, out-of-the-box, novel, and realistic ways). Understanding more of how these building blocks influence each other and citizens' cognitive processes when proposing ideas for a R&I agenda might enable us to better identify, interpret, and use the distinctive and meaningful societal knowledge that emerges.

Relevance to EC Policy
The results of this study show that citizens' collective wisdom on waste management, policy, and research issues legitimize various EC priorities on urban waste. Citizens were concerned about excessive packaging, and would like fewer and better packaging materials, such as fully biodegradable packaging and alternative plastics that can be fully recycled without the loss of quality. Furthermore, citizens want more easily recyclable and reusable products, and more recycling points at convenient locations. They also want producers to be more rigorously regulated, taking responsibility for the lifespan and recycling of the products that they manufacture. This supports current EC initiatives to encourage industries to reduce waste [38]. Our study also confirms that, in many areas, citizens have knowledge of the challenges facing waste disposal in Europe, and they agree with many current EC policies.
Besides legitimizing current EC priorities for research and regulations on urban waste, the results of this study also highlight the contribution of EU citizens in complimenting current R&I priorities with regards to MSW management. First, European citizens addressed the crucial factor of convenience in the household, and they indicated that there is a clear need for devices to facilitate the sorting and compacting waste at home, or technologies that allow waste to be used as a resource in the household. This area of MSW management seems neglected by current research. Furthermore, citizens want incentives to separate their waste, such as deposit systems and reward schemes. More communication and education, especially for children, on the topic of waste management is necessary, as citizens were largely unaware of waste pathways and would feel more engaged in the process if they felt that information was more widely available. Finally, citizens indicated that specific technology systems using chips, electronic tags, and apps may help to motivate people to recycle. The results of the FGs can therefore be relevant for input into (1) national, regional, and EU policy-making processes; (2) national, regional, and EU research programs; and (3) rethinking governance options.
The results of the project were primarily used by the EC to draft the call for proposals on 'Waste: A Resource to Recycle, Reuse, and Recover Raw Materials-Towards a Near-Zero Waste Society' within the Horizon 2020-Climate action, Environment, Resource Efficiency and Raw Materials-Work Programme 2014-2015 [39] for which approximately 161 million Euro was made available. Within this, five of the seven topics (see Table 5) explicitly state to respond to the EU research priorities identified within VOICES, with specific reference to the projects' website. This illustrates the commitment of the EC to embed the results of the citizen consultations in the process of EC R&I agenda-setting. However, it remains less clear to what extent the specific outcomes of VOICES have actually inspired researchers to draft proposals. Furthermore, the project provided insights into citizens' perspectives on barriers and concerns related to waste management, which policy-makers at the local, national, and EU level could use in order to develop EU-level waste management policies. It is unclear to what extent this was realized in practice.

Methodological Considerations
The process of the FGs was evaluated by the FG participants, the independent advisory board, and two external evaluators. In general, the participants enjoyed the FGs and considered them well organized. Many participants said that they had learned a lot about MSW. More than half of the participants claimed that the event had made them 'change their views' or that their view had been 'strengthened'. Furthermore, they appreciated the initiative of the EC to consult them, and were curious to find out what would be done with their input. Some participants raised doubts as to whether their input would have an impact, while many expressed the hope that it would influence the EC R&I agenda. The advisory board expressed satisfaction with the process, arguing that the well-structured FG approach was very effective in gaining information about the European citizens' general (lack of) knowledge on waste management. This was supported by the external evaluators, who concluded that the FG design was very rigorous.
Although overall the moderators successfully led the FGs and followed the script, there was some diversity in the quality of the FG moderation between different EU member states. In some cases, moderators let the discussion digress towards more general issues of sustainability. Occasionally, moderators became engaged in the conversation to the extent that they were almost bringing in their own ideas for R&I. In a few other cases, moderators did not probe participants sufficiently to identify the reasons for proposing certain ideas. However, we found that the results of these FGs were not significantly different compared to the others.
With respect to participants, the aim to achieve diversity was realized. We did not apply exclusion criteria, e.g., citizens who could be considered stakeholders, such as an active member of a civil society organization concerned with waste or an employee of a waste management facility. Citizens with such a stakeholder 'hat' would be more knowledgeable on the topic, have clear pre-formulated standpoints, and could thereby strongly steer discussions in the FGs. Although we did not apply such exclusion criteria, the moderators did not come across such behavior in the FGs, nor did we observe this in the transcripts of the FGs. While we cannot say with certainty that no participants were involved that could be considered stakeholders, we are convinced this did not considerably influence the results of the FGs.
Despite the fact that VOICES was able to collect and analyze all data within six months (necessary to meet EC deadlines), it is recommended that such a rather short time frame should not be adopted for future similar EU wide consultations. The time frame was short for such a large study, allowing less time for flexibility and contemplation. It is proposed that future similar endeavors should take roughly one year to collect and analyze all data. However, in terms of cost-efficiency and feasibility, the EC should consider whether it is necessary to consult citizens of all EU member states for future R&I agenda-setting initiatives. Such initiatives are not only costly to execute, they also require a large team of researchers to analyze the enormous amount of data. Alternatively, a sample of for example ten countries could be considered, representing (contextual) variety within the EU in terms of the area (or system) under consideration.
That being said, and as noted by the projects advisory board, some ideas with great potential were mentioned in only a few FGs. Consequently, not all of the participants were exposed to these 'rare' ideas when conducting the priority-setting exercise during the FG. This brings us to the question of whether it would have been better to organize a separate priority-setting activity after the FGs in which all of the different ideas voiced in the FGs were included-either at the national or EU level, as has been successfully done in various national research agenda-setting processes with patients [14,40]. Needless to say, these design choices have significant consequences for the logistics of the public consultations.

Conclusions
The present research identified a number of barriers and concerns, as well as ideas for R&I related to MSW management from the perspective of EU citizens. In this way, VOICES opened the floor for citizens to think of desired future R&I directions that are relevant to them, while taking their own experience with the system into account, as well as the guiding visions of the EU. The research also explored the quality, in terms of innovativeness and feasibility, of citizens' proposed ideas to realize a zero-waste society. While many individual ideas were not considered innovative and/or feasible, participants did prioritize more feasible ideas, indicating that common sense knowledge enables citizens to roughly estimate feasibility. In addition to supporting some of the current EC priorities, citizens also proposed various new ways to strengthen current MSW management (e.g., innovations to enhance convenience in household waste management).
We argue that citizen involvement in EC R&I agenda-setting across the EU is feasible, and leads to meaningful input. Future large-scale, EU-wide public consultations aimed at informing R&I agendas need to take contextual differences into account. This leads to a more complex process of interpreting the results and extracting relevant knowledge, but this prevents the outcomes (i.e., R&I agenda) to be less steered towards countries that have more developed and innovative systems, and be more inclusive to all. In this process, specific attention should be devoted to the distinction between outcomes that appeal to the EU level, and outcomes that foster more region-specific policy-making. At the same time, these EU-wide studies enable us to get a bigger picture of the problem(s) at stake and allow us to integrate existing solutions across contexts. For a representative priority-setting, it is advised to consider extending the agenda-setting process with a separate quantitative phase (e.g., using questionnaires). This enables EU citizens to prioritize topics other than the ones they identified themselves.