Next Article in Journal
Greening the Ivory Tower: A Review of Educational Research on Sustainability in Post-Secondary Education
Previous Article in Journal
¨ A Dilemma of Abundance: Governance Challenges of Reconciling Shale Gas Development and Climate Change Mitigation
Article Menu

Export Article

Sustainability 2013, 5(5), 2233-2251; doi:10.3390/su5052233

The Pragmatic Collective Interest as the Product of Civic Deliberation: The Case of Pesticide Management in Belgium
François Mélard * and Marc Mormont
Department of Environmental Sciences & Management, University of Liège, 6700–Arlon, Belgium
Author to whom correspondence should be addressed; Tel.: +32-63-230-901; Fax: +32-63-230-818.
Received: 19 March 2013; in revised form: 29 April 2013 / Accepted: 6 May 2013 / Published: 16 May 2013


: Through the issue of pesticide management in Belgium, this article offers an empirical and conceptual grasp on what Ulrich Beck called the second-order reflexive modernity; that which is exercised among citizens when they are confronted with threatening and uncertain situations. To achieve this, we use two case studies of two public policy instruments, which we offer to the public for discussion: food product labelling, and the modelling of toxic effects linked to pesticide use. To this end, we organised two focus groups designed to encourage discussion, composed of citizens/practitioners. The results obtained plead in favour of a collective deconstruction-reconstruction of these tools and can lead to what we propose calling a “pragmatic collective interest.” This “pragmatic collective interest” can take the form of a new set-up or new associations that enable the coexistence of conflicting propositions and points of view, and a suspension of efforts to hierarchize causes and required solutions.
participatory research; pesticide; food; deliberation; reflexivity; governance

1. Introduction

This article aims to highlight original proposals made by citizens/practitioners when engaged in a participatory set-up (dispositif in French) with a deliberative orientation (a deliberative focus group). We conceive of these as the expression of what we have chosen to call a “pragmatic collective interest” so that it may be contrasted with other similar notions (like common interest) that are associated more with representative democracy. We deduce it from the results of two projects related to the issue of pesticide management in Belgium. The article is structured so that the concept of “pragmatic collective interest” is highlighted through lessons drawn from these case studies. It will begin by outlining the context of Belgian pesticide management and risk evaluation, particularly the associated management style. We then introduce the reader to the deliberative option and its methodological consequences. We will examine two case studies, for which a public policy instrument is used: the case of food product labelling (the “label” project), and the case of risk modelling linked to the use of pesticides (the “PEPAM” project [1]). In both cases, an introduction to the project and its general development is presented, followed by their analysis. We will conclude the article by summarising the argument and some of its conclusions.

2. The Problem

The reduction of pesticide use has featured on the EU’s policy agenda for years. National programs moving in this direction exist in some countries and are encouraged by the EU authorities. The main problem with these programs is generally described as a lock-in effect [2,3]. This lock-in effect occurs when the use of a technology—in this case, pesticides—is associated with a whole series of other techniques, with other knowledge, and with other practices (particularly those related to consumption) that make it difficult to implement the change. In fact, pesticides are associated with so many other elements of the socio-economic system that replacing them implies modifying all of these components. This results in a persistent inertia in the socio-technological system.

Nevertheless, the pesticide use reduction imperative emerges mainly from health concerns that are increasingly regarded as environmental health risk factors. These risk factors have been part of the history of environmental policy since the publication of Rachel Carson’s book [4] and after a great deal of civic action; they are now clearly recognized by the scientific literature, even if there remains some controversy with regard to their evaluation. The pesticides’ risk assessment faces two major difficulties: it deals with diffuse sources, and long-term effects difficult to perceive and to measure (latent risks) [5]. From this it follows that they represent so-called modern risks: risks that result from the use of technology, and are burdened with substantial uncertainty. They therefore present a twofold problem: first, that these risks present the problem of trust in scientific and technological expertise; second, that their management interferes directly with the economic and technological trajectories of the sectors concerned.

2.1. Dealing with Modern Risks

Modernity has always dealt with the risks that its innovations have engendered. It is therefore not the risks themselves that distinguish our current situation. However, the first modernity, or rather what Ulrich Beck calls the first-order reflexive modernity, dealt with these risks in some ways [6]. The perspective was that of a linear development, towards more progress and the control of nature and risks. From this perspective, dealing with risks involves a rational approach that proceeds on the basis of the following assumptions:


analysis of the dynamics is possible and controllable, enabling the effects of actions to be predicted;


evaluating the underlying values of each option enables the desired objectives to be defined in a legitimate manner;


it is then possible to implement strategies and control their effects.

The key to this rationalization model is eliminating uncertainty and ambivalence, and controlling interference. It makes reductions by selecting relevant elements, by isolating causal chains to identify and distribute responsibilities. This is the model that underpins modern science, technological development and social organization, that is to say our society’s differentiation into specialized subsystems that are managed in a relatively autonomous manner.

This approach is brought into question by the emergence of modern risks of the kind that result from the use of pesticides. Thus the risks linked to pesticides are now evaluated through marketing authorization procedures in Europe. These authorizations are awarded by authorities based on files presented by the producers after completion of toxicity studies according to standardized procedures. The European Food Safety Authority [7] recently published a scientific opinion that emphasizes the strong limitations of current assessment methods. These methods are criticized because they are ultimately far removed from real ecological conditions, in which pesticides are applied in weak doses, but persistently, and they do not act alone but in conjunction with other products; they are used in a variety of ways by a wide range of users with unequal competence levels; they disperse into a multitude of connected environments (air, water, organisms, trophic chains). Moreover, these procedures, which are supposed to “control” risk, in fact lead industry to search for new compounds, which might be less dangerous under laboratory conditions, but entail uncertainties in the real world [8].

This being the case, from a second-order reflexivity perspective [9], dynamics such as these cannot be handled with traditional risk-reduction approaches. What is therefore required is to get more thought directed at knowledge creation and problem management. In the use and development of pesticides, several rationalities coexist and interconnect: that of the production facilities, that of agronomists and farmers, even that of consumers. The improvements are ambiguous and cause mutual interference between plant health and human health, as well as between different conceptions of quality. It is no longer a matter of resolving a clear-cut problem, but rather of a second-order governance that consists of “a procedural approach towards reflecting the interdependencies, understanding aggregate effects of specialised concepts and strategies, and engaging in the modulation of ongoing societal developments by establishing links, organizing problem-oriented communication and interaction among distributed steering activities” [10].

2.2. Policy and Research in Belgium

Belgium is a small country, with a very open economy. Its highly developed farming industry is well established on the international market. Many sectors of this industry strongly depend on the use of pesticides whose presence in water sources and foods is a recognized fact. Even if statistical data are poor, Belgium is one the biggest consumer of pesticides by ha of arable land. It was in reaction to the pressure of EU policy and environmental groups that the Belgian government adopted a policy in this area in the early 2000s. According to the European Strategy, Belgium opted for reducing the risks linked to pesticides, making its policy a first-order reflexivity since, as we shall see, it assumes an objective evaluation of these risks. At the same time, however, no doubt tempted by use of the economic instruments of environment management, it appears that the government wanted to explore the role consumers could play in a reduction policy through their consumption choices.

2.3. The Deliberative Option

The authors of this article were the scientists in charge of two projects: the “Label” project, which studied the role of labeling in a pesticide reduction policy, and the “PEPAM” project, on a public examination of a statistical model for evaluating risks linked to pesticide use. In these two cases, the authors proposed to make use of deliberative methods to explore these two questions. Using models based on small-group deliberation is nothing new in social sciences [11]. Originally intended for the analysis of message reception, then for the evaluation of innovations in the health sector [12], these methods are enjoying renewed interest because of the public’s involvement in decision making. This occurs as an example in technology assessment, which is specifically based on the consensus conference method, because of the recognition in social sciences of the role the public could potentially play in research policy, such as through patient associations [13].

Habermas can be recognised as one of the principal authors to have given meaning to deliberation. He has already long stressed the possible democratic fruitfulness of deliberation that does not consist of negotiation between interests—something that reduces society to a strategic game—but is also not adherence to a community with predefined values. He defends an action type that is neither strategic nor communal, which he calls communicative action [14], action that aims at mutual understanding. This understanding results from discussion between different, equal perspectives seeking to establish agreement on the basis of reason, but not an instrumental reason. It is a reason that can be said to simply emerge from the discussion, out of the individuals’ involvement in situ ations they have experienced that make sense; they can recognise one another instead of objectifying each other through science, technology and economic or political markets.

Habermas recently made the link [15] between the deliberative methods mentioned above and his conception of deliberation, thus lending a kind of theoretical or philosophical support to methods that had until then been considered research methods or decision-making aids.

There is consistency between second-order reflexivity and deliberative methods. Even if Habermas saw deliberation as first and foremost a matter of norms, his reference to situations experienced as opposed to abstract systems is quite reminiscent of the question of what kind of knowledge needs to be taken into account—knowledge being, in this case, inextricably interwoven with norms. Deliberation is therefore one possible avenue with which to experiment, in order to reconstruct systems of knowledge and norms.

In contemporary societies, standards and knowledge are particularly interlinked in public policy instruments. This is because these instruments rely heavily on knowledge of risks and benefits, as well as on standards relating to their distribution. These instruments are not exogenous; they are derived from ethical knowledge and standards, but then they serve as cognitive frameworks and prescriptions, and they structure social relationships [16]. Thus they constitute a prime area for second-order reflexivity. This is why it is public policy instruments that are here subjected to public deliberation.

3. Material and Methods

In the two research projects cited, we opted for research methods inspired by these deliberative methods, using an intermediary format between a focus group and a consensus conference. For a given time, a small group of 10 to 15 citizens—laypeople—were assembled to explore and discuss a question, and their informed opinion is expected to contribute to the knowledge on which the decision would be based. All the presentations by stakeholders and the discussions by citizens were recorded, typed and used as a corpus for conventional qualitative coding procedures.

One of the principal critics often made against Habermas’s notion of deliberative democracy is that it presupposes a kind of discursive equality among individuals. It is true that when a matter like pesticides is concerned, ordinary individuals are not on an equal footing with scientists or specialised producers. It is therefore necessary to implement a discussion set-up that is suitable for making participants able to discuss situations that are complex and infused with instrumental rationalities.

In both cases, this set-up is based on the following principles:

  • the participants are in no way stakeholders or interest group representatives, but instead citizens who are “interested” in the question;

  • a first work phase consists of giving participants a wide range of information, on the one hand from scientific or professional experts so they can become sufficiently informed, on the other hand from stakeholders (specifically manufacturers, administrators, pressure groups, etc.) so they understand the range of positions already established);

  • the next work phases are dedicated to a discussion that includes an explanatory stage, a discussion stage and a position-taking stage, without any consensus being required, but in a way that encourages heightened expression;

  • the facilitators are sociologists whose task is to encourage democratic expression, by managing the discussion, keeping an eye on the minority voice, and ensuring final propositions are concluded.

All presentations and discussions were recorded to generate research material that would later be analysed by the sociologists, who would seek to extract a reflexive analysis that highlights the originality of the participants’ propositions. In a sense, this is a matter of bringing out the discussion’s internal logic, which is only partially apparent in the final propositions. This is very hard to convey through quotations from these discussions, which are not linear. The participants shift from one topic to another, change points of view and then revise their opinion, working in a manner that could be called “rhizomatic” [17]. This is why we only summarily quote extracts that mark positions and shifts in the discussion.

3.1. The “Label” Project

3.1.1. Introduction and Development

The first project—the Label [18] project—concerned the question of determining whether or not product labelling could influence the development of agricultural sectors, due to the public’s clear preference for products with such guarantees as “integrated pest management” (IPM) or “pesticide-free” [19], as already exist marginally on the market.

In the European context, labelling for food is quite usual for local products, for organic products and, more marginally, for integrated pest management. The labels are regulated by Europe through a common framework and guaranteed by a certification process that checks compliance at the farm level. The labels are not neither public nor private: they are owned by collectives of producers and supervised by European institutions. Labels concern a small part of the total agricultural production but provide significant added value to producers. Private labels exist too but without public control and regulation. Consumers are more and more faced with a multiplicity of labels which can create some confusion and distrust [20].

Financed for three months in the context of a large national research program on sustainable development in the food industry, this project aimed to examine, in a multidisciplinary manner, the opportunity to link this kind of labelling to a pesticide reduction policy. It brought together three teams: (1) an agronomy team specialising in the phytosanitary control and the impact of pesticides, who evaluated the various existing labels from a risk perspective; (2) a team of economists who evaluated the economic relevance of these labels, and finally (3) a team of sociologists who specifically studied the question of the public’s perception of labels, employing a focus group methodology. Members of the public were recruited by means of direct contact, through different types of business (supermarkets, local shops, producer/consumer networks), ensuring an adequate level of social diversity. The development of the deliberative group can be summarized by a series of four phases.

The first phase brought together two groups of a dozen citizens recruited in markets and supermarkets. In its first two parallel sessions, a first piece of information was given to them, on labelling systems and the risks associated with pesticides. The discussion consisted of identifying which questions and themes the participants deemed relevant. This survey went beyond the scope of the question raised, revealing a wide range of knowledge among the participants. They began by questioning the reliability of monitoring and then proceeded to consider not only the relevance of organic foods but also the inequality effects that labels could have on small producers and consumers alike. Then they considered the question of food-related responsibilities: are they exclusively a matter for the state, or of the individual? Finally, they addressed the diversity of the different kinds of quality guarantee and the confusion this can engender, then stressed the issue of consumer information. The three subsequent phases brought together eighteen of the consumers who took part the first session.

In the second phase, they were introduced to four producers: an organic farmer, another who used integrated pest management, another from a local label, and finally a farmer certified by EurepGAP, a form of oversight and traceability implemented by the European big retailers. One point common to all these speeches particularly affected the consumers, namely, these four producers’ acknowledgement of past mistakes in the food industry, and of the excessive use of pesticides. The discussion also brought to light the complexity of food distribution channels and heavy requirements made on producers in terms of distribution and consumer demands for “quality.” Next, the participants made a clear distinction between labelling indicating respect for compulsory standards (imposed either by the law or by buyers) and that based on voluntary adherence by groups of producers.

The third phase was introduced by four representatives from organizations promoting the labels presented by the producers in the previous session. The participants then engaged in a lengthy discussion about labels in terms of public guarantees or simple private approaches. Thus were raised, on the one hand, the question of trust and, on the other hand, that of the contents of available guarantees in terms of health. In counterpoint, there also arose the question of proximity networks and the alliances possible between producers and consumers.

The fourth and final phase involved only the participants, and focused on the propositions they would be making through the research process, since the scientists who had organized the project intended to include these propositions in their report. In a surprising way, the discussion took the form of a process of conceiving a “quality sign” capable of legibly integrating the different requirements they deemed necessary for describing a product. Finally they proposed that public spending be redirected, so that research and consumer education would be favoured as the principal objectives of a food policy.

3.1.2. Analysis

This first case presents the participants with a question that remains relatively close to their everyday concerns, because everyone knows at least one quality label, because they are all consumers and because the question addresses the very relevance and/or legitimacy of these instruments.

To answer the question of a specific label’s relevance for consumers, the participants first assume an exploratory disposition: they seek to understand what a label is, what standards have been adopted, how much credibility they can be granted, and what relationships these establish between producers, distributors and consumers. They even consider other possible strategies, like one participant who asked if they should perhaps be seeking better pesticides. But this exploration (whose aim is understanding) is constantly fed by questions through which the participants attempt to grasp the consequences of these systems for consumers and agricultural producers. Thus they raise the question of equity in terms of the exclusion of some segments of the population. And then they examine everyone’s responsibilities: those of the consumers and/or the state.

From this emerges a variety of viewpoints from which a judgment may be made. Employing a classic evaluation logic they ask themselves, for example, whether or not this kind of labelling is effective and reliable, and what benefits they bring.

Effectiveness itself can be evaluated from several points of view. It is of course a matter of knowing whether or not labelling reduces risks, but it is also trying to know if, for example, the widespread development of organic foods would allow the world population to be fed. As for reliability, the participants certainly questioned the nature of existing oversight (particularly in relation to international trade), but they also questioned the validity of some labels that differ slightly from legal standards.

These questions can stem from what we have called a first-order evaluation, in which the problem is defined and can be circumscribed by identifying causes and effects, the possible targets of an action. But they also bring about a series of shifts in framing through which they question both the very logic behind labelling in terms of its social consequences, as well as the reasons why these problems arise.

Here, we will only deal with two themes that reflect the discussions, chosen because they appear to lead to the conclusions and recommendations that followed.

In principle, labels strike the participants as an inadequate response to risks. They are somewhat sceptical of their reliability; they see them as representing costs for producers and consumers, thus introducing inequalities into a food industry that should be beneficial for everyone. They are also aware, however, from their own experience of consumers and their understanding of producers, that pesticide use is ultimately a result of earlier choices that entail irreversibilities. Crop selection methods aimed at increasing productivity make this practice inevitable insofar as consumers favour choice criteria (such as fruit appearance) that encourage it. Several participants made reference to foods they knew as children in order to critically evaluate the technological “advances” that make pesticides indispensable. While questioning label promoters, they also recognize what distribution companies characterize as requirements on farmers, demands based on consumer perception, consumers they mislead through their marketing tactics. When participants express their disappointment that the quality “commercial presentation” is favoured over the quality “nutritional,” the producers’ and promoters’ reactions are lively. They argue that the “commercial” quality is not their desire but that of consumers, who have a preference for what is less expensive and is bought with the eyes only. This divergence of views then produces a result that is instructive at the level of the twofold interpellation aroused in the consumers. On the one hand, they become aware that they—and/or other consumers—do in fact act this way, and this leads them to question their own way of doing things and recognize that they must accept some share of the responsibility. On the other hand, however, they also stress the fact that they feel “manipulated” or led to act in this way, pointing generally to developments, power relationships, constraints and various social phenomena that have contributed to the advent and the pervasiveness of the consumer society to which they belong.

However, the critic elaborated reaches a limit:

“But I think it’s a very good thing for the big retailers to have a label anyway, for people who don’t have the possibility of going to a small farmer. It’s very good to have some kind of security. But this should be done without raising the price since not everyone can afford the luxury of labelled products. The unemployed have rights like everyone else; they can’t spend so much money on organic vegetables.”

This new reference to social equity will lead them to search for a kind of pragmatic proposal.

The deliberation actually reveals that beyond apparently unequivocal criteria (meaning those that can be translated into technical standards), the food issue has multiple anchor points and each evaluation criterion, upheld by a given participant, must be set against another criterion. Thus the desire expressed by some participants to favour proximity networks over the internationalisation of markets must be set against both the feasibility of local agriculture (what about coffee and cacao growing?) and the rights of farmers in the south to export in order to ensure their development.

However, with regard to pesticides, the participants will gradually establish distinctions between organic agriculture, integrated pest management (IPM) and what, for lack of a better term, they called “reasoned” agriculture. This classification enables them to highlight a difference they consider significant, namely the producer’s approach. If organic and IPM production are accepted as they are and are clearly distinguished from other forms, this is because the participants oppose them to others that make reasonable use of pesticides according to current legal requirements or private standards, but do not make the extra efforts that are part of IPM and organic production. This distinction shows the importance, for consumers, of engagement on the part of producers, not just delegation to an organisation that regulates their work. They emphasize this distinction particularly by making reference to their own practices; they wish to preserve freedom of choice while also stressing the effort that must be made to ensure the quality of one’s diet. Women in particular make reference here to the care they invest in feeding their children.

From a first-order logic perspective, confidence is not a problem since it is supposed to follow automatically from the rational, effective solutions implemented. Therefore, against a standards, regulations and guarantees logic, they contrast that of mutual understanding and engagement. Then it is the producers’ effort and autonomy that are rewarded by a label and are a source of trust, and also protect farmers from the excessive influence of industry.

Thus, they reformulate the question by placing the focus on relationships between actors: the different actors’ responsibilities and the modes of communication between them become the central issue instead of the issue of the impersonal, standardised control of legal instruments and procedures. The participants formulate various proposals. For example, they suggest putting producers in contact with salaried distribution personnel so that products are better handled and consumers are better informed.

At the end of the deliberation, when asked to formulate their proposals, the participants attempt to translate their pragmatic options into an operational solution that satisfies a series of requirements that emerged as legitimate. They set about—with the enthusiasm that all creativity stimulates—imagining a logo that could incorporate all these multiple requirements. First, they call for a simple unconfusing quality sign:

“there must be only one label, one single label that says various things, organic is good, but local produce is good too and so on.”

Next, this logo must be legible because:

“there is a vast amount of advertising everywhere you look […] consumers get lost in the face of that mass of information.”

It has to be a logo

“that catches your eye: it must be able to be visual”.

This label must also emphasize the diversity of means of production and must therefore be all-encompassing while distinguishing organic food. It should be able to highlight local products. This logo should make these distinctions without “persuading” since “people shouldn’t be urged to always buy more; the logo shouldn’t push people into consumption.” In other words, the label should favour information over goading, leaving “everyone free to choose, but they must be given the information.”

Pragmatism is translating into the desire to create a “simple drawing, [...] that should give three types of information, the environmental aspect, the pesticides aspect and the geographical aspect.”

The participants will therefore design a logo that incorporates, collects, and summarises several different interests without reducing them to a common principle. The symbol they propose is a circle divided in three, one section indicating the product’s origin, the second indicating pesticide use (organic, IPM, etc.) and the third indicating the environmental dimension. The whole symbol would be crowned with an independent certification mark.

  • Industrial/artisan production

  • Product origin

  • Environmental approach

This solution, which proceeds by addition, is not the product of compromise between particular interests but rather the expression of a “collective pragmatic interest,” meaning an interest that collects and combines several different interests without reducing them to a common principle, that is, without being reductive for the sake of efficiency. The participants supply another way of solving the effectiveness paradox while “ecologizing” it, if we make reference to what Isabelle Stengers conceives of as the “ecology of practices,” meaning “the creation of a local interweaving between divergent interests, an interweaving between heterogeneous entities that remain heterogeneous, that link without unifying, that compose without subjecting the terms that enter into composition to a common interest” [21].

This is a second-order governance logic that attempts to limit or thwart the effects of specialised rationality (one single incorporative logo) and assembles a set of actors and systems into only one instrument.

In summary, the participants displayed three competencies: the ability to systematically comprehend complexity, which they resolved by finding a bypass via consumers (i.e., themselves) and through a reframing of the food industry; the ability to integrate various points of view without exclusion or hierarchy, therefore creating a diverse collectivity; and finally, a creative and pragmatic competency through the conception of a simple and effective logo that could bring about change.

3.2. The PEPAM Project

3.2.1. Introduction and Development

The second project—the PEPAM [22] project—concerns a public evaluation of the risks linked to pesticides. This project was very different from the labelling project since it was concerned with generating discussion about a complex technical-scientific model. The model in question—PRIBEL (Pesticide Risk Indicator for BELgium)—was developed in Belgium in the early 2000s and was chosen to evaluate the risks associated with pesticide use (insecticides, fungicides and herbicides). The model called PRIBEL) is based on the statistical distribution of risk events and on their aggregation at the national level. The different “risk compartments” (indicators) are: consumers, applicators workers on the fields, birds, aquatic organisms, subterranean waters, earthworms and bees [23]. The aggregation was made at the Walloon region scale, for the acute risk for bees. It implies the following requirements:


The amount of pesticide used (per crop per pesticide active substance and per use), for a given year;


The pesticides’ toxicological parameter (LD50);


A map of agricultural areas;


The spatial distribution of crops per agricultural area.

Those parameters are the scenarios of the aggregation model. Aggregation is performed through the modelling of the statistical distribution of risks of individual risk events, and then the calculation of the p90 per agricultural region (more information on the statistical functioning of the PRIBEL model can be found at [24]).

Given the very controversial nature of the causes linked to the high bee mortality rate at the time when the PEPAM project was being set up, it was decided that the model would be opened to discussion through a description of how the “bees” indicator works. The stakes linked to this indicator are important, since it should ultimately enable the authorization or exclusion of compounds present in pesticides on the Belgian market. It is important to note the twofold challenge of the PEPAM project: on the one hand, to stimulate the interest of a panel of citizens/practitioners and give them the ability to deliberate over an evaluation device that, though far removed from their everyday experience, has implications that would impact their own practices; on the other hand, to give the expert/modeller the ability to explain the uncertainties that surround the evaluation of the PRIBEL model, and then draw from the citizen/practitioner panel’s deliberations lessons that might influence his own practice.

This project was originally initiated by an engineer in charge of evaluating the modelling of these risks, who was at that time taking a degree in the sociology of science. His primary motivation was to examine the validity, from the public’s perspective, of choices made in the process of modelling. In a sense, this researcher introduces a second-order reflexivity by questioning his own practice, which he recognizes entails a heterogeneous set of choices (ethical, political, social and economical) that could remain obscure in scientific expertise. He obtained an unallocated budget from the aforementioned durable development policy research program. To this was added a small budget from a public utility foundation (The King Baudouin Foundation). This project brought together two academic teams (from Ghent University and the University of Liège) as well as a national centre specialising in food safety (CERVA–the Veterinary and Agricultural Research Centre): eight researchers organized and supported the experiment, assisted by a professional facilitator. The participants from the public included three citizens recruited to represent consumers, as well as two amateur horticulturalists, three beekeepers and three farmers with different agricultural systems. A panel of contributors (farm union, industry, environmentalists, entomologist) were asked to explain to the participants their positions on the question of pesticides and the PRIBEL model. The deliberative focus group took place over a weekend in three phases.

The first phase was dedicated to introducing the context and the PRIBEL pesticide management model. It was on this occasion that the modeller explained the functioning of the model, the choices involved in its conception and the uncertainties that surround the difficult question of aggregating data for the sake of an indicator. One of the information sources that most attracted the citizen panel’s attention was a list the modeller had made of around twenty choices that had driven the conception of the model [25]. The ensuing discussions focused on PRIBEL and on pesticide use and its effects “in reality.” An initial observation made by the panel was that the pesticides—like the model that was supposed to evaluate their associated risks—in order to work properly, must satisfy requirements that are difficult to meet. Thus, in the case of the functioning of PRIBEL, this presupposes databases that, due to their size and quality, must currently be regarded as incomplete. Moreover, the claim that the model aggregates the set of measures with the ultimate aim of producing a single figure (the indicator) inspires mistrust in some citizens: the reduction and simplification that this aggregation induces leaves it incapable of taking account of the “ecosystemic” and complex character of the reality in the field. In the same way, pesticides raise questions not only because of the continuous development of their toxicity (acute vs. chronic, the synergetic effect, the switch to newer, more powerful compounds), but also particularly because of the great variability and inadequacy of some of the practices of those who use them: these products are dangerous if they are not used correctly (“always use a slightly higher dose than what is marked in the instructions”). More generally, the panel’s accounts attest to the existence of uncertainty surrounding the behavior of farmers (the quality of spraying work), beekeepers (with the development of pesticides and the combined effect of varroa mites, “beekeepers have become incompetent”), bees (“we don’t know how bees work, how they gather pollen”), consumers (when they are inclined to choose products according to their appearance rather than for their taste).

These objections lead to the need to take more account of the living, of the field and its actors, which is precisely that from which the PRIBEL model attempts to free itself through its statistical and aggregative scope.

This being the case, one position is recommended: that some degree of vigilance should be cultivated:

“[...] you have to keep being vigilant to constantly make adjustments and perfect the tool. [It is necessary to aim for] adaptability and speed of reaction and reassessment.”

The second phase was built around presentations made by six participants (stakeholders and experts) on the validity and consequences of using the PRIBEL model; representatives from the health ministry, the phytopharmaceuticals industry, a farmer’s union, an apiculture federation, and an environmental NGO; and an academic entomological team. The aim was to assist the panel’s deliberation through the confrontation of these different representatives’ points of view. What attracts the panel’s attention is the increasing complexity of the issues related to pesticides, not to discredit or fear it, but to carry it at arm’s length (“now I see that it’s still a quite extraordinary complexity, I also see the issues, the snags, the urgent need to react [...] and the immensity of the task as well, and I think there’s an urgent need to disseminate information, increase the awareness of regular people, and we need to mobilise a more adult population to avoid catastrophes. I think it’s a problem.”

This vigilant position assumes a certain ability to hesitate, a concern for getting all of the participants’ different points of view to coexist (something that, as we shall see, will signal the manifestation of a “collective pragmatic interest”):

“I notice that people who go into studies [scientists], they go in what I’d call an upward sense, but they don’t go... they don’t move sideways to try and see if there are other impacts, other parameters involved as well. [...] So one of them will take insecticides and pesticides in one direction but it’s just that, sometimes, maybe only by taking a step sideways can we maybe go in another direction to find new systems. [...] if someone pushes you, instead of counteracting it, well... give me a push and we’ll move in the same direction and we’ll arrive at another result.”

This pushes people to encourage the management that has already been done, but also to question its purpose (“what direction will it take? Measures are being taken, that’s fine, but what for?”).

It was a chance for the panel to discuss their opinions on the place of pesticides in their agricultural system. These positions were synthesised by the organisers that same evening (do we need pesticides? “We can do without pesticides”, “We can do without pesticides in the long term”, “We can’t do without pesticides”), so that they could be returned to the panel the following morning.

Thus the third session was dedicated to creating three citizen sub-groups able to argue and construct propositions around these three positions, then summarised them in a plenary session. These two-pronged deliberations dealt with (a) increased dependence upon pesticides; (b) the redefinition of the mode of management that results from this and (c) the relevance of the PRIBEL model. These deliberations concluded with recommendations touching on the roles and practices of different designated key players and the use of the PRIBEL model.

3.2.2. Analysis

The sense of urgency that guided both policy-related interest in the PEPAM project (addressing how to manage the bee decline and the use of pesticides) and scientific interest (stating an opinion about the working of the PRIBEL model) caused the panel to slow down in a way: before examining the model itself, it was necessary to place it in perspective in relation to the initial policy injunction represented by the federal pesticide reduction plan. The increased complexity generated turns on the types of questions the panel wishes to promote. They are of two types: (a) of what can a public pesticide management policy consist? and (b) can one do without pesticides?

First its relevance as a management tool is questioned. In the eyes of the participants, although the model deserves to exist, is it the only possible or desirable one? And can this model be modified? How valid is the data, knowing that it came mostly from industry? To what extent does it simplify? Would it not be more realistic to evaluate the impacts? Is the cumulative effect of weak doses taken into account? All of these questions express the desire not to get locked into one definition of the problem, the definition implied from the outset by the PRIBEL model. Thus, whereas this definition is the result of an established policy choice (to shift from a policy of reducing the volume of pesticides used to a risk evaluation policy), in the course of their deliberations, the participants divided into three positions with regard to how pesticides fit into our agricultural system: “yes, we can do without pesticides,” “yes, in the long term,” “no we can’t do without pesticides.” Although we will not be detailing the arguments supporting these positions here, we note that what allows these contrasting positions to coexist is the essential attitude that underpins the exploration of the issue: “[…] the panel did not express any objection to working in these [three] sub-groups. The participants could choose theirs and accepted the coexistence of the three very different groups. This, therefore, confirms that they felt comfortable with how their positions were classified on Saturday. It also confirms what was previously emphasised concerning the specificity of the citizen posture—for those who played the game—citizens are prepared to explore the question, to make different perspectives converge in order to gain a more complex understanding. Citizens do not want merely to assert one point of view at any cost, to settle the matter or arbitrate conflicting positions” ([22], p.32).

This re-examination of policy positions relating to our coexistence with pesticides enabled citizens to conduct an initial reframing of the issue and enumerate a number of criticisms and propositions concerning the practices of some of the key players involved in pesticide management. Thus the search for the necessary trust between research partners (between beekeepers and entomologists) is invoked, openness by scientists to local realities and the need to vary solutions according to these is pleaded, investment in the production of independent scientific data (not issuing directly from industry) is demanded, consumer responsibility is encouraged, the paradoxical responsibility of citizens with regard to their political representatives’ lack of personal investment is evoked, the complex situation in which farmers find themselves—being at the same time the creators of the pollution stemming from the intensive use of pesticides and the victims of a restrictive technical–financial system—is highlighted. These various competing analyses and points of view, expressed in the course of the deliberative focus group, testify to the problem’s complexity, which those who promote the use of the PRIBEL model are attempting to reduce.

Although this time we are positioned on the side of the organizers of the deliberative focus groups, the organisers can—under certain conditions [21,26]—place themselves in a situation in which they are learning and opening their techniques democratically [27]. This is particularly noticeable at two points in the process: at the preparatory stage and when the modeller draws lessons from the citizens’ deliberative process [28].

Although a set of indicators may appear obvious to the modeller, when speaking to the public outside the laboratory, a minimum of explanation and illustration is assumed. These explanations proved to be necessary, not only for his fellow organisers (researchers in social sciences), but also for the citizens who would be deliberating. The constitution of both the deliberative group and of the corresponding introduction of his work becomes a key issue. On this occasion, the modeller must consider several questions: how does one describe PRIBEL? What is the right question to address to this new group so that they can make constructive propositions? The modeller himself is surprised by how the set-up he contributed to build affects him: it becomes necessary to clarify the hypotheses underlying the creation of PRIBEL so that the citizens “can understand its fundamental principles, how it works, its limitations, the hypotheses chosen, how it will be used, its developmental possibilities, the databases it feeds.”

For the modeller, this need to explain should not necessarily be subordinated to the need to justify the choices made: “The objective is not for the modeller to justify the limits of his model in such a way that the citizens accept it but to explain them so that citizens, on that basis, can take a position regarding their acceptability” ([22], p. 24). Consequently, the modeller presents himself more as a practitioner of the modelling than as a spokesperson for the PRIBEL model [29]. We can postulate that it is, in fact, this distinction which reveals itself to be the gradient through which the group evaluates the legitimacy and productiveness of each person’s contribution to the deliberative or collective work.

Second-order governance can also be expressed through cross learning. Cross learning made possible by the manifestation of a “pragmatic collective interest.” It is manifested through a change of perception expressed here by the expert. This learning was revealed on two levels: one concerning possible adaptations of the model, and another concerning a change in the modeller’s interpretation of his own model’s role. We will take a moment to examine this second point.

It must be stressed—according to the expert in question—that several key exchanges with certain citizens on the panel constituted decisive moments. Learning resulted from gaps between the modeller’s perception of his own model and the citizens’ perceptions and propositions. Each time, these propositions took the form of a metaphor: one citizen thought the PRIBEL model should be conceived as a sieve, another saw it as a “dashboard.” Each of these metaphors draws attention to the two distinct issues that stem from the use of the PRIBEL model in a pesticide use risk-reduction policy. “The sieve makes me see the model more as a tool that is used to select pesticides, rather than calculate the risks”; even though the modeller is completely absorbed in the objective of technically evaluating the model (measuring the abstract risk and the accompanying reduction required), from this perspective, the model becomes an instrument aiming to select compounds and therefore discriminate between different agricultural processes. “The sieve retains some things, but others pass through,” are released into nature). Consequently, they can interact, even though they were placed in the sieve (i.e., the model) separately”: this second interpretation of the sieve metaphor points to “unforeseen consequences of using the model.” Even though the modeller considers himself to be conscious of the issue of synergies between products, he had always conceived of the issue of evaluating through this model and that of synergies as two separate things. Thus, by acting at the level of acute toxicity, “at least some of the risks are removed.” However, as the modeller admits:

“The image of the sieve made me see things differently. What if the removal of certain products created a kind of product selection that happened to have very strong synergies, stronger than the previous product synergies? In that case, we might be increasing the risk, in unknown proportions... It is the whole issue of risks that are unknown, but possible. It had never occurred to me that my work could increase these unknown risks.”

The second model metaphor, comparing it to a “dashboard,” points to another issue similar to the first. After a panel member made this suggestion, the modeller—by subsequently immersing himself in literature relating to the use of scorecards for corporate balance sheets—understood that his model was used by federal authorities in a standards context: the withdrawal or renewal of product authorisations, or even precautionary measures for the use of these products. Once again, the model is here taken to mean something very different from what had prevailed in the academic world: “the model becomes a practical tool (for decision making and policy management) rather than a heuristic means.”

These two metaphors manifest concern for keeping the different dimensions linked to the model intact. As in the case of labels, these metaphors manifest that “collective pragmatic interest.” Each of them supplements the initially “heuristic” design favoured by the modeller.

This pragmatic collective interest led the modeller to alter his assessment approach. This had the consequence of motivating him—as a scientist—to take an interest in literature on indicators in other fields (economic, sustainable development, etc.) in order to “better situate [his] modelling work in the context of how it is used for policy.” This led him, in his work on his thesis, to search for ways of incorporating socioeconomic, ethical and legal indicators into these evaluation models.

4. Discussion and Conclusion

In the process of amending the notion of risk and attempting to make this one of the central issues for thought on the future of our societies, Beck proposes to take an interest in second-order reflexivity, in which citizens might play a new role, that of forces determining their destiny in situ ations of strong uncertainty, a role that had until then been entirely in the hands of those in power and their experts. What the present article has attempted to propose, by means of the environmental issue linked to pesticide use, is a way of understanding the second-order reflexivity at work on the occasion of two experimental situations. These propositions focus on the description of a process (shifts in framing, increases in complexity, reformulations and lessons learned), and also its result (the emergence of a “collective pragmatic interest”).

In terms of the process, the citizen deliberation reveals laypeople’s ability to grasp problems—including scientific and technical notions—and then go beyond the established framework by pinpointing connections and systemic effects. But the process does not end at this deconstruction. The participants, when using their knowledge and practical experience, when combining them instead of separating them, show that they are capable of reformulating the question and producing propositions that shift the question. Far from disputing science, they most often call for a more soundly based science that involves more interaction, both between disciplines and with non-scientists. They often call for a cultural reconstruction, in the sense that they call into question the definition of what constitutes food, what constitutes agriculture. This reconstruction is most often based on an emotionally charged personal experience, such as when mothers make reference to the how they feed their families. The distribution of responsibilities seems to be a key issue for them, since this requires people to say precisely why they act a certain way and what the action’s meaning is.

It seems to us that a second, more original conclusion in our research is to be found in what we have called the “pragmatic collective interest.” In the case of labels, the issue of pesticides is introduced by means of the information to be communicated to consumers, to help orient their choice. It is as consumers and citizens that the deliberative focus group’s participants are led to deconstruct the device applied to them. In a sense, it is a matter enhancing their user status in a different way: they become co-designers of this set-up. In the case of the opening of a pesticide management statistical model to the public, the situation is very different and leads to a collective pragmatic interest that is different, supported by actors no less different: some of the experts themselves. In fact, the question submitted to these shrewd participants is, in principle, far removed from their everyday experience; the statistical model presented for discussion is not usually intended for citizens but is much more meant for stakeholders and, especially, their experts. Nevertheless, the deliberative focus group process made it possible to subject certain aspects of PRIBEL to deliberation: the wider context of its use (its place in the federal pesticide reduction policy); more fundamentally, the place of pesticides in agriculture; and also PRIBEL’s effects on policy and technology in the process of registering compounds. In our opinion, it was on this last dimension that the pragmatic collective interest expressed itself best. The pragmatic collective interest consists of holding together—in the context of an argument or deliberation, even in a technical device—points of view that are (potentially) divergent, but whose pragmatic cohabitation helps keep the reflection open.

We believe this pragmatic collective interest results from the participants’ competencies and from the deliberative set-up implemented. By relying on laypeople, by using experts and stakeholders as simple informers, this set-up actually cuts participants off from the real world and creates a fiction, that of a kind of parliament. It is, however, a productive fiction. By ensuring equality of participation, by organizing and finalizing the discussion, the set-up establishes a space in which these competencies can unfold. By not seeking consensus but through mutual understanding searching for approaches to solutions, the deliberation ends up being a search for solutions that are pragmatic but not exclusive, one that strives not to exclude any potentially concerned interest. The propositions get their collective character from this desire to take into account.

As long as the collective pragmatic interest is not the definition of a universally shared common good in the name of which one hierarchises and divides (in the sense of a representative working in, and drawing his or her legitimacy from, a representative democracy), this instance of the coexistence of potentially conflicting arguments and points of view—which collectively establishes a situation of hesitation, without making this a weakness—is extremely invaluable. We think that this is the special, though not exclusive, work of people acting as practitioners and not as representatives. This pragmatism—which consists in defining the situation not according to grand principles but according to situations as they present themselves, through what one sees and experiences—can also be expressed in a second way. The pragmatic collective interest cannot be decreed; it is observed from what has been rendered accountable. Thus, the deliberation on agri-food labels or evaluation models attests to the desire to desectorialise the technical side of policy. Labels show that there can be technology in political will, just as the models demonstrate that there is a political element in technical choices.

The final conclusion we were able to draw concerns the impact of these projects on policy. Given that the aim was to open two different public policy instruments to deliberation (one intended for consumers, the other for experts)... and that the public assembled consisted of that constructed by the deliberative focus group set-up (groups of citizens assembled by the researchers), what might be the relevance of our experiments and their lessons if we consider them in the context of public policy? This question is all the more legitimate in that it was asked each time by the participants of our deliberative focus groups: “What/who will benefit from the results of our deliberations?”

In the case of risk evaluation, it is the expert’s reception of the results (the modeller in charge of PRIBEL evaluation) that can testify to this collective pragmatic interest. From this perspective, it is indeed the modeller’s learning process, as well as his desire to get all of the dimensions and tensions emphasised by the participants to hold together, that testifies to it. In fact it is he who—through his status as expert and evaluator—has taken the lead and can best translate and convey the PRIBEL model’s uses and users. This was given concrete practical expression in two documents he produced. The first is a lengthy document that explicitly presents participants’ contributions to elements of PRIBEL (and to its potential effects) and also presents the conclusions he has drawn from these for his evaluation work. Secondly, and more fundamentally, there was the doctoral thesis he was in the process of writing, which concerns the increasing complexity of modelling, and attempts to take account of dimensions that were originally lacking (political, ethical, etc....). Although the participants of the deliberative focus groups carried out articulation and reformulation work, the conception work that emerged from this was more the creation of the modeller [30].

In the case of labels, the return to public policy is more complicated. What we gained by separating the participants from the real world and establishing a kind of parliament, we lost due to the length of the connections between results and future policy instruments. Thus it may be fitting to change the status of these instruments in some way: shift them from the status of instruments to that of both social and technical experimentation materials [16,31].

Conflict of Interest

The authors declare no conflict of interest.

References and Notes

  1. The acronym of the project stands for: “Feasibility of a Participatory Modelling Process for Pesticides Risk Assessment” (cfr. Section 3.2.).
  2. Wilson, C.; Tisdell, C. Why farmers continue to use pesticides despite environmental, health and sustainability costs. Ecol. Econ. 2001, 39, 449–462. [Google Scholar]
  3. Van Loqueren, G.; Baret, P. How agricultural research systems shape a technological regime that develps genetic engineering but locks out agroecological innovations. Res. Policy 2009, 38, 971–983. [Google Scholar] [CrossRef]
  4. Carson, R. Silent Spring; Houghton Mifflin: Boston, MA, USA, 1962. [Google Scholar]
  5. Kuchler, F.; Ralston, K.; Unnevehr, L. Reducing pesticide risks to us food consumers: Can agricultural research help? Food Policy 1997, 22, 119–132. [Google Scholar] [CrossRef]
  6. Beck, U. Risk Society: Towards a New Modernity; Sage publications: London, UK, 1992; p. 260. [Google Scholar]
  7. EFSA Panel on Plant Protection Products and their Residues (PPR). Scientific Opinion on the science behind the development of a risk assessment of Plant Protection Products on bees (Apis mellifera, Bombus spp. and solitary bees). EFSA Journal 2012. [CrossRef]
  8. Karabelas, A.J.; Plakas, K.V.; Solomou, E.S.; Drossou, V.; Sarigiannis, D.A. Impact of european legislation on marketed pesticides—a view from the standpoint of health impact assessment studies. Environ. Int. 2009, 37, 1096–1107. [Google Scholar]
  9. Beck, U. The Reinvention of Politics: Rethinking Modernity in the Global Social Order; Blackwell Publisher Ltd.: Oxford, UK, 1997. [Google Scholar]
  10. Voss, J.-P.; Kemp, P. Reflexive Governance for Sustainable Development—Incorporating Feedback in Social Problem Solving. In Proceedings of the ESEE Conference—Special Session on Transition Management, Lisbon, Portugal, 14–17 June 2005.
  11. Stewart, D.W.; Shamdasani, P.N. Focus Groups: Theory and Practice; Sage: Thousand Oaks, CA, USA, 2007. [Google Scholar]
  12. Kitzinger, J. The methodology of focus groups: The importance of interaction between research participants. Sociol. Health and Illness 1994, 16, 103–121. [Google Scholar]
  13. Callon, M.; Lascoumes, P.; Barthe, Y. Acting in an Uncertain World: An Essay on Technical Democracy; The MIT Press: Boston, MA, USA, 2011; p. 298. [Google Scholar]
  14. Habermas, J. The Theory of Communicative Action; Beacon Press: Boston, MA, USA, 1984. [Google Scholar]
  15. Habermas, J. Political communication in media society: Does democracy still enjoy an epistemicdimension? The impact of normative theory on empirical research. Commun. Theory 2006, 16, 411–426. [Google Scholar] [CrossRef]
  16. Lascoumes, P.; Le Galès, P. Understanding public policy through its instruments-from the nature of instruments to the sociology of public policy instrumentation. Governance 2007, 20, 1–21. [Google Scholar]
  17. Deleuze, G.; Guattari, F. A Thousand Plateaus: Capitalism and Schizophrenia; Continuum: London, UK/New York, NY, USA, 2004. [Google Scholar]
  18. Van Huylenbroeck, G.; Steurbaut, W.; Mormont, M.; Pussemier, L. Sustainability of Certified Production Systems: The Case of Labels in the Food Sector; Belgian Science Policy: Bruxelles, Belgium, 2006. [Google Scholar]
  19. Cannell, E. European Famers Plough Ahead. Pesticides News 2007. [Google Scholar]
  20. Phillips, P.W.B.; McNeill, H. Labeling for gm foods: Theory and practice. AgBioForum 2000, 3, 219–224. [Google Scholar]
  21. Stengers, I. La Vierge et le Neutrino. Les Scientifiques Dans la Tourmente; Les empêcheurs de penser en rond: Paris, France, 2006; p. 283. [Google Scholar]
  22. Piñeros Garcet, J.; Deblonde, M.; Mélard, F.; Louviaux, M.; Goorden, L.; Mormont, M. Feasibility of a Participatory Modelling Process for Pesticides Risk Assessment—PEPAM; Research contract N° 0A/00/027; Belgian Science Policy: Brussels, Belgium, 2006; p. 83. [Google Scholar]
  23. Here are the associated class of pesticides according to thoses compartments : fungicides pose the highest risk for consumer (e.g., thiram, fenpropimorph, epoxiconazole), operator (e.g., fentin hydroxide, mancozeb, fluazinam and an important soil disinfectant is methyl bromide) and earthworms (e.g., mancozeb, fenpropidin, fentin hydroxide); insecticides persuasively for birds (e.g., aldicarb, carbofuran and carbosulfan), bees (e.g., vamidothion, chlorpyrifos, imidacloprid) and aquatic organisms (e.g., flufenoxuron, lindane, endosulfan), and herbicides for groundwater (e.g., lenacil, atrazine, isoproturon) (Vergucht et al. 2007).
  24. Vergucht, S.; Piñeros Garcet, J.; Pussemier, L.; Steurbaut, W. Development of a Pesticide Risk Indicator for the Evaluation of the Belgian Reduction Plan. In Towards a Safer Food Supply in Europe; Van Peteghem, C., de Saeger, S., Daeseleire, E., Eds.; Belgian Science Policy (BELSPO): Brussels, Belgium, 2007. [Google Scholar]
  25. These decisions are just as much political as they are technical (for example: “Consider solely risks linked to agricultural pesticides”, “Zero danger for certain products”, “no spatial relationship, no bee displacement, to pesticide displacement”, “no ecosystem-type effect”).
  26. Thiery, O.; Cerf, M. Penser la Recherche Participative Comme une Pratique-proposition de Diagnostique. In Dynamique des Savoirs, Dynamiques des Changements; Béguin, P., Cerf, M., Eds.; Octarès: Toulouse, France, 2009; pp. 29–50. [Google Scholar]
  27. We say “opening” insofar as a true democratisation of the technical choices assumes a commitment by the public authorities to take account of these openings (see next section).
  28. The modeller/project initiator gave this introspective work concrete expression by writing a document dozens of pages long, rich in details and reflections. This working document was not published, but has been used in the present section as a source for the quotations. One part of this document is also available in the PEPAM report (“Analysis from a modeller perspective”) and in his doctoral thesis, (currently being written).
  29. Which is, in the present case, all the more comfortable since the modeller belongs not to the model conception team (FOCUS-Gw), but to the evaluation team (APECOP scientific consortium).
  30. This link between new learnings in behalf of experts and the manner in which these modify their future practices is rarely discussed (if not with their colleagues in the privacy of their inner sanctums). However, in uncertain situations for which the experts have no readymade solutions to propose, the conditions of their creation, especially of their learning, is essential. What the deliberative focus group enabled to come to light was the character of the posture adopted by the modeller in order to give life to this pragmatic collective interest. Because of the necessary interdependence between experiments conducted in the course of this deliberative focus group, we could say that this pragmatic collective interest is jointly the product of the participants and the modeller, since this collective interest is shared by all members of the deliberative focus group set-up.
  31. Mélard, F. Écologisation: Objets et Concepts Intermédiaires; P.I.E. Peter-Lang: Bruxelles, Belgium, 2008. [Google Scholar]
Sustainability EISSN 2071-1050 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top