Sustainability is widely recognized today as a guiding idea for societal and economic development. With the growing importance of sustainability, there is also a growing need for reliable and comparable approaches that allow measuring the sustainability of a given object, e.g., a product, service, process, or an enterprise. In response to this demand, a plethora of approaches [1
] has emerged to provide sustainability assessments (SAs). The SA field has origins in previously established methods like industrial ecology, LCA [3
], policy impact evaluation [4
], and has been increasingly established and institutionalized in the form of political commissions and declarations, such as the Commission on the Measurement of Economic Performance and Social Progress [5
] and the Istanbul Declaration [6
] and “standards” like Bellagio STAMP [5
]. Spurred by research, the sophistication, standardization, and specialization of SA approaches are improving. However, little attention has been paid to how research projects navigate the challenges of selecting and customizing SA approaches for their unique research purposes.
In considering general aspects of sustainability assessments, it is important to note that there is no interdisciplinary approach to identifying standards nor a consistent basis for SA [7
]. The field of SA is highly diversified, cuts across sectors, products, processes and lifestyles/consumption patterns so researchers need to know what sustainable practices are, products, firms, sectors, policies, countries. Sustainable development is multidimensional and complex, so it is not possible to have a single indicator; furthermore, SA is informed by different understandings of sustainable development, which makes the operationalization of sustainable development and SA more complicated. Moreover, SA has to face the normative content of sustainable development: When trying to design and implement sustainable development, sooner or later there are conflicts of interest that have to be dealt with, but dealing with tradeoffs can hardly be addressed by science. This means that the plurality of approaches to SA can be meaningful because of the very different tasks, goals, subjects, objects, reference levels and contexts of SA on the one hand and because of normative assumptions and underlying values as we work out in the study.
Nevertheless, SA is something that most researchers in sustainability have to “do”. In the course of our respective projects, we identified a gap in the literature when it comes to a practice-oriented understanding of the main challenges that researchers face when they engage with SA in diverse contexts. This led to our research question “What are common challenges that researchers are facing in using SA approaches?”
In the past decades, a wide variety of SA tools has been produced [8
]. In this context, the vague and diverse definitions of sustainability itself remains a problem. There are various sustainability assessment tools that not only consider different perspectives or indicators but also differ in their methodology. As such, the following attempt to define characteristics for sustainability assessments can only be rather generic.
According to Sala et al., “Sustainability assessment (SA) is one of the most complex types of appraisal methodologies. Not only this does entail multidisciplinary aspects (environmental, economic and social), but also cultural and value-based elements. Besides, SA is usually conducted for supporting decision making and policy development in a broad context” [9
Additionally, the wide field of SA makes it an easy target for lawmakers, companies or other interest groups to pride themselves with implementing or acting upon sustainable assessments or tools, without the capability of measurement or adequate comparison. Still, there is a certain interest in keeping the definition of sustainability as broad as it is, in order to keep the great variety of stakeholders and their areas of action included in the discourse. As Pope et al. puts it: “[T]he concept of sustainability has the potential to not only keep everyone at the table, but to provide the catalyst for reflexivity and a deliberative space, or axis around which discussion can occur” [8
1.1. Goals of Sustainability Assessments
Sustainability assessments can, thus, only be defined in a somewhat generic manner. Likewise, the goals and targets of SA follow a broad approach.
One definition of overall goals in sustainability assessment is provided by Verheem: “The goal of sustainability assessment is to pursue that “plans and activities make an optimal contribution to sustainable development” [10
A more elaborate outlook is given by Waas et al. on how SA support decision-making by addressing four purposes:
Substantive: Describing “the achievement of the intended purposes of the SA”;
Normative: Focusing on “the achievement of normative goals—i.e., can stakeholders learn, improve their knowledge and change their views”;
Procedural: “[C]onsideration of SA process aspects and the establishment of SA, procedures and policy”;
Transactive: “[T]he achievement of intended purposes with minimal resources and time or in other words efficiency” [11
1.2. Precise Subject of Sustainability Assessments
The conceptual framework can be established with two main dimensions and various sub-dimensions, one being the sustainability concept (including social, economic and ecologic aspects) and the other the decision-making context. The decision-making context is particularly important to bring the theory of sustainability into practice and involve the different stakeholders identified [8
]. However, there is no general conformity about the types and quantity of dimensions to encompass the SA.
When deciding about developing an ex ante or ex post tool, a helpful differentiation might be to review whether the means of the assessment tool are to guide and direct decision-making by predicting effects before the implementation. If the technique rather requires a retracing look upon things, so, for example, the assessing of processes, an ex post tool is required. According to many authors in this field, to design sustainability assessment in the context of decision-making, an ex ante tool is the most common approach [8
When defining the precise subject of sustainability assessment, there are different frameworks that can be considered. The different SA tools can be categorized as product-related assessment tools, project-related assessment tools and country-related assessments, as well as an indicator-related approach. The following Table 1
illustrates these categories and gives examples for established tools.
lists a wide range of tools. Some of the tools from the environmental, the economic and the ecologic sections can be seen as sources or methodological foundations for integrated tools addressing sustainable development or as parts of sustainability indicator sets (e.g., Ecological Footprint, LCA or MIPS). A couple more tools, especially in the non-existing column for enterprise-related assessment, could easily be added (diverse management systems, as well as self-assessment tools, for an overview, see Reference [14
]). Irrespective of the existing, multi-faceted instruments, the projects involved in the study had to “tailor” their tools to their respective contexts as described in Section 3
The assessment of sustainability impacts is a complex scientific issue that immediately becomes normatively loaded. For that reason, reflective and critical discussions, as well as exchanges on strategies are needed to further develop scientific research and practical experience and to obtain greater comparability. This paper addresses the gap resulting from the little attention that has been paid to how research projects navigate the challenges of selecting and customizing SA approaches for their unique research purposes.
In that context, a case study comparison can make a contribution by developing approaches for the subject of SAs (structuring diversity; generating comprehensive results, comparison of different key aspects in the field of sustainability) and identifying academic gaps.
Therefore, this paper provides relevant input for a discussion that needs to be further scientifically investigated, by serving as a methodological basis while specific strategies are presented and comparatively discussed.
It provides an exploration of the context-specific conditions of SA through a comparative case study design of three research projects within a German research program on the sustainable economy. Each case study explores the different approaches, methodologies, as well as difficulties and similarities that researchers face in “doing” SA.
Against this background, the following research questions guided the comparative analysis:
Are there similarities and similar difficulties in doing SA in different settings and projects and how can we learn to overcome the diverse challenges in SA? By answering the research question, we aim to create awareness and to improve the ability of researchers to transparently modify and customize generic SA methodologies to their research contexts.
To address our research questions, it was important to take a view beyond the horizon and analyze different projects that deal with SA in various contexts regarding their SA goals, lessons learnt, etc., to see if there is “one best way” to do SA. We develop our investigation along a comparison of three different projects.
describes the methods and materials used, where we detail the comparative case study design, the research context and the analyzed projects. In Section 3
, we then present the results obtained from applying the methods for each of the three cases compared. Based on a comparison of the results, we discuss the main commonalities and differences in Section 4
, before we briefly summarize our findings in Section 5
In Section 3
, we compared and analyzed the SAs in the three chosen projects regarding the external and internal features and challenges they have to deal with. In Section 4
, we discuss these results against the background of our SA experiences and further scientific literature. It should be noted here that the field of sustainability research in general and SA, in particular, is very diverse. There are established approaches, such as Bellagio STAMP, which are more on the macro level. “The authors of the STAMP had three distinct audiences in mind that could benefit most from applying the Principles in their monitoring and assessment practice: The communities involved in developing alternative metrics systems, the communities focused on integrated assessment and reporting, and those practicing project or policy-focused evaluation.” [5
]. So the principles formulated there are relevant to sustainability research, and in particular, to SAs at the level of sustainability policies, but they are of rather limited use at the level of individual projects we have presented, which ask how SA can be put into practice in different contexts on the level of sustainable products, consumption and lifestyles. We consider this level as equally important. As several of the Bellagio STAMP principles seem to match with our results, we do integrate references throughout this discussion section. Other principles (e.g., defining scope and indicators) may address issues that are certainly relevant to the projects we look at, but they are not at the center of our discussion of similar challenges and will, therefore, not be discussed in this Section. Regarding key challenges, such as the clarification of the main purpose of SA—deliberation or assessment—the disclosure of values of those involved, our discussion focuses on other issues than guiding principles as Bellagio STAMP (or goes beyond the aspects mentioned there). Our findings indicate that existing systems and guiding principles suggested for doing SA, such as the Bellagio STAMP principles, are helpful and could benefit from more explicitly requiring transparency on normative value judgements in order to pay justice to broad participation and to enable effective communication.
4.1. External/ Context Related Challenges
In some publications in the field of SA [82
], the question of the main purpose of SA—deliberation or assessment—is discussed. The fact that in numerous others [11
] this question it is not explicitly stated, seems to be a bit surprising given the assumption that—especially in transdisciplinary intervention research—the participants of a SA do not necessarily have a shared understanding (or maybe even a concrete idea at all) of sustainability. Moreover, in Bellagio STAMP, the postulation “not to lose sight of the purpose of the assessment” is rather a side aspect of principle 2 (essential considerations) that points in the direction of the problem we have raised.
Our three projects are not excluded from the critical finding that the reflection about the main purpose of SA—deliberation or valuation—would deserve more attention in many SA situations, even though deliberation about sustainability and how to operationalize it played a major role for the tool development in our projects. In this sense, the project approaches are also in line with principle 1 (guiding vision) of Bellagio STAMP, which also calls for the development of a common understanding of sustainability among the actors involved [5
]. In the cases where SA is not a routine operation in an established setting, it is more than a technical accounting process since the fundamental questions ‘‘sustaining what, for whom and why?” [91
] should be discussed with the stakeholders. Nevertheless, there still is some need for methodological innovation to conceptualize and implement deliberation processes in the SA context with stakeholders [83
]. The necessity and kind of deliberation that frames SA, of course, depends on the context the SA is embedded: “[F]ormal or informal, legally prescribed or voluntary, science-driven or policy-driven” [83
]. Irrespective of a more or less strong orientation at higher-level regulatory systems or political decisions, all three examined projects have a rather informal (in the sense of not executing a legal act or similar), voluntary and science-driven focus. This results in a certain indeterminacy which underlines the meaning of deliberation. The demand for a conscientious deliberation process in the three projects is further increased by the fact that the projects leave the paths of scientific routines with their approaches. Therefore, the authors encourage researchers and practitioners “doing” SA to plan adequate time and personal resources and methodological preparation to carry out a thorough deliberation process with the participating stakeholders. For the three presented projects, the deliberation process within the scientific project teams and with stakeholders is an integral part of the project and SA design—even if this aspect was not explicitly addressed in the context of the conception of the projects. In this respect, the approaches of the three projects are in line with Principle 7 (broad participation) of Bellagio STAMP [5
]. At this point, the potential for improvement could be brought by an approach that is more guided by theory and more stringently oriented towards methods of systematic and controlled integration of deliberation into the process of SA development. Some literature [83
] with a focus on methods, designed for different SA contexts, may help researchers with this challenge.
In Section 3
, data quality, data reliability and data availability have been identified as key challenges in all three project contexts. In this, we conform to many other authors. For example, Guinée lists this problem first [3
], Finkbeiner et al. call it a “big challenge” [20
], Sala et al. make the important statement that “uncertainties associated to results are barely discussed in literature, whereas, they could be extremely high due to data quality and availability” [92
], and the EPA states that “data availability will, in part, determine the necessary tool” [93
]. The Bellagio STAMP claim that “data quantity and quality continue to be serious problems” [5
Wiek et al. mention the “lack of resources for primary data collection” [94
], which can be seen as an important reason for the problematic data situation in many funded projects. In the NAHGAST project, the lack of (mainly ecological) data for the NAHGAST calculator indeed led to the leaving out of many ingredients that are used in gastronomy, which was by far the most mentioned criticism by practitioners. Whether this problem can ever be solved, given the high costs of collecting primary data and ever-scarce funding resources, seems to be as uncertain as to the data situation in many projects. On the other hand, the problem of the lack of mutual disclosure of data, which has already been addressed in this article, could more likely be solved within the scientific system by a higher degree of transparency among all those involved, at least where no economic utilization constraints prevent disclosure. In the quest for maximum transparency in terms of data, our results are in line with the corresponding claim of Bellagio STAMP in principles 5 (transparency) and 6 (effective communications) [5
]. In addition to the demand for openness and transparency, however, we did not find concrete solutions at this point in the further.
Regarding reliability and feasibility, individual decisions need to be made by the evaluator, which was identified as another challenge. Suh et al. confirm our SA experiences that the “decisions on inclusion or exclusion of processes in analysis (the cutoff criteria) are typically not made on a scientific basis” [82
]. They further state, that “the requirement of deciding which processes could be excluded from the inventory can be rather difficult to meet because many excluded processes have often never been assessed by the practitioner, and therefore, their negligibility cannot be guaranteed.” [82
] According to Suh et al., the neglect of capital goods can lead to a significant underestimation in LCI.
Clune et al. underline the necessity of common functional units in food LCAs, to simplify the comparison of reports, avoid misrepresentation and strengthen the validity of comparisons [95
]. Hunkeler et al. underline the fact that “an LCC practitioner might be confused regarding how to make the most appropriate choices” [96
]. Jeroen Guinée [97
] emphasizes not least, because of his presentation at the LCA Food 2018 in Bangkok: “LCA: [E]verything is relative, and nothing is certain”. Every SA, as was shown in the three projects and the previously mentioned literature, is always subjectively and individually influenced, which makes the comparison between different studies very complicated.
4.2. Assessment-Internal Challenges
As stated in Section 3
, values of the involved actors have an influence on the SA. According to Dijk et al., this problem is inherent in contemporary society: “Lack of consensus on ends or means is directly associated with the plurality in values and interests of today’s society” [83
]. The particularly important role that values play in the field of nutrition is shown by Alrøe et al. [88
], while the aspect of making actor’s values transparent is considered only indirectly and rather marginally in Bellagio STAMP (principle 5 transparency: “[E]xplain the choices, assumptions and uncertainties determining the results of the assessment”) [5
All three projects conduct (at least among others) research in the field of nutrition. Everyone is affected by nutrition and their own opinion that is, among others, influenced by habitual aspects in social milieus, cultural, philosophical and ethical backgrounds [98
], as well as (dis)information in media [100
]—the latter mostly about health aspects. Sometimes the topic of nutrition is downright ideological [101
]—the discussion of what people are “allowed” to eat is often felt as an encroachment on personal lifestyle [102
]. Within the three projects, we tried to deal with this challenge by creating a discourse that is based on a solid scientific foundation. However, making values explicit was no part of the SA tool development agenda, which could be a sensible task for further SA projects. This may be all the more relevant when researchers are not familiar with the cultural backgrounds of the participating practitioners, as is not only the case when doing research in emerging and developing countries [104
], but also within the ethnically often very diverse teams in the catering sector.
In the context of values, it should be mentioned that, of course, indicators from different dimensions are affected differently by underlying values. Ecological and health-related indicators, on the one hand, can be derived from the current state of research on ecological and health-related questions. This area of research is partially based on normative objectives, such as the 1.5 degree target of the Paris Agreement, the concept of reducing the consumption of resources by a factor of ten [105
] or the endeavor to prevent health damage. However, these normative foundations are broadly accepted by the scientific community. The situation in the area of assessment of social indicators is quite different, e.g., for the question: Which extent of ingredients needs to be produced in an acceptable manner, for both humans and animals, for a meal to be sustainable. In this respect, there is a lack of scientifically approved target values, which is why a higher level of subjectivity is included in that context. To balance the subjectivity of the approach, transparent stakeholder dialogues on the social target values should be conducted, as was done in the NAHGAST project.
The discussion about normative assessment principles underlines that in SAs—in all contexts –normativity is an inevitable and integral part actors have to deal with, because they necessarily need to make assumptions and work with heuristics. SA should, therefore, be a joint, stakeholder-based deliberation, with increased transparency—when it comes to normativity.
Therefore, we view SA not only as tools for measuring, but also to encourage scientists, and above all, practitioners getting involved with the topic of sustainability. The obtained knowledge can lead to a change in behavior, for example, in terms of consumption. This shows the interaction between SA and values: One the one hand, SA projects are influenced by the actor’s values and assumptions, on the other hand, the SA could have an effect on the user’s values in turn.
In this article, a ‘most different case study approach’ was carried out for identifying common challenges researchers are facing in using SA approaches and how to overcome them. We envisaged to emphasizing practice-oriented aspects for the benefit and encouragement of other researchers and practitioners. The analysis indicates that doing SA is important across different contexts, but that it is especially relevant to identify the specific purpose, challenges and additional benefit of doing SA within each context. Stemming from the limitations of our most different approach, the need to conduct further, more in-depth case studies becomes apparent—if possible, in the form of a separate research project, instead of an additional task to the actual research work.
On the basis of the deliberative analysis of the research topic, the three projects created different assessment tools and models that supported context-specific interventions in the respective field of research. Thus, the tools and models fulfilled two functions: Firstly, they carried out the SA of the respective item. Secondly, they served the deliberative discourse among the participating scientists and practitioners on the nature of sustainability, in relation to the respective object of observation. This can motivate actors to deal with content and methodological issues of the subject and lead to individual and organizational learning. Even though this aspect was rather implicit in the projects, the cases indicate that a substantial occupation with the case-specific character of sustainability helps to manage the challenge of gaining a common understanding of the researchers and the stakeholders about sustainability. Researchers should integrate sufficient resources for this deliberation process in their projects. Further research potential, especially regarding methods for designing deliberation processes, can be assumed in this area.
Regarding normative assessment principles, measurable criteria have to be derived from the normative idea of sustainability. This is why sustainability evaluations always have a normative basis and depend on the societies’ and the actors’ values. Those values vary over time and are influenced by external impulses. Indicators from different dimensions are affected by underlying values differently. Some normative (especially ecologic and health-related) foundations are broadly accepted by the scientific community, while the situation in the area of assessment of social indicators is quite different. Hence, the case studies indicate that transparent disclosure of underlying values has been underestimated so far and show the importance of a context-dependent definition of sustainability. The discursive integration of stakeholders in the tool designing process can help to deal with their underlying values and to enhance the assessment’s objectivity. Again, a need for further practice-oriented methods must be emphasized.
Furthermore, although all assessment tools aim to measure sustainability in general and in their specific focus area, they all encounter problems with data reliability and data availability on different (indicator) levels. Therefore, within further research, it is crucial to work on data availability. In order to achieve this, each SA is encouraged to create transparency on which existing data were used and ensure the data availability that was generated through the SA for further research. Data disclosure plays an important role with regard to comparability and validity assessments. Therefore, there should not only be a call for the availability of primary data, but rather more transparency in the publication of the results (such as for methods of syntax, allocation, assessment indicators, as well as database, software, etc.) is required. In that sense, using the same database and applying the same methods can still lead to different results, since different parameters or different weighting of these individual parameters are stored within different software. Only comprehensive transparency regarding the chosen indicators—and the definition of the scope for a feasible and appropriate SA under the circumstances of data availability and time restrictions—can lead to an improved understanding of the sustainable steps taken and the data used. Hence, comparability can be achieved, when everyone avoids hiding data and information needed to retrace the sustainability assessment; the comparison and disclosure of different methods and their problems in terms of triangulation can lead to an overarching and deeper understanding of how to achieve sustainability goals, for specific problems and in general. Nevertheless, our case study confirms that the creation of primary data, especially ecological data, is one of the most important and urgent needs for further research in the field of SA to ensure the validity of SA results.
One reason for the difficulties encountered might be that literature does not provide standardized steps for developing and carrying out SA. The goal is to promote sustainability by taking different views and aspects into account and combining them with manifold and suitable methodologies. As different as the settings and methods might be, the development of different approaches to assessing sustainability leads to useful results in various contexts. Accordingly, SA projects should be encouraged to combine different methods to cover the whole spectrum of sustainability and face the challenge of how to integrate various methods.