Conceptualizing the Effectiveness of Sustainability Assessment in Development Cooperation

Sustainability assessment has emerged as a key decision-support process in development cooperation in response to the growing acknowledgement of the impacts of global change. This paper aims at conceptualizing the effectiveness of sustainability assessment as applied in development cooperation, by focusing on the sustainability assessment practice by actors of the official Belgian Development Cooperation. The conceptualization of the effectiveness of sustainability assessment is synthesized in a set of issues and concerns, based on semi-structured interviews. The paper highlights the specificity of sustainability assessment in the development cooperation sector (e.g., through the cultural and discursive compatibility dimensions of assessment in a North-South context). Effectiveness is inherently linked to the expected functions of sustainability assessment in the decision-making process, which include fostering organizational change, shaping contextually adapted framings of OPEN ACCESS Sustainability 2015, 7 5736 sustainability and operationalizing the sustainability transition. These findings highlight the relevance of a discourse-sensitive approach to sustainability assessment if one is to strengthen its credibility and legitimacy.


Introduction
There is a growing sense of urgency for integrating sustainability into development cooperation.This urgency could be attributed to the rise of environmental issues on the international political agenda, the visible and projected consequences of climate change in the South [1], and the acknowledgement of the interrelationships between environmental vulnerability, aid efficiency and poverty reduction, particularly in developing countries [2].Sustainable development (used synonymously in this paper with sustainability, as in [3]) is defined as development that meets the needs of the present, while safeguarding the Earth's life-support systems, on which the welfare of current and future generations depends [4].Due to the different interpretations of sustainability, it may be challenging to translate the concept into action, although the development and use of indicators and assessment tools allows the concept to be operationalized in various settings [5].
While there is a general acceptance that sustainability should guide development cooperation [6,7], advocating the use of sustainability as a decision-making strategy requires processes that allow operationalization of sustainability for policy makers, project managers and development partners.Designing an assessment framework e.g., sustainability assessment is one such process, which may be useful in this context [8].Sustainability assessment is not a prescribed process as yet (contrary to the codified nature of environmental impact assessment and strategic environmental assessment).It is rather an orientation of practice [9] that allows for many degrees of freedom in developing context-specific definitions, methods and applications.
The study of the effectiveness of sustainability assessment has recently gained new attention from scholars and practitioners alike [10][11][12][13][14][15][16][17].This has been accompanied by the acknowledgement of the political nature of sustainability assessment, as well as the recognition of the expectations, framings and roles of a variety of involved actors, and the learning implications of sustainability assessment [18].
However, the effectiveness of sustainability assessment in development cooperation has rarely been looked into.Though [12] and [14] have dealt with sustainability assessment effectiveness in a comprehensive way, there is currently little research on the development co-operation context.In times when the functions of development cooperation are being reviewed [6], and when sustainability challenges (in particular climate change and inequality [1,19]) have come to dominate the international agenda, it is timely to reflect upon the effectiveness of sustainability assessment in development co-operation.This paper aims to bridge this gap and focuses squarely on sustainability assessment as a decision-guiding strategy in development co-operation [20,21].Development cooperation is defined here as financial and/or technical aid given by governments and other agencies to support the economic, social, environmental and political development of developing countries (inspired by the definition proposed by the Organization for Economic Cooperation and Development [6]).
We aim to develop a qualitative conceptualization of the effectiveness of sustainability assessment in development cooperation.A case study approach was chosen as it has the capacity to connect practical complex events to theoretical abstractions [22], and as it acknowledges the importance of context, which is key in shaping the conceptualization of sustainability assessment effectiveness.This paper focuses on a particular example of a sustainability assessment tool, known as the "KLIMOS toolkit" as applied by the Belgian Development Co-operation between 2010 [23].This institutional context influenced the framing of sustainability in the KLIMOS toolkit, which could be termed as prudently reformist following [24].
This paper includes the empirical identification of concerns and needs regarding sustainability assessment effectiveness among assessment users (in our case officials from the Belgian Development Cooperation).This is the first step in the development of a set of sustainability assessment effectiveness criteria.Our empirical research was guided by three main questions.(i) How can the effectiveness of sustainability assessment in development cooperation be conceptualized based on the identification of users' concerns and needs?(ii) What modifications are necessary in the framework of Bond et al. (2012) [12], to suit the needs of the development cooperation context?(iii) How can future sustainability assessment practice and effectiveness be improved for development cooperation?
A full analysis of the concerns and needs of other actors (including partner organizations in the South, scientists, etc.) is beyond the scope of this paper.However, we are aware of the importance of including all stakeholders' perspectives [25] in the next steps towards a comprehensive assessment framework of sustainability assessment effectiveness.The body of the paper starts with a brief outline of sustainability assessment theory, which is then followed by the methodology, the results and the discussion.The conclusion includes recommendations on future research.

Sustainability Assessment and Its Effectiveness
The origins of sustainability assessment are partly found in an extension of the scope of classical environmental assessment.Hence sustainability assessment is sometimes referred to as "the third generation of impact assessment" [9].In a North-South context, the adoption of sustainability as a key goal of human development has led to a repositioning of impact assessment, with sustainability now invariably seen as its ultimate purpose [9,10].
Sustainability assessment might be interpreted in a variety of ways depending on the context and on actors' expectations [26,27].Although sustainability assessment has been institutionalized in a number of jurisdictions worldwide (such as in England, Western Australia, Belgium, Switzerland and South Africa), this has happened under different names, for different purposes and with different scopes of application [8,28].
Consequently there is no clear definition of what a "good" sustainability assessment process entails despite the existence of generic process quality criteria such as the Bellagio principles [11].This makes it important to conceptualize the effectiveness of sustainability assessment.Effectiveness refers to the question of whether something works as intended and meets the purposes for which it was designed [12].Bond et al. (2012) [12] define four categories of sustainability assessment effectiveness, i.e., procedural, substantive, transactive and normative (see Table 1), which, along with the consideration of how pluralism and learning are addressed, provide a framework for evaluating sustainability assessment practice.

Effectiveness Category Key Question
Procedural Have appropriate processes been followed that reflect institutional and professional standards and procedures?

Substantive
In what ways, and to what extent, does sustainability assessment achieve its intended objectives?
Transactive Were resources (human, financial) efficiently used in the sustainability assessment?

Normative
In what ways, and to what extent, do the involved actors learn, improve their knowledge, modify their perspectives on sustainability and adjust their policy choices during the sustainability assessment process?
These categories reflect the diversity of functions that sustainability assessment can (sometimes simultaneously) perform in decision-making.First, sustainability assessment can be a tool and process to generate information.It is then seen as an objectifying process, providing instrumental knowledge that can underpin specific decisions [29].Second, moving beyond mere information generation, sustainability assessment can also structure complexity by providing a systematic approach to deal with the intrinsic complexity of multidimensional societal challenges as well as with institutional complexity [29].Third, sustainability assessment can be a forum for deliberation thereby contributing to a shared understanding of sustainability in a specific context, acknowledging the plurality of values at stake [30].Fourth, sustainability assessment can also foster attitude shifts in decision-makers, thereby creating opportunities for policy change [29,31].
These potential functions in turn reflect the discourses shaping sustainability assessment.A sustainability assessment framework can favor particular discourses of both environmental governance and sustainability, which points to the need for sustainability assessment practitioners and users to be aware of the shifting patterns of dominant and marginalized discourses and of their associated stakeholders [27].There has been a boom in scientific literature focusing on sustainability assessment [28], and a comprehensive review of the literature lies outside the scope of this paper.However, it is important to acknowledge the political nature of sustainability assessment [32], as well as to be aware of the diversity of expectations, approaches [33] and tools [34] used, in order to grasp the concept of sustainability assessment effectiveness.

Expert Interviews
We opted for expert interviews (n = 10) as the main method to assess and conceptualize the effectiveness of sustainability assessment as used by the official Belgian Development Cooperation entities.This is due to the importance of context in understanding sustainability assessment effectiveness, especially in the pilot phase, and given the number (n = 15) of pilot applications of sustainability assessment (in our case the KLIMOS toolkit) in 2010-2013.The methodology focused on understanding rather than on statistical generalization.The term "experts" in research methodology refers to "persons responsible for development, implementation or control of solutions, strategies or policies" [35].Experts usually have privileged access to information about groups of persons and/or decision processes and have a high level of aggregated and specific knowledge that is otherwise difficult to access [36].The KLIMOS toolkit users are here termed as experts, having applied sustainability assessment on a range of interventions (e.g., project proposals).
The face-to-face interviews were individual.To reduce bias, we did not provide the respondents with any a priori definition of sustainability nor of sustainability assessment.Instead they were encouraged to elaborate on their own perception and interpretation of the effectiveness of the sustainability assessment(s) that they were directly or indirectly involved with.
The interviewees (n = 10) were selected based on their expertise and role in the KLIMOS Toolkit application in 2010-2013.Most were officials working for the official Belgian Development Cooperation entities, others were external academic experts who were directly involved in the development of the toolkit (see Table 2).The interviews were performed between September 2013 and May 2014, in Brussels, Belgium and Montreal, Canada.The interviewees ideally had a broad understanding of the KLIMOS Toolkit context.The officials were involved in the practical application of the tool, whereas the academic experts had more of an informed outsider's view.In order to improve the validity of the research outcomes, conclusions will be based on the inter-subjectivity of the responses [10]: the agreement or consensus between the interviewees about the conceptualization of the effectiveness of the KLIMOS Toolkit.The interviews were semi-structured, allowing for elaboration on certain topics.The basis of the questionnaire was repeatable and systematic.We performed systematizing expert interviews [37]: multiple experts were interviewed to be able to compare and aggregate data to gain process knowledge.
The interviews were semi-structured based on the framework of [12], refined in [14].The conceptualization of substantive effectiveness in the questions is adapted from [17].Table 3 provides the structure of the interviews (leaving out the questions pertaining to demographics and job content of the interviewee).The interviewees were encouraged to respond to the question with their particular practical sustainability assessment experience in mind, and were asked to reflect on elements and concepts that they considered being of key importance.This elaboration would result in in more detailed data that would enrich the identification and interpretation of concerns and needs, and in subsequent steps, criteria.Table 3.General structure of the interviews.The KLIMOS Toolkit).

General question (G)
G-Q1: Do you have any additional remarks, suggestions concerning the effectiveness of sustainability assessment?

Content Analysis
In order to complement the data gathered through the interviews, draft and final project documents as well as completed KLIMOS toolkit "screening notes" were analyzed qualitatively, mainly to identify indications for substantive effectiveness such as possible modifications in project content after the application of the sustainability assessment.It is less straightforward to identify indicators that allow the assessment of procedural, substantive, transactive or normative effectiveness in project documents.When available, project documents and screening notes were assessed from individual archives provided by the respondents.
Synthesizing the interview data into a set of concerns and needs inevitably involves a degree of interpretation.In order to ensure the transparency and traceability of the interpretation by the interdisciplinary team of authors, all interviews were transcribed and subsequently coded for recurrent terms and ideas along the pre-established categories of effectiveness (see Section 1).A second qualitative analysis of the transcripts was performed without the pre-set categories in mind.

Results
The interviews yielded insights that allowed the synthesis of 27 concerns and needs regarding sustainability assessment effectiveness in a development cooperation context (Table 4).Procedural effectiveness concerns were centered on the adaptability and flexibility of the sustainability assessment procedure.Issues of timing, training and integration with intra-and extra-organizational decision-making processes were highlighted, as well as differences in expectations and capacity between technical field staff and office staff.
Ten concerns and needs regarding substantive effectiveness were synthesized from the interview transcripts (see Table 4).Substantive effectiveness answers the question: "did the sustainability assessment lead to tangible change?"(in a project proposal and/or "on the ground").We focus here on concerns that were not necessarily shared by everyone, but which point out interesting views on the purpose of sustainability assessment.Enforcing the outcomes of the sustainability assessment, i.e., ensuring that the proposed changes are realized, is included.This is because some respondents felt that a noncompulsory sustainability assessment (such as the current KLIMOS toolkit) will otherwise not be substantively effective.This raises an important question regarding the function of sustainability assessment and points at the characteristics of decision-making itself.There is, however, no linear relationship between sustainability assessment outcomes and the ultimate decision (e.g., on funding a project proposal or not, based on the project's expected sustainability implications).
The transactive effectiveness concerns include time, skill, financial resources and specification of roles.The respondents emphasized this last point.This also led to the identification of an additional concern on communication among the actors implementing the sustainability assessment throughout the various phases of the project cycle in the different organizations involved.This focus on exchanging experiences stresses the importance of sustainability assessment-induced learning.
The questions aimed at appreciating normative effectiveness were intentionally broad and triggered wide-ranging responses yet were also considered by some respondents as a little overwhelming.The normative criteria proposed in Table 4 go beyond the generic criteria, and include both organizational, discursive and cultural alignment, all aspects that can are considered of paramount importance in a cross-cultural international cooperation environment.The issue of attribution was especially difficult to solve: i.e., ascertaining whether a particular (re)action was actually caused by the introduction and/or application of sustainability assessment or whether it was the result of a more general momentum towards sustainability.This was especially apparent with regard to the "reversal of unsustainable trends" question.Some interviewees stated that the very existence of a sustainability assessment process was in itself a quality label for the organization, airing the idea that realizing a sustainability assessment is a good thing to do regardless of the quality of its implementation.
The learning dimension of the application of sustainability assessment was highlighted by most respondents.It was referred to e.g., as "mindset creation" and includes both individual and organizational learning, confirming [12,28].Individual learning was experienced by a range of respondents, yet was presented as necessarily related to a broader-formal-sustainability learning effort (e.g., through staff training sessions) that could not be attributed to nor provided solely by sustainability assessment.The respondents acknowledge that learning is a key mechanism in improving practice over time, as also stated by [12].
The need for regular iteration of sustainability assessment application was stressed (as people in the administration often change positions, as the political support for sustainability (assessment) might shift, etc.), reflecting a genuine concern for the long-term anchoring of the sustainability assessment process in the organization(s).Most respondents did not elaborate further on their preferences regarding the prescriptive role of sustainability assessment (i.e., what it ought to realize at the organizational level).
The alignment of the purpose and the structure of a sustainability assessment process with the dominant sustainability discourses or framings of the organization (in our case the official Belgian Development Cooperation) was considered key by the respondents.This reflects the importance of perceived compatibility between the organizational culture and the way sustainability assessment-triggered changes are framed.Beyond discourses and policies, the respondents acknowledged the importance of a certain alignment with the cultural references of the partner countries in the South, when dealing with sustainability.This aspect was considered key to realizing normative effectiveness.
While the use of the four effectiveness categories was helpful in structuring the interviews and in analyzing the outcomes, logically the interviewees did experience effectiveness as a multidimensional concept within which the category boundaries were not absolute.This is particularly important given that the flexibility of the sustainability assessment exercise was seen as key by the respondents in discussing procedural effectiveness.At the same time, the potentially contradictory requirement of clarity of purpose was emphasized.Whether the sustainability assessment is presented and used as a process to support decision-making or as an evaluative process (even with possible financial (funding) consequences) makes a considerable difference in its perception (e.g., by actors dependent on development cooperation funds as mentioned by three respondents) as well as in the required type of procedure.Some concerns and needs were characterized as being relevant for various categories of effectiveness (e.g., adequate timing; clear specification of roles) indicating the cross cutting character of some of the concerns, as well as the somewhat artificial subdivision of multidimensional sustainability assessment effectiveness.The porous boundaries between the subdivisions of effectiveness also reflect the linkages between a sound and efficient procedure and a substantively and normatively effective assessment process.In the same vein, some sustainability assessment effectiveness concerns were challenging to categorize.These include linking sustainability assessment to the broader North-South policy dialogue on the changing conceptualization of development cooperation (although this issue was mentioned by two respondents as an element of substantive effectiveness, it embodies a normative effectiveness element as well).

Discussion
The concerns and needs regarding sustainability effectiveness in development cooperation proposed in this paper are a useful addition to the generic effectiveness criteria in the existing literature.When comparing the identified concerns and needs regarding sustainability assessment effectiveness in development cooperation (Table 4) with the set of generic criteria proposed by [12] and [14], it is apparent that there is a lack of an explicit mention of stakeholder involvement in Table 4.
While the transactive criteria are broadly similar, although with a stronger focus on communication in the development cooperation sector, there are some noteworthy differences between the generic substantive criteria and the concerns and needs proposed in Table 4.While the generic criteria of [14] stress public consultation, this is left unmentioned by our respondents.They instead focus more on the alignment between the sustainability assessment approach and the needs of the organization and its partners, without making public consultation explicit, unless one considers transparency to entail consultation and participation requirements.The dominant concern issued by the interviewed development cooperation practitioners, which is mostly ignored in the generic criteria, concerns the required adaptability and flexibility of sustainability assessment process: An adapted procedure that matches the needs of different actors (e.g., users in North and South, officials and non-governmental partners, technical staff and policy-makers), as well an adaptation to the project cycle and openness with regard to various sustainability framings.
We would also like to highlight that the elements set out in Table 4 should be interpreted as "key concerns and needs related to the effectiveness of sustainability assessment in development cooperation, as proposed by the users of a sustainability assessment tool".The proposed concerns can be interpreted from a prescriptive perspective.If so, they embody the ideal characteristics of an effective sustainability assessment and they contribute to answer the question: "What should an effective sustainability assessment look like?"But the proposed concerns can equally be interpreted from a descriptive perspective, and can hence contribute to an-ex ante or ex post-evaluation of the effectiveness of a specific sustainability assessment exercise.In that case the concerns contribute to answer the question: "How effective was/will be this specific sustainability assessment exercise?" Statements such as "sustainability assessment contributes to the translation of sustainability into understandable concepts and actions" and "sustainability assessment should contribute to action and reinforce policy coherence" hint at the perceived role of sustainability assessment in organizational learning.Sustainability assessment can indeed contribute to steering organizations towards sustainability, through a process whereby aligned individuals and groups congruently learn and modify their mental models (cf. the abovementioned "mindset creation") and actions [38].These phrasings are reminiscent of sustainability transition thinking, which entails a change from one state of a system to another and is described as a multi-actor, multi-level and multi-phase movement spanning several decades [39].While the Belgian Development Cooperation has not officially taken up sustainability transition thinking, the concept is increasingly used in Belgian official documents [40].Transition as an ultimate aim of sustainability assessment was mentioned by a few respondents, although there are few empirical foundations that link the gradualism of sustainability assessment to the transformative approach of transition experiments [41].
The stated purpose of sustainability assessment may influence the perceived and real "ownership" of the process in a development cooperation context [42], and resonates with the more general question on the purpose of sustainability assessment effectiveness criteria which can be prescriptive or descriptive.Decision-making is a messy process subject to many influences [30], and the information generated by a sustainability assessment is only one of many factors influencing the final decision.Sustainability assessment is a decision-supporting exercise, which has no bearing on the final decision.Furthermore, a sustainability assessment will only have the potential to generate change if its message is considered legitimate and necessary by policy-makers, as well as by other societal actors), and if other influencing factors are taken into account.Huston (2008) [43] cites ideology, interests and institutional context among other decisioninfluencing factors.
Without delving deeper into the functions of sustainability assessment in decision-making (for which we refer the reader to [30]) it is key to consider the enforcement concern with caution.Enhancing the sustainability assessment process with regard to transparency and accountability may already be a first step, as suggested by the respondents.
Few of the interviewed sustainability assessment users had major remarks on the current framing of sustainability in the sustainability assessment, although they were aware of the multi-interpretability of the concept.This situation did not necessarily reflect a lack of sensitivity to other sustainability discourses.This is because discursive compatibility between North and South was mentioned as a concern, reflecting the realization of different worldviews exhibited by Northern and Southern development partners [44].
The differences in sustainability framing surfaced in particular when dealing with tradeoffs and prioritization of "brown" versus "green" sustainability issues.While brown development projects typically address urgent and tangible local sustainability issues (e.g., clean water, environmental health, climate change adaptation), "green" development projects address regional or global sustainability issues focusing on global public goods (biodiversity, climate change mitigation) [45].Sustainability assessment is ideally expected to provide guidance in balancing these two legitimate approaches towards sustainability in development cooperation.Furthermore, the dominant perspective stating that sustainability assessment should not be isolated from the broader policy dialogue on development and environment issues, pointed towards the importance of policy coherence.The ongoing international climate negotiations, the Sustainable Development Goals and the momentum that these events create, were mentioned by the respondents as being intrinsically linked to small-scale initiatives such as the KLIMOS toolkit sustainability assessment.

Conclusions and Recommendations
The dense qualitative data generated through the interviews allowed us to propose a set of 27 concerns and needs regarding sustainability assessment effectiveness.Further research will allow the concerns to be synthesized into criteria, and will require the involvement of other involved actors beyond the direct users of sustainability assessment (e.g., non-governmental actors in North and South).Based on the gathered interview data, the generic effectiveness framework of Bond et al. (2012) [12] is complemented and customized for development co-operation by focusing on inter-cultural compatibility and on communication between the various involved actors in the assessment process, as well as by a focus on linkages between the various stages of the project cycle.
Together the concerns and needs related to procedural and transactive effectiveness (embodying organizational change), and the concerns and needs related to substantive and normative effectiveness (embodying the framing and the realization of sustainability) provide a sustainability assessment framework which is of direct practical use for both descriptive and normative purposes.
In conceptualizing the effectiveness of sustainability assessment in development cooperation, the respondents stressed the fact that it is critical to focus on an adaptable and flexible sustainability assessment procedure, including alignment with organizational objectives and including openness with regard to a variety of interpretations of sustainability.This awareness of a plurality of sustainability interpretations as well as of a wide range of expectations regarding sustainability assessment is present among the respondents, and procedural flexibility is seen as a strategy to deal with this.This apparently contrasts with the codified nature of "classical" impact assessment processes such as environmental impact assessment (EIA) and strategic environmental assessment (SEA) which are as a rule "stricter" when it comes to addressing trade-offs, pointing to the risk of overly flexible sustainability assessments as stressed by [46].
While the empirical findings of our research contribute to a better understanding of the effectiveness of sustainability assessment in development cooperation, future research is needed on the potential and actual effectiveness of sustainability assessment in development cooperation in:  fostering organizational change towards sustainability.While organizational change is determined by internal motivation, organizational capacity and external context, the introduction and application of sustainability assessment is influenced by and influences all these factors.Through its focus on learning, sustainability assessment can contribute to gradual organizational change, yet empirical evidence on this topic in development cooperation organizations (bilateral and multilateral agencies, non-governmental organizations, etc.) is lacking. framing sustainability in a development cooperation context.By translating sustainability into manageable bits of information (including indicators), sustainability assessment can determine how sustainability is understood, framed and realized in the multi-actor and inter-cultural context of development cooperation.Setting sustainability goals operationalizing and contextualizing a sustainability vision is a key step in translating that framing into manageable decision-supporting information as proposed by [26].Sustainability assessment can also contribute to discourse reflection and formation [47].Stimulating decision-makers and other stakeholders to critically reflect upon and to reconsider their perspectives on sustainability may enhance the chance of sustainability assessment to influencing decision-making.However, here again, systematic empirical evidence on the actual experience of sustainability assessment-mediated sustainability framing in development cooperation is mostly absent. research on the role of sustainability assessment in realizing the sustainability transition through changing normative principles and through their operationalization on the ground is needed.
The capacity of sustainability to capture the major challenges that development cooperation aims to address at its modest level (such as widespread poverty, climate change, resource depletion, etc.) by providing a decision-support framework will need to be continually improved and fine-tuned.Research on the functions and on the effectiveness of sustainability assessment will be instrumental in realizing this.
The process of decision-making, including decision-support processes such as sustainability assessment, ultimately determines a decision's legitimacy and credibility.While sustainability assessment might be instrumental to boosting the legitimacy of development cooperation by explicitly linking it to the broadly shared agenda of sustainability in times of global change, the data gathered indicate that the interviewed users of sustainability assessment in development cooperation have a deeper, more ambitious understanding of its potential.The next step following this research project will entail the translation of the identified concerns and needs related to sustainability assessment effectiveness into synthetic criteria that can be applied on practical sustainability assessment exercises.
: What is the (administrative) status regulating the use of sustainability assessment (in casu: The KLIMOS toolkit) in your organization?P-Q2: What is your appreciation of that status?P-Q3: What is the procedure for applying sustainability assessment (in casu: The KLIMOS toolkit) in your organization?P-Q4: What is your appreciation of that procedure?P-Q5: What are possibilities for the future administrative status and procedure of sustainability assessment (in casu: The KLIMOS toolkit)?Substantive effectiveness (S) S-Q1: Did decision-makers in the Belgian Development Cooperation entities consult (seek information from) the sustainability assessment (in casu: The KLIMOS toolkit screening notes and/or database) during the decision-making process (e.g., during the various stages of the project cycle)?S-Q2: To what extent, and how, did the sustainability assessment (in casu: The KLIMOS toolkit) function as a reference during the decision-making process?Was it used to develop, review and/or discuss the project and possible alternatives?S-Q3: Did the sustainability assessment (in casu: The KLIMOS toolkit screening note and/or database) influence subsequent decision-making or other impact assessment exercises?If so, how?S-Q4: Did the KLIMOS Toolkit Screening/Database educate actors involved in the decision-making process about the sustainability implications of the project proposal?S-Q5: Did the actors involved in the decision-making process alter their vision of the project proposal due to the sustainability assessment exercise (in casu: The KLIMOS toolkit)?S-Q6: What was the most important contribution of the sustainability assessment (in casu: The KLIMOS toolkit) to the decision-making process (e.g., regarding project proposals-In particular or generically)?S-Q7: Was the content of the final project proposal modified due to the application of the sustainability assessment (in casu: The KLIMOS Toolkit)?S-Q8: If yes, how was the project proposal modified, and what are the possible indications pointing to a causal relationship between the sustainability assessment (in casu: KLIMOS Toolkit) and the modification of the project proposal?S-Q9: When you have indicated that the project proposal has been altered because of the SA (in casu: The KLIMOS toolkit) how significant is this modification?Transactive effectiveness (T) T-Q1: Has acquiring the requisite skills and training the necessary staff for implementing the sustainability assessment (in casu: The KLIMOS toolkit) constituted a big burden or not?T-Q2: Were responsibilities clearly defined and allocated and were tasks undertaken by the most appropriate people?T-Q3: Could you elaborate on the time and financial resources used to realize the application of the sustainability assessment (in casu: The KLIMOS toolkit)?Normative effectiveness (N) N-Q1: Did the application of the sustainability assessment (in casu: The KLIMOS toolkit):  Reverse prevailing (unsustainable) trends? Integrate key intertwined factors affecting sustainability? Minimize non-transparent tradeoffs?N-Q2: What do you think of the framing/interpretation of sustainability in the sustainability assessment (in casu: and 2013 (KLIMOS is a Dutch acronym reflecting the focus on climate & development cooperation).The KLIMOS toolkit was co-developed by academics grouped in the "KLIMOS Research Platform on Climate Change & Development Cooperation", and officials from the Belgian Development Cooperation.The official Belgian Development Cooperation (Directorate for Development Cooperation & Humanitarian Aid of the Belgian Ministry of Foreign Affairs, Foreign Trade & Development Cooperation and the Belgian Technical Cooperation) decided to fund and supervise the development of a customized sustainability assessment tool and to pilot its application on a range of development projects.This initiative is rooted in the Development Cooperation Act (dated 19 March 2013), which stipulates in Article 11, § 2 that the Belgian Development Cooperation will integrate the environment as a cross-cutting issue in all its interventions

Table 4 .
Concerns and needs regarding the effectiveness of sustainability assessment applied in development cooperation.
procedural effectiveness P1: Adaptability: The sustainability assessment procedure is flexible and adaptable, ideally modular, to fit requirements and structures of various organizations and actors."Any sustainability assessment tool should acknowledge the need of the variety of users who are expected to use it (e.g., technical staff and desk officers)."P2: Clarity of purpose: Is the sustainability assessment aimed at decision-support or at evaluation? "Failing to clarify our main purpose will make it difficult to convince our partners (such as non-governmental organizations) to collaborate in completing a sustainability assessment.What's in it for them?"P6: External tiering: Vertical integration of sustainability assessment with other processes, templates and databases at different decision-making levels in the partner organizations/partner countries "The sustainability assessment process and outcomes should be linked with the international standards of the OECD-Development Assistance Committee, such as the "Rio markers", which indicate if an cooperation initiative targets environmental objectives."
).This would make it more relevant, and everyone would see the instrument as-Partly-Its own."S4:Adaptation to the project cycle:Sustainability assessment stages linked to the organizational flowchart, as well as to monitoring schemes "Ideally, the questions should be related to the stages of the project cycle, which often have their own appointed official per project."S5: Quantitative target setting, indicators and monitoring in the sustainability assessment "Without measurable data, how will we convince anyone of the use of this exercise?"S6: Operationalization of sustainability: The sustainability assessment reflects the multidimensionality of sustainability; sustainability is linked to local concerns in the South ("green" and "brown" environmental issues) "Sustainability does not mean the same to everyone here in our department, let alone when one talks to people in our partner countries.Any sustainability assessment should acknowledge this fact."S7: Incorporation of changes: Into project proposals (link between the assessment and the content of the proposal), possibly linked to a sustainability/environmental action plan, reflecting the ability of sustainability assessment to guide trade offs "When the contents of your project proposal have been modified (and became more sustainable), then your sustainability assessment has been successful."

S8: Enforcement of the outcomes of sustainability assessment
: e.g., link sustainability assessment outcomes with financial consequences "We should not only make sustainability assessment compulsory, we should make a satisfactory outcome a condition for funding (at least in the long run)."S9: Transparency: Visibility of the consideration of the sustainability assessment outcome in the decision-making process "Everyone should know we are using this tool, it is good for our accountability and reputation."T4: Helpdesk: Providing timely ad hoc support during the sustainability assessment process "A helpdesk would allow us to ask specific questions to experts on short notice."T5: Clear outline of responsibilities: Division of roles for all actors, in all involved partner organizations "Who is responsible for performing the assessment, who will enforce its outcomes?Will I get feedback from the other departments?"