Next Article in Journal
Transferring the Cost of Wage Rigidity to Subcontracting Firms: The Case of Korea
Previous Article in Journal
The Use of Oral Histories to Identify Criteria for Future Scenarios of Sustainable Farming in the South Yangtze River, China
Article Menu

Export Article

Sustainability 2016, 8(9), 861; doi:10.3390/su8090861

Article
Can We Improve Indicator Design for Complex Sustainable Development Goals? A Comparison of a Values-Based and Conventional Approach
1
Department of Environmental Science and Engineering, Fudan University, Shanghai 200433, China
2
Values and Sustainability Research Group, University of Brighton, Brighton BN2 4GJ, UK
3
Wageningen University and Research Centre, Wageningen 6700 HB, The Netherlands
4
Department of Environmental Science & Engineering, Fudan Tyndall Centre, Fudan University, Shanghai 200433, China
*
Author to whom correspondence should be addressed.
Academic Editor: Marc A. Rosen
Received: 18 April 2016 / Accepted: 22 August 2016 / Published: 30 August 2016

Abstract

:
A conceptual framework was constructed for United Nations’ complex Sustainable Development Goal (SDG) Target 4.7 focusing on education for sustainable development (ESD), and used to analyse the usefulness and character of indicators produced from a values-based approach called ESDinds, compared to a UN process. The analysis shows that the latter generated very few indicators concerning the wider aspects of knowledge such as ‘critical thinking’ or ‘learning to learn’. The values-based approach, created for a different purpose, produced complementary if not better coverage of Target 4.7, including finely-developed concepts for competencies and less tangible aspects. It is suggested that the UN process would benefit from ESDinds design elements such as intersubjective and slightly disruptive elements, purposeful contextualisation at group level, and a holistic and inductive consideration of values. The use of a reference ‘fuzzy framework’ of slightly generalised proto-indicators suited for deep contextualisation locally is recommended, rather than any rigid global-level indicator with unclear local value. It is recommended that ESD practitioners immediately develop localised interpretations of valid measures for whatever final Target 4.7 indicator is selected by the UN, as this localisation process will itself cause important learning towards local ESD achievements.
Keywords:
Sustainable Development Goals; sustainability indicators; indicator design; values-based indicators; education for sustainable development; education for sustainability

1. Introduction

The creation of the 2030 Sustainable Development Goals (SDGs) responded to the call by United-Nations Secretary-General Ban Ki-Moon for “the most inclusive and transparent development agenda the world has ever seen” ([1], p. 4). People in 194 countries contributed to the goals through a systematic process of 88 national consultations, 11 thematic dialogues, and a global online survey with over 8.5 million participants [1,2]. The resulting goals are also distinctive in being applicable to all nations, regardless of gross domestic product or geographical location [3] which makes them a marked improvement on the Millennium Development Goals, which were widely criticised for ‘ghettoising’ development as an issue for the global South ([4], p. 1184). The SDGs represent the first explicit acknowledgement, at the level of global goals, of the interconnectedness of the challenges surrounding sustainability and the corresponding need for integrated problem-solving [3].
While the SDGs are in many ways an important accomplishment, fundamental questions remain around how their implementation will be monitored and evaluated at all levels, from the global to the most immediately local. It is difficult to overstate the importance of identifying relevant and valid indicators. The 2015 Millennium Development Goals (MDGs) Report has explicitly acknowledged that “what gets measured gets done” (p. 10, [5,6]). The choice of indicators to measure progress towards the 169 SDG targets will contribute substantially to shaping national policies, strategies and action plans in most UN Member States, from now until 2030.
As noted by Gudmundsson [7] and subsequent authors [8,9,10,11], the contributions made by indicators to sustainable development go far beyond their instrumental uses in providing data that may inform policy formation. In addition to their significance in shaping conduct, their mere existence alters awareness, shapes attitudes and directs resources for the justification of decisions. These symbolic implications of indicators may be found entirely independent of, and even in conflict with, the empirical data which the indicators generate [8]. As an example, the use of gross domestic product (GDP) as a symbolic indicator of economic growth may be used to justify inaction or ‘business-as-usual’ policies that can directly block the achievement of ecologically-oriented goals [12].
The MDGs report calls for a “data revolution” through the use of the phrase “together we can measure what we treasure” ([5], p. 13). This recognises the intimate relationship between goals, the values from which they spring, the indicators by which they are recognised, and the tools selected for their assessment. While deeply evocative, the use of the first person plural in the statement ‘we can measure what we treasure’ implies singular identity and vision. This rhetorically compelling assertion erases the diversity of values and ideologies (at times incompatible) that characterise discussions bearing on sustainable development (SD) within institutions, whether global or local. Hopwood, for example, maps more than 30 different views on SD within a two-dimensional space—positioning them from ‘low environmental concern’ through ‘techno-centred’ to ‘eco-centred’ on the horizontal axis, and from low to high concern for socio-economic justice on the vertical axis [13]. Thus although the need is acknowledged for indicators to accommodate plural viewpoints, to measure what ‘matters’ and to involve all those willing to participate, there is no clear pathway yet known to do this. The UN’s approach may be the best under the circumstances, but it does not claim to be optimally designed.
In contemplating the many possible starting points towards designing effective working measures for the SDGs, the authors brought in a further dimension: the ‘missing’ fourth pillar. Without endorsing a particular definition of SD, we note assumptions and biases that the UN approach appears to perpetuate, which contribute to the marginalisation of those dimensions of sustainability that are not found within the established three dimensions of the economic, the environmental and the social. Any developments constrained to those three dimensions are insensitive, at minimum, to cultural/aesthetic dimensions, e.g., general discussions of cultural integrity and vitality [14,15,16] and specific discussions in relation to Indigenous communities [17]; the role of the arts in sustainability [18]; political/institutional dimensions, e.g., `good governance’ [19,20]; and religious/spiritual dimensions [21,22,23]. While in many ways these excluded dimensions are mutually distinct, they intersect in their shared interest in the category of values, in whatever manner these are understood. Although the domain of values certainly does not cover the full range of those excluded dimensions, it provides an entry point for bringing those dimensions into discussion with the three ‘pillars’ that appear to be more amenable to traditional forms of conception and measurement [24].
Values have historically been excluded from the realm of indicator development because they are perceived as intangible and difficult to measure [24,25,26]. However, it is possible to operationalise values when they are conceptualised intersubjectively [27]. That is, rather than viewing values as static latent traits possessed by individuals, they may become visible in groups when operationalised through systematic processes in clearly defined practical contexts, such as their normal working practice. An intersubjective approach to the design of indicators for values has recently been piloted in a variety of settings [24,27,28,29,30,31,32,33,34,35,36], including for formal evaluation. It typically involves an intersubjective process of values elicitation at the level of short statements about what is ‘valuable, worthwhile and meaningful’ to the group, e.g., “Mistakes are understood as opportunities to learn” coupled with gently disruptive probing to force conscious recognition of tacit knowledge, to denaturalise socially indicated responses, and finally to agree on local indicators. This inductive values- based approach to developing indicators differs from the conventional one used by the UN in that it starts with an open question (“What is ‘important’ to you about your group work?”); it proceeds intersubjectively; and it is purposely disruptive in ways that provoke discussion and contributions from all members until some resolution is reached. The indicators were generated by working groups (‘doers’) relating to their work/practice (‘doing’), rather than by individual leaders or national or regional representatives in an abstract and intellectual mode.
In this paper the authors present an exploratory study of the indicators produced via this values-based approach compared to the conventional approach used by the UN in order to test their influences to the operationalisation of complex SDG targets. To do this the values-based indicators developed by twenty-one organisations to evaluate their broadly-defined ‘ESD’ programs are compared to those obtained through the UN process for SDG Target 4.7 which is focused on ESD. The analysis involved first constructing a conceptual framework for Target 4.7 using the method of Hak et al. [37,38], and using it to systematically compare, via coding, the two sets of indicators for their usefulness for operationalising in terms of their potential relevance (item validity), and coverage (sampling validity).
The following sections discuss principles of indicator design, as background to the subsequent description of the conventional approach currently used within the UN SDG process and the values-based approach that we have studied.

1.1. Emerging Principles of Indicator Design

The development of indicators for the SDGs should be systematic and informed by theory [37,38] and should be undertaken in a way that goes beyond ‘what can be measured using currently available methods’ in order to measure ‘what should be measured’ ([39], p. 295, p. 296). The strong appeal by McCool and Stankey that we go beyond the convenient [39] constitutes recognition that the development of indicators is informed by both social and political negotiations [40]. Were indicator development purely technical, there would be no risk that the convenient could be substituted for the valid.
Hák et al. [37] advocate the creation of coherent conceptual frameworks for SDG targets, as a necessary condition for ensuring the validity of linkages between indicators and the facts they are to represent. If the conceptual framework behind SDG targets is not internally coherent, that incoherence will compromise their indicators (see also [40]). This echoes pre-SDG indicator development guidelines such as those included by Dalal-Clayton and Bass in their resource book Sustainable Development Strategies [41], advocating the creation of a ‘framework of parts and aims’ as the first essential step in designing an indicator-based assessment. In this context, ‘parts’ are understood as dimensions, elements or themes, and ‘aims’ as goals, objectives, principles or criteria. These authors suggest that using a framework of parts and aims as the foundation for indicator development has numerous advantages, e.g., avoidance of measuring the same part twice or omitting essential parts; highlighting parts for which no measurable indicator exists; and showing the relationships between different parts and appropriate weightings for each of them in the development of indices (see also [7,8]).
Dalal-Clayton and Bass [41] further recommend the development of a framework of indicators that is systemic (organised to illustrate key features of the system and subsystems), hierarchical in the sense that the parts are organised into a series of levels, logical in the sense of being a series of means and ends (e.g., ‘we measure A by measuring B, B by measuring C…’), and readily communicable to non-specialists. More recently, Lozano, Llobet and Tideswell [42] have emphasised the importance of examining the interlinkages between different indicators within the framework.
Assuming that a given indicator is empirically defensible, it must also be recognised as socially relevant (e.g., policy relevance, relevance to specific stakeholder groups, or applicability at the appropriate level) [37]. Optimising indicators for both the empirical and the social, in turn, requires clear conceptions of key terms such as ‘sustainable’, ‘efficient’ or ‘substantial’ [38]—terms whose ambiguity may have been useful and intentional when the targets were negotiated, but which then pose challenges for operationalisation. These issues can be avoided in narrowly defined SDG areas, but are problematic with complex areas which involve intermeshed concepts, such as ESD.

1.2. The Conventional UN Approach to Developing SDG Indicators

The task of deciding what should be measured to evaluate the SDGs at a global level was delegated to the ‘Inter-Agency and Expert Group on Sustainable Development Goal Indicators’ (IAEG-SDG) by the United Nations Statistical Commission in March 2015. This group consists of representatives from the national statistical offices of 28 ‘member’ countries (consisting of seven each from Africa and Asia, two each from Oceania and the Caribbean, one from North America, three from Central and South America, and five from Europe) [43], as well as observers from non-member countries, regional commissions, and regional and international agencies (e.g., non-governmental organisations) who can provide technical advice and support [44].
In its March 2016 report, demonstrating the extent to which indicator development is as much socio-political as it is empirical, the IAEG-SDG describes the initial step in the process of indicator development as involving an “open consultation… with all countries, regional and international agencies, civil society, academia and the private sector” from August to September 2015 ([44], p. 6). This was followed by an initial coding of all proposed indicators as either ‘green’ (general agreement) or ‘grey’ (more discussion required) by the panel. Two limited public consultation processes were then launched—the first, on the ‘green’ indicators, open for only three days in November 2015 [45], and the second, on the ‘grey’ indicators, for seven days in December 2015 [46]. While the consultations received little or no attention in the mainstream media and academic journals, their output represents a snapshot of the international discourse surrounding SDG indicators at the crucial launch time of the goals. They are noteworthy not only in the sense of who has contributed and what has been said, but also—as we discuss below—because of what and who has not been included.
It is apparent from the March 2016 report and related IAEG-SDG documents [44,45,46] that the IAEG-SDG approach to indicator design did not begin with the decomposition of the target into a framework of parts and aims, as advocated by Dalal-Clayton and Bass [41]. Rather, individuals and organisations were invited to comment on draft indicator proposals issued by the IAEG-SDG, and submit their recommendations for alternatives. This raises the possibility that proposed indicators might have been informed as much by participants’ cultural backgrounds, institutional affiliations and interests as by the functional need. Insofar as there is no opportunity for critical cross-examination or acknowledgement of the full range of concepts to be covered, the entire process invites contributors to formalise and uncritically advance their own perhaps entrenched ways of thinking about what constitutes ‘SD’.

Operationalizing ‘ESD’ through the Conventional Approach

The strategic importance of education as a main Sustainable Development Goal (Goal 4) has been widely acknowledged, and its two-way linkages with other goals outlined, in numerous United Nations flagship reports—to the extent that the goals can be implicitly organised as a conceptual map with education at the centre [47].
In this paper we focus specifically on operationalising Target 4.7, the target which relates most explicitly to education for sustainable development (ESD):
SDG Target 4.7:
By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development.
We have chosen to focus on this SDG target because its great breadth provides an excellent overlap of subject domain with that available from the values-based approach that we wish to consider. In examining Target 4.7 we do not adopt a position on what is meant by either ‘sustainable development’ or ‘education for sustainable development’ and we fully acknowledge that the term ‘ESD’ is highly controversial and that its usefulness has been contested, especially by environmental educators [48]. Nevertheless, we accept that the language of ESD has become so entrenched—not only in the SDGs themselves but also in a vast body of United Nations literature, particularly around the UNESCO 2004–2014 ‘Decade of Education for Sustainable Development’—that it would be unrealistic to imagine that current objections alone could secure its replacement with a less politically loaded term (such as ‘education for sustainability’). We expand further on these points, with relevant supporting literature, in Appendix A.
While some targets for other SDGs also have relevance to ESD, especially Target 12.8 (“ensure that people everywhere have the relevant information and awareness for sustainable development and lifestyles in harmony with nature”) and Target 13.3 (“improve education, awareness-raising and human and institutional capacity on climate change mitigation, adaptation, impact reduction and early warning”), these are very much narrower in focus than Target 4.7. The complexity of Target 4.7 not only provides better domain overlap for our comparison (see Section 1.3 below), but also epitomises both the challenges of, and the crucial need for, a systematic and theoretically-grounded approach to conceptualisation and operationalisation in the development of complex SDG indicators.
The initial attempt by the IAEG-SDG [44] to operationalise this target, using one indicator, took a very narrow interpretation that focused specifically on scientific knowledge:
Candidate Indicator (UN-IAEG) for Target 4.7:
Percentage of 15-year old students enrolled in secondary school demonstrating at least a fixed level of knowledge across a selection of topics in environmental science and geoscience. The exact choice/range of topic will depend on the survey or assessment in which the indicator is collected (Dis-aggregations: sex and location (and others where data are available).
This indicator is problematic not only because of the narrow scope, but also its promotion of a transmissive rather than a transformative view of ESD (c.f. [48]). The recognition by the IAEG members that this indicator does not fully reflect the concepts found in the target (i.e., lacks sampling validity) is evident in their decision to call for proposals for alternative indicators [46].
A total of 114 separate responses to the public consultation were received for Target 4.7, of which 83 came from civil society organisations (CSOs), mainly non-governmental organisations; 16 from United Nations agencies or national UN-related organisations; nine from national statistical offices; five from universities; and one from a working group specifically convened for the purpose of reflecting on SDG indicators (the ‘Adolescent Girl and SDG Indicators Working Group’, which was comprised of representatives from the UN Foundation and six CSOs). Several were replicated identically, but the 71 unique indicators form one set for our comparison and we label them the ‘IAEG-SDG’ responses set, produced through the conventional, UN, approach. Below we analyse their relevance and coverage for Target 4.7.

1.3. Values-Based Approaches to Indicator Design

The second approach and indicator set that we will analyse originated with the ESDinds project (ESDinds: Development of Values-Based Indicators and Assessment Tools for Civil Society Organisations Promoting Education for Sustainable Development), a project funded by the European Commission’s Seventh Framework Programme from 2008 to 2011 under the specialist funding scheme ‘Research for the Benefit of Specific Groups: Civil Society Organisations’ (BSG-CSO) [49]. The two-year project was constructed as a consortium of two academic partners (university research groups) and four civil society organisations promoting ESD in non-formal contexts:
(a)
the Alliance of Religions and Conservation, a secular organisation working with 11 major faiths;
(b)
the Earth Charter Initiative, a global network of organisations promoting the principles of the Earth Charter, with its secretariat in Costa Rica and strong representation from the global South;
(c)
EBBF (ethical business building the future), a global learning community initially registered as a non-governmental organisation in France;
(d)
People’s Theater (sic), a small German organisation.
The consortium evolved from earlier collaborations and partnerships, and consisted of organisations that viewed their ESD provision as broadly ‘values-based’—defining ‘values’ in the sense of “principles or standards of behaviour”, i.e., ethical, spiritual or moral values, as well as “[people’s] judgement of what is important in life” [50]. The overarching aim was to develop project-level indicators and tools that would not only capture the values and priorities of the participating civil society organisations (CSOs) in their work, but also help them to identify, evaluate and communicate less tangible ESD dimensions. While the ESDinds process did not set out to produce indicators that covered Target 4.7, we show that its execution within organisations with express interest in ESD was sufficient to do this (see Table 1).
The methodology of the ESDinds project is described fully in the final report [51] and by Podger et al. [52] and is usefully understood to be an example of ‘Research through Design’ [28]. In brief, the initial process of developing draft indicators used a multiple case study approach, incorporating diverse methods of data collection—key informant semi-structured interviews, informal interviews, participant observation, and document collection—to explore what group participants found valuable, meaningful and worthwhile in the context of their values-based ESD projects. Six organisations contributed in this way, all affiliated to the four partner CSOs. Thematic content analysis of these large data sets was based around a codebook derived from values and indicators literature, and involved tests for inter-rater reliability and member checking within all participating CSOs. During the analysis, the wording of each draft indicator was negotiated so that participating organisations found them to be comprehensible, measurable, relevant and locally valid, while attempting to achieve a level of abstraction that would make them generalisable to other contexts. This was in accordance with the ‘bottom-up and top-down’ approach presented as an example of good practice in indicator development by Reed, Fraser and colleagues [53,54,55,56,57]. The draft indicators from the different organisations were then clustered intersubjectively to produce a set of 177. This was later reduced to a final reference set of 125 applicable to multiple contexts, including but not limited to non-formal ESD [58], via field trials in 15 further organisations, and that set is centred on ESD practice. We thus analyse it for relevance and coverage of Target 4.7, denoting it the ESDinds set, and the approach as ‘values-based’. This process produced not rigid indicators, but ‘proto-indicators’—reflecting the concept of a `prototype’ in design literature, to refer to concise statements that can serve as templates or triggers for the local development of immediately relevant measurable indicators [33].
Proto-indicator sets have since been developed in a simplified manner for other contexts, including secondary schools [28,30,31,32], universities [29,59] and community-university research partnerships [34], All the sets have significant overlap, and their indicators reflect the values-based approach. However, none of those have yet achieved the same level of validation through field testing as the ESDinds set.

1.4. The Overlap of the Two Domains

The aim of this study is to compare indicators produced from the two approaches to explore their intrinsic differences with respect to useful operationalisation of Target 4.7. Ideally, the two would be developed in parallel, but without time and funds to do this we have made use of what is available, and noted areas where comparisons would not be appropriate or need extra caution. Table 1 summarises these, and is referred to in the subsequent design of analysis, presented in Section 2.2.2. The aim is to identify aspects of the indicator sets or the processes by which they are generated which suggest that further research would be useful.

2. Methods

In this section we briefly outline the approach used to develop a conceptual framework for Target 4.7 and then describe the analysis conducted to compare the IAEG-SDG and ESDinds indicators, from the conventional and values-based approaches respectively, in terms of their usefulness to the operationalisation of this target. Reference is made to their relevance and coverage of different components which would impact on the overall item validity and sampling validity of any final indicator(s) developed.

2.1. Creating a Conceptual Framework for Target 4.7

The process of creating a conceptual framework for Target 4.7 proceeded in three stages: (1) defining two sub-targets; (2) subdividing the skills sub-target, by identifying broad categories of sustainability competencies derived from academic and professional literature; and (3) identifying smaller clusters of skill competencies within those categories where applicable, again with direct reference to literature. A broad overview is given below and further details in Appendix B.

2.1.1. Top Level: Sub-Targets

Hak et al. ([37], p. 570) provide an example of the process of defining sub-targets by breaking down Target 12.3, “By 2030 achieve sustainable management and efficient use of natural resources”, into two sub-targets, “sustainable management of natural resources” and “efficient use of natural resources”. Each of these, in turn, can be subdivided, e.g., efficient use of raw materials (comprising fuels, minerals, metals and biomass), efficient use of food, efficient use of water, and efficient use of land. With each recursively created level, the sum of all the sub-targets constitutes the super-ordinate.
The application of the same approach to Target 4.7 is frustrated by the lack of natural points where it may be cleanly separated into mutually exclusive and collectively exhaustive sub-targets. In principle, the target can be split into two overall dimensions relating to learning outcomes—all learners acquire the knowledge needed to promote sustainable development, and all learners acquire the skills needed to promote sustainable development.
  • Sub-Target 1: All learners and educators involved in organised learning activities, both formal and non-formal, acquire knowledge needed to promote sustainable development.
  • Sub-Target 2: All learners and educators involved in organised learning activities, both formal and non-formal, acquire skills needed to promote sustainable development.
A case could also be made for including a third or alternative sub-target that can be operationalised as multiple process indicators, e.g., “All learners enrolled in formal and non-formal organised learning activities receive education for sustainable development and sustainable lifestyles” (or human rights, gender equality, peace and non-violence, etc.), on the grounds that in certain countries it may be logistically difficult or impossible to assess learning outcomes for ESD. Where this is the case, the only way of collecting meaningful data would be to focus on the provision of relevant learning activities. In this paper, however, we focus on operationalising the target as it is currently worded, with its focus on learning outcomes, only returning to this alternative sub-target in the discussion.

2.1.2. Second Level Categories: Competencies

We realised there was no need to conceptualise in great detail `knowledge that enables people to promote sustainable development’ since actions are local and contexts vary, resulting in the knowledge required to promote ‘SD’ (however it might be defined) being idiosyncratic to each locality. Attempting to specifically catalogue this knowledge for a global level of application would be counter-productive, hence the need for localisable assessments such as the ‘Sustainability Literacy Test’ (SuLiTest) developed by UNEP and other agencies. On the other hand, this same SuLiTest is helpful as an initial proxy measure for SD-related knowledge at a general level, and we advocate its use as a starting point even though it has significant limitations, focuses on a minimum level of basic knowledge, and requires further required to develop higher-level assessments [60].
In contrast, we are able to develop great detail in the conceptualisation of skills for Sub Target 2. Despite being largely overlooked by the IAEG-SDG in their indicator development process, the skills dimension features prominently in ESD in both professional and academic literature on ESD—usually framed in terms of ‘sustainability competencies’. We thus expanded our framework in the skills dimension through a sub-study of significant works relating to skills that enable people to promote sustainable development. Ultimately we produced a hybrid framework which covered the key aspects of the competencies discussed elsewhere. It was comprised of two accepted conceptual frameworks, one from professional literature [61,62] and the other from academic literature [63]. relating to Both are framed in terms of ‘sustainability competencies’, also referred to as ‘ESD competencies’ (although we note that the term ‘competency’ as defined by Rychen and Salganik ([64], p. 51) includes not only skills but also other cognitive and non-cognitive components, such as attitudes). Appendix A sets out the justification for, and caveats associated with, the use of the competencies lens (and of these specific references) within a broader literature review on ESD, while Appendix B provides further detail on the actual construction of our conceptual framework. The key concepts of the two frameworks used are summarised below:
UNESCO/UNECE model: A seminal report to the United Nations Educational, Scientific and Cultural Organisation (UNESCO) by the International Commission on Education for the Twenty-First Century, chaired by the former European Commission President Jacques Delors, distinguishes four `pillars’ for education in the twenty-first century ([61,62,63,64,65,66,67]). To these four pillars, which have been applied specifically to ESD as the foundation of the United Nations Economic Commission for Europe (UNECE) report Learning for the Future: Competencies in Education for Sustainable Development [61], a fifth was later added by UNESCO (see Appendix A), resulting in the following set of competencies [62]:
  • learning to know (e.g., learning to learn, developing critical thinking, acquiring tools for understanding the world, and understanding sustainability concepts and issues);
  • learning to do (e.g., acquiring technical and professional training and applying learned knowledge in daily life);
  • learning to be (e.g., seeing oneself as the main actor in defining positive future outcomes, developing self-identity and self-knowledge, acting with autonomy and personal responsibility);
  • learning to live together (e.g., understanding other people’s values and traditions, cooperating with people, celebrating diversity and coping with conflict);
  • learning to transform oneself and society (e.g., building non-discriminatory societies, reducing ecological footprint, integrating sustainable lifestyles and promoting social solidarity) [62].
Wiek model: Through a systematic literature review, Wiek et al. [63] identify five broad categories of key competencies for sustainability:
  • systems thinking competence, the ability to analyse the dynamics of complex social-ecological systems;
  • anticipatory competence, i.e., the ability to create, analyse and evaluate what they term ‘rich pictures’ of the future;
  • normative competence, also referred to as value(s)-focused thinking, which focuses on “the ability to collectively map, specify, apply, reconcile, and negotiate sustainability values, principles, goals, and targets” ([63], p. 10);
  • strategic competence, which ensures that learning is translated into effective policies, programs and action plans; and
  • interpersonal competence, which refers to the ability to motivate, enable and facilitate participatory sustainability research and collaborative problem-solving, to celebrate diversity, and critically evaluate different positions and perspectives. This is regarded as a cross-cutting issue, as the involvement of multiple actors in sustainability problems makes it essential for collective strategies to be employed in trying to solve them.
While the literature on competencies in ESD reveals a complex, multifaceted and rather low-consensus picture comprising both affective and cognitive elements (see Appendix A), we find that most of this complexity is covered within the two models described by UNESCO/UNECE and Wiek et al., respectively. For the practical purpose of this paper, which requires a framework adequate to roughly compare potential relevance (item validity) and coverage (sampling validity) of indicators, we have combined the main concepts of these two models into a single eight-category framework of ESD competencies, described in detail in Appendix B and illustrated in Figure 1, (1)–(8) below.

2.1.3. Third Level: Specific Clusters of Competencies

Of the eight broad competency categories, five contained further distinctive sub-dimensions, as shown in the bottom half of Figure 1, closely following the text of a UNECE infographic [68]. In order to balance rigour (in the sense of including a sufficient number of codes to provide a valid and reliable recognition of all analytically relevant data) with usability, we worked at the level of clusters. Our conceptual framework for Target 4.7 thus consisted of a Sub-Target for Knowledge with only one level of division of concepts, and a Sub-Target for Skills with three levels of concepts of competencies (which are shown in Figure 1).

2.2. Exploring the Usefulness of the Operationalisation of Target 4.7 through Two Contrasting Approaches

In considering how to compare the outputs from the two approaches for generating indicators, and to identify important aspects which can feed into future research design, attention must be paid to the incomplete overlap of approach domains. Had the aim of this work been to evaluate two approaches across equivalent domains, we could have done tests of relevance (item validity) and coverage (sampling validity) of the two sets of indicators using statistics. However, the authors realised it was more useful to obtain a detailed and rich overview of interesting aspects of not only the indicators but also the processes which produced them. For example, although we have described the non-conventional approach taken by ESDinds as ‘values-based’, it also has clear aspects of intersubjectivity, holism and local-level operationalisation, which might prove more important than the values lens itself. Similarly the IAEG-SDG process could be described as involving individual representatives of organisations, rather than working groups of members; direct copying of indicator proposals by other respondents; and a stronger motivation for the participating organisations to push political agendas, such as sexuality or HIV/AIDS education, rather than focusing on the specific Target 4.7.
Another consideration was whether to focus on ‘ideal single indicators’ which could potentially cover the entire remit of Target 4.7, or the combined contribution of the output set of indicators from each approach in terms of its ‘helpfulness towards operationalisation’. Since the two approaches favoured these differently, both were taken forward so as not to bias either: both provide pathways to producing good coverage.
There were 110 responding organisations in the IAEG-SDG process, providing a total of 114 indicator proposals, but a number of these proposals were repeated multiple times. Our approach required considering only the 71 unique indicators (retaining those with small variations in wording as separate items). The ESDinds set contained 125 proto-indicators which had already been clustered from others, and thus were already unique. Due to the large number of indicators in each set (71 and 125 respectively), and the complexity of the conceptual framework for Target 4.7 (see Figure 1), we developed a coding and scoring system to evaluate relevance and coverage for both individual indicators and sets of indicators. We outline this below, and then use the summary Table 2 to illustrate where less direct considerations must be used, where a lack of domain overlap occurs.

2.2.1. Codebook Development and Scoring Methodology of Content Analysis

In order to facilitate the content analysis of both datasets, we developed a codebook from the conceptual framework outlined above. The scoring systems are given in Table 2 and Table 3. By comparing the content of each indicator proposal (IAEG-SDG) or proto-indicator (ESDinds) text with the text in the codebook, an initial subjective judgement of fit was made by the first author, and the coded datasets were subsequently checked by the other authors.
We used the codebook to analyse the relevance of individual indicators for Sub-Target 1 (Knowledge) and Sub-Target 2 (Skills) respectively, and then to explore the coverage of sub-targets, both at the level of individual indicators and across each dataset as a whole:
  • In evaluating relevance for Sub-Target 1 (Knowledge) at the level of individual indicators, we scored an indicator as 1 (‘relevant’) if it explicitly measured the acquisition of knowledge in relation to either (a) at least one of the eight content areas mentioned in the target; or (b) the Sustainability Literacy Test. For this purpose, the terms ‘understanding’ and ‘proficiency’ were taken as synonymous with ‘knowledge’. Indicators that measured knowledge acquisition only in the limited sense of environmental science and/or geoscience (as included in the original indicator proposal, which the IAEG-SDG had already acknowledged as inadequate for operationalising the target) and those that did not measure knowledge acquisition at all were scored as 0 (‘not relevant’) for Sub-Target 1. Table 2 illustrates and summarises the scoring methods.
  • To evaluate relevance for Sub-Target 2 (Skills) at the level of individual indicators, ‘proficiency’ was not taken as synonymous with ‘skills’, but the word ‘skills’ was not essential in itself for scoring as 1. Rather, the coder made a judgement, on the basis of the whole indicator text, as to whether the indicator would measure the acquisition of skills in one or more competency domains. Indicators measuring related aspects—such as the creation of appropriate environments for skill acquisition, the existence of policies or structures intended to support ESD skills development, or the percentage of schools providing a certain type of education—were not scored as 1 unless there was clear evidence that the indicator requires the measurement of skill-related learning outcomes for specific individuals or groups in those settings.
  • In our evaluation of coverage of Sub-Targets at the level of individual indicators, we highlighted the specific content areas (for Sub-Target 1: Knowledge) or competency domains and sub-domains (for Sub-Target 2: Skills) that were hit by each indicator, scoring 1 for ‘yes’ and 0 for ‘no’ in each case. (Although there is no particular reason or advantage for a given indicator to score against more than one category, those scoring highly might be considered more potentially useful as a main indicator for Target 4.7, whereas others might have low coverage but perhaps provide innovation in how to deal with a specific and potentially challenging aspect of ESD.)
  • In assessing coverage of Sub-Targets the level of the whole dataset, we assigned an overall category score of 1 point per content area (for Sub-Target 1: Knowledge) or competency domain (for Sub-Target 2: Skills) that we judged to be partially covered, and 2 points per content area or competency domain that we judged to be systematically or rigorously covered. The definitions of partial coverage and systematic/rigorous coverage are provided in Table 2 and Table 3.
One complication occurred in the above process: it was found that one of the Skills sub-domains—namely L2K_CON (‘Understand sustainability concepts and issues’)—was so close to a description of Knowledge that was not possible to clearly allocate indicators to it. A distinction was implied between knowledge and understanding, but in practice the indicators did not provide such distinctions: for example, it seemed to attract most of the same qualifiers as for Knowledge. We thus excluded this sub-domain completely from analysis.

2.2.2. Consideration of Domain Overlaps and Non-Overlaps

A summary of domain differences between the IAEG-SDG and ESDinds indicator development approaches was provided in Table 1. The impact of those differences on the analysis for different components of the conceptual framework is illustrated by the shading shown in Table 2 and Table 3 above. Indicators arising from ESDinds were formed through the aggregation of more specific indicators, because part of the process of forming a transferable set across multiple organisations involved this. One consequence is that any natural mention of a specific knowledge area would have been subsumed into any parallel indicator relating to the practice of that knowledge—i.e., a related competency. Thus, with the coding rules as defined above, the ESDinds dataset is not expected to score in Sub-Target 1 (Knowledge) at all, as an artefact of its construction, meaning that the entire Sub-Target cannot be used for comparisons.
Similarly, the ESDinds approach was focussed on the present and near-future for the groups involved, and not the far future, and thus anticipatory competencies are not expected to be covered in its output indicators.
Both of these domains could easily be incorporated into future work using an ESDinds-type approach, with very little extra effort. Thus, they cannot be properly considered to be zero scores. However, in this retrospective study and with the specified scoring system, only zero scores are expected for Knowledge. The IAEG-SDG indicators were still processed, to allow consideration of its overall relevance of the approach to both Sub-Targets (Knowledge and Skills).

3. Results

In the sections that follow, we present overview summaries of the study findings; compare the item validity and sampling validity of the outputs of the two approaches where their domains overlap (i.e., Skills, other than Anticipatory Competencies); and compare sampling validity across each dataset as a whole at the third level of the conceptual framework, i.e., coverage of competency sub-domains. Specific examples from the coded text of indicators from both conventional and values-based approaches can be found in Appendix C, which readers would benefit from browsing. Full tables of coverage analysis results are presented in Appendix D, respectively illustrating the coverage of Sub-Target 1 (Knowledge) by the IAEG-SDG consultation responses (Figure D1); the coverage of Sub-Target 2 (Skills) by the IAEG-SDG consultation responses (Figure D2); and the coverage of Sub-Target 2 (Skills) by the ESDinds proto-indicators (Figure D3). The full coded datasets for IEAG-SDG and ESDinds are provided as Tables S1 and S2, respectively, in the Supplementary Material.

3.1. Overview of Relevance (Item Validity)

Table 4 provides an overview of the number of useful indicators for the operationalisation of Target 4.7, as produced by the two approaches. It is striking that a significant number of them were not useful for either the Knowledge or the Skills sub-targets. Given the specific request of the IAEG for responses to focus on Target 4.7, greater relevance might be expected. The second most striking point is that the UN-IAEG approach produced mostly knowledge-based indicators—but many of the respondents had expertise in ESD, and should have been aware of the skills aspects, so it is perhaps surprising that those were underrepresented. Thirdly, the ESDinds approach produced a large number of indicators generally (in the Skills category only—as predicted in Section 2.2.2). Given the original, local question they were designed to address, it is surprising that so many are relevant.

3.2. Overview of Coverage (Sampling Validity)

For Sub-Target 1 (Knowledge), the majority of relevant responses from the IAEG-SDG process covered only one or two content areas, with the most common ones being ‘gender equality’ and ‘appreciation of cultural diversity’ (see Figure D1 in Appendix D). It was notable that most of the candidate indicators from the IAEG-SDG process were concerned only with the acquisition of knowledge about sexual and reproductive health. These focused on measuring the outcomes of comprehensive sexuality education through two standard responses, proposed by UN agencies and repeated multiple times by other contributors. Two responses focused only on human rights. Some other responses mentioned knowledge in geoscience and environmental science—content areas mentioned in the target—but lacked specificity about what to assess. An exceptional contribution was provided by Brookings USA, which included detailed proposals relating to many knowledge domains (and skills, as discussed below). Only one response, from the United Nations Environment Programme (UNEP), called for universal application of the more holistic Sustainability Literacy Test [60] across the tertiary education sector. As anticipated, none of the ESDinds values-based proto-indicators mentioned knowledge acquisition at all. This was an expected artefact of the methods used to aggregate them into a set—see Section 2.2.2.
For Sub-Target 2 (Skills), there were far fewer relevant indicators from the IAEG-SDG process, and they generally had poor coverage of Target 4.7 (see Figure D2 in Appendix D). A number of UN consultation responses referred to the concept of “life skills based HIV [human immunodeficiency virus] and comprehensive sexuality education”, and some made a brief mention of, for example, vocational skills. Only a minority (n = 22, or 31%) included proposals for indicators that would actually measure the acquisition of skills relevant to content areas outlined in the target. In particular, it is striking that the coverage and number of indicators for the Sub-Target: Skills cluster of “Learning to Know” is extremely low (see Figure D2 in Appendix D), in contrast to the high coverage found in the Sub-Target: Knowledge (see Figure D1). This suggests a narrow view of the nature of knowledge, dominated by content rather than ways of knowing such as critical thinking or learning to be a self-driven learner and thinker. We explore this more fully in the Discussion section below. It is also interesting to note the unusually high number of hits for the sub-domain L2L_DIV within the competency domain of ‘learning to live together’, i.e., “Tolerate, respect, welcome, embrace, and even celebrate difference and diversity in people.
The results for the same Sub-Target 2 (Skills) from the ESDinds process had a very different profile (see Figure D3 in Appendix D). As expected, most ‘hit’ only one competency sub-category, as they were designed to be specific. However, as a set they produce very good coverage of Target 4.7, and in particular in the area that the IAEG-SDG process was weak i.e., in “Learning to Know”. Interestingly, like the other approach there is a large number of diversity-related indicators, but unlike it, there are also a large number relating to L2K_ACT (converting knowledge into action) and L2L_PAR (participation). It is also interesting that both approaches did not produce any indicators relating to the Anticipatory competencies category.
The overall coverage score for the full IAEG-SDG dataset was 7 out of a possible 16 points, representing partial coverage of all competency domains other than Anticipatory Competencies. For ESDinds, the overall coverage score was 9, representing systematic/rigorous coverage of Learning to Be and Learning to Transform Oneself and Society and partial coverage of all the other competency domains, with the exception of Anticipatory Competencies. The breakdown of results by sub-domain is shown in Table 5.

3.3. Sensitivity Analysis

Before proceeding to deeper discussion, we perform an analysis on the data to ensure that it is not inappropriately sensitive to small variations in the process by which each was created. To this end we removed the most significant contributor to each set of indicators. For the IAEG-SDG data it was noted that one particular respondent provided particularly useful and considered indicators—Brookings USA. These indicators achieved high coverage and relevance, and in that respect were anomalous to the other contributions. For a sensitivity test this single, most significant respondent’s contributions were removed, and the coverage obtained by the IAEG-SDG approach was re-calculated, as in Table 6 below. Similarly, the highest contributing indicator in the ESDinds set was removed. Table 6 shows that the IAEG-SDG score reduced from 7 to 5, where the highest possible score was 16. The ESDinds set performed better, reducing only from 9 to 8.

4. Discussion

4.1. Comparisons across Approaches

Figure 2 facilitates a broader comparison of the contribution of the two approaches to the coverage of Target 4.7 in the domain where they overlap—Skills. This in turn raises interesting questions about the intrinsic nature of the approaches and their potential use in future sustainability processes. Of particular interest is the low number of IAEG-SDG indicators in Learning to Know; the high absolute number of ESDinds indicators generally and in particular in Learning to Do and Learning to Be; and the fact that the highest relative number for both approaches is in Learning to Live Together.

4.2. Inside and Outside the (Traditional) ‘Box’

The questions posed by the UN’s SDG consultation and the ESDinds approach covered similar domains, but the UN consultation started with one proposed indicator and asked respondents to suggest alternatives—with the implication that any operationalising indicators returned would be appropriate to that given target, and that the responses would be in the context of traditional indicator types. The ESDinds project, despite having the same aim of generating indicators, began with the question of what practitioners find `meaningful, worthwhile and valuable’ in relation to their ESD work. This was usually done with a focus on successful projects that they had completed in the past, and the values underlying their success. Asking specifically for new candidate indicators may predispose respondents to thinking about indicators and data collection methods that are already familiar to them; but asking specifically about what is meaningful, worthwhile and valuable to people may help them to think outside the traditional indicator ‘box’. In this latter case, they may find it easier to identify less tangible dimensions of ESD, which usually fall outside the realm of evaluation and assessment [24]. The approach brought about useful results which could help to achieve the shift, noted by McCool and Stankey [39], from ‘what can be measured, using currently available methods’ to ‘what should be measured’. Figure 2 shows that ESDinds returned many indicators in Learning to Do, Learning to Be and Learning to Transform, which are not traditionally well covered by indicators.
As a way of portraying this, a phrase cloud is given below (Figure 3), which gives more of a feel for the nature of the indicators than the histogram in Figure 2. The smallest font indicates one indicator and the largest font represents 17 relevant indicators—’actor and thinker’ from ‘Learning_to_Do’ as generated by the ESDinds process. Less tangible areas such as ‘participate and cooperate’ and ‘self-actualise’ show up clearly.

4.3. Tacit Knowledge and the Values-Action Gap

The ESDinds indicators proved to cover many of the less tangible dimensions of ESD. One useful way of thinking about this result is that values elicitation, especially when based around reflection on collective projects that have been successfully completed, may provide access to tacit knowledge that has not previously been expressed in words. Tacit knowledge accumulates through shared contexts of practical experience [69,70] but may never be formally articulated unless—as in the ESDinds project and more recent adaptations of its methodology—it is actively elicited. The accumulation of tacit knowledge provides one explanation for a value-discourse gap—a situation in which people are already enacting a particular value in their day-to-day actions, but do not bring it into conversation [28]. A ‘value-discourse’ gap is the converse of the previously recognised ‘value-action gap’ [71], also known as an ‘environmental values/behaviour gap’ [72] or ‘attitude-action gap’ [73], in which people espouse a particular value in their discourse but fail to carry it forward into action [28]; see also [74,75].
The ESDinds approach has shown it can produce a large number of indicators covering areas which are generally considered to be much less tangible (see Appendix C for examples), and may make accessible tacit knowledge about the types of skills that are required for sustainable development. Such knowledge might not be foregrounded in traditional indicator development consultations, because of the existence of a value-discourse gap in the respondent organisations: aspects that are important in practice may not always be articulated in words.

4.4. Current SDG Discourse

The concept of value-discourse gaps takes on a greater significance if we regard the IAEG-SDG public consultation as a snapshot of contemporary global political discourse around the SDGs, and the Target 4.7 responses as an indication of how ESD is being discussed within the ‘SDGs community’. Remembering that the IAEG-SDG responses included 22 with relevance to Sub-Target 1 (Knowledge), it is noteworthy that only one was found to be relevant to the broader ‘ways of knowing’- such as critical thinking, or applying learning to life—described in the Skills, Learning to Know competencies. It seems that even though the target is about ESD, the SDG community is still focussing on the more familiar domains of the cognitive ‘learning to know’, e.g., in the sense of receiving knowledge, rather than challenging received knowledges or producing new systemic knowledges. Such traditional knowledge is characterised by an outward focus and readily measurable outcomes, e.g., those tested through examinations. The more affective and inward-facing dimensions of ‘learning to know’—as well as many other sustainability competencies like ‘learning to live together’, ‘learning to be’ and ‘learning to transform self and society’ and the ‘normative’ competencies that allow people to explore questions of values and ethics, and to critique prevalent definitions of SD itself—are frequently overlooked [28] (see also [21,22,25]). This is evident in the responses obtained through the IAEG-SDG approach, itself representing current discourse. This should not be taken to mean that such competencies are not valued, but rather that people’s individual tendencies to value them do not yet appear to have been translated into targeted conversations around how to facilitate their acquisition through education, and thus certainly not how to measure the success or failure of such efforts.
The ESDinds approach was designed to address this issue, and the indicators produced suggest its success. A case in point is L2B_SEL, “Develop one’s personality, self-identity, self-knowledge and self-fulfilment”, which had seven indicators proposed from ESDinds in comparison to the two from the IAEG-SDG approach. Thus, values-based approaches to indicator development may both support and promote an expansion of the global discourse on ESD to include these ‘inner’, ‘heart-centred’, or even `spiritually-oriented’ aspects that have eluded traditional approaches to indicator development. Such an expansion has been previously advocated in principle, although not demonstrated in terms of indicator development, by authors in both education [76,77,78] and sustainable design [79,80,81,82].
The absence of indicators relating to ‘learning to know’ and systems thinking in the IAEG-SDG public consultation is a matter of concern for two reasons: first, because the consultation can be seen as a reflection or indication of the current state of discourse, and second, because of its implications for the future. In terms of the state of discourse, it infers that meta-learning—i.e., having an awareness of the phenomenon of learning in general and of one’s own learning habits in particular, and taking control of one’s learning [83,84]—appears not to be a salient issue for the respondents. In a similar way, it suggests that systems thinking may not yet have been embedded in the way in which these respondents approach complex sustainability problems. In terms of implications for the future, the likelihood is that if an issue is not covered by the consultation responses, it is unlikely to be represented in the final indicator. Bearing in mind the old adage that “what gets measured gets done” [6], this would constitute a missed opportunity to embed a more radical shift in modes of education—from a transmissive to a transformative paradigm [48,78]—and to push schools, colleges and universities to produce graduates who are not only critical and creative `systems thinkers’, but also effective lifelong learners. This, in turn, has serious implications for the future of sustainable development, in that knowing about SD in a theoretical sense is very different from having the skills and motivation to take action, which in turn is not the same thing as actually taking action.

4.5. The Need for Localisable Indicators in ESD

Whichever indicator(s) the IAEG-SDG might ultimately select to operationalise Target 4.7, there will be a subsequent question of data collection that needs to be considered carefully. At one extreme, data for most indicators could be collected through a national-level survey, or even a simple ‘tick-box’ exercise within the respective government departments. This could easily become the default option, especially where resources are limited. At the other extreme, local institutions (such as teachers’ colleges, universities and schools) could take the indicator as a starting point for a variety of initiatives designed to report on the achievement of the SDG target within their day-to-day practice. These could include, for example, creating new assessment criteria and rubrics for assessing learning outcomes at the individual or group level [29]; auditing existing curricula, internal policies, organisational structures and codes of conduct in relation to the indicator; and/or evaluating the extent to which these internal policies are implemented, or guidelines adhered to, in practice.
The authors would argue that while it is important for the final indicator to be standardised to the point of comparability across institutions and countries, it must also allow for localisation in its application if it is to be perceived as relevant (a condition for engagement) and produce locally valid measures. In addition, local institutions should be strongly encouraged to develop their own initiatives to set up localised operationalisations, especially if the final indicator does not fully operationalise the skills dimension—as seems likely, on the basis of the responses. Without targeted action, there is a high risk that global conversations about ESD will end up focusing primarily on the transmissive model of education—focusing on acquiring particular kinds of knowledge [48]—and that the opportunity to embed transformative, competency-based approaches could be missed.
It may be helpful, in light of this discussion, to revisit Figure 1—the conceptual framework created in this paper. We propose this conceptual framework as a tool in its own right for use by local institutions (e.g., universities, schools, and civil society organisations) in their efforts to develop locally-valid ESD indicators, and apply them to the evaluation of organisational structures, policies, curricula, teaching practices and learning outcomes. The authors are currently developing new work in this area.

4.6. Salience and Emerging Breadth of Concepts

The existence of diverse candidate indicators or proto-indicators for a particular category, within a given context, may suggest that the category has been widely thought about and discussed within that group of people. In the ESDinds case, for example, we have noted that the category “L2D_ACT: Be an actor as well as a thinker” was used to code 17 different proto-indicators, in comparison to only three in the SDGs set. It is unsurprising but encouraging that this category is highly salient to the respondents, who are practitioners promoting ESD within the non-formal education sector—specifically in civil society organisations—within project contexts.
Other possible explanations for multiple indicators in a given category are that the category itself is difficult to differentiate from other categories within the framework; that it is too broad to be useful; or that the indicator development process is biased in favour of that category. In the case of L2D_ACT, for example, there is a significant overlap with two other categories whose wording inherently implies a shift from the realm of thought to the realm of action, namely “L2L_PAR: Participate and co-operate with others in increasingly pluralistic and multi-cultural societies” and “L2T_FOO: Promote behaviours and practices that minimise our ecological impact on the world around us”. However, even among those uniquely coded as L2D_ACT within ESDinds, there are several different manifestations. These include, for example, behaving in a way that is consistent with one’s words; actively helping the organisation to meet its goals; investing one’s own time and resources in activities that benefit the environment or society; and being motivated and productive in one’s work. This may suggest that there is a need for further conceptualisation of what is meant by “being an actor as well as a thinker”, which can most usefully be done at the local level. Such an observation reflects the point that the ESDinds items are proto-indicators: they serve as triggers or prompts for localised processes of reflection, conceptualisation and operationalisation, leading ultimately to indicators that are both salient and measurable at the local level.
The category with the highest overall number of indicators within the IAEG-SDG consultation dataset was “L2L_DIV: Tolerate, respect, welcome, embrace, and even celebrate difference and diversity in people”. In this case, the multiplicity of indicators is attributable largely to the wording of Target 4.7 itself, which explicitly references this competency: “Ensure that all learners acquire the knowledge and skills needed to promote SD, including, among others…appreciation of cultural diversity”. Thus, any indicators which replicated the wording of the target automatically scored a point for L2L_DIV. However, the same category was also associated with multiple indicators for ESDinds, where a number of different practical manifestations of welcoming, embracing and celebrating diversity were identified—e.g., accommodating different learning styles, giving people flexibility to do things within agreed boundaries, or providing different groups with equal access to information and decision-making. This highlights the possible need for further conceptualisation of L2L_DIV with reference to the literature on, and practice of, equality and diversity in both education and employment settings. As before, this may be most appropriately done at a local level.

4.7. Whose Targets? Whose Indicators?

Regardless of whether explicitly values-based approaches are used, our findings also open up a broader debate about indicator development in relation to the SDGs. Revisiting the United Nations call to “measure what we treasure” [5] in the light of this study, it becomes evident that what is not discussed is the meaning of ‘we’ in this statement. It appears to imply that a single set of values are being used as the benchmark for determining what is ‘treasured’, thereby setting a normative standard for indicator design—which seems to have gone unchallenged in this instance, but has previously been the subject of critique [39,85]. A more generous interpretation might suggest multiple definitions of ‘we’, and hence a nuanced and multi-subjective/intersubjective understanding of values and indicators—echoing the position of Ostrom [86] on the need for polycentric approaches, as well as Saith’s critique of the Millennium Development Goals (MDGs) [4]. However, neither the MDGs Report nor any subsequent document relating to measurement and assessment for the SDGs incorporates an explicit call for polycentric, decentralised approaches to SDG indicator development. Rather, as for the MDGs, the official process has been oriented towards identifying a single global indicator (or, if absolutely necessary, two or three indicators) for each of the 169 SDG targets.
Bell and Morse [12] highlight the ‘tyranny of methodology’ which is inherent in any attempt to reduce a multiplicity of stakeholder perspectives to a very limited set of global indicators. Whatever is ultimately produced by the IAEG-SDG as the indicator (or set of indicators) for Target 4.7 at a global level, we speculate that it is unlikely to satisfy any of its observers. Furthermore, we have illustrated in this paper that it is impossible for a single indicator or even a handful of indicators to provide valid measures of progress towards such a broad, multi-faceted, and intermeshed target, in the sense of covering the whole of the content domain [87,88].

4.8. Democratisation and Pluralism in Indicators

In highlighting the challenges of operationalising complex concepts such as ‘education for sustainable development’, we neither question the significance of the IAEG-SDG indicators and indicator design processes, nor the importance of global and national monitoring in a more general sense. Our intent is, rather, to suggest the relevance of supporting democratisation of the indicator development process. This would mean acknowledgement that designing indicators of progress towards complex goals like the SDGs is best served by a complementary processes involving both expert groups and international institutions and by local values and priorities at a variety of different levels.
The inductive development of indicators from what groups identify as immediately valuable, worthwhile and meaningful in their diverse contexts would have important implications for engagement in and the democratisation and diversification of global sustainability and development agendas as a whole. The inclusion of such indicator development processes in international conversations could draw increasing attention to diverse ways of seeing and understanding the world—leading, potentially, to the wider recognition of plural ‘dynamic sustainabilities’ in place of the IAEG’s single, static conception of sustainable development, implicit in the use of the definite article to refer to ‘the’ knowledge and ‘the’ skills required to promote SD. This is already an acknowledged issue for Indigenous communities, in the light of their long history of disenfranchisement and ongoing collective actions for international recognition. Indeed, the importance of creating SD indicators grounded in Indigenous values and world-views was formally noted in a 2006 technical report of the Permanent Forum on Indigenous Issues [17]. However, a trend towards democratisation and diversification could also open up the possibility of creating a place at the SDG table for other sustainability discourses that do not sit comfortably with the implicitly neoliberal concept of sustainable development [89]. These include, for example, the interface of the arts and sustainability [18]; work on ‘design for sustainability’ positioned in opposition to ‘design for sustainable development’, which demands a fundamental shift in values rather than a mere `technical fix’ of contemporary industrial societies [80,81,82,90]; Stephen Sterling’s positioning of ‘sustainable education’ in opposition to ‘ESD’ [77], grounded in similar concepts and arguments to those of ‘design for sustainability’; and the ecological philosophy of David Abrams, which posits that members of the ‘more-than-human’ community of life should be treated as perceiving subjects rather than as objects for human consumption [91].
Rather than advocating for the immediate dissemination and use of the specific proto-indicators created through the ESDinds project, our concluding action is to highlight the pragmatic usefulness of inductive values-based approaches in operationalising ESD targets at the project level, and—in light of this—to call for increased and wider engagement in national, regional and local initiatives generally, in order to complement the official IAEG-SDG process. In view of recognised difficulties in defining terms such as ‘global citizenship’ in a global way (and assuming that the same competencies will be equally important everywhere), we propose that the key point is perhaps the process of creating values-based indicators, as part of a global trend towards the democratisation of the SDGs.

5. Conclusions and Recommendations

A conceptual framework was constructed for SDG Target 4.7 and used to analyse the usefulness and character of indicators produced from a values-based approach compared to the UN approach. The findings suggest that the current SDG community, and within it those focusing on ESD, still conceive of or model knowledge in a narrow, traditional way relating to content—to the extent of not being able to suggest relevant indicators for different ways of knowing, such as critical thinking or `learning to learn’. Similarly, indicators for many of the less tangible areas of ESD which are not already easily covered by traditional indicator approaches were not produced by the ESD community involved in the UN’s IAEG-SDG process. Although this is likely to be a reflection of current discourse in that community, it is also likely to be a reflection of the IAEG’s consultation process, which appears to have difficulty generating non-traditional indicators. However, the consultation process did produce candidate indicators which partially covered most of Target 4.7.
The ESDinds approach produced complementary and occasionally better coverage of Target 4.7. It was specifically designed to produce indicators covering the less tangible areas of ESD, and the work presented here provides evidence that it did so, as well as opening up a discourse which included finely-developed concepts in several of the less tangible areas. Although the underlying premise in the ESDinds approach is to drive the process by considering what is locally valuable, worthwhile and meaningful, it is not clear to what extent to which its other aspects—e.g., its intersubjective and slightly disruptive elements, its purposeful contextualisation at group level, and its consideration of values in a holistic manner—contribute to those results. However, it seems likely that all three of these elements do contribute, because the resulting indicators produced included several representing group values in action, aspects not usually articulated, and aspects often considered intangible—which were not produced in the IAEG-SDG process.
Although it may initially seem unlikely that the ESDinds approach could contribute in practical ways to an indicator development program for complex targets, such as that of the IAEG-SDG, there are in fact some ways in which it could. Firstly, if the ESDinds-type approach were run by a small number of groups prior to wider consultation, it could produce candidate indicators (or even Targets) of those rare categories, which could then be used as prompts—both for the expert groups and wider public. Secondly, the IAEG-SDG process might be enhanced if contributors were advised to develop their candidate indicators through ‘disruptive’ or at least intersubjective, consultations at group level, rather than relying on single representatives of organisations. Lastly, it appears that even the ESD community which is cognizant of less traditional output types such as competencies (compared to narrow knowledge) did not propose related indicators, suggesting a hegemony of sorts which hinders untraditional measures or evidence. To get around this, the IAEG-SDG could encourage novel or non-standard ideas like those reflecting ‘heart’ or ‘culture’, or preferably produce examples of them for consultation. It certainly would need to try to include them when deciding on Targets and indicators, as they may be rare but they are valid—as made evident in our conceptual framework of Target 4.7.
Finally, the ESDinds type of approach of developing indicators from the shared values of a group, but articulating them in individualised action-related and context-related examples, may serve to relieve the tension between different indicator types. The ESDinds indicators are slightly generalised versions of specific examples that are important at the group level, and may be transferable to other groups, who can then localise them in their own context. The recommendation from this for future indicator development is to build in a ‘fuzzy framework’ of indicators, which allow localisation. For example, the indicator “People are not afraid to make mistakes” can be scored using evidence deemed locally to be relevant. Furthermore, whatever final indicator is constructed by the IAEG-SDG for Target 4.7, the authors would suggest that asking groups around the world to immediately consult and develop their own local interpretations and scoring systems as this would not only lead to more valid and useful indicators, but would also build awareness and capacity in ESD that produced a wealth of ideas to share with less creative groups. This could become an important part of the process of increasing ESD itself.
Lastly, the work presented here makes it clear that much more research is needed to explore the ESDinds process to better understand how it produces measures of that long-sought ‘intangible’: sustainability. Such work is likely to reveal new knowledge not only for sustainability, but also for other linked disciplines.

Supplementary Materials

The following are available online at www.mdpi.com/2071-1050/8/9/861/s1, Table S1: Full coding of responses to IAEG-SDG public consultation on indicators for Target 4.7 of the Sustainable Development Goals; Table S2: Full coding of ESDinds/WeValue indicators.

Acknowledgments

The research described in this study was jointly funded by the University of Brighton and through support for M.K.H. from the China National Thousand Talents program. The original project ‘ESDinds: Development of Indicators and Assessment Tools for Civil Society Organisations Promoting Education for Sustainable Development’ was funded by the European Commission Seventh Framework Programme (FP7), Environment strand, under the funding scheme ‘Research for the Benefit of Specific Groups: Civil Society Organisations’, grant number 212237.

Author Contributions

M.K.H. initially suggested work with the SDGs through the lens of values-based indicators; G.B. designed and performed the investigations and conducted a preliminary analysis; all authors together reviewed the initial findings, re-designed aspects of the investigation, analysed the resulting data, and wrote the paper.

Conflicts of Interest

Two of the authors (G.B. and M.K.H.) were members of the research team that developed the original ESDinds indicators. However, these indicators are in the public domain and not subject to any form of intellectual property protection (www.esdinds.eu), and the authors do not gain any commercial benefit from their dissemination, use or application. The funding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:
CSEComprehensive sexuality education
CSOsCivil society organisations
EFAEducation for All
ESDEducation for Sustainable Development
ESDindsa project funded by the European Commission’s Seventh Framework Programme from 2008 to 2011 under the specialist funding scheme ‘Research for the Benefit of Specific Groups Civil Society Organisations’ (BSG-CSO) [49].
HIVHuman immunodeficiency virus
IAEG-SDGInter-Agency and Expert Group on Sustainable Development Goal Indicators
IUDIntrauterine device
MDGsMillennium Development Goals
QLQualitative
QNQuantitative
SDSustainable development
SDGsSustainable Development Goals
UNECEUnited Nations Economic Commission for Europe
UNEPUnited Nations Environment Program
UNESCOUnited Nations Educational, Scientific and Cultural Organisation

Appendix A. Additional Background on ESD and ESD/SD Competencies

The language of Goal 4 responds to criticisms levelled at the earlier Millennium Development Goals concerned with education, which focused on net primary school enrolment and gender parity and failed to mention quality or to recognise that education continues beyond formal schooling [4]. While the current goal is thought to be an improvement, the term ‘quality’ is an example of the ambiguous wording critiqued by Lu and colleagues [38].

Appendix A.1. ‘Education for Sustainable Development’ (ESD)

Two decades ago, there were already over 300 definitions of ‘sustainable development’ and ‘sustainability’ [92]; now the number may be in thousands. An important manifestation of this ambiguity is the deep division within the environmental education community about the appropriateness, or otherwise, of the term ‘education for sustainable development’. Although the UNESCO definition of ESD refers to an education that “empowers learners to take informed decisions and responsible actions for environmental integrity, economic viability and a just society, for present and future generations, while respecting cultural diversity” [93], the word ‘development’ remains inherently problematic for many. It evokes long histories of paternalistic, centrally organised, and often environmentally and socially destructive economic policies [89]. The term ‘education for sustainable development’ is often felt to imply an endorsement of neoliberal economic growth agendas and the corporate globalisation of education, and a corresponding negation of non-growth-based ideologies such as Arne Naess’s [94] concept of `deep ecology’. Indeed, the United Nations resolution which launched UNESCO’s ‘Decade of Education for Sustainable Development’ has been criticised for lacking any explicit reference to ecology or the environment [48].
Yet however academics might problematize ESD and call for its replacement with less loaded terms such as ‘education as sustainability’ or ‘sustainable education’ [77], or even ‘education consistent with Agenda 21’ [95], there are both pragmatic and ideological reasons for keeping ESD at the forefront of discussions about SDG indicator development.
On the pragmatic side, the success of the UN Decade of ESD has resulted in the concept becoming firmly embedded in global discourses: not only in the SDGs themselves, but also in the Muscat Agreement adopted at the 2014 Education for All conference, and intergovernmental declarations on climate change, biodiversity, sustainable consumption and production, and many other issues [96]. The question of its measurement thus has enormous practical significance.
On the ideological side, any attempt to create a single, universally applicable indicator to represent the entire domain of ESD can be viewed as a way of reinforcing global hegemonies and silencing debate, and as “fundamentally ‘mis-educative’ and anti-democratic”; and by remaining silent on this issue, we effectively become complicit in it. This is particularly true when the proposed indicator frames ESD in terms of disseminating scientific information (transmissive education), rather than empowering learners as critical, creative and proactive change agents (transformative education) (c.f. [48]).

Appendix A.2. Justification of Competencies Approach and Review of Literature on (E)SD Competencies

The term ‘competency’ has been defined by Rychen and Salganik ([64], p. 51) as “the ability to successfully meet complex demands in a particular context through mobilisation of psychosocial prerequisites (including both cognitive and non-cognitive aspects)”, and as “a complex action system encompassing cognitive skills, attitudes, and other non-cognitive components“ (ibid). We have chosen to adopt a competencies approach because, as illustrated by the examples below, this framing has become widespread throughout academic and professional literature on ‘skills for promoting SD’. However, there are some important caveats associated with the use of this term, owing to its historical roots in “competencies-based education”—which was often narrowly defined as education that sought to prepare people for particular jobs or functions—and its association with discourses of quality and accountability ([97], p. 85). One such caveat is that in listing specific competencies that students might acquire, we must not lose sight of the interconnections between them, nor fall into the trap of conceptualising learning as segments of knowledge that can be ordered in a hierarchical sequence. Another is that focusing on competencies may imply that knowledge is static and can only be ‘acquired’, rather than being something that learners themselves can produce as autonomous thinkers (ibid, pp. 94–95). A third important point is that competencies can only be measured indirectly, as they are manifested in very specific contexts of behavior and action [64,98].
A seminal report to the United Nations Educational, Scientific and Cultural Organisation (UNESCO) by the International Commission on Education for the Twenty-First Century, chaired by the former European Commission President Jacques Delors, distinguishes four ‘pillars’ for education in the twenty-first century. These are learning to know (e.g., learning to learn, developing critical thinking, acquiring tools for understanding the world, and understanding sustainability concepts and issues); learning to do (e.g., acquiring technical and professional training and applying learned knowledge in daily life), learning to be (e.g., seeing oneself as the main actor in defining positive future outcomes, developing self-identity and self-knowledge, acting with autonomy and personal responsibility), and learning to live together (e.g., understanding other people’s values and traditions, cooperating with people, celebrating diversity and coping with conflict) [62].
The ‘Delors Report’ has had a substantial impact on education policy and practice worldwide, and constituted a major theme in more than 50 global conferences [99]. These four pillars, which have been respectively correlated to ‘domain competencies’ (or ‘subject competencies’), ‘methodological competencies’, ‘personal competencies’ and ‘social competencies’ [65,66], have been applied specifically to ESD as the foundation of the United Nations Economic Commission for Europe (UNECE) report Learning for the Future: Competences in Education for Sustainable Development [61]. In this report, these domain, methodological, personal and social competencies are applied across three ‘fields’, namely `taking a holistic approach’, ‘envisioning change’ and ‘achieving transformation’, and respectively framed for ESD educators rather than students as follows: ‘The educator understands…’, ‘The educator is able to…’, ‘The educator is someone who…’ and ‘The educator works with others in ways that…’. This report incorporates much more detail than the original UNESCO framework of competencies. To these four main competencies a fifth, learning to transform oneself and society (expanded as minimizing ecological footprints, integrating sustainable lifestyles, creating gender-neutral and non-discriminatory societies, and respecting the Earth and life in all its diversity) was added on the advice of Latin American educators and the United Nations Children’s Emergency Fund (UNICEF) and later officially adopted by UNESCO [67].
The Delors-inspired Learning for the Future report is cited in the 2012 report of the 2009–2014 Decade on Education for Sustainable Development, Shaping the Education of Tomorrow, which notes that a key change during the Decade was the growing recognition not only of the contested nature of ESD, but of “the importance of the capacities, skills, competencies and qualities people need to contribute to transitioning towards a more sustainable world” ([100], p. 22). The same report also cites a 2011 UNECE evaluation which highlights the “need for distinction between SD competence (e.g., citizen’s capacities to contribute to sustainable living both professionally and personally) and ESD competence (e.g., an educator’s capacity to help people develop SD competence through a range of innovative teaching and learning practices)”([101], p. 23).
The Comenius-2 funded CSCT project ‘Curriculum, Sustainable Development, Competences, Teacher Training’ identifies three overall competencies for sustainable development, namely teaching/communicating, reflecting/visioning and networking, overlaid on five domains of professional competencies (respectively values and ethics, action, knowledge, systems-thinking and emotions) which mirror the Delors ‘pillars’ to some extent, and can be applied to teachers as individuals, within their institutions, and in wider society [65].
Wiek et al. ([63], p. 212) have conducted a systematic review of both academic and grey literature on sustainability competencies, in which they critique the existing literature for its lack of rigor—particularly because it tends to refer to competencies in list form, rather than grounding them in a theoretical understanding of sustainability and highlighting interlinkages in the form of a conceptual framework. Their systematic review of 28 academic sources (journal papers and books) and 15 grey literature sources (reports and White Papers) resulted in the identification of five main clusters of key competencies for sustainability:
(a)
systems thinking competence, the ability to analyse the dynamics of complex social-ecological systems;
(b)
anticipatory competence, i.e., the ability to create, analyse and evaluate what they term ‘rich pictures’ of the future;
(c)
normative competence, also referred to as value(s)-focused thinking, which focuses on “the ability to collectively map, specify, apply, reconcile, and negotiate sustainability values, principles, goals, and targets” (p. 10);
(d)
strategic competence, which ensures that learning is translated into effective policies, programs and action plans; and
(e)
interpersonal competence, which refers to the ability to motivate, enable and facilitate participatory sustainability research and collaborative problem-solving, to celebrate diversity, and critically evaluate different positions and perspectives. This is regarded as a cross-cutting issue, as the involvement of multiple actors in sustainability problems makes it essential for collective strategies to be employed in trying to solve them.
Wiek et al. argue that these five categories, which they term key competencies, are essential for sustainability and should therefore be distinguished from ‘basic’ competencies such as critical thinking or communication skills in a more generic sense.
More recent work largely reiterates similar points, rather than making substantive new contributions to the definition of ESD competencies. Murga-Menoyo, for example, echoes Wiek et al. (without citing them directly) in differentiating ‘generic traditional competencies’ (such as problem-solving, critical and self-critical skills, or the ability to work in a team) from ‘generic ompetencies for sustainability’. Murga-Menoyo states that generic traditional competencies, which they characterise as being either instrumental, interpersonal or systemic, are a prerequisite for the achievement of the latter. Table A1, below, correlates the post-2012 references [102,103,104] with the framework developed by Wiek et al. [63]. Within the domain of critical thinking, Cebrián and Junyent specifically highlight the importance of reflecting critically on the values, beliefs and worldviews underpinning current ways of knowing and understanding, and co-constructing new shared meanings (see also [77,105]).
Table A1. Sustainability competencies identified by Wiek et al. [63] and more recent authors.
Table A1. Sustainability competencies identified by Wiek et al. [63] and more recent authors.
Category from Wiek et al. (2012)Related Framings from Recent LiteratureReferences
Systems-thinking competenciesCritical contextualisation of knowledge (interrelating social, economic and environmental issues, local and global levels)Murga-Menoyo [102]
Problem solving, critical thinkingCebrian and Junyent ([103], p. 2769) Education for All (EFA) Global Monitoring Report 2013–2014 ([104], p. 295)
Working and living with complexityCebrian and Junyent [103]
Strategic competenciesSustainable use of natural resources and prevention of negative social and environmental impactsMurga-Menoyo [102]
Action competence: decision-making, participation and action for changeCebrian and Junyent ([103], p. 2769)
Interpersonal competenciesCommunication, co-operation […], conflict resolution, leadership and advocacyEducation for All (EFA) Global Monitoring Report 2013–2014 ([104], p. 295)
Participation in community processes towards sustainabilityMurga-Menoyo [102]
Managing emotions and concernsCebrian and Junyent [103]
Establishing interdisciplinary dialoguesCebrian and Junyent [103]
Normative competenciesValues clarificationCebrian and Junyent [103]
Enacting sustainability values and applying ethical principles in both professional and personal contextsMurga-Menoyo [102]
Anticipatory competenciesVisioning futures or alternative scenariosCebrian and Junyent [103]
For conceptual simplicity and ease of communication, the Wiek model could be entirely contained within that of UNESCO/UNECE. Systems-thinking competencies, for example, might be seen as a subset of ‘learning to know’; anticipatory competencies as a subset of ‘learning to transform oneself and society’; and normative competencies as a subset of ‘learning to be’. The Wiek categories of interpersonal and strategic competencies can respectively be viewed as synonymous with ‘learning to live together’ and ‘learning to do’. We would argue, however, that the political importance of the largely neglected (sub-)categories of systems-thinking, anticipatory and normative competencies negates any pragmatic advantage gained by subsuming them within larger categories. Operationalisation of systems-thinking, anticipatory and normative competencies could have wide-reaching implications, especially in terms of conceptual use of any resulting indicators—inspiring people to change the ways that they think, plan and reflect on values.

Appendix B. Creation of a Conceptual Framework for Target 4.7

In this section, we provide further detail on the creation of the conceptual framework utilised in this article, to supplement the information provided in the main text.

Appendix B.1. Top Level: Identifying Sub-Targets

By analogy with the example provided by Hak et al. [37], the first stage of framework development is the breakdown of the target into sub-targets. As noted in the main text, this cannot be achieved simply by splitting it directly into two blocks (all learners acquire the knowledge needed to promote sustainable development, and all learners acquire the skills needed to promote sustainable development), as the wording of the target poses some inherent conceptual challenges. These are as follows:
(1)
The use of the term ‘all learners’ is problematic in operational terms because, in conjunction with Goal 4’s call for “lifelong learning opportunities for all”, it effectively implies assessing the entire population of the planet. To address this, we focus on individuals who are engaged in some form of organised learning activity, whether formal or non-formal. We acknowledge that in an ESD context the domain of ‘non-formal education’ may include activities in which the learning element is implicit, such as reforestation projects and wildlife surveys, as well as more structured learning activities such as ‘Forest Schools’ and ‘Wildlife Clubs’ [33].
(2)
By focusing on learning outcomes, we are implying that educators must themselves possess relevant knowledge and skills and be encouraged to put them to appropriate use, in order to facilitate learners to acquire them. This also applies, to some extent, to individuals who manage educational institutions and develop curricula.
(3)
The wording of the target is ambiguous in relation to the long list of content areas—human rights, gender equality, etc. It is apparent, however, that these are not intended to be ends in themselves, but means through which knowledge and skills might be acquired. The various content areas are suggested to be necessary but not sufficient for the acquisition of the knowledge and skills required to promote SD, as conveyed by the phrase “among others”. It may be the case that there are other necessary content areas, or that the overall picture is a holistic one, in which the ‘overlaps’ between the various content areas—which may ultimately be undefinable—constitute the space in which the most transformational learning occurs. There may also be ways, as we discuss later, of acquiring the relevant knowledge and skills without passing through any of the eight specific content areas.
(4)
The use of the definite article (the knowledge and the skills needed to promote sustainable development) implies the existence of a discrete, bounded body of knowledge and a clearly definable set of skills, in turn relating to a single unambiguous concept of sustainable development. As this position contradicts any reasonable representation of the state of agreement on these points, we would dispute the inclusion of ‘the’, and have omitted it from our proposed sub-targets. It has been widely acknowledged, for example, that the definition of sustainable development is a ‘wicked’ problem, characterised by “a lack of clarity, uncertainty, ambiguity, high risk, and limited understanding” ([59], p. 100).
Taking all of these points into consideration, we propose two sub-targets for operationalisation as outcome indicators for Target 4.7:
  • Sub-Target 1: All learners and educators involved in organised learning activities, both formal and non-formal, acquire knowledge needed to promote sustainable development
  • Sub-Target 2: All learners and educators involved in organised learning activities, both formal and non-formal, acquire skills needed to promote sustainable development

Appendix B.2. Second Level (Skills Dimension Only): Broad Competency Domains

As discussed in Appendix A above, we have chosen to develop the second level of the conceptual framework with reference to literature on ‘sustainability competencies’, also referred to as ‘ESD competencies’. The term ‘competency’ has been defined by Rychen and Salganik ([64], p. 51) as “the ability to successfully meet complex demands in a particular context through mobilisation of psychosocial prerequisites (including both cognitive and non-cognitive aspects)”, and as “a complex action system encompassing cognitive skills, attitudes, and other non-cognitive components“ (ibid).
We have selected two systematic models, namely the UNESCO/UNECE model (derived from the four ‘pillars of learning’ of the Delors Report [62] with the subsequent addition of ‘learning to transform oneself and society’ [67], as detailed in Appendix A), and the model developed by Wiek et al. [63] on the basis of a systematic review of academic literature. As discussed above, these models were chosen on the grounds that they collectively encompass most of the other literature on ESD competencies, including more recent work. In combining them, we note that the Wiek category of ‘interpersonal competencies’ can be seen as largely equivalent to the UNESCO/UNECE ‘learning to live together’, and ‘strategic competencies’ as equivalent to ‘learning to do’. Thus, we have condensed these into a single category in each case, creating an eight-category rather than a ten-category conceptual framework.

Appendix B.3. Third Level: Specific Clusters of Competencies

To create the third level of the framework, we utilised a UNESCO infographic [68] in which the five broad competency domains are broken down into specific clusters of competencies. Thus, for example, the domain of ‘learning to live together’ incorporates competencies in coping with situations of tension, violence and conflict; accepting and celebrating diversity and difference; responding constructively to cultural diversity and economic disparity around the world; participating and co-operating with others in a multi-cultural society; and developing an understanding of other people and their histories, cultures, etc. These, which we refer to as ‘sub-domains’, are shown in full in Figure 1. For the purpose of our framework, we assigned each of them a unique code, consisting of a designator for the domain as a whole (e.g., ‘L2L’ for ‘learning to live together’) followed by a three-letter code for the sub-domain itself. These abbreviations are widely used in the results tables in Appendix D.
It is important to note that there is considerable overlap between the sub-domains in the UNESCO/UNECE framework, and it is not always meaningful to attempt to differentiate them. This lack of specificity is relevant to our analysis in that an indicator may be found relevant to more than one cluster, not because the indicator itself lacks precision, but because the clusters to which it is relevant are poorly mutually distinguished. Notwithstanding these concerns, we regard the model as a useful initial approximation of a detailed conceptual framework for ESD competencies, which can be further enhanced and clarified by future work.
Another key issue is that some of the items in the framework may be contested or controversial, such as the sub-domain ‘Acquire universally shared values’ under the pillar/theme ‘Learning to Be’, which evokes challenging questions. Which values are deemed to be universally shared? Who is (or should be) responsible for making such decisions? What happens (or should happen) when the values of a specific individual or group come into conflict with the ‘universal’ set? In this paper, we have opted to evade this controversy by replacing the problematic ‘shared universal values’ with the less sensitive ‘common global values’. This latter term relates to the analysis by Burford et al. [13], which illustrates references to global values in different United Nations documents.

Appendix C. Examples of ESDinds Proto-Indicators with Applicable ESD Competency Codes

DomainCompetency CodeSample Proto-Indicator Tagged with This Code
Learning to beL2B_AUTBe able to act with greater autonomy, judgment and personal responsibility Everyone feels responsibility for their part of the work
L2B_DEFSee oneself as the main actor in defining positive outcomes for the futurePeople have a sense of power that they can effect change
L2B_EXPEncourage discovery and experimentationPeople are not afraid to make mistakes (linked to ‘Mistakes are understood as opportunities to learn and improve’)
L2B_SELDevelop one’s personality, self-identity, self-knowledge and self-fulfillmentIndividuals express their own opinions
L2B_VALAcquire shared universal valuesPeople reflect on their values
Learning to doL2D_ACTBe an actor as well as a thinkerPeople invest their own time and resources in activities that benefit the environment or society
L2D_APPApply learned knowledge in daily lifePeople have replicated a project or approach in other contexts
L2D_CREBe able to act creatively and responsibly in one’s environment[Code not used]
L2D_ISSUnderstand and act on global and local sustainable development issues[Code not used]
L2D_TRAAcquire technical and professional training[Code not used]
Learning to knowL2K_CONUnderstand sustainability concepts and issuesPeople understand the complexity of natural systems
L2K_CRICritical thinkingPeople reflect critically on what is necessary to learn
L2K_LEALearn to learnPeople approach their personal development with an attitude of learning
L2K_TASAcquire a taste for learning throughout lifePeople approach their personal development with an attitude of learning
L2K_TOOAcquire tools for understanding the world[Code not used]
Learning to live togetherL2L_CNFBe able to cope with situations of tension, conflict, exclusion, violence and terrorismDifferences of opinion are acknowledged and valued through dialogue
L2L_DIVTolerate, respect, welcome, embrace, and even celebrate difference and diversity in peoplePeople appreciate the differences in others
L2L_ECNRespond constructively to the cultural diversity and economic disparity found around the world[Code not used]
L2L_PARParticipate and co-operate with others in increasingly pluralistic and multi-cultural societiesPeople introduce ideas to each other with respect, humility and patience
L2L_UNDDevelop an understanding of other people and their histories, traditions, beliefs, values and culturesPeople are able to suspend their own standpoints during dialogue and listen to those of others
Learning to transform oneself and societyL2T_DEMPromote democracy in a society where peace prevailsPeople take part in decisions that may affect them directly
L2T_FOOPromote behaviors and practices that minimise our ecological footprint on the world around usLong-term commitments to care for the natural environment are adhered to
L2T_GENWork towards a gender-neutral, non-discriminatory societyPeople take conscious action to improve the ways colleagues are treated
L2T_LIFDevelop the ability and will to integrate sustainable lifestyles for ourselves and othersPeople act to reduce the organisation’s environmental impact and/or restore the natural environment
L2T_RESBe respectful of the Earth and life in all its diversityPeople respect nature and the wider community of life (human and non-human)
L2T_SOLAct to achieve social solidarity[Code not used]
ANTAnticipatory competencies: collectively analyze, evaluate, and craft rich ‘‘pictures’’ of the future related to sustainability issues and sustainability problem-solving frameworks[Code not used]
NORNormative competencies: collectively map, specify, apply, reconcile, and negotiate sustainability values, principles, goals, and targetsPeople can identify applicable ethical values in a given context
SYSSystems-thinking competencies: collectively analyze complex systems across different domains (society, environment, economy, etc.) and across different scales (local to global)People understand the complexity of natural systems

Appendix D. Full Results of Coverage Analysis for IAEG-SDG Consultation Responses and ESDinds Proto-Indicators

Figure D1. Coverage of Sub-Target 1 (Knowledge): Extent to which the submission made by each of the respondents in the IAEG-SDG process on Target 4.7 of the Sustainable Development Goals is relevant to each of the eight content areas named in the target. For the full coded text of the responses, see Table S1 in Supplementary Materials.
Figure D1. Coverage of Sub-Target 1 (Knowledge): Extent to which the submission made by each of the respondents in the IAEG-SDG process on Target 4.7 of the Sustainable Development Goals is relevant to each of the eight content areas named in the target. For the full coded text of the responses, see Table S1 in Supplementary Materials.
Sustainability 08 00861 g004
Figure D2. Coverage of Sub-Target 2 (Skills) by the IAEG-SDG consultation: Extent to which the relevant indicators generated through the IAEG-SDG public consultation process provide coverage of the eight domains of the Skills conceptual framework developed in Figure 1 above, where the abbreviations for each sub-domain are explained in full. The term ‘CI Code’ is an abbreviation of ‘Candidate Indicator Code’, which refers to the codes used to designate the unique candidate indicators proposed in the IAEG-SDG consultation responses. Candidate indicators that differed by only a few words were designated as 01a, 01b, etc. For the full coded text of the responses and candidate indicators, see Table S1 in Supplementary Materials.
Figure D2. Coverage of Sub-Target 2 (Skills) by the IAEG-SDG consultation: Extent to which the relevant indicators generated through the IAEG-SDG public consultation process provide coverage of the eight domains of the Skills conceptual framework developed in Figure 1 above, where the abbreviations for each sub-domain are explained in full. The term ‘CI Code’ is an abbreviation of ‘Candidate Indicator Code’, which refers to the codes used to designate the unique candidate indicators proposed in the IAEG-SDG consultation responses. Candidate indicators that differed by only a few words were designated as 01a, 01b, etc. For the full coded text of the responses and candidate indicators, see Table S1 in Supplementary Materials.
Sustainability 08 00861 g005
Figure D3. Coverage of Sub-Target 2 (Skills) by the ESDinds proto-indicators: Extent to which the relevant indicators generated through the IAEG-SDG public consultation process provide coverage of the eight domains of the Skills conceptual framework developed in Figure 1 above, where the abbreviations for each sub-domain are explained in full. For the full coded text of the proto-indicators, see Table S2 in Supplementary Materials.
Figure D3. Coverage of Sub-Target 2 (Skills) by the ESDinds proto-indicators: Extent to which the relevant indicators generated through the IAEG-SDG public consultation process provide coverage of the eight domains of the Skills conceptual framework developed in Figure 1 above, where the abbreviations for each sub-domain are explained in full. For the full coded text of the proto-indicators, see Table S2 in Supplementary Materials.
Sustainability 08 00861 g006

References

  1. United Nations Millennium Campaign. We the Peoples, Celebrating 8 Million Voices: The MY World Global Survey for a Better World; United Nations: New York, NY, USA, 2015.
  2. United Nations Development Group. A Million Voices: The World We Want—A Sustainable Future with Dignity for All; United Nations Development Group: New York, NY, USA, 2013.
  3. United Nations. Transforming Our World: The 2030 Agenda for Sustainable Development; United Nations: New York, NY, USA, 2015.
  4. Saith, A. From universal values to Millennium Development Goals: Lost in translation. Dev. Chang. 2006, 37, 1167–1199. [CrossRef]
  5. United Nations. The Millennium Development Goals Report; United Nations: New York, NY, USA, 2015.
  6. Henshaw, J.M. Does Measurement Measure up? How Numbers Reveal and Conceal the Truth. In Measurement in Business: What Gets Measured Gets Done; Johns Hopkins University Press: Baltimore, MD, USA, 2006; pp. 55–65.
  7. Gudmundsson, H. The policy use of environmental indicators—Learning from evaluation research. J. Transdiscipl. Environ. Stud. 2003, 2, 1–12.
  8. Grainger, A. Forest sustainability indicator systems as procedural policy tools in global environmental governance. Glob. Environ. Chang. 2012, 22, 147–160. [CrossRef]
  9. Rosenström, U. Exploring the policy use of sustainable development indicators: Interviews with Finnish politicians. J. Transdiscipl. Environ. Stud. 2006, 5, 1–13.
  10. Rosenström, U.; Kyllönen, S. Impacts of a participatory approach to developing national level sustainable development indicators in Finland. J. Environ. Manag. 2007, 84, 282–298. [CrossRef] [PubMed]
  11. Hinkel, J. “Indicators of vulnerability and adaptive capacity”: Towards a clarification of the science-policy interface. Glob. Environ. Chang. 2011, 21, 198–208. [CrossRef]
  12. Bell, S.; Morse, S. Sustainable development indicators: The tyranny of methodology revisited. Cons. J. Sustain. Dev. 2011, 6, 222–239.
  13. Hopwood, W.; Mellor, M.; O’Brien, G. Sustainable development: Mapping different approaches. Sustain. Dev. 2005, 13, 38–52. [CrossRef]
  14. Hawkes, J. The Fourth Pillar of Sustainability: Culture’s Essential Role in Public Planning; Common Ground Publishing Pty Ltd. in association with the Cultural Development Network (Vic): Melbourne, Australia, 2011.
  15. UNESCO. Mexico City Declaration on Cultural Policies; UNESCO: Paris, France, 1982.
  16. Nurse, K. Culture as the Fourth Pillar of Sustainable Development; Commonwealth Secretariat: London, UK, 2006.
  17. UN-PFII. Report of the Meeting on Indigenous Peoples and Indicators of Well-Being, Ottawa, 22–23 March 2006; United Nations Permanent Forum on Indigenous Issues (UN-PFII): New York, NY, USA, 2006.
  18. Kagan, S. Art and Sustainability: Connecting Patterns for a Culture of Complexity; Transcript Verlag: Bielefeld, Germany, 2011.
  19. Spangenberg, J.H. Institutional sustainability indicators: An analysis of the institutions in Agenda 21 and a draft set of indicators for monitoring their effectivity. Sustain. Dev. 2002, 10, 103–115. [CrossRef]
  20. Pfahl, S. Institutional sustainability. Int. J. Sustain. Dev. 2005, 8, 80–96. [CrossRef]
  21. Clugston, R. Ethical Framework for a Sustainable World: Earth Charter Plus 10 conference and follow up. J. Educ. Sustain. Dev. 2011, 5, 173–176. [CrossRef]
  22. Hedlund-de Witt, A. The rising culture and worldview of contemporary spirituality: A sociological study of potentials and pitfalls for sustainable development. Ecol. Econ. 2011, 70, 1057–1065. [CrossRef]
  23. Interreligious Statement Towards Rio+20. Towards Rio+20 and beyond—A Turning Point in Earth’s History; Southern African Faith Communities’ Environment Institute: Kalk Bay, South Africa, 2012.
  24. Burford, G.; Hoover, E.; Velasco, I.; Janouvkova, S.; Jimenez, A.; Piggot, G.; Podger, D.; Harder, M.K. Bringing the ‘missing pillar’ into Sustainable Development Goals: Towards intersubjective values-based indicators. Sustainability 2013, 5, 3035–3059. [CrossRef]
  25. Dahl, A.L. Achievements and gaps in indicators for sustainability. Ecol. Indic. 2012, 17, 14–19. [CrossRef]
  26. Hitlin, S.; Piliavin, J.A. Values: Reviving a dormant concept. Ann. Rev. Sociol. 2004, 30, 359–393. [CrossRef]
  27. Burford, G.; Velasco, I.; Janouskova, S.; Zahradnik, M.; Hak, T.; Podger, D.; Piggot, G.; Harder, M.K. Field trials of a novel toolkit for evaluating ‘intangible’ values-related dimensions of projects. Eval. Prog. Plan. 2013, 36, 1–14. [CrossRef] [PubMed]
  28. Burford, G.; Hoover, E.; Dahl, A.; Harder, M.K. Making the invisible visible: Designing values-based indicators and tools for identifying and closing ‘value-action gaps’. In Responsible Living: Concepts, Education and Future Perspectives; Thoresen, V.W., Doyle, D., Klein, J., Didham, R.J., Eds.; Springer International: Cham, Switzerland, 2015; pp. 113–134.
  29. Burford, G.; Hoover, E.; Jarvis, D.; Harder, M.K. Assessing group learning processes: Reflections on using values-based indicators as assessment criteria with final-year undergraduates. In Evolving Experiences: Articles from The Learning and Teaching Conference 2013; Wisker, G., Marshall, L., Greener, S., Canning, J., Eds.; University of Brighton: Falmer, UK, 2014; pp. 58–67.
  30. Dahl, A.; Harder, M.K.; Mehlmann, M.; Niinimaki, K.; Thoresen, V.; Vinkhuyzen, O.; Vokounova, D.; Burford, G.; Velasco, I. Measuring What Matters: Values-Based Indicators. A Methods Sourcebook. PERL Values-Based Learning Toolkit 1. 2014. Available online: http://iefworld.org/fl/PERL_toolkit1.pdf (accessed on 23 August 2016).
  31. Dahl, A.; Harder, M.K.; Mehlmann, M.; Niinimaki, K.; Thoresen, V.; Vinkhuyzen, O.; Vokounova, D.; Burford, G.; Velasco, I. Discovering What Matters: A Journey of Thinking and Feeling. Activities Developed with Students, for Students. PERL Values-Based Learning Toolkit 2. 2014. Available online: http://iefworld.org/fl/PERL_toolkit2.pdf (accessed on 23 August 2016).
  32. Dahl, A.; Harder, M.K.; Mehlmann, M.; Niinimaki, K.; Thoresen, V.; Vinkhuyzen, O.; Vokounova, D.; Burford, G.; Velasco, I. Growing a Shared Vision: A Toolkit for Schools. Activities for Organisational and Staff Development. PERL Values-Based Learning Toolkit 3. 2014. Available online: http://iefworld.org/fl/PERL_toolkit1.pdf (accessed on 23 August 2106).
  33. Harder, M.K.; Velasco, I.; Burford, G.; Podger, D.; Janouskova, S.; Piggot, G.; Hoover, E. Reconceptualizing ‘efffectiveness’ in environmental projects: Can we measure values-related achievements? J. Environ. Manag. 2014, 139, 120–134. [CrossRef] [PubMed]
  34. Hoover, E. Starting from Values: Evaluating Intangible Legacies; University of Brighton: Brighton, UK, 2015.
  35. Podger, D.; Piggot, G.; Zahradnik, M.; Janouskova, S.; Velasco, I.; Hak, T.; Dahl, A.; Jimenez, A.; Harder, M.K. The Earth Charter and the ESDinds initiative: Developing indicators and assessment tools for civil society organisations to examine the values dimensions of sustainability projects. J. Educ. Sustain. Dev. 2010, 4, 297–305. [CrossRef]
  36. Podger, D.; Velasco, I.; Amezcua Luna, C.; Burford, G.; Harder, M.K. Can values be measured? Significant contributions from a small civil society organisation through action research evaluation. Action Res. 2013, 11, 8–30. [CrossRef]
  37. Hák, T.; Janoušková, S.; Moldan, B. Sustainable Development Goals: A need for relevant indicators. Ecol. Indic. 2016, 60, 565–573. [CrossRef]
  38. Lu, Y.; Nakicenovic, N.; Visbeck, M.; Stevance, S. Five priorities for the UN Sustainable Development Goals. Nature 2015, 520, 432–433. [CrossRef] [PubMed]
  39. McCool, S.; Stankey, G. Indicators of sustainability: Challenges and opportunities at the interface of science and policy. Environ. Manag. 2004, 33, 294–305. [CrossRef] [PubMed]
  40. Tamas, P. Systematic review of methods in low-consensus fields: Supporting commensuration through ‘construct-centered methods aggregation’ in the case of climate change vulnerability research. PLoS ONE 2016, 11. [CrossRef]
  41. Dalal-Clayton, B.; Bass, S. Sustainable Development Strategies: A Resource Book; Earthscan: London, UK, 2002.
  42. Lozano, R.; Llobet, J.; Tideswell, G. The Process of Assessing and Reporting Sustainability at Universities: Preparing the Report of the University of Leeds; UNESCO and Universitat Politechnica di Catalunya: Barcelona, Spain, 2016.
  43. United Nations Economic and Social Council. Inter-Agency and Expert Group on Sustainable Development Goal Indicators. 2016. Available online: http://unstats.un.org/sdgs/iaeg-sdgs/members (accessed on 23 August 2016).
  44. Inter-Agency and Expert Group on Sustainable Development Goal Indicators. Report of the Inter-Agency and Expert Group on Sustainable Development Goal Indicators to the 47th Session of the United Nations Statistical Commission; United Nations Economic and Social Council: New York, NY, USA, 2015.
  45. IAEG-SDG Open Consultation on Green Indicators. 2015. Available online: http://unstats.un.org/sdgs/iaeg-sdgs/open-consultation-2 (accessed on 19 July 2016).
  46. IAEG-SDG Open Consultation on Grey Indicators. 2015. Available online: http://unstats.un.org/sdgs/iaeg-sdgs/open-consultation-3 (accessed on 19 July 2016).
  47. Vladimirova, K.; Le Blanc, D. How Well Are the Links between Education and Other Sustainable Development Goals Covered in UN Flagship Reports? A Contribution to the Study of the Science-Policy Interface on Education in the UN System; DESA Working Paper No. 146, ST/ESA/2015/DWP/146; Department of Economic and Social Affairs: New York, NY, USA, 2015.
  48. Jickling, B.; Wals, A.E.J. Globalization and environmental education: Looking beyond sustainable development. J. Curric. Stud. 2008, 40, 1–21. [CrossRef]
  49. European Commission. Seventh Framewok Programme Guide for Applicants: Cooperation: Theme 6: Environment (Including Climate Change); Research for the Benefit of Specific Groups: Civil Society Organisations (BSG-CSO). FP7-ENV-2007-1; European Commission: Brussels, Belgium, 2007.
  50. Oxford English Dictionary. Definition of ‘Values’; Oxford English Dictionary: Oxford, UK, 2013.
  51. ESDinds. ESDinds: The Development of Values-Based Indicators and Assessment Tools for Civil Society Organisations Promoting Education for Sustainable Development; ESDinds Project Consortium Led by University of Brighton: Brighton, UK, 2011.
  52. Podger, D.; Hoover, E.; Burford, G.; Harder, M.K. Revealing values in a complex environmental program: A scaling up of values-based indicators, in press. J. Clean. Prod. 2015. [CrossRef]
  53. Fraser, E.D.; Dougill, A.J.; Mabee, W.E.; Reed, M.; McAlpine, P. Bottom up and top down: Analysis of participatory processes for sustainability indicator identification as a pathway to community empowerment and sustainable environmental management. J. Environ. Manag. 2006, 78, 114–127. [CrossRef] [PubMed]
  54. Reed, M.; Fraser, E.D.G.; Morse, S.; Dougill, A.J. Integrating Methods for Developing Sustainability Indicators to Facilitate Learning and Action. Ecol. Soc. 2005, 10, 1–6.
  55. Reed, M.S.; Dougill, A.J. Participatory selection process for indicators of rangeland condition in the Kalahari. Geogr. J. 2002, 168, 224–234.
  56. Reed, M.S.; Dougill, A.J.; Baker, T.R. Participatory indicator development: What can ecologists and local communities learn from each other? Ecol. Appl. 2008, 18, 1253–1269. [CrossRef]
  57. Reed, M.S.; Fraser, E.D.G.; Dougill, A.J. An adaptive learning process for developing and applying sustainability indicators with local communities. Ecol. Econ. 2006, 59, 406–418. [CrossRef]
  58. Values and Sustainability Research Group, University of Brighton. WeValue: A Values-Based Approach. Available online: http://blogs.brighton.ac.uk/wevalue/the-approach/ (accessed on 23 August 2016).
  59. Ribeiro, M.M.; Hoover, E.; Burford, G.; Buchebner, J.; Lindenthal, T. Values: A bridge between sustainability and institutional assessment-a case study from BOKU University. Int. J. Sustain. High. Educ. 2016, 17, 40–53. [CrossRef]
  60. Carteron, J.-C.; Decamps, A. The Sustainability Literacy Test: Can Universities Be Sure They Are Producing Sustainability Literate Graduates? One Year Report Presented on the Occasion of the World Conference on Education for Sustainable Development; Higher Education Sustainability Initiative: Nagoya, Japan, 2014.
  61. United Nations Economic Commission for Europe (UNECE). United Nations Economic Commission for Europe (UNECE). Learning for the Future: Competences in education for sustainable development. In United Nations Economic Commission for Europe Strategy for Education for Sustainable Development; ECE/CEP/AC.13/2011/6; Programma Leren voor Duurzame Ontwikkeling: Utrecht, The Netherlands, 2011.
  62. Delors, J.; Al Mufti, I.; Amagi, I.; Carneiro, R.; Chung, F.; Geremek, B.; Gorham, W.; Kornhauser, A.; Manley, M.; Quero, M.P.; et al. Learning: The Treasure within Report to UNESCO of the International Commission on Education for the Twenty-First Century; UNESCO: Paris, France, 1996.
  63. Wiek, A.; Withycombe, L.; Redman, C.L. Key competencies in sustainability: A reference framework for academic program development. Sustain. Sci. 2011, 6, 203–218. [CrossRef]
  64. Rychen, D.S.; Salganik, L.H. Defining and Selecting Key Competencies; Hogrefe & Huber Publishers: Ashland, OH, USA, 2001.
  65. Sleurs, W. Competencies for ESD (Education for Sustainable Development) Teachers: A Framework to Integrate ESD in the Curriculum of Teacher Training Institutes; CSCT Project (Comenius 2.1 project 118277-CP-1-2004-BE-Comenius-C2.1): Brussels, Belgium, 2008.
  66. Erpenbeck, J.; von Rosenstiel, L. (Eds.) Handbuch Kompetenzmessung. Erkennen, Verstehen und Bewerten von Kompetenzen in der Betrieblichen, Pädagogischen und Psychologischen Praxis [Handbook of Competency Measurement. Recognising, Understanding and Valuing Competencies in Professional, Pedagogical and Psychological Practice]; Schäffer, Poeschel: Stuttgart, Germany, 2003.
  67. Combes, B.P. The United Nations decade of education for sustainable development (2005–2014): Learning to live together sustainably. Appl. Environ. Educ. Commun. 2005, 4, 215–219. [CrossRef]
  68. Oladottir, H. UNESCO Education for Sustainable Development 2005–2014. Available online: http://menntuntilsjalfbaerni.weebly.com/uploads/6/2/6/2/6262718/unesco_5_pillars_for_esd.pdf (accessed on 23 August 2016).
  69. Sanders, E.B.-N.; Stappers, P.J. Convivial Toolbox: Generative Research for the Front End of Design; BIS Publishers: Amsterdam, The Netherlands, 2012.
  70. Polanyi, M. The Tacit Dimension, Revised ed.; University of Chicago Press: Chicago, IL, USA, 2009.
  71. Blake, J. Overcoming the ‘value-action gap’ in environmental policy: Tensions between national policy and local experience. Local Environ. 1999, 4, 257–278. [CrossRef]
  72. Kennedy, E.H.; Beckley, T.M.; McFarlane, B.L.; Nadeau, S. Why we don’t ‘walk the talk’: Understanding the environmental values/behaviour gap in Canada. Res. Hum. Ecol. 2009, 16, 151–160.
  73. Kollmuss, A.; Agyeman, J. Mind the gap: Why do people act environmentally and what are the barriers to pro-environmental behavior? Environ. Educ. Res. 2002, 8, 239–260. [CrossRef]
  74. Schlater, J.D.; Sontag, M. Toward the measurement of human values. Fam. Consum. Sci. Res. J. 1994, 23, 4–25. [CrossRef]
  75. Rescher, N. Introduction to Value Theory; University Press of America: Lanham, MD, USA, 1982.
  76. Sterling, S. Higher education, sustainability and the role of systemic learning. In Higher Education and the Challenge of Sustainability: Problematics, Promise, and Practice; Corcoran, P.B., Wals, A.E.J., Eds.; Kluwer Academic Publishers: Dordrecht, The Netherlands, 2004.
  77. Sterling, S. Sustainable Education: Revisioning Learning and Change; Green Books: Cambridge, UK, 2001.
  78. Wals, A.E.J.; Jickling, B. ‘Sustainability’ in higher education: From doublethink and newspeak to critical thinking and meaningful learning. Int. J. Sustain. Higher Educ. 2002, 3, 221–232. [CrossRef]
  79. Fuad-Luke, A. Re-Defining the purpose of (sustainable) design: enter the design enablers, catalysts in co-design. In Designers, Visionaries and Other Stories: A Collection of Sustainable Design Essays; Chapman, J., Gant, N., Eds.; Earthscan: London, UK, 2007; pp. 18–55.
  80. Walker, S. Design Redux. In Designers, Visionaries and Other Stories; Chapman, J., Gant, N., Eds.; Earthscan: London, UK, 2007; pp. 56–75.
  81. Wood, J. Relative Abundance: Fuller’s Discovery that the Glass Is Always Half Full. In Designers, Visionaries and Other Stories: A Collection of Sustainable Design Essays; Chapman, J., Gant, N., Eds.; Earthscan: London, UK, 2007; pp. 96–115.
  82. Walker, S. Sustainable by Design: Explorations in Theory and Practice; Earthscan/James and James: London, UK, 2006.
  83. Biggs, J.B. The role of meta-learning in study process. Br. J. Educ. Psychol. 1985, 55, 185–212. [CrossRef]
  84. Meyer, J.H.F.; Shanahan, M.P. Developing metalearning capacity in students—Actionable theory and practical lessons learned in first-year economics. Innov. Educ. Teach. Int. 2004, 41, 443–458. [CrossRef]
  85. Lele, S.; Norgaard, R.B. Sustainability and the scientist’s burden. Conserv. Biol. 1996, 10, 354–365. [CrossRef]
  86. Ostrom, E. Polycentric systems for coping with collective action and global environmental change. Glob. Environ. Chang. 2010, 20, 550–557. [CrossRef]
  87. Onwuegbuzie, A.; Witcher, A.E.; Collins, K.M.T.; Filer, J.D.; Wiedmaier, C.D.; Moore, C.W. Students’ Perceptions of Characteristics of Effective College Teachers: A Validity Study of a Teaching Evaluation Form Using a Mixed-Methods Analysis. Am. Educ. Res. J. 2007, 44, 113–160. [CrossRef]
  88. Onwuegbuzie, A.; Johnson, R.B. The validity issue in mixed research. Res. Sch. 2006, 13, 48–63.
  89. Robinson, J. Squaring the circle? Some thoughts on the idea of sustainable development. Ecol. Econ. 2004, 48, 369–384. [CrossRef]
  90. Walker, S.; Giard, J. General introduction: Design for Sustainability—A Reflection. In Handbook of Design for Sustainability; Walker, S., Giard, J., Eds.; Bloomsbury Academic: London, UK; New York, NY, USA, 2013; pp. 1–12.
  91. Abrams, D. The Spell of the Sensuous: Perception and Language in a More-Than-Human World; Pantheon: New York, NY, USA, 1996.
  92. Dobson, A. Environmental sustainabilities: An analysis and a typology. Environ. Politics 1996, 5, 401–428. [CrossRef]
  93. UNESCO. What Is Education for Sustainable Development? Available online: http://en.unesco.org/themes/education-sustainable-development/what-is-esd (accessed on 29 May 2016).
  94. Naess, A. The shallow and the deep, long-range ecology movements. Inquiry 1973, 16, 95–100. [CrossRef]
  95. Smyth, J. Is there a future for education consistent with Agenda 21? Can. J. Environ. Educ. 1999, 4, 69–82.
  96. UNESCO. Aichi-Nagoya Declaration on Education for Sustainable Development; UNESCO: Paris, France, 2014.
  97. O’Donoghue, T.; Chapman, E. Problems and prospects in competencies-based education: A curriculum studies perspective. Educ. Res. Perspect. 2010, 37, 85–104.
  98. Soucek, V. Flexible Education and New Standards of Communication Competence. In Economising Education: The Post-Fordist Directions; Kenway, J., Ed.; Deakin University Press: Geelong, Australia, 1993; pp. 33–72.
  99. Tawil, S.; Cougoureux, M. Revisiting ‘Learning: The Treasure Within’: Assessing the Influence of the 1996 Delors Report. In UNESCO Education Research and Foresight: Occasional Papers; UNESCO: Paris, France, 2013.
  100. UNESCO. Shaping the Education of Tomorrow: 2012 Full-Length Report on the UN Decade of Education for Sustainable Development; UNESCO: Paris, France, 2012.
  101. UNECE. Learning from Each Other: Achievements, Challenges and Ways Forward: Second Evaluation Report of the Implementation of the UNECE ESD Strategy; UNECE: Geneva, Switzerland, 2011.
  102. Murga-Menoyo, M. Learning for a sustainable economy: Teaching of green competencies in the university. Sustainability 2014, 6, 2974–2992. [CrossRef]
  103. Cebrián, G.; Junyent, M. Competencies in Education for Sustainable Development: Exploring the Student Teachers’ Views. Sustainability 2015, 7, 2768–2786. [CrossRef]
  104. Education for All. Teaching and Learning: Achieving Quality for all. Education for All 2013–2014 Global Monitoring Report; UNESCO: Paris, France, 2014.
  105. Wals, A.E.; Corcoran, P.B. Sustainability as an Outcome of Transformative Learning. In Drivers and Barriers for Implementing Sustainable Development in Higher Education; Holmberg, J., Samuelsson, B.E., Eds.; UNESCO: Paris, France, 2006; p. 103.
Figure 1. Original conceptual framework for Target 4.7 of the Sustainable Development Goals, created by combining models developed by Wiek et al. [63] (competency domains shown in grey), and United Nations Educational, Scientific and Cultural Organisation (UNESCO)/United Nations Economic Commission for Europe (UNECE) [62] (competency domains shown in colour with the respective sub-domains below them).
Figure 1. Original conceptual framework for Target 4.7 of the Sustainable Development Goals, created by combining models developed by Wiek et al. [63] (competency domains shown in grey), and United Nations Educational, Scientific and Cultural Organisation (UNESCO)/United Nations Economic Commission for Europe (UNECE) [62] (competency domains shown in colour with the respective sub-domains below them).
Sustainability 08 00861 g001
Figure 2. A comparison of the contribution of the two approaches to the coverage of Target 4.7 in the most appropriate domain—Skills. It also illustrates the different competency profiles of the two approaches, which deserve reflection.
Figure 2. A comparison of the contribution of the two approaches to the coverage of Target 4.7 in the most appropriate domain—Skills. It also illustrates the different competency profiles of the two approaches, which deserve reflection.
Sustainability 08 00861 g002
Figure 3. A phrase cloud illustrating relative frequencies of production of indicators for different competencies for the traditional and values-based approaches. The font size indicates the number of times an indicator was produced for that competency, by the ESDinds process (cool colours) and by the IAEG-SDG process (warm colours) respectively. The phrases in the cloud are abbreviations of the full-text descriptors for the sub-domains as shown in Figure 1, e.g., ‘participate and co-operate’ is an abbreviation of L2L_PAR, “participate and co-operate with others in increasingly pluralistic and multi-cultural societies’.
Figure 3. A phrase cloud illustrating relative frequencies of production of indicators for different competencies for the traditional and values-based approaches. The font size indicates the number of times an indicator was produced for that competency, by the ESDinds process (cool colours) and by the IAEG-SDG process (warm colours) respectively. The phrases in the cloud are abbreviations of the full-text descriptors for the sub-domains as shown in Figure 1, e.g., ‘participate and co-operate’ is an abbreviation of L2L_PAR, “participate and co-operate with others in increasingly pluralistic and multi-cultural societies’.
Sustainability 08 00861 g003
Table 1. A summary of factors in the two approaches which indicate domain differences or similarities for consideration in the comparison analysis.
Table 1. A summary of factors in the two approaches which indicate domain differences or similarities for consideration in the comparison analysis.
Values-Based Approach (ESDinds) (N = 125)UN-IAEG Conventional Approach (N = 114)
Who was asked?Members of 6 groups, civil society organisations (CSOs): their answers were used to construct the setAll countries; regional and international agencies, civil societies, academia and the private sector were asked.
Members of a further 15 CSOs: their answers were used to reduce the set by clustering overlapping responses
Who answeredAs above 114 groups answered, of which 83 CSOs, 16 UN Agencies or related, nine national statistics offices, five universities and one specific working group.
Question responded to“What is worthwhile, valuable and meaningful to you about your work?”“Please examine the Candidate Indicator given for SDG Target 4.7 and propose any alternatives you think might be more appropriate.”
Boundaries which are contextually implied:The question was with respect toThe question was with respect to
aA local groupaMostly national or international groups
bNon-formal educationbFormal or non-formal—not specified
cAssessment of ESD of a collectivecAssessment of ESD of individuals
dLocal SD focusdLocal or global—not specified
eThe present situation and near futureePresent, near or far future—not specified
fPractice (as opposed to knowledge)fPractice or knowledge—not specified
gHolistic approach to assessmentgNon-holistic approach to assessment
hESD decomposition via local valueshESD deconstruction via outputs
iObtained intersubjectively within the groupiProcess within groups unknown but likely to be non-inter-subjectively
Table 2. Scoring method used across the conceptual framework for Sub-Target 1 (Knowledge) of Sustainable Development Goals (SDG) Target 4.7 for relevance and coverage. Note that this Sub-Target is not covered in the values-based approach (ESDinds).
Table 2. Scoring method used across the conceptual framework for Sub-Target 1 (Knowledge) of Sustainable Development Goals (SDG) Target 4.7 for relevance and coverage. Note that this Sub-Target is not covered in the values-based approach (ESDinds).
RelevanceCoverage
Holistic knowledge
OR1 point2 points
The Sustainability Literacy Test
CONTENT x8:1 point if AT LEAST ONE content area is assessed1 point per content area judged to be PARTIALLY covered and 2 points per content area judged to be SYSTEMATICALLY OR RIGOUROUSLY covered
1. Education for Sustainable Development
2. Education for sustainable lifestyles
3. Human rights
4. Gender equality
5. Culture of peace and non-violence
6. Global citizenship
7. Appreciation of cultural diversity
8. Culture’s contribution to sustainable development
NONE of the above0 points0 points
Table 3. Scoring method used across the conceptual framework for Sub-Target 2 (Skills) of SDG Target 4.7 for relevance and coverage. The coloured area indicates domains not covered in the values-based approach case- studied in this work (Anticipatory Competencies).
Table 3. Scoring method used across the conceptual framework for Sub-Target 2 (Skills) of SDG Target 4.7 for relevance and coverage. The coloured area indicates domains not covered in the values-based approach case- studied in this work (Anticipatory Competencies).
RelevanceCoverage
1 point awarded1 point per competency domain that is PARTIALLY covered, i.e., the indicator covers AT LEAST ONE of its sub-clusters
COMPETENCIES x8:if ANY of these competency domains are assessed
L2D Learning to do/Strategic Competencies 2 points per competency domain that is SYSTEMATICALLY OR RIGOROUSLY covered, i.e., the indicator covers ALL of its sub-clusters
L2K Learning to know
L2B Learning to be
L2T Learning to transform oneself and society
L2L Learning to live together/Interpersonal
Normative Competencies1 point awarded if ANY of these competencies are assessed (only if not already awarded for UNECE competencies)1 point per competency domain judged to be PARTIALLY covered *.
Anticipatory Competencies
Systems Thinking Competencies 2 points per competency domain that the coder judges to be SYSTEMATICALLY OR RIGOROUSLY covered *
NONE of the above0 points0 points
* According to the description by Wiek et al.
Table 4. An overview of the numbers of useful indicators for SDG Target 4.7 produced by the two indicator development approaches (conventional approach exemplified by Inter-Agency and Expert Group on Sustainable Development Goal Indicators (IAEG-SDG) public consultation, and values-based approach exemplified by ESDinds).
Table 4. An overview of the numbers of useful indicators for SDG Target 4.7 produced by the two indicator development approaches (conventional approach exemplified by Inter-Agency and Expert Group on Sustainable Development Goal Indicators (IAEG-SDG) public consultation, and values-based approach exemplified by ESDinds).
IAEG-SDG Public Consultation Responses (Conventional Approach)ESDinds Proto-Indicators (Values-Based Approach)
Total N = 71 itemsTotal N = 125 items
Number of indicators with:
item validity score of 0 (not useful):27 (38%)56 (45%)
item validity score of 1 (useful for only ONE Sub-target):3069
Sub-Target 1 (knowledge acquisition only)22n/a
Sub-Target 2 (skills acquisition only)869
item validity score of 2 (useful for both sub-targets)14n/a
Table 5. Number of indicators coded in each competency domain for candidate indicators from the IAEG-SDG public consultation (‘IAEG’) and proto-indicators from the ESDinds project (‘ESDinds’). Coverage is scored as 0 (none), 1 (partially covered), or 2 (fully or rigorously covered) for each domain.
Table 5. Number of indicators coded in each competency domain for candidate indicators from the IAEG-SDG public consultation (‘IAEG’) and proto-indicators from the ESDinds project (‘ESDinds’). Coverage is scored as 0 (none), 1 (partially covered), or 2 (fully or rigorously covered) for each domain.
Learning to DoLearning to KnowLearning to BeLearning to TransformLearning to Live TogetherNORANTSYSTOTAL SCORE
L2D_ACTL2D_APPL2D_CREL2D_ISSL2D_TRAL2K_CRIL2K_LEAL2K_TASL2K_TOOL2B_AUTL2B_DEFL2B_EXPL2B_SELL2B_VALL2T_DEML2T_FOOL2T_GENL2T_LIFL2T_RESL2T_SOLL2L_CNFL2L_DIVL2L_ECNL2L_PARL2L_UND
IAEG-SDG (full)3201410002201221320129042401
Coverage IAEG-SDG111111017
ESDinds (full)#10001210712962811413#0#2501
Coverage ESDinds112211019
Table 6. Sensitivity analysis for coding of candidate indicators from the IAEG-SDG public consultation (‘IAEG’) and proto-indicators from the ESDinds project (‘ESDinds’). Coverage is scored as 0 (none), 1 (partially covered), or 2 (fully or rigorously covered) for each domain. In each case the most critical item (MCI) has been removed from the analysis. The figures for the full dataset are also included for the purpose of comparison.
Table 6. Sensitivity analysis for coding of candidate indicators from the IAEG-SDG public consultation (‘IAEG’) and proto-indicators from the ESDinds project (‘ESDinds’). Coverage is scored as 0 (none), 1 (partially covered), or 2 (fully or rigorously covered) for each domain. In each case the most critical item (MCI) has been removed from the analysis. The figures for the full dataset are also included for the purpose of comparison.
Learning to DoLearning to KnowLearning to BeLearning to TransformLearning to Live TogetherNORANTSYSTOTAL SCORE
L2D_ACTL2D_APPL2D_CREL2D_ISSL2D_TRAL2K_CRIL2K_LEAL2K_TASL2K_TOOL2B_AUTL2B_DEFL2B_EXPL2B_SELL2B_VALL2T_DEML2T_FOOL2T_GENL2T_LIFL2T_RESL2T_SOLL2L_CNFL2L_DIVL2L_ECNL2L_PARL2L_UND
IAEG-SDG (full)3201410002201221320129042401
Coverage IAEG-SDG111111017
IAEG-SDG (MCI removed)2100400001100221220016020300
Coverage IEAG-MCI101111005
ESDinds (full)#10001210712962811413#0#2501
Coverage ESDinds112211019
ESDinds (MCI removed)#10001210712962811413#0#2500
Coverage ESDinds-MCI112211008
Sustainability EISSN 2071-1050 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top