Assessing the Impact of ESD: Methods, Challenges, Results

Round 1

Reviewer 1 Report

Thank you very much for a very well written, interesting and important review paper. I have a few things that came to my mind when I read the manuscript. 

• In several places in the manuscript you refer to ESD as "the need for everyone to have the knowledge and skills to meet the challenges of tomorrow". Why tomorrow? The environmental crisis are here and now and in the ESD literature there is a lot written about taking action in authentic problems in education. The way of seeing education that it is only for tomorrow risks of not empowering students with the competences you are describing. Which brings me in to the next issue, action competence. 

• It is not clear to me how you chose the competences for sustainability in section 4.1. There is a lot written I recent years about action competence for sustainability as an important outcome of ESD. It is not clear to me why some competence concepts are included and some not?

• You need to me more detailed about your method. It is not clear how you did your literature review? Why are some relevant papers in here and some not? You need to describe this. Actually, I miss  some important papers that makes me wonder about your "systematic review". For example from this journal, The effectiveness of education for sustainable development  by Boeve-de Pauw, Gericke Olsson & Berglund, 2015. There are also other recent review studies that I think you at least need to relate to somehow. For example the Environmental education and K-12 student outcomes: A review and analysis of research, by Ardoin and colleagues 2018 and Developing students' action competence for a sustainable future: A review of educational research, by Chen & Liu (2020).
  
  In addition, you need to describe "the collection of perceptions from two experts". How did you do this and how do you use this information in your review?
  
  
• I would like a little bit more critical reflection on how you define ESD in terms of holism and pluralism. There is actually a lot of critique of pluralism. See for example articles by Helen Kopnina. Her article Revisiting Education for Sustainable Development (ESD): Examining Anthropocentric Bias Through the Transition of Environmental Education to ESD, could be a good start for this.
• Finally Check the references and the reference list once more. For example, on page 21 you refer to Gericke, Olsson and Berglund. I cannot see this reference in the reference list (or when I did a search).

I wish you good luck in your work with the revision of this manuscript and I am looking forward to see the final paper published.

Author Response

Response to Reviewer 1 Comments

Thank you for your insightful comments, which have been very valuable in improving our paper.

Point 1: In several places in the manuscript you refer to ESD as "the need for everyone to have the knowledge and skills to meet the challenges of tomorrow". Why tomorrow? The environmental crisis are here and now and in the ESD literature there is a lot written about taking action in authentic problems in education. The way of seeing education that it is only for tomorrow risks of not empowering students with the competences you are describing. Which brings me in to the next issue, action competence.

Response 1: Thank you for this strong point on the need to not only use this article to focus on tomorrow but also on the present, which we have taken into account with modifications throughout the document.

Point 2: It is not clear to me how you chose the competences for sustainability in section 4.1. There is a lot written I recent years about action competence for sustainability as an important outcome of ESD. It is not clear to me why some competence concepts are included and some not?

Response 2: We thank you for your comment, which made us realize that it was necessary to stress that our list was not meant to be exhaustive but rather to show the variety of the existing frameworks, which we have now done in section 4.1.

Point 3: You need to me more detailed about your method. It is not clear how you did your literature review? Why are some relevant papers in here and some not? You need to describe this. Actually, I miss  some important papers that makes me wonder about your "systematic review". For example from this journal, The effectiveness of education for sustainable development by Boeve-de Pauw, Gericke Olsson & Berglund, 2015. There are also other recent review studies that I think you at least need to relate to somehow. For example the Environmental education and K-12 student outcomes: A review and analysis of research, by Ardoin and colleagues 2018 and Developing students' action competence for a sustainable future: A review of educational research, by Chen & Liu (2020).

Response 3: In full agreement with your comment on our lack of clarity, we have expanded the Materials and Methods section to better explain our methodology. We have also added new references, including Boeve-de Pauw, Gericke Olsson & Berglund, which we were familiar with but were not certain about including in this paper as it is mainly focused on the specific efficiency of pluralism and holism in ESD, and Ardoin & al., which is more focused on technical characteristics of studies than on their conclusions. Nonetheless, as you rightfully put it, they are a must in recent ESD literature. We thank you for the reference to Chen & Liu, which we discovered through your comment and have included in our study.

Point 4: In addition, you need to describe "the collection of perceptions from two experts". How did you do this and how do you use this information in your review?

Response 4: We thank you for your comment, which has led us to expand our Materials and Methods section with text on how we used this information in our review.

Point 5: I would like a little bit more critical reflection on how you define ESD in terms of holism and pluralism. There is actually a lot of critique of pluralism. See for example articles by Helen Kopnina. Her article Revisiting Education for Sustainable Development (ESD): Examining Anthropocentric Bias Through the Transition of Environmental Education to ESD, could be a good start for this.

Response 5: We thank you and completely agree regarding the need for the addition. A section on the critique of the two predicates has been added and it actually echoes one of your recommended articles (Boeve-de Pauw, Gericke Olsson & Berglund, 2015).

Point 6: Finally Check the references and the reference list once more. For example, on page 21 you refer to Gericke, Olsson and Berglund. I cannot see this reference in the reference list (or when I did a search).

Response 6: We apologize for the missing references. We have added it and checked that the necessary references are now in the reference list.

Thank you once again. We greatly appreciate the time and effort that you have dedicated thus far to providing feedback on our manuscript.

Reviewer 2 Report

Dear authors

In my opinion, the paper contains an interesting study of case. However think that it lacks an introduction where to put very important aspects such as:

• The limits of previous research that justify the need of the article.
• Please consider adding a purpose statement and research questions.

The authors explain defining impact measurement in many detail, but the specific details of Materials and methods are not detailed, while these are the most interesting part of the context. Materials and methods are short and very poor. I’m sure that this section allows for much more than a few lines.

At the bottom of page 18, maybe you should change Jane and John to Jane and Jill.

Description of results in 6.Conclusions is not detailed and comprehensive and statements are not as in-depth as it would desirable.

Kind regards

Author Response

Response to Reviewer 2 Comments

Thank you for your insightful comments, which have been very valuable in improving our paper.

Point 1: I think that it lacks an introduction where to put very important aspects such as: The limits of previous research that justify the need of the article.

Response 1: Thank you for your recommendation. We have expanded our Introduction with the suggested points. We have also created a more natural transition into the Materials and Methods section, which we have also expanded.

Point 2: Please consider adding a purpose statement and research questions.

Response 2: We very much appreciated this suggestion. We have added a clear research question to the introduction, which we now revisit in the conclusion. We have also clarified the overarching purpose of our paper in both the abstract and the introduction.

Point 3: The authors explain defining impact measurement in many detail, but the specific details of Materials and methods are not detailed, while these are the most interesting part of the context. Materials and methods are short and very poor. I’m sure that this section allows for much more than a few lines.

Response 3: We have expanded the Materials and Methods section to better explain our methodology. Thank you for encouraging this very necessary modification.

Point 4: At the bottom of page 18, maybe you should change Jane and John to Jane and Jill.

Response 4: Thank you very much for pointing out this error, which has been duly corrected.

Point 5: Description of results in 6.Conclusions is not detailed and comprehensive and statements are not as in-depth as it would desirable.

Response 5: We acknowledge this weakness in the original version of the paper and have greatly enhanced the Conclusions section drawing on our reformulated research question.

Thank you once again. We greatly appreciate the time and effort that you have dedicated thus far to providing feedback on our manuscript.

Reviewer 3 Report

This article aims to address the impact of education for sustainable development through a literature review. The article is interesting and an advance in our current knowledge on an emerging topic such as ESD.
However, I would like to make a series of observations.
Regarding the writing of the article, I miss well-defined research objectives and questions, and a more solid and clearly expressed methodology for systematic review. Which would help in an interpretation of the results and more solid conclusions if the writings had discussed it. This will attract a larger audience, as this topic is interesting and meaningful.
There are previous reviews on EDS published in the journal, which it would be advisable to mention in your manuscript.

Author Response

Response to Reviewer 3 Comments

Thank you for your insightful comments, which have been very valuable in improving our paper.

Point 1: Regarding the writing of the article, I miss well-defined research objectives and questions, and a more solid and clearly expressed methodology for systematic review.

Response 1: Thank you for your recommendation. We have added a clear research question to a newly constructed introduction. We have also clarified the purpose of our paper and its objectives in both the abstract and the introduction. We have also expanded the Materials and Methods section to more clearly express our methodology.

Point 2: Which would help in an interpretation of the results and more solid conclusions if the writings had discussed it. This will attract a larger audience, as this topic is interesting and meaningful.

Response 2: We acknowledge this weakness in the original version of the paper and have greatly enhanced the Conclusions section drawing on our reframed research question.

Point 3: There are previous reviews on EDS published in the journal, which it would be advisable to mention in your manuscript.

Response 3: Thank you for this valuable piece of advice. We have referenced three additional relevant articles published in the journal.

Thank you once again. We greatly appreciate the time and effort that you have dedicated thus far to providing feedback on our manuscript.

Reviewer 4 Report

The article raises an issue present in the field of education and, by extension, in ESD. The authors "aim to address the doubts about the impact of ESD by 1) shedding light on the methods and good practices for measuring this impact; 2) underlining the specificity of the data to be collected in the context of these methods of measurement; and 3) outlining the existing conclusions of impact studies dedicated to ESD that have served to highlight the limits and challenges for accurate measurement". However, in my opinion, none of these objectives is achieved in the article. 
As far as the theoretical framework is concerned, the authors ignore the trajectory of environmental education, its theoretical and practical contributions to which ESD is the heiress. Some of its previous contributions elaborated by UNESCO or in some of the summits promoted (Tblisi, e.g.) by this institution are collected without mentioning the paradigm of environmental education. 
The article starts from some questionable theoretical foundations. The contributions of environmental pedagogy are much richer than what is presented. See UNESCO, Tilbury, D. Benayas, J. etc.
 If the visions provided as Pluralistic and holistic are recovered from UNESCO documents, it should be cited. It contradicts the usual presence in the specialized literature of the systemic approach and interdisciplinarity.   On the other hand, environmental education already raised in 1977 in Belgrade Charte the need to train the students to enable them for action, it cannot be considered as a novelty. The omission of all this trajectory and its meaning in practice is not understood. 
In some points, there is no clarification between the internal evaluation of the programs themselves and the assessment of the impact of the program itself (point 3). It's valuable to take into account elements, such as coherence. The impact measurement measures are not specific to the ESD (point 3) and are presented generically.  There should be more linkage and contextualization in the ESD to respond to the proposed objectives. 
The article also ignores the change of model implied by the Bologna Process, especially in educational proposals design's, based on competencies, and in their evaluation. To propose traditional assessment methods as a starting point ignores the evolution that has taken place in the last decades in all educational stages. Section 4. 
The analysis of the specific literature on the evolution of the impact of ESD, one of the most relevant objectives, is restricted to a section that needs greater systematization to understand the contributions of these studies. 
The conclusions would merit a cross-cutting analysis of previous contributions. 

Author Response

Response to Reviewer 4 Comments

Thank you for your insightful comments, which have been very valuable in improving our paper.

Point 1: The article raises an issue present in the field of education and, by extension, in ESD. The authors "aim to address the doubts about the impact of ESD by 1) shedding light on the methods and good practices for measuring this impact; 2) underlining the specificity of the data to be collected in the context of these methods of measurement; and 3) outlining the existing conclusions of impact studies dedicated to ESD that have served to highlight the limits and challenges for accurate measurement". However, in my opinion, none of these objectives is achieved in the article.

Response 1: We thank you for your comment, which made us realise that our objectives needed to be reformulated to better reflect the purpose of our research. We have therefore clarified that our aim is to foster scaled ESD research and initiatives by offering a better understanding of the doubts that surround its potential impact and that we are doing this by 1) shedding light on the methods and good practices for assessing this impact [in section 3, and more precisely 3.1 (❡2), 3.2.1, 3.2.2]; 2) underlining the specificity of the data to be collected in the context of these methods of assessment [in section 4, and more specifically 4.2]; 3) outlining the existing conclusions of impact studies dedicated to ESD that have served to highlight the limits and challenges for accurate assessment [in section 5].

Point 2: As far as the theoretical framework is concerned, the authors ignore the trajectory of environmental education, its theoretical and practical contributions to which ESD is the heiress. Some of its previous contributions elaborated by UNESCO or in some of the summits promoted (Tblisi, e.g.) by this institution are collected without mentioning the paradigm of environmental education.

Response 2: We would like to stress that we are very thankful for this comment, which has led us to include clarifications regarding the articulations between Environmental Education (EE) and ESD. We agree that this trajectory is an important and exciting topic, and we believe that it is deserving of substantial development that perhaps goes beyond the scope of this specific manuscript.

Point 3: The article starts from some questionable theoretical foundations. The contributions of environmental pedagogy are much richer than what is presented. See UNESCO, Tilbury, D. Benayas, J. etc.

Response 3: We thank you for highlighting this. We have UNESCO and Tilbury in our references and we have used them both in our presentation of the theoretical foundations of ESD. We have tried to keep this part – already quite long in our Introduction – focused on the essentials in order to concentrate more on the scope targeted within the paper.

Point 4: If the visions provided as Pluralistic and holistic are recovered from UNESCO documents, it should be cited. It contradicts the usual presence in the specialized literature of the systemic approach and interdisciplinarity.

Response 4: We regret not making this reference to UNESCO clearer in the beginning and we thank you for raising the issue to us. We have made the necessary changes in the body of the text and in the reference list to correct this.

Point 5: On the other hand, environmental education already raised in 1977 in Belgrade Charte the need to train the students to enable them for action, it cannot be considered as a novelty. The omission of all this trajectory and its meaning in practice is not understood. 

Response 5: Thank you for this very welcome reminder. We have tried to efficiently make note of your recommendation in a newly added paragraph.

Point 6: In some points, there is no clarification between the internal evaluation of the programs themselves and the assessment of the impact of the program itself (point 3). It's valuable to take into account elements, such as coherence.

Response 6: We regret not making it clear enough that we have only focused on the assessment of the impact of the program itself and not on internal evaluation.

Point 7: The impact measurement measures are not specific to the ESD (point 3) and are presented generically. There should be more linkage and contextualization in the ESD to respond to the proposed objectives.

Response 7: We thank you for your comment. In sections 3 and 4 we tried to make the necessary linkages and relevant contextualization with ESD.

Point 8: The article also ignores the change of model implied by the Bologna Process, especially in educational proposals design's, based on competencies, and in their evaluation. To propose traditional assessment methods as a starting point ignores the evolution that has taken place in the last decades in all educational stages. Section 4. 

Response 8: We are very grateful for your highlighting of the Bologna Process, which we have added to our manuscript, though only briefly considering its relative remoteness from the core of our research (yet indeed an important topic to mention).

Point 9: The analysis of the specific literature on the evolution of the impact of ESD, one of the most relevant objectives, is restricted to a section that needs greater systematization to understand the contributions of these studies.

Response 9: We thank you very much for your comment, which we did our best to answer with an extended section on our Materials and Methods.

Point 10: The conclusions would merit a cross-cutting analysis of previous contributions. 

Response 10: We thank you for raising this key point, which we have done our best to answer in a newly extended conclusion.

Thank you once again. We greatly appreciate the time and effort that you have dedicated thus far to providing feedback on our manuscript.

Reviewer 5 Report

Measuring the impact of ESD is a very important topic. There is clearly a need for studies which summarise the ways impact is measured as well as develops the measuring further. I also think the manuscript makes many good observations about the topic. Thus, I think the manuscript is well worth publishing.

I think the paper could be improved. The beginning of introduction is clear and well referenced. However, on page 5 references are missing, even when the text mentions sources, such as Financial Times ranking factors from year 2020.

I think the second section should not be titled "Materials and Methods" as methods are discussed in the third section. I would include the rationale from the study from the end of the first section to this section and rename it as "rationale and research questions". I think this section as well as the next section should be written in first tense instead of using passive voice when you talk about what you did in this study.

In the third section there are several sub-sections in which you describe how the analysis was carried out. Some of the sub.sections contain barely any references. The number of bulleted lists are rather high in the later sections. I would like you to reconsider, is the use of bulleted lists needed in every case.

Fourth and fifth section presents the results. I think the discussion could be even more focused if you would structure it using the research questions. Fifth section could be reserved for making a short summary of main results and making conclusions. I do not think including Figures 3 and 4 really improve the paper at all. (If they are included, they should be marked as tables rather than figures.) Permission to use Miller's figure (Figure 2) should maybe be obtained or the figure should be redrawn because of the copyright issues.

In summary, I think the manuscript makes a relevant and important contribution to the discussion about the measuring the impact of ESD. However, the manuscript could be much improved by paying more attention to follow the structure and practices of scientific writing. I hope the writers will consider the advice I have given above to improve their manuscript.

Author Response

Response to Reviewer 5 Comments

Thank you for your insightful comments, which have been very valuable in improving our paper.

Point 1: The beginning of introduction is clear and well referenced. However, on page 5 references are missing, even when the text mentions sources, such as Financial Times ranking factors from year 2020.

Response 1: Thank you for pointing out our error. We have checked the document and added references where we found them to be lacking, including the reference to the 2020 Financial Times ranking factors.

Point 2: I think the second section should not be titled "Materials and Methods" as methods are discussed in the third section. I would include the rationale from the study from the end of the first section to this section and rename it as "rationale and research questions". I think this section as well as the next section should be written in first tense instead of using passive voice when you talk about what you did in this study.

Response 2: Thank you for your recommendation. We believe we have clarified your doubts by expanding our Introduction section and adding new elements to the Materials and Methods section, which has been reframed. We have looked into the use of the first person tense and, although not used in every case, we have made several modifications throughout the document.

Point 3: In the third section there are several sub-sections in which you describe how the analysis was carried out. Some of the sub.sections contain barely any references. The number of bulleted lists are rather high in the later sections. I would like you to reconsider, is the use of bulleted lists needed in every case.

Response 3: Thank you for your insight. While it is certainly true that some sections draw heavily on a single source, we have sought to include additional references. We have reworked several bulleted lists in the document to reflect your very useful comment, which we will take into account in the future as well.

Point 4: Fourth and fifth section presents the results. I think the discussion could be even more focused if you would structure it using the research questions. Fifth section could be reserved for making a short summary of main results and making conclusions.

Response 4: We acknowledge this weakness in the original version of the paper and have greatly enhanced the Conclusions section drawing on our reformulated research question. Thank you.

Point 5: I do not think including Figures 3 and 4 really improve the paper at all. (If they are included, they should be marked as tables rather than figures.) Permission to use Miller's figure (Figure 2) should maybe be obtained or the figure should be redrawn because of the copyright issues.

Response 5: Thank you for your useful comment. We have removed Figures 3 & 4. Upon further analysis of the use of Figure 2, we have decided to remove it as well.

Thank you once again. We have made a great effort to improve the structure and rigor of the document. We greatly appreciate the time and effort that you have dedicated thus far to providing feedback on our manuscript.

Round 2

Reviewer 1 Report

Dear authors,

You have done an excellent work in improving the manuscript with regards to my previous comments. I  am now satisfied with your work and wish you good luck with the publication!

Reviewer 4 Report