The Evolution of Educating for Sustainable Development in East Asia: A Bibliometric Review, 1991–2023

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article does an up-to-date state of the art in the topic of education for sustainable development in the East Asia area. One of the main contributions of the paper was to be not only that it reports the advances on this topic in that region but also that it compares it with other world regions, although this is less evident after reading the paper.

Main comments

The paper is clear and well written, no unreported issues detected, however regarding the limitations addressed by the authors:

1)      To avoid the “large number of Chinese scholars with the same names” it was not possible to use specific author codes such as ORCID or SCOPUS ID? This would have allowed to conduct author citation/co-citation analyses.

2)      As authors recognize the review neither synthesized findings nor evaluated the quality of individual studies. This makes the contribution of the review more superficial.

Minor issues

Line 70 repetition “Yet, scholars have yet”

Line 104 space missing “theliter-”

Line 406 is first time CSR appears and no explanation is given (I assume it refers to corporate social responsibility, but it should be stated.

Line 657 instead of “These findings and compatible” must say “These findings are compatible”

Author Response

Please see attached file with our response

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper documents and analyses the scope and topical foci of research on education for sustainable development (ESD) in East Asia; the authors make a good argument for that focus. The paper draws on just articles indexed in Scopus (a wider search, using other databases, would strengthen the claim to systematicity: the argument re analytical convenience is not a compelling one!). It finds that base consistent with the global literature, with content clustered around studies of environmental education and gradually over time expanded to include sociocultural, economic, and integrated perspectives on sustainability. It exposes three key themes: Environmental Sustainability in Education, Social Sustainability in Education, and Teaching and Learning for Sustainability (how does the first differ from the last? L128 suggests the first is classroom based while the second might not be, but I wonder if that’s not necessarily a useful distinction? The whole paper focuses on ESD, so I wonder if this set of distinctions should be ‘backseated’?). It finds that recent research focuses particularly on carbon/energy issues, technology-enhanced learning for sustainability, empirical research methods for studying ESD, and, to a lesser degree, studies of the economic pillar of sustainability in education. As such, it includes a number of important findings, but on a really quite restricted evidence base, and giving those limitations a higher profile would enhance the robustness of the paper. Particular points include:

·       L48. Note that ‘quality education’ (SDG4) does not necessarily include a focus on ESD – that depends entirely on the definition of ‘quality education’, so this argument needs refinement.

·       The paper draws on the UNESCO definition of ESD as ‘Education for sustainable development (ESD) gives learners of all ages the knowledge, skills, values and agency to address interconnected global challenges including climate change, loss of biodiversity, unsustainable use of resources, and inequality.

·       However, search terms only included stems related to education for sus- tainability and education for sustainable development, rather than e.g. citizenship, poverty, equity, which again significantly reduces the scope. Books and book chapters were also excluded – that seems very restricted.

·       However, the PRISMA protocol is included, which supports transparency.

·       L190 exposes the classification as adopting an environmental, socio-cultural, economic, or multi-pillar perspective toward sustainability, and that might be more helpful than the ‘educational’ distinctions focused on in l128 on.

·       L299: perhaps revisit the use of the word ‘exponential’? We need more than 2 data points to establish that!

·       Section 3.2: I’m not sure how useful this analysis is: as the authors suggest, it’s virtually self-fulfilling given the search terms, and distinguishing between ‘ESD’ and ‘education for sustainable development’ seems almost perverse. I suggest deleting this section – it seems to fall into the trap of ‘analysing because one can’, without adding significant meaning, though some of the finer distinctions are worth noting, e,g. by phase of education.

·       Figure 6: I’m not sure the titles here are all helpful: within education, perhaps environmental sustainability, social sustainability and ‘process and structural approaches’ (or something better)? Each of the first necessarily involves teaching and, hopefully, learning, whereas not all the red cluster themes do, directly.

·       Some excellent visuals, that really support understanding of evolution and structures within the field – although care needs to be taken, e.g. I Figure 7, that the diagram is comparing consonant variables – I’m not sure the nature of research tools belongs in the same diagram as the focus of the research? The methodological approach probably deserves its own focus, since in important ways it constrains the sorts of learning from the associated research.

·       Overall, the limitations are significant, including in relation to research quality. Given that limitations, taken in conjunction with the use of just Scopus and a very restricted set of search terms, the authors should I think frame their findings much more contingently – ‘indicative’ of wider findings rather than anything stronger.

·       L546-547: ‘the broad purpose of this review was to identify distinctive features of the knowledge base on ESD in East Asia’ . I’m not sure it does that – if the authors think it does, perhaps they could usefully identify where and how.

·       L552: ‘niche’ topic? Clearly not – it has become very mainstream, so perhaps use ‘formerly-niche’?

·       L589-591: ‘Indeed, the exponential growth of this literature has rendered obsolete the data reported in bibliometric reviews of ESD published in 2018, 2019, and 2020 [13, 14, 16, 21]’. Why? Use of those might establish the claimed ‘exponential’ number of publications – please use that term only if strictly accurate.

·       L598-604: care needed – this suggests that a better representation would be achieved if e.g. PNG published in numbers comparable to China, which is clearly unreasonable as an expectation. However, it might be constructive to point explicitly to the important perspectives that are likely being missed if, e.g., nations particularly vulnerable to climate change are under-represented in the knowledge base.

·       L662-665. I think there are other important implications of this: scholars, and policy makers, cannot possibly continue to engage with such a volume, let alone if patters of publication persist. That suggests that quality, and so selection for publication, needs to take a higher profile. The point about a need for enhanced cross-nation collaboration is a really important one!

·       Finally, I suggest strengthening the profile for the finding about empirical studies: that is important, and while such studies are usually small, synthesising a large number in robust ways is important for answering ‘what works?’

 

 

 

Author Response

Please see our responses to the comments in the attached file.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The paper seems somewhat improved, but the response is really not compelling. 'Because other studies have done this' or 'the technology does not allow for this' are not robust or scholarly arguments, so that there is room for significant further strengthening with fairly minimal further revisions: scholarly work is constructively critical, and this  paper could still easily be considerably strengthened in that regard. These suggestions are offered in the hope that this paper can become a really strong one in the field - I hope they help.

For example: 

 

You say you wanted to compare with previous publications - fine. But that is no reason not to critique the very obvious (and profound) limitations of using just the two search terms. 

 

l113-114: unlike Google Scholar, Scopus can export bibliographic meta-data in a format that can be analyzed by bibliometric software programs. Perhaps use that productively to suggest ways in which another approach could complement the 'press the button' approach used here? WoS, Scopus and GS do indeed target different sets of journals, and patterns in one will not necessarily be reflected in the others - so the limitations do need further attention. 

 

Section 3.2: you do not I think address my argument that this section adds nothing meaningful to the paper (and in fact, weakens it by appearing to include software output because one can, not because it is useful to do so).

 

l403: 'the red cluster focuses on teaching methods about sustainable development' - and actually, much else. Similarly, for the other two clusters. The paper gives the impression of being servant of the software, not master. With this, or any reasonably sophisticated  software, implicit as well as explicit ideas need considering: the researcher has to add in a brain! So your response to my point 6 rather suggests you are content not to engage intelligently with the software output, and the paper would be considerably strengthened if you did (I appreciate other similar papers do not - yours can be stronger!). Are there ways of removing those other terms, or otherwise backgrounding them? As I said before, research methodology does not fit in any of them, so the researchers need to critique that and devise a constructive approach to tackling it. I didn't, if you read my comments, suggest ignoring it, but rather, giving it a separate treatment, since as you say, the shifts in methodological approaches are themselves telling. 

 

My point 12: you say 'this is not a problem that can be solved with current technology'. Maybe not, so a constructive response finds another approach to this subtle problem (pro rata expectations are not entirely valid, since the range of voices should be heard, and especially those most affected by the issues)

 

l546: 'Thus, it is unlikely that this limitation greatly impacted the  results'. This really is a non-sequitur: what you have gives a lower bound to recent publications, but there is no reason why there should not be 2 or 3 times as many out there given your methodology - or conversely, that previous single-engine approaches did not massively  underestimate numbers and they have now moved (as publications tend to) to 'clump' in given publications captured by your methods. So a less definite statement is needed. (Do you, incidentally, know where this paper will be listed?)

Author Response

Response to Reviewer 2: Round 2

'Because other studies have done this' or 'the technology does not allow for this' are not robust or scholarly arguments, so that there is room for significant further strengthening with fairly minimal further revisions: scholarly work is constructively critical, and this  paper could still easily be considerably strengthened in that regard. These suggestions are offered in the hope that this paper can become a really strong one in the field - I hope they help.

We have addressed all the points that the reviewer has raised in this response memo. There does remain one point of disagreement concerning the value of the section on co-word analysis. For this section, we have restated the rationale for why and how we did. We do not feel that the value of the article rises or falls on this section, but feel that we would wish to keep it in the paper.

1. For example: You say you wanted to compare with previous publications - fine. But that is no reason not to critique the very obvious (and profound) limitations of using just the two search terms. l113-114: unlike Google Scholar, Scopus can export bibliographic meta-data in a format that can be analyzed by bibliometric software programs. Perhaps use that productively to suggest ways in which another approach could complement the 'press the button' approach used here? WoS, Scopus and GS do indeed target different sets of journals, and patterns in one will not necessarily be reflected in the others - so the limitations do need further attention. 

There is no “perfect” database for bibliometric analysis, but either Scopus or WOS are the standards in bibliographic analysis (Pranckutė 2021).  They have wider coverage than most databases, and substantial (but not complete) overlap with each other (Gusenbauer 2022; Martin-Martin et al 2018, 2021).  In particular, Scopus is known to work well for various sustainability issues (e.g., Guo et al 2021), and cites [13-17, 21], already cited in the article. 

 The goal of capturing every relevant goal is indeed a worthy one that all authors conducting systematic reviews should seek to strive for. This is perhaps most critical in a meta-analytic review where the omission of several potentially relevant and important articles in a sample of 25 or 50 studies could have a significant impact. However, in bibliometric reviews synthesize meta-data extracted from thousands of studies rather than empirical findings. Thus, the impact of “omitted documents” is quantitatively different. The exception is where the authors are reporting a table of top-cited authors or documents, where such omissions could yield a distorted result (indeed, we have seen this in multiple bibliometric reviews). However, for the “high ground view” of overall patterns, this approach using Scopus seems to work well, even if a few little details are missed.

 Nonetheless, as noted above, we agree that the authors should strive toward providing a comprehensive view of the topic or field. In order to gain empirical insight into this issue, we ran a similar search in WoS in order to compare the results. The goal was not to repeat all of the analyses, which would have necessitated a time-consuming article-by-article screening for relevance before reanalyzing the data in VOSviewer. Screening the Scopus list took 10 hours (not quite press the button). Instead, we examined several descriptive analyses in WoS to see if the data trends were similar to the Scopus trends reported in the article.  Subsequently, we slightly revised the paragraph at the start of section 2.1, and inserted the following text into the Limitations.

 “Nonetheless, recognizing the limitations of relying on a single database, the authors conducted a similar search in the Web of Science and compared the results of several descriptive analyses with those obtained from Scopus. Although Scopus generated substantially more documents, the two databases yielded a similar pattern of results. For example, the Scopus and Web of Science publication trajectories were very similar. The scholars authoring the most documents in this literature were essentially the same and ranked similarly. This pattern also described the geographical distribution of the literature. The top five contributors were in the same rank order, just with different numbers of documents. Of course, these similarities do not discount the possibility that slightly different findings could emerge from the analysis of a merged dataset, but this comparative enhanced confidence in the findings reported in the article. Nonetheless, this suggests a challenge for developers to expand the utility of the bibliometric software.”

 In summary, we believe note that both Scopus and WOS are fairly (not perfectly) representative of the higher quality international literature, even though they do not fully overlap.  Two different representative samples tend to yield similar results on large datasets, even though the parameter estimates are not exactly the same. The quick test we did on WOS confirms this.

 Gao, S., Meng, F., Gu, Z., Liu, Z., & Farrukh, M. (2021). Mapping and clustering analysis on environmental, social and governance field a bibliometric analysis using Scopus. Sustainability, 13(13), 7304.

Gusenbauer, M. (2022). Search where you will find most: Comparing the disciplinary coverage of 56 bibliographic databases. Scientometrics, 127(5), 2683-2745.

Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177.

Martín-Martín, A., Thelwall, M., Orduna-Malea, E., & Delgado López-Cózar, E. (2021). Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations. Scientometrics, 126(1), 871-906.

Pranckutė, R. (2021). Web of Science (WoS) and Scopus: The titans of bibliographic information in today’s academic world. Publications, 9(1), 12.

 Section 3.2: you do not I think address my argument that this section adds nothing meaningful to the paper (and in fact, weakens it by appearing to include software output because one can, not because it is useful to do so). For example: l403: 'the red cluster focuses on teaching methods about sustainable development' - and actually, much else. Similarly, for the other two clusters. The paper gives the impression of being servant of the software, not master. With this, or any reasonably sophisticated  software, implicit as well as explicit ideas need considering: the researcher has to add in a brain! So your response to my point 6 rather suggests you are content not to engage intelligently with the software output, and the paper would be considerably strengthened if you did (I appreciate other similar papers do not - yours can be stronger!). Are there ways of removing those other terms, or otherwise backgrounding them?

Possibly there is some disagreement on “focuses on” vs. “includes”.  It is obvious that all three of the color clusters include lots of things, but they do have a focus on what we named them.  Saying “the red cluster focuses on teaching methods about sustainable development” seems more informative than “the red cluster has lots of different topics, and includes some nodes on teaching methods about sustainable development”, or earlier, “the green cluster focuses on lots of different topics, and includes nodes on environmental education”, etc. 

 The more general issue is whether there is any value to giving an overview of topic areas in the ESD literature.  For true experts in the field who have followed the literature in detail for a number of years, possibly not.  But sustainability and ESD is a rapidly growing field (Fig. 2 in the article), new researchers are coming in rapidly.  An overview of the patterns in general topic areas seems to have value for most readers. 

 For example, our PhD students use such bibliometric articles to obtain an overview of key trends, thematic lines of inquiry, and current topics of interest. Bibliometric temporal maps are useful for discouraging a research topic that shows up with a big node in the old side of the temporal map, if there seems to be nothing new in the student’s initial thinking about a potential topic.

 We prefer not to eliminate this discussion.  Most journals quite rightly do not like figures/tables with little discussion supporting them.  The first author recently completed a critical evaluation of bibliometric reviews in education. The issue of what the reviewer called “not applying your brains to the chart/table/map” was one of the key problems that we critiqued. However, we respectfully disagree that this applies to this analysis.

 We revised and condensed the early part of this section.  Mainly, we took out discussion about some of the mechanics and simple listing of node names.  We have not changed the latter part of the section – it is not describing mechanics or data so much, but discussing what the data means for East Asian research on these issues.

 Third author here: I once didn’t think bibliometric articles were very useful.  I cited them occasionally, mainly because reviewers would criticize me for not citing them.  But now I see that they are sometimes pretty good at giving an overview of whatever field they cover (So, they are useful in introductions to articles, not so useful later when we get into conceptualization.) 

 Partly the bibliographic articles have gotten a little better now than the early ones, partly I simply understand their particular role in the literature a little better.  They are quite useful in getting new PhD students and junior faculty up to speed quickly as they start research in a field.  They are also useful for old people like me when I want to take some of what I know very well into a slightly new area. 

 But however useful bibliometric work is for mapping what is going on in a field, I am still quite skeptical that it can be used for major conceptual breakthroughs.  When we are able to examine conceptual issues in the whole article, possibly.  We already do that now in building conceptual frameworks from a set of articles that have useful relevant discussion.  Software can handle big data better than brains, applying software to big full-article datasets might give some additional conceptual insights. 

 However, except for open access, at present only keywords and abstracts are available from most major publishers.  (and using only OA would invite its own potential bias problems.)  Keywords, abstracts, authors, citations, etc., are sufficient for general overview of the literature, networks of authors, co-citation, etc., but are not sufficient for getting into much depth in conceptualization. We do publish quite a lot on conceptualizing issues in education and sustainability, as well as some bibliometric work, but we don’t yet see how the bibliometric work can be used for deeper conceptualization with current data access. We would be very happy to get some additional thinking from the reviewer on this.  Maybe we are missing something.

3. As I said before, research methodology does not fit in any of them, so the researchers need to critique that and devise a constructive approach to tackling it. I didn't, if you read my comments, suggest ignoring it, but rather, giving it a separate treatment, since as you say, the shifts in methodological approaches are themselves telling.

We respect your opinion and agree that many bibliometric reviews lack any useful analysis beyond the presentation of the charts and maps, as if they are the point of the paper. However, we disagree that this is the case with this paper or section 3.2. Before executing the co-word analysis, we developed a thesaurus file with 205 discrete keywords. This entailed considerable decision-making as to what ought to be included on the map, or not. We could have excluded the research keywords but chose not to because “sometimes” they highlight useful patterns, despite not being topics per se.

In Figure 6, the research methods did not group as a cluster; instead, they appear as individual nodes associated with the topics they have been most frequently used to study. We chose not to eliminate them because their location on the map could be of interest to readers. This contrasted with the following analysis (Figure 7) where their temporal emergence highlighted trends in the use of advanced quantitative designs and methods research methods being used to study the impact of ESD in East Asia.

 I think part of the reason for different viewpoints here is disagreement over clustering methods.  The reviewer seems to want methodology articles to group together, rather than being scattered across the different clusters. VOSViewer uses a version of modularity-based clustering, which is common in much bibliometric software (Van Eck & Waltman, 2023; Waltman et al., 2010; Yan et al., 2011).  Essentially, rather than clustering based on similarity of the characteristics of the items, it clusters based on the network of connections between items. 

 It is quite right that methodology articles are scattered across the map in different clusters.  But this not a failure of the software to identify clusters, and not an indication that our interpretation of the green, red, blue clusters is wrong.  Rather, it indicates that the various topics about methodologies do not form a coherent set, i.e., there is little network of connections between them.  Specific methodologies tend to be associated with specific sustainability or education topics.  Figure 7 was already changed from the original submission to the revised figure 7 to show this more clearly.  In this revision, we added brief comment about this on p. 15, lines 508 and then line 534.

 Van Eck, N. J., & Waltman, L. (2023). VOSviewer version 1.6.19. University of Leiden.

Waltman, L., Van Eck, N.J., & Noyons, E.C.M. (2010). A unified approach to mapping and clustering of bibliometric networks. Journal of Informetrics, 4(4), 629–635.

Yan, E., Ding, Y., & Jacob, E. K. (2012). Overlaying communities and topics: An analysis on publication networks. Scientometrics, 90(2), 499-513.

 My point 12: you say 'this is not a problem that can be solved with current technology'. Maybe not, so a constructive response finds another approach to this subtle problem (pro rata expectations are not entirely valid, since the range of voices should be heard, and especially those most affected by the issues)

It seems that we may have misinterpreted your earlier comment #12. We added the following paragraph, highlighted on page 17. Hopefully this is more on-target.

Nonetheless, despite the significant increase in the size of the East Asian literature, the under-representation of studies from some of East Asia’s smallest and poorest societies is problematic. This suggests that we know the least about the countries with the lowest capacity to address sustainability challenges through education. Future integrative and narrative synthesis reviews could be employed to highlight the challenges faced by these societies.

5. l546: 'Thus, it is unlikely that this limitation greatly impacted the results'. This really is a non-sequitur: what you have gives a lower bound to recent publications, but there is no reason why there should not be 2 or 3 times as many out there given your methodology - or conversely, that previous single-engine approaches did not massively underestimate numbers and they have now moved (as publications tend to) to 'clump' in given publications captured by your methods. So a less definite statement is needed.

Taking note of your concern, we deleted the last sentence: “Thus, it is unlikely that this limitation greatly impacted the results.” We do however, believe that the difference in the size is not due to under/over-estimation of the size of the databases extracted in prior reviews. Rather, we continue to believe that it is due to the rapid increase observed in the size of the sustainability literature. Interestingly, this trend is consistent in reviews published in Scopus and WoS. Moreover, quick Scopus search of the general sustainability literature found that almost 70% was published in the past six years.