Review Reports - Tracking Biliteracy Skills in Students Attending Gaelic Medium Education: Effects of Learning Experience on Overall Reading Skills

Round 1

Reviewer 1 Report

Consider having the paper, or at least parts (e.g., Introduction, Study 1, Methods) edited by someone with specific focus on Standard Academic English. Just a few examples:

Lines 28-29: ". . . some degree of dyslexia with lifelong problem" seems awkward. Maybe reword.

Line 33: "who" should be changed to "that"

Line 34: "isolate" should be "isolated"

Line 34: "little" should be changed to "few"

Line 34: "to consistent" should be changed to "for consistent"

Line 79: should be "Graded Intergenerational Disruption Scale"

Line 129: "That is . . ." is a fragment; also, avoid split infinitives in formal writing

Line 234: "narratives" not capitalized before "task" (or anywhere else), but "Humour" is capitalized in line 248 before "task" and nowhere else. Just a note on consistency.

Line 296: "Reading for All" . . ."All" is capitalized here but not in other places (e.g.,261, 284, 285, 299, 342, 436, 437, 507, 576). Also, italicized in some places and not in others.

Line 356: "In term" should be "In terms"

Line 639: Comma needed after "expected"

Line 691: "While whether"

Also:

Line 22: For keywords, "Gaelic, medium education" might be better without the comma - "Gaelic medium education" though "Gaelic" might be listed separately.

Overall, the study is well organized and detailed.

-The case is clearly made for why there is a need for the study (e.g., GME teachers perceive their students do better, there is a need for more literacy research in GME settings).

-The background information is sufficient to understand the context (e.g., history of GME, discourse comprehension/factors affecting reading for understanding)

-The methods section is clear and easy to follow, and the procedures section was very clear and succinct; it would be interesting (though not necessary) to know more about the background of the Gaelic version for the Reading for all (e.g., Is it a translation? Is it written by native speakers? Is it inclusive of Gaelic cultural information or information relevant to the test takers?).

-The scatter plots might be harder to see in print, but as this is open access, readers are able to zoom in

-Clear description and organization in Study 2 as well; one question - if some of the same assessments were used on the same participants, could priming have affected the scores?

-Clear discussion

The references at the end are not all in alphabetical order (e.g., McEwan-Fujita)

Author Response

Thanks for your careful reading of the manuscript. Your comments have been very precious to make it more readable. Thanks for the positive comments on the study. We have made all the grammatical changes suggested and we had a second full read (two of the authors are native speakers, hopefully this should be enough!).

Lines 28-29: ". . . some degree of dyslexia with lifelong problem" seems awkward. Maybe reword.

This has been reworded in “ some form of dyslexia with long-term effects”.

Line 33: "who" should be changed to "that"

Changed. Thanks

Line 34: "isolate" should be "isolated"

Changed. Thanks

Line 34: "little" should be changed to "few"

Changed. Thanks

Line 34: "to consistent" should be changed to "for consistent"

Changed. Thanks

Line 79: should be "Graded Intergenerational Disruption Scale"

Changed. Thanks

Line 129: "That is . . ." is a fragment; also, avoid split infinitives in formal writing

Changed. Thanks

Line 234: "narratives" not capitalized before "task" (or anywhere else), but "Humour" is capitalized in line 248 before "task" and nowhere else. Just a note on consistency.

Changed and made consistent across occurrences.

Line 296: "Reading for All" . . ."All" is capitalized here but not in other places (e.g.,261, 284, 285, 299, 342, 436, 437, 507, 576). Also, italicized in some places and not in others.

Changed and now consistent. Thanks a lot.

Line 356: "In term" should be "In terms"

Changed, thanks.

Line 639: Comma needed after "expected"

Line 691: "While whether"

Also:

Line 22: For keywords, "Gaelic, medium education" might be better without the comma - "Gaelic medium education" though "Gaelic" might be listed separately.

Very good suggestion, thanks. We add "Gaelic" and remove the comma.

Overall, the study is well organized and detailed.

-The case is clearly made for why there is a need for the study (e.g., GME teachers perceive their students do better, there is a need for more literacy research in GME settings).

-The background information is sufficient to understand the context (e.g., history of GME, discourse comprehension/factors affecting reading for understanding)

We have included a short paragraph, mentioning that the test has been conceived in Gaelic first and then adapted to English. Thanks for asking, this was an important detail overlooked by the authors.

-The scatter plots might be harder to see in print, but as this is open access, readers are able to zoom in

-Clear description and organization in Study 2 as well; one question - if some of the same assessments were used on the same participants, could priming have affected the scores?

The scores of study 1 were adopted for study 2. Participants were not retested.

-Clear discussion The references at the end are not all in alphabetical order (e.g., McEwan-Fujita)

Thanks for looking at this

Reviewer 2 Report

First I would like to congratulate the authors on undertaking this important research. This is a very valuable project; however, the paper itself needs a lot of work. The fundamental problem is that there is almost no discernible narrative thread. The paper feels like a collection of references and data thrown together without a guiding purpose. I am sure that that is not how it is in the minds of the authors, but they need to do much more to convey to their readers what they are trying to show, why they made the choices they made, and how the different parts of the paper all tie together in a coherent story. In short, the authors need to tell us a clear story. What is the point of this paper? Anything that does not work directly to further that story needs to be rewritten or cut.

The introduction is case in point. The authors run through a list of references to other research that is not well explained and that does not appear (as presented) to relate to the study directly. How are these different theories and concepts briefly noted related to the research that follows? And how are they related to each other? Did they inform the design of the assessment in some way? Take, for instance, the reference to Joshua Fishman’s work. As a sociolinguist, I am very familiar with his Graded Intergenerational Scale for Threatened Languages, with what the authors mean by “stage 6”, and with the debates surrounding these concepts, but the mention these ideas is so brief and decontextualised, that I am sure that they will be mostly lost on education researchers who are not familiar with Fishman’s work, just as the brief references to literacy research were largely lost on me.

I believe the authors are using this reference to argue for the importance of immersion education in language revitalisation, and thereby, the importance of their own study, but they need to make this clearer, and drop any jargon that does not relate to their point. There really is no need to mention the Graded Intergenerational Scale or stage 6 to make their point with reference to Fishman’s work. They should use this space to briefly but clearly explain why Fishman thinks home/community transmission of a threatened language is so important, and why others have argued that successful immersion education also is critical. They should assume that they are writing for a broad readership of various specialists in a range of fields, and make sure that every concept and theory is well explained and woven into a strong narrative that supports the purpose of their paper.

They also need to explain their sampling in much more detail, and discuss how their sampling might impact their measurements and the validity of their results. Where did the children come from: Highlands, Islands, Lowlands, urban, rural? The location of the schools might have a significant impact on the results, and they should be able to do this without jeopardising anonymisation. Their pool of adults in both sections of the study appear to be convenience samples, and they offer no evidence that their samples are representative of any particular group: proficient adults native speakers educated in English, proficient adult native speakers educated in Gaelic, proficient adult new-speakers who acquired Gaelic as adults, proficient adult new-speakers who acquired Gaelic in primary or secondary, or adults still learning at various levels. If they are going to use these adults at all, they need to give much more detail of who they are, and how representative they may be of different classes of Gaelic speakers.

I am particularly concerned about the tests of adults in general. They need to be much more fleshed out in the first instance, but also, it is not at all clear how they relate to the tests of school children. They feel tacked on. Is this perhaps a separate paper? Would it be better to focus on the children in this paper and write a separate, much more developed paper about the adults? That might help clarify their narrative.

I also believe that there is an epistemological question at the centre of this paper that is not sufficiently addressed. The authors claim that their new assessment tool allows them to compare reading ability between Gaelic and English, but do not discuss how this is even possible. How do you match reading difficulty in one language to another? This is a very complex, fraught theoretical and practical question that the authors do not address at all. The authors need to give much more detail in the methods for how they know that the Gaelic and English versions of their assessment tool are measuring the same things.

In general, the purpose of the paper is better explained in the discussion than the introduction. They should try to tell us their story three times. They should make sure that the aims of the paper are clear from the outset, tie everything in the body of the paper to these aims, and then summarise how they have achieved these aims at the end.

This is more a note for the editors, but I should say that I am not a statistician, so to a degree, I had to take the very involved but also terse and unelaborated discussions of the different statistical tests on faith. It is be important that at least one of the reviewers at some point in the process has expertise in designing statistical tests to properly evaluate these sections.

650-63 If the authors can justify it (above), this is the most important result of this study, and it would be valuable to extend the discussion in this section to include more context and other research on GME outcomes, for instance:

Dunmore, Stuart S. (2017). “Immersion education outcomes and the Gaelic community: identities and language ideologies among Gaelic medium-educated adults in Scotland.” Journal of Multilingual and Multicultural Development, 38(8), 726–741.

Nance, Caire. (2014). “Phonetic variation in Scottish Gaelic laterals.” Journal of Phonetics, 47, 1-17.

O’Hanlon, Fiona (2012) 'Celtic-medium education and language maintenance in Scotland and Wales: language use, ability and attitudes at the primary to secondary school stage,' In Nancy R McGuire & Colm Ó Baoill (Eds.), Rannsachadh na Gàidhlig 6, Obar Dheathain: An Clò Gàidhealach, 323–354.

742-7 This is a particularly clear example of the problems with the sampling. Given a sample of adult learners at various stages of their language-learning project, it is not surprising that many of them don’t read as well in Gaelic as they do in English. The authors have only shown that Gaelic learners are learning Gaelic. They would need to limit their sample to proficient new speakers and then assess the difference in Gaelic and English reading ability. But then, what is “proficient” and how would one access who is fully proficient and who is still learning? The authors are caught in a tautology. It might be more meaningful to assess new speakers who have finished a particularly advanced Gaelic course, or who use Gaelic in a particular way, Gaelic professionals perhaps, or teachers, but without more information about all their samples, it is difficult to say what their conclusions actually mean.

Some issues of language:

It is purely an issue of style, but “kids” feels very informal, and I might use “children”;

The word “class” confused me occasionally, since it is often used next to SIMD in the analysis. Might “grade” be clearer, or “school-year” perhaps, to better distinguish school class from social class;

I would recommend arriving at a single set of names for the children in GME and EME and sticking to them in all text, tables and figures. Using “Gaelic” and “English” as a shorthand for these two groups is confusing since the authors are also measuring Gaelic and English reading ability, and it is also inaccurate since all GME students also speak, and are learning, English. The authors could simply use GME and EME, or they could define even more accurate names at the beginning, such as “bilingually-educated children” (BEC) and “monolingually-educated children” (MEC) and stick to those acronyms throughout.

Proofing errors:

66 in order to [reach?] full fluency and literacy

117 estimations on [of?] pupil attainment

396 Reading for All test. [the italics missed the last L in “All”]

I hope I haven’t discouraged the authors, because this really is an important piece of work, and with careful revision, I am sure that it will make a valuable contribution to the literature, and I am also very pleased to see that this assessment is being developed. I am certain that it will be of real, practical use to Gaelic educators as they work to provide the best possible education for their students.

Author Response

Q1: First, I would like to congratulate the authors on undertaking this important research. This is a very valuable project; however, the paper itself needs a lot of work. The fundamental problem is that there is almost no discernible narrative thread. The paper feels like a collection of references and data thrown together without a guiding purpose. I am sure that that is not how it is in the minds of the authors, but they need to do much more to convey to their readers what they are trying to show, why they made the choices they made, and how the different parts of the paper all tie together in a coherent story. In short, the authors need to tell us a clear story. What is the point of this paper? Anything that does not work directly to further that story needs to be rewritten or cut.

The introduction is case in point. The authors run through a list of references to other research that is not well explained and that does not appear (as presented) to relate to the study directly.
How are these different theories and concepts briefly noted related to the research that follows? And how are they related to each other? Did they inform the design of the assessment in some way? Take, for instance, the reference to Joshua Fishman’s work. As a sociolinguist, I am very familiar with his Graded Intergenerational Scale for Threatened Languages, with what the authors mean by “stage 6”, and with the debates surrounding these concepts, but the mention these ideas is so brief and decontextualised, that I am sure that they will be mostly lost on education researchers who are not familiar with Fishman’s work, just as the brief references to literacy research were largely lost on me.
I believe the authors are using this reference to argue for the importance of immersion education in language revitalisation, and thereby, the importance of their own study, but they need to make this clearer, and drop any jargon that does not relate to their point.

There really is no need to mention the Graded Intergenerational Scale or stage 6 to make their point with reference to Fishman’s work. They should use this space to briefly but clearly explain why Fishman thinks home/community transmission of a threatened language is so important, and why others have argued that successful immersion education also is critical. They should assume that they are writing for a broad readership of various specialists in a range of fields, and make sure that every concept and theory is well explained and woven into a strong narrative that supports the purpose of their paper.

A1: Thanks a lot for your comments. We have now deleted the GIS reference and hopefully made the main aim of the study clear. This is a short report to present data on the development of a reading comprehension test and not much to discuss sociolinguistics implications or models. We have made this point neat in the introduction and focused only on the literature on biliteracy and reading comprehension in particular. Thanks again for pointing this out. This is a psycholinguistics study aiming at developing a reading assessment for Gaelic and not to capture the reading abilities of all GME school population. Further studies will require a larger data set.

Q2 They also need to explain their sampling in much more detail and discuss how their sampling might impact their measurements and the validity of their results. Where did the children come from: Highlands, Islands, Lowlands, urban, rural? The location of the schools might have a significant impact on the results, and they should be able to do this without jeopardising anonymisation.

A2: This is a pilot study, as now more data across Scotland and Ireland (with an adapted version of the test) will be collected in the next couple of years. We are not expecting mayor differences due to the location as the current data recorded are in line with data from other populations. We have included a short paragraph stating that this is a pilot study on a sample of students from two GME schools, one of them being the largest provider in Scotland. Further evidence will be collected in the future to consider if location of the school has an impact on the results.
Thanks for the comment.

Q3: Their pool of adults in both sections of the study appear to be convenience samples, and they offer no evidence that their samples are representative of any particular group: proficient adults native speakers educated in English, proficient adult native speakers educated in Gaelic, proficient adult new-speakers who acquired Gaelic as adults, proficient adult new-speakers who acquired Gaelic in primary or secondary, or adults still learning at various levels. If they are going to use these adults at all, they need to give much more detail of who they are, and how representative they may be of different classes of Gaelic speakers.

A3: We have now considered to remove the adults’ sample from this study and to focus on the GME students for this paper. Thanks for the comment. More data on adults soon!

Q4: I am particularly concerned about the tests of adults in general. They need to be much more fleshed out in the first instance, but also, it is not at all clear how they relate to the tests of school children. They feel tacked on. Is this perhaps a separate paper? Would it be better to focus on the children in this paper and write a separate, much more developed paper about the adults? That might help clarify their narrative.

A4: Yes! Thanks a lot for this comment. We have now excluded the adults from this study and managed to collect more systematic data for another paper. Thanks again, this was an excellent suggestion.

Q5: I also believe that there is an epistemological question at the centre of this paper that is not sufficiently addressed. The authors claim that their new assessment tool allows them to compare reading ability between Gaelic and English, but do not discuss how this is even possible. How do you match reading difficulty in one language to another? This is a very complex, fraught theoretical and practical question that the authors do not address at all. The authors need to give much more detail in the methods for how they know that the Gaelic and English versions of their assessment tool are measuring the same things.

A5: The study has been conceived and designed following a well tested model of reading comprehension, the construction-integration model (Kintsch, 1988). This is presented in paragraph 1.2 of the present study. This model has been tested in several populations and languages. The current study is on reading comprehension and not reading aloud (a very different component of the reading abilities) and there is no reason to assume different discourse models for different languages. For this reason the stories have been developed and designed with a focus on daily activities with no culture biases and then adapted to English (and other languages) matching several language properties. Since the reviewer has expressed a clarification on this matter, we have included an extra paragraph in section 1.2 but we are confident that this is not an issue for this study, as the main aim was to develop a reading assessment for Gaelic first and then to look at competences across languages.

Thanks again. We hope the paper is now suitable for publication in the open access journal. A follow up paper with data from adults will be drafted to better discuss the matter of adults learners.

Round 2

Reviewer 2 Report

The authors have made significant additions to the text and I feel that I can now recommend the paper for publication.

Author Response

We are extremely grateful to the reviewer for the accurate reading and suggestions. We have now followed all the recommended changes highlighted in the pdf version. The manuscript is not cleared and more readable.

Thanks.