A Review of Meta-Analyses of Correlation Coefficients on L2 Reading Comprehension

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

 A Review of Meta-Analyses of Correlation Coefficients on L2 2 Reading Comprehension

 

 

The study goal of this meta-analyses was to review 14 meta-analysis of primary studies that have investigated the relationship between L2 reading comprehension and reading related variables, published between 2011 and 2023.

 

The structure is clear, and it is easy to read and understand.

 

Nevertheless, there are some issues that should be addressed.

 

About the data bases used to find the articles another specialized education database such as ERIC should have been included.

 

The search string to find the reviewed articles is not indicated and there is not a guide for the systematic review such as for example PRISMA. The review question, inclusion criteria and search strategy are not included either.

 

Table 1. Summary of Study Features:

I miss the date of publications for each study presented in this table. The study by Park was published in 2010 so it does not meet the criteria to be included in this review (between 2011 and 2023)

 

What does it mean that the publication type was NR (Not reported)?  I assume that a study cannot be reviewed if it is not published. I am sorry but I do not understand this concept.

 

Table 2. Summary of Study Features Continued:

The L1 and L2 languages are not reported in 4 studies out of 14, and 7 studies out of 14 the L1 and L2 languages were various. So, the results could be underpowered because languages are very different in characteristics and compare studies with varied languages and in some cases studies that do not report the language, should be noted as a limitation of this study. Besides, studies that do not identified the L1 and L2 languages (4 of them) should not be included in this review.

 

Table 3. It would be interesting to indicate some of the test used to measure the indicated variables (i.e. phonology, orthography, etc.) how they were treated or measure in each study, because the variation in measures implies variation in results.

 

The results with the comparisons of the relationship between L2 reading comprehension and reading related variables, seem to compare the studies by Author & Author 1, 2 with other studies. Giving a high preponderance to these studies compare to others.

 

Another table should be included with the quantitative results of the reviewed studies to better understand the trends.

 

The discussion is centered on the overlapping among primary studies across meta-analysis and this was not a goal of the present review.

 

The conclusion that the “value of an independently conducted meta-analysis heavily outweighs that of a replicated meta-analysis” is not a research goal of the present study and it is not treated and well-founded. So, it should be omitted.

 

The comment on the study by Lee et al. is not understood in the context of a discussion.  Also, the remark on the 2 of the 14 meta-analyses centered on L1 Chinese participants only is not well understood either.

 

A clear limitation of this study is that the 14 reviewed studies are so different among them that it is not possible to reach a clear conclusion.

 

The Discussion should be rewritten and oriented to the study goals.

Author Response

Dear reviewers, thank you very much for your astute and insightful comments which helped us substantially revise our manuscript. Our responses to each of your comments are provided in blue in this document and the revisions responding to your comments also appear in blue print in the file containing the revised manuscript.

Author & Author

Reviewer 1:

The study goal of this meta-analyses was to review 14 meta-analysis of primary studies that have investigated the relationship between L2 reading comprehension and reading related variables, published between 2011 and 2023.

Authors’ response: Thank you for the correction provided below which noted that Park (2010) was published in 2010. The text has been revised accordingly.

The structure is clear, and it is easy to read and understand.

Nevertheless, there are some issues that should be addressed.

About the databases used to find the articles another specialized education database such as ERIC should have been included.

Authors’ response: Thank you for this comment. We understand that PRIMO is a relatively new discovery system and realize that it would be beneficial to offer further explanation. ExLibris PRIMO database is a one-stop servicing, supra-database that pulls search results from all scholastic databases such as ERIC, LLBA, PsycINFO, ScienceDirect, Web of Science, etc. that are available through the library. For this reason, we used PRIMO instead of using individual databases. In addition, we used Google Scholar as it is another comprehensive search tool which we believed would supplement our literature search. Once the initial pool of eligible studies was gathered, we also searched for more eligible studies through citation chasing.

I first learned about the PRIMO discovery system through a faculty orientation at my university several years ago. Since this was a few years ago and memories decay, I verified my initial understanding of the PRIMO database by consulting my university’s reference librarian, Ms. Krystal Dean. She confirmed that my understanding of PRIMO is indeed correct. Please see below for our chat transcript for Ms. Dean’s response which also includes websites that provide further information about PRIMO. Guest87058579 is the ID assignment to me during this chat session and Krystal Dean was the reference librarian whom I consulted during this chat session.

This is the chat transcript from () on Apr 10 2024, 08:46pm:

1. 20:41:48Guest87058579: Hi, I have a quick question about PRIMO database.
2. 20:42:10Krystal Dean: Sure! I will do my best to answer
3. 20:43:34Guest87058579: Hi Krystal, yes, I am revising my meta-analysis for a journal and one of the reviewers asked why I didn't use databases like ERIC and LLBA and instead used PRIMO. My understanding is that PRIMO is like a supra-database which includes all such smaller databases like ERIC, LLBA, or PsychInfo. Is my understanding correct?
4. 20:44:44Krystal Dean:  PRIMO is a discovery system that pulls results from all of the databases available through the library. 
5. 20:45:05Guest87058579: awesome! Is there any citation that I can use to support that information?
6. 20:45:16Guest87058579: any websites or print citation?
7. 20:46:00Krystal Dean: Here is some info from the ExLibris website: https://knowledge.exlibrisgroup.com/Alma/Product_Documentation/010Alma_Online_Help_(English)/060Alma-Primo_Integration/010Introduction
8. 20:46:05Krystal Dean: Here is the product page for PRIMO: https://exlibrisgroup.com/products/primo-discovery-service/
9. 20:46:13Guest87058579: Awesome! Thank you so much!
10. 20:46:16Krystal Dean: Of course!

Read our privacy policy.

The search string to find the reviewed articles is not indicated and there is not a guide for the systematic review such as for example PRISMA. The review question, inclusion criteria and search strategy are not included either.

Authors’ response: Thank you for your comment which we incorporated into the revised draft. For the literature search, we used combinations of the following keywords with no restrictions on search date (i.e., publication dates of eligible studies): second language, foreign language, L2, reading comprehension, reading, meta-analy*, correlat*, relation*, variable* (By adding an asterisk * after the root word means you are asking the database to search for any possible ending after the asterisk). By using this method, which did not include specific names of reading-related variables (e.g., phonological awareness, vocabulary knowledge, metacognition, etc.), we aimed to gather quantitative meta-analyses that examine the relationship between reading comprehension and a wide range of reading-related variables. Once we gathered the preliminary pool of eligible studies, we examined the literature review of the studies to further identify eligible, quantitative meta-analysis. As a result, a total of 55 potentially eligible studies in the initial pool were reduced to 14 studies which made into the final pool. This information was added to the revised manuscript.

Table 1. Summary of Study Features:

I miss the date of publications for each study presented in this table. The study by Park was published in 2010 so it does not meet the criteria to be included in this review (between 2011 and 2023).

Authors’ response: Thank you very much for alerting us to this error. Yes, the study by Park was indeed published in 2010 and was corrected accordingly in the revised draft of the paper. To be clear, as noted above, we had not set restrictions on the publication dates of eligible studies. In other words, our goal was not to examine studies that were published between 2010 and 2023 for some rationale. Rather, when we collected eligible studies, they turned out to have been published during this period.

What does it mean that the publication type was NR (Not reported). I assume that a study cannot be reviewed if it is not published. I am sorry but I do not understand this concept.

Authors’ response: This simply means that the authors themselves of the included meta-analysis did not report whether the primary studies were published (e.g., journal articles, book chapters) or fugitive literature (e.g., thesis, dissertation, ERIC document, arXiv document). We would like to share that in the meta-analysis research community, it is customary to the second type of documents “unpublished” or “fugitive” literature. For prospective readers who might not be familiar with this convention, we added this information to the “Note” appearing under Table 1.

Table 2. Summary of Study Features Continued:

The L1 and L2 languages are not reported in 4 studies out of 14, and 7 studies out of 14 the L1 and L2 languages were various. So, the results could be underpowered because languages are very different in characteristics and compare studies with varied languages and in some cases studies that do not report the language, should be noted as a limitation of this study. Besides, studies that do not identified the L1 and L2 languages (4 of them) should not be included in this review.

Authors’ response: Yes, we agree that for a more fine-grained synthesis, it would be ideal to review meta-analyses conducted on comparable languages or at least meta-analyses all of which report included L1s and L2s. However, as shown in our study pool, this is far from the reality of the research domain; many important and carefully conducted meta-analyses sometimes simply report that they included primary studies involving various L1s and L2s (e.g., Ke et al., 2021) or do not report which L1s and L2s were involved at all (e.g., In’nami et al., 2022). Although the authors of such meta-analyses did not explicitly state their rationale for this decision, we have a few conjectures in their defense (if you will); first, it is surprisingly common that some large-scale primary studies investigating L2 proficiency (even those conducted by testing organizations such as the Educational Testing Service) do not themselves identify which exact L1s and L2s were involved. When the primary study authors do not report details like this, meta-analysts are sometimes successful in collecting further information by contacting the authors, but more often than not, unsuccessful. In such a case, the meta-analysts do not have any other option but to simply report the involved languages to be “various” or “not reported” against their intention; the second rationale is based on our experience of observing numerous moderator analyses based on linguistic characteristics (e.g., crosslinguistic distance between L1 and L2 (e.g., Author & Author, [2014, 2022]) or language type [L1 vs. L2 as in Dong et al. [2020]) which repeatedly showed that they are largely statistically insignificant (e.g., only 1 out of dozens of moderator analysis using crosslinguistic distance between L1 and L2 barely reached a significant level in Author & Author [2014] and none reached a significant level in Dong et al. [2020a, 2020b,]). While this may seem counterintuitive at first, in a way it is an indication that the relationship between reading comprehension and reading-related variables tends to be more universally convergent than divergent. We wholeheartedly agree, however, that the studies that involved both L1 and L2 in their correlation data must be flagged so that the readers are accurately informed. We have added a marking to those studies in the tables and also discussed it as a caveat in the Notes attached to Tables 3 and 4.

Table 3. It would be interesting to indicate some of the test used to measure the indicated variables (i.e. phonology, orthography, etc.) how they were treated or measure in each study, because the variation in measures implies variation in results.

Authors’ response: We absolutely agree that this would have been a point worth discussing. But unfortunately, most of the meta-analyses we reviewed in the study did not report such detailed information on measurement instruments used in primary studies. As a result, we regret that we did not have enough information to synthesize across meta-analyses.

The results with the comparisons of the relationship between L2 reading comprehension and reading related variables, seem to compare the studies by Author & Author 1, 2 with other studies. Giving a high preponderance to these studies compare to others.

Authors’ response: Thank you for the astute observation. Yes, we unfortunately ended up having this feature because we chose our own meta-analyses as a ruler (if you will) throughout the review. Our rationale for this decision is that compared to other meta-analyses, ours tended to be broader in scope (e.g., number of studies included, inclusivity regarding languages, ages, proficiency levels, operationalization of reading variables, etc.,) and therefore, logically, would make good starting points for comparisons with other, smaller-scope studies. The decision was purely based on this fact and this fact only, and not at all motivated by to self-promote our studies.

Another table should be included with the quantitative results of the reviewed studies to better understand the trends.

Authors’ response: Thank you very much for this comment. This is a great idea and we created two new tables and replaced the original Tables 3 and 4 with them. The new tables provide effect sizes reported by the primary meta-analyses so that readers can easily view not only which reading variables were included in each meta-analysis but also what the effect sizes were.

The discussion is centered on the overlapping among primary studies across meta-analysis and this was not a goal of the present review.

Authors’ response: Thank you for this comment. To explain our rationale, we thought the significant lack of overlap in primary studies across meta-analyses of similar goals and inclusion criteria indicates a serious methodological weakness in the research domain, namely, problems with the rigor or literature search by meta-analysts. Since the primary data of a meta-analysis are primary studies, great divergence in primary study pools across meta-analyses can be likened to missing data and this is clearly a gap in the research domain. One of the research questions asks whether there are gaps in the current research domain, and this is our rationale for discussing this problem in the review.

 

The conclusion that the “value of an independently conducted meta-analysis heavily outweighs that of a replicated meta-analysis” is not a research goal of the present study and it is not treated and well-founded. So, it should be omitted.

Authors’ response: Thank you for this comment. We also agree with the reviewer’s comment and have deleted this part from the conclusion.

The comment on the study by Lee et al. is not understood in the context of a discussion.  Also, the remark on the 2 of the 14 meta-analyses centered on L1 Chinese participants only is not well understood either.

Authors’ response: We are afraid that we do not understand this comment. Could the reviewer specify which page of the paper this comment entails and what remark he/she is referring to?

A clear limitation of this study is that the 14 reviewed studies are so different among them that it is not possible to reach a clear conclusion.

Authors’ response: Thank you for this comment. While we do agree that there is substantial divergence among the primary study pools included in the meta-analyses reviewed in our paper, we consistently noted in our discussion of findings that the convergence in the strength of the relationship between reading comprehension and each of the reading-related variables was still strong across studies. In other words, strong convergence in the key findings (e.g., the strength of the relationship between reading comprehension and reading-related variables) despite the divergence in primary study pools actually further affirms the overall findings rather than weakening them. This is how we interpret the overall findings we reported from pp. 6-19. I hope this explanation helps the reviewers see our point of view and interpretation of the findings.

The Discussion should be rewritten and oriented to the study goals.

Authors’ response: Thank you for this comment. We included research questions to more clearly convey our study goals, and removed our discussion on replications of meta-analyses to streamline the discussion.

 

Reviewer 2 Report

Comments and Suggestions for Authors

The authors wrote a clear and well-written review report on the correlates of L2 reading comprehension. I have two major issues, that make me a bit hesitant about the significance of the contribution to the field:

- No explicit information is provided about the search terms and search procedures. Although the authors state that a meta-analysis of meta-analyses was not their intention, they also show that there is only small overlap in primary studies in the reviewed meta-analyses. The authors state that they aimed to synthesize literature that 'most interested readers are likely to come in contact' with. However, with their review they show that there is much variation at this point. Considering this fact, I would suggest to provide more explicit information about the search procedure. What were the exact search terms? How many articles were screened? How was 'reading-related variables' defined? etc.

- The meta-analyses reviewed focus on both L1 and L2 predictors of L2 reading comprehension. In the current version of the paper it is sometimes hard to make the distinction which correlations are described. I would suggest a clearer/more structured distinction between the two types of relations. Related to that, the authors also describe a meta-analysis including both L1 and L2 predictors (Dong et al.) and state if type of language moderated the effect sizes in this meta-analysis. Since there is very little overlap in primary studies, it is not clear to what extent the conclusion about the possible influence of type of language can be considered in line with results of the other described meta-analyses.

 

Two minor points that might deserve attention:

- In the first paragraph of the introduction, reading strategies are categorized as a meta-cognitive skill. However, there is no consensus in the literature if reading strategies should be considered cognitive or meta-cognitive skills.

- p. 7, line 5 "31 correlations included in Author and Author [2]  should be [1] I think. 

Author Response

Reviewer 1:

The structure and organization of the work is acceptable, but the selected papers do not have a solid theoretical foundation, so the conclusions are limited and this fact should be reported.

I do not agree with the statement presented at the end of page 20 "Perhaps, through these meta-analyses, we are indeed uncovering the true nature of the relationship between reading related variables and reading comprehension"

Rather, the paper presents an approach to the study of the relationships between L2 reading comprehension and reading-related variables. This approach seems to me valuable, but from there to conclude that "you have uncovered the true nature between reading-related variables and reading comprehension" it seems to go too far.

Limitations of your study should be reported.

Authors’ response: Thank you for these valuable comments. We fully agree with your assessment of the problems of our paper and revised it by removing the problematic statement and also by adding limitations of the study. The revised section is highlighted in yellow.

Reviewer 2:

The points I raised in my review of the previous version of the manuscript were all sufficiently addressed in the revisions and response of the authors.

Authors’ response: Thank you very much for your help throughout the review process!

 

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The structure and organization of the work is acceptable, but the selected papers do not have a solid theoretical foundation, so the conclusions are limited and this fact should be reported.

I do not agree with the statement presented at the end of page 20 "Perhaps, through these meta-analyses, we are indeed uncovering the true nature of the relationship between reading related variables and reading comprehension"

Rather, the paper presents an approach to the study of the relationships between L2 reading comprehension and reading-related variables. This approach seems to me valuable, but from there to conclude that "you have uncovered the true nature between reading-related variables and reading comprehension" it seems to go too far.

Limitations of your study should be reported.

 

Author Response

Reviewer 1:

The structure and organization of the work is acceptable, but the selected papers do not have a solid theoretical foundation, so the conclusions are limited and this fact should be reported.

I do not agree with the statement presented at the end of page 20 "Perhaps, through these meta-analyses, we are indeed uncovering the true nature of the relationship between reading related variables and reading comprehension"

Rather, the paper presents an approach to the study of the relationships between L2 reading comprehension and reading-related variables. This approach seems to me valuable, but from there to conclude that "you have uncovered the true nature between reading-related variables and reading comprehension" it seems to go too far.

Limitations of your study should be reported.

Authors’ response: Thank you for these valuable comments. We fully agree with your assessment of the problems of our paper and revised it by removing the problematic statement and also by adding limitations of the study. The revised section is highlighted in yellow in the attached file.

 

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The points I raised in my review of the previous version of the manuscript were all sufficiently addressed in the revisions and response of the authors.

Author Response

 

Limitations of your study should be reported.

Authors’ response: Thank you for these valuable comments. We fully agree with your assessment of the problems of our paper and revised it by removing the problematic statement and also by adding limitations of the study. The revised section is highlighted in yellow in the attached file.

Reviewer 2:

The points I raised in my review of the previous version of the manuscript were all sufficiently addressed in the revisions and response of the authors.

Authors’ response: Thank you very much for your help throughout the review process!

 

Author Response File: Author Response.pdf