Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors are taking on an important issue: synthesizing literature in relation to data literacy which is an area of study everyone seems to be writing about but there is very little clarity as to what exactly it is, how it is taught, or what the learning outcomes are of teaching it in K-12 educational settings. This paper holds a lot of promise in helping move the field forward as its research questions and goals are important for the field. However I have issues with how the paper has been framed and the methods employed that limit the usefulness of the results. I think these issues are addressable but will require reworking the paper and likely going back to the analysis.     

 

Let me start with issues related to the theoretical framing of the paper. I am struggling with your definition of data literacy as it does not seem much different than that of statistical literacy. Wild and Pfannkuch included real world contexts as an integral part of their framework in terms of shuttling between the contextual sphere and the statistical enquiry so that is no new addition and though they do not highlight ethics in the same way it is present so at best it seems like the only additions are the holistic consideration of ethics. That still begs the question why did you choose the Wolff framework over others? It does not come from educational research even though that is the focus of your study. There is no clear rationale and though you do state there are others you provide no rationale as to your choice of frameworks. Furthermore, the others you point to but do not discuss do come from educational research which is the focus of your study. This is important as it frames your entire study. This is also problematic as a simple google search will point to a huge number of different definitions of data literacy used across a myriad of fields. You seem to be oversimplifying what is currently an extremely ill defined and murky space which you are then using to drive your study. 

 

A minor point related to your background literature is that when looking at curriculum in other countries you point out not many list data literacy however your definition is not really different from statistical literacy which is commonly present in most countries curriculum this seems like a problematic oversight in your review of the literature. I would suggest including some consideration of statistical literacy and how that appears in the curriculum as well.  

 

The area where I see the most significant issues are in terms of your methodology. I have a number of questions that remain unanswered after reading the paper that I think it would be important to address to clarify what you did in your study. For example, how many people sorted the articles to determine exclusions and how much overlap was there between those determining which articles to retain and which to exclude? Why do you need EC3? You are interested in K-12 education uses of data literacy so why further narrow your selection to STEM only topics as all topics include data literacy in some form? I could also use further clarification as to how you have operationalized EC4. Does this mean a paper that did not focus on all aspects of an investigative process would not be included? I could also use more examples of how you operationalized EC1 as your definition does not go much beyond those in statistical literacy. How did ethics play a role in your decision to include or exclude articles. 

Otherall my main concern here is that a lot of articles have been excluded which makes me question how broadly useful this analysis is to the field at large. A clear example of this is that a quick google search found a systematic review very similar to your was just published in the International Journal of STEM education (https://doi.org/10.1186/s40594-024-00517-z)  and they include nearly three times as many articles as your review did. There are also several articles that seem like they should have been included given your search terms such as Conceptions and perspectives of data literacy in secondary education by Engida Gebre https://doi.org/10.1111/bjet.13246 or Philip, T. M., Olivares-Pasillas, M. C., & Rocha, J. (2016). Becoming racially literate about data and data-literate about race: Data visualizations in the classroom as a site of racial-ideological micro-contestations. Cognition and Instruction, 34(4), 361–388. Also there is a whole issue of SERJ in 2022 that addresses data science education that has many articles which seem like they should fit your criterion but have not been included in your references.  

 

For coding the articles there needs to be substantially more discussion in terms of how the codes were chosen and where they are coming from. Some are rather straightforward such as the country of the first author and publication year. However, ones like mentioned competencies are much less so as several of these competencies are not mentioned earlier such as data visualization, data organization, data retrieval, and evaluation. It is not clear where these are coming from or how you are operationalizing them to be able to understand the findings that have resulted from the coding. Also you have an N/A section for many. In the case of mentioned competencies from your framework I am wondering how these articles then got through the exclusion criterion as one of your criteria was that the articles fit your theoretical framework.   

 

It is hard for me to evaluate the results in depth as it is still unclear for me what some of the categories mean and what articles have been included versus excluded. Some minor points though are that in Figure 3 the x axis needs a label. Also in table three it would help if you clarify what the numbers represent. Also as I assume these are counts it would be helpful to have some percentages as well such as row or column percentages to compare occurrences across categories. 

 

I do think this paper has potential as the topic is an important one. However I think until the issues I have raised above are addressed the paper’s potential impact will be limited.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

I want to thank the authors for selecting an important topic. I enjoyed reading the manuscript. I would like to raise the points that I found crucial. The introduction does not prepare the readers for the research questions of this study. The authors emphasize the importance of interdisciplinary teaching of data skills in K-12 education but asked a completely different group of research questions:

RQ 1: Which data literacy skills are considered important in K-12 education?

RQ 2: How can specific skills be strengthened in K-12 Education to improve learners' data literacy?

As clear from the research questions, data-literacy at K-12 is the main problem of this study and I could not see why this study is limited to K-12. As also mentioned by the authors, “as a large number of articles addressed higher education instead of the K-12 education that is the focus here,” maybe, the researched terms are not appropriate for K-12. Therefore, I agree with the authors’ suggestion that “…the search strategy may not have identified all relevant papers, particularly if key terms were not present in the title, keywords, or abstract” . In conclusion, excluding the university level from this study stands as a crucial issue on its own.

 

The study also needs to convince the reader why this study is needed by showing how this new study builds on existing review studies such as:

Cui, Y.; Chen, F.; Lutsyk, A.; Leighton, J.P.; Cutumisu, M. Data Literacy Assessments: A Systematic Literature Review. 538 Assess. Educ. Princ. Policy Pract. 2023, 30, 76–96, doi:10.1080/0969594X.2023.2182737.

Ghodoosi, Bahareh, Tracey West, Qinyi Li, Geraldine Torrisi-Steele, and Sharmistha Dey. 2023. “A Systematic Literature Review of Data Literacy Education.” Journal of Business & Finance Librarianship 28 (2): 112–27. doi:10.1080/08963568.2023.2171552.

 

Regarding the methodology, here are the questions that the authors may use to make this section stronger and clearer.

-Table 1 could become concise by removing some of these very understandable exclusion criteria, such as “6. Papers published before 2014”.

-Why such a low agreement between the authors at the initial phase of coding?

“ The analysis showed substantial agreement for these six articles. Cohen's Kappa ranged between .47 and .84, with an average of .69,”

-Provide a rationale for the coding categories such as country, publication year, journal type, research methods, etc. As a reader, I am not convinced why these features are crucial to be selected and coded. By looking at the research questions, it is difficult for me to see how these variables are the most important stuff. Besides, I am really wondering how categories are formed for these variables. To give an example, how did you decide the journal type of an article among Teaching/ Learning Data Science/ Math Education ?

-3 of the studies are netiher Elementary, Middle or High School? Then, why are these articles in this scoping review? Have you reached the researchers of these articles and asked for more information? If not, isn’t it better to exclude them from this review study?

In brief, there seems to be an issue of misalignment between research questions and the dimensions examined in the review process. If you have a section called “3.2 Study Characteristics” in the study, then why not a research question specific to these findings? I would suggest the authors to work on coherence throughout the paper.

 

A minor issue:

The abbreviation EC should be given in full form when it appears first in the manuscript.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I appreciate the attention the authors have taken to addressing the reviewer comments. The manuscript is much improved and communicates the findings in a much clearer and trustworthy manner. I have only two minor comments for consideration.

1) I would like to see the coding and exclusion document share https://osf.io/bcdre/?view_only=f7019ac2741e427fbb8c804f29064e65 as part of the final paper as I think this significantly improved the trustworthiness of the work and I appreciate the attention to document the authors have taken here.  

2) Since the authors reframed their paper to draw on Lee et al.'s framework I would encourage them to revise their discussion to connect back to that framework to bring those ideas full circle.

Author Response

Thank you for the prompt feedback on the paper once again. We are pleased that you appreciate and recognize our work on it. Finally, we have now incorporated the two new comments into the paper (blue text):

Comment 1: I would like to see the coding and exclusion document share https://osf.io/bcdre/?view_only=f7019ac2741e427fbb8c804f29064e65 as part of the final paper as I think this significantly improved the trustworthiness of the work and I appreciate the attention to document the authors have taken here.  

Response 1: This is an important point, and ensuring transparency in our work is also in our interest. The link to the document has now been mentioned under 'Supplementary Material,' ensuring that all readers of the paper can access this document (see p. 15, ll. 560-562).


Comment 2: Since the authors reframed their paper to draw on Lee et al.'s framework I would encourage them to revise their discussion to connect back to that framework to bring those ideas full circle.

Response 2: To further address the competence framework by Lee et al. (2022) in the discussion, several paragraphs have been added, making specific reference to this model (see Section 4, p. 14, ll. 488-491; Section 4, p. 14, ll. 499-501).

Reviewer 2 Report

Comments and Suggestions for Authors

The author addressed the comments and made important additions/clarifications to the manuscript. 

Comments on the Quality of English Language

The language is a bit involuted. I would prefer a clear language so that the readers can read it with less hassle. 

Author Response

Thank you for the prompt feedback on the paper once again. We are pleased that you appreciate and recognize our work on it. In response to your comment, we have revised the English language of the paper. We hope that some sections are now clearer and the flow of reading is not disrupted. The changes can be found in blue text at the following sections/ pages:
Section 1.2.1, p. 4, ll. 172-174
Section 1.2.2, p. 5, ll. 207-209
Section 2.1, pp. 6-7, ll. 260-263
Section 2.2, p. 7, ll. 277-280
Section 3.2.2, p. 12, ll. 403-405
Section 4, p. 14, ll. 480-486
Section 4, p. 15, ll. 526-528
Section 5, p. 15, ll. 553-556