Assessing Portuguese Elementary School Students’ Scientific Literacy: Application of the ALCE Instrument
Round 1
Reviewer 1 Report
Thank you for the opportunity to review your manuscript: " Assessing Portuguese elementary school students' scientific literacy: application of the ALCE questionnaire”. It describes the scientific literacy level of Portuguese students at the end of grade 9 and verify how different variables are related to the scientific literacy level.
In Portugal, besides PISA and TIMMS, there are not many research studying students’ scientific literacy, which reinforces the pertinence of the study. The introduction, methodology and results sections are clear, covering all the necessary aspects.
Below I present a list of aspects that I believe need further reflection and reformulation.
Line 53 and 54
Not “Essential Apprenticeship” but “Essential Learnings”. This correction needs to be done throughout the article.
Line 54
Profile of Students Leaving Compulsory Education… Please, use the translation used by the Portuguese minister of education – “Students’ Profile by the End of Compulsory Schooling” (see https://cidadania.dge.mec.pt/sites/default/files/pdfs/students-profile.pdf)
Results - I believe that the introduction of some subtitles could improve the results
Line 133-135
I have some concerns regarding the variable - Which course do you intend to choose in Secondary Education? (Science and Technology Course/Socio-economic Sciences Course/Languages and Humanities Course/Visual Arts Course/Professional Courses). Professional Course is not at the same level as science and technology course. There are some professional courses that are extremely related to science and technology…
Line 143
“Curricular Guidelines for Physical and Natural Sciences” – which document is this? Please, you should reference this document.
Line 205
I believe the legend is not correct.
Line 226-231
Without knowing the instrument used, it is impossible to assess this analysis. For instance, we need to be sure that the cognitive demand of the questions is the same regardless of the scientific area. Maybe it would be important to add the instrument in supplementary/ appendix material.
Line 268-269
I think it would be worth exploring this idea a little more... What are these differences in contexts?
Line 304 and 314
Please, indicate the meaning of the error bars (what metric is being shown).
Line 348
“The results of the ANOVA involving the scientific literacy level and the variable socio-economic level of the students” – in fact, the variable is not the socio-economic level of students, but of the school students attend.
Line 376
What do you mean by “school composition”?
Line 386-387
“higher so-cioe-conomic status” - correct
Line 421-422
“and the interest in answering the questionnaire, for example” – what do you mean by this? It would be important to understand the validity of the instrument. What was the context of the application? We know it was online, but all students who received the questionnaire answered it? Who has sent the e-mail?
Despite not being able to identify spelling or grammar mistakes, the wording is sometimes confusing, maybe as a result of a word for word translation.
Therefore, this paper would benefit from a linguistic revision.
Author Response
Dear Reviewer,
Thank you for the analysis and the feedback with suggestions that you have sent us. We are sure that, after the changes that have been made, the text of the article is much more robust and consistent.
Please see the attachment.
We look forward to receiving the final report.
Yours sincerely,
The authors
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The article presents a very preliminary research work, in which it is proposed to answer eight research questions, through the analysis of the responses to a questionnaire.
This questionnaire does not appear in the text at any time. What the authors say is that there are 34 items of which 9 serve to assess the sociodemographic profile of the sample. Some of the questions are of the "Do you like..." type, which in my opinion does not offer relevant information to obtain reliable conclusions in relation to the level of scientific literacy, if not, at most, they could be answers indicative of perceptions .
On the other hand, to analyze the results, the authors establish arbitrary punctuation strata without justifying, even giving the same value to two answers (zero points to incorrect answers and to "don't know").
The graphs do not have axes to help interpret them.
The Methodology section should be divided into smaller sections where the characterization of the mixture (Sample), the research method used (Method) is established. And therefore, the Results section should be sectioned in the same way as the research objectives, in order to clarify the results obtained and to provide evidence of the answers to the research questions. I recommend the same for the conclusions section.
On the other hand, in the Results, they are not limited to exposing the evidence found, but this statement is also the discussion of the results. I recommend separating it into two sections, which would make it easier to understand which are the objective evidence obtained from the investigation and which are the comments. In the conclusions section, the data found in the results should be contrasted with the specialized bibliography.
In general, throughout the entire article there are excessive verbatim quotes.
And the bibliography provided is mostly from the Portuguese context, so the international part could be expanded.
Author Response
Dear Reviewer,
Thank you for the analysis and the feedback with suggestions that you have sent us. We are sure that, after the changes that have been made, the text of the article is much more robust and consistent.
Please see the attachment.
We look forward to receiving the final report.
Yours sincerely,
The authors
Author Response File: Author Response.pdf
Reviewer 3 Report
This study presents a survey of the scientific literacy of Portuguese 9th-grade students and examines which variables explain the variance of the performance. The aim is to collect data for educational development, complementing the Portuguese results of international studies (PISA, TIMSS). The topic is relevant to both system-level assessment and science education. The study is well structured. The aims and research questions are clear. I suggest a few clarifications to better understand the details and to make the research more widely useful.
The sample consisted of 9th-grade students at the end of the 3rd cycle of Portuguese Basic Education. More information could be shared on Portuguese science education and its stages.
Who can be considered as scientifically literate based on the results of the test? (Research question 2). It is not possible to directly compare the ability categories based on quartiles with the ability levels used by PISA.
The Avaliação da Literacia Científica Essencial (ALCE) questionnaire assesses students' knowledge by asking questions at different cognitive levels on different science topics. The instrument is not a questionnaire, but a test with an attached background questionnaire. In the title of the article and the description of the instrument, the word "questionnaire" is used, while the units of the instrument are called "subtests" (e.g. Table 1).
For each of the three subtests, it would be important to present a sample item to get an idea of the nature of the tasks.
For true-false items, the probability of a random correct answer is high. To check whether the test worked reliably on this sample, the results of the reliability and IRT analysis should be reported. The reliability should also be calculated for the subtests.
In the Conclusions section, it would be useful to make recommendations for the development of primary science education based on the results. It should also highlight, beyond the Portuguese context, the relevance of this research in the international context of educational research. Further research possibilities could also be discussed.
Author Response
Dear Reviewer,
Thank you for the analysis and the feedback with suggestions that you have sent us. We are sure that, after the changes that have been made, the text of the article is much more robust and consistent.
Please see the attachment.
We look forward to receiving the final report.
Yours sincerely,
The authors
Author Response File: Author Response.pdf
Reviewer 4 Report
This was an interesting paper, on the whole well written with only minor typographical errors to be addressed (e.g. student's / students' ).
The context and method were well explained, and to a large degree the results well presented. It would be helpful to identify the age of 3rd cycle students for non-Portuguese readers at an earlier stage in the paper (though this can be seen in the mean age of respondents later).
In the results section, lines 294-313, there is a slight confusion of language for the likert scales, e.g. dislike / like little - neither of which terms appears on the likert graph. These discrepancies should be tidied up. If the two lower likert categories have been combined to a new "dislike" group then this should be made explicit.
The conclusion that opting for vocational courses predicts scientific literacy (line 411) is not valid - it IS valid to say that the two are correlated but where is the evidence of cause and effect? It could well be that students with good scientific literacy are more likely to choose these courses, so cause and effect are the other way round.
This paper confirms earlier studies so is not a major contribution to knowledge. It would be strengthened by a more nuanced discussion about what educators might do with the findings, e.g. whether early conversations about what students like or might choose to study later could support intervention to raise scientific literacy before the end of the 3rd cycle. maybe there are systemic reasons why this would not be possible, but a study like this that is rooted in classrooms surely has the potential to have an impact on practice, yet this is not discussed.
English is good, with only minor errors to be addressed.
Author Response
Dear Reviewer,
Thank you for the analysis and the feedback with suggestions that you have sent us. We are sure that, after the changes that have been made, the text of the article is much more robust and consistent.
Please see the attachment.
We look forward to receiving the final report.
Yours sincerely,
The authors
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
The manuscript has been improved. It can be published in the present form.
Author Response
Thank you.
Reviewer 3 Report
The proposed changes have been made by the authors.
If Table 4 is inserted, the results presented in the table should be interpreted in the text or referred to it later, for example in subsection 3.2 (line 390).
Author Response
We thank you for the observation. This information was indeed missing and therefore the results presented in Table 4 have been presented and referenced in the first paragraph of each subsection of the Results and Discussion session. We hope this is sufficient for a good understanding of the article and, consequently, for its publication.
