The Effectiveness and Sustainability of Tier Diagnostic Technologies for Misconception Detection in Science Education: A Systematic Review

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The topic of the article is very relevant, as in recent decades students have increasingly neglected proven sources of knowledge, preferring to obtain them from the Internet, for example, from Wikipedia. The article is well structured. It is written in simple and understandable language and is based on a fairly large number of articles retrieved from the Web of Science (WOS) and Google Scholar platforms. The analysis of the articles considered in the Review is professional and reasoned. Conclusion: The article should be published.

Noted shortcomings.

1. When referring to an article, its number should be indicated immediately after the author's last name, for example, Channel [8]. Otherwise, ambiguity may arise, as, for example, in line 108.

2. Pay attention to link 13 in line 150

3. It may be correct to indicate which researchers are meant in line 214

4. In line 252, a link to the corresponding article should be made.

5. Did I understand correctly that ML in paragraph 3.3.4 means "mobile learning"? It's better to provide explanations for abbreviations.

Author Response

Dear Reviewer,

Thank you very much for your careful and thorough review of my manuscript. Your valuable comments are incredibly important to us! Below are my responses to your comments:

Comment 1: When referring to an article, its number should be indicated immediately after the author's last name, for example, Channel [8]. Otherwise, ambiguity may arise, as, for example, in line 108.

Response 1: We have addressed this issue. For example, Caleon and Subramaniam [34] introduced the confidence tier...

Comment 2: Pay attention to link 13 in line 150.

Response 2: We have made the necessary modifications, and the issue has been corrected in line 295.

Comment 3: It may be correct to indicate which researchers are meant in line 214.

Response 3: The new expression has been presented in line 359.

Comment 4: In line 252, a link to the corresponding article should be made.

Response 4: We have added the reference in line 478.

Comment 5: Did I understand correctly that ML in paragraph 3.3.4 means "mobile learning"? It's better to provide explanations for abbreviations.

Response 5: "ML" refers to machine learning (ML), and this clarification has been added in line 542.

Wishing you a joyful Spring Festival, all the best, and good fortune in the Year of the Snake!

Regards,
Gaofeng Li

Reviewer 2 Report

Comments and Suggestions for Authors

In my opinion the article is more of an essay than a research paper. The authors themselves acknowledge that ‘the results and discussion sections are not exhaustive’ (line 49).

According to the narrative, a systematic review of the literature is carried out, but this method is not mentioned and reference is only made to content analysis.

As for the documents reviewed, I consider that something has gone wrong either in the search or in the selection, as the number of final articles to be reviewed is very small considering the time span established.

The results are not presented in the way that a systematic literature review or content analysis requires. It would be necessary to explain the software used for this analysis, the system of categories that has emerged, how these have been defined, and if desired, the number of occurrences of each one of them. In addition, the results are often evidenced using verbatim quotations.

The perspective of qualitative research is to be critical, to identify where research in this field converges, but above all to identify dilemmas.

Reading the article gives me the feeling that the positioning or conclusions were pre-established. There is no discussion, which is the key to this type of research. There is no detection of tensions. Nor are there any recommendations for the possible improvement of practice, and this is key because qualitative research aims to transform and improve what it analyses.

 

For these reasons, it gives me the feeling of being more of an essay than a research article. Perhaps turning it into an essay would make it more honest and more coherent.

Author Response

Thank you for your constructive comments, which have breathed new life into our paper. We have made significant revisions to the manuscript based on your feedback. If there are any aspects you are not satisfied with, we are more than willing to make further adjustments.

comment 1：According to the narrative, a systematic review of the literature is carried out, but this method is not mentioned and reference is only made to content analysis.

response 1：Thank you for your valuable comments. We have provided a clearer description of the systematic literature review method in the methods section. We appreciate your feedback.

comment 2：As for the documents reviewed, I consider that something has gone wrong either in the search or in the selection, as the number of final articles to be reviewed is very small considering the time span established.

response 2：The reality of how tiered diagnostic instruments uncover misconceptions is as outlined. This review is specifically focused on SSCI journals.

comment 3：The results are not presented in the way that a systematic literature review or content analysis requires. It would be necessary to explain the software used for this analysis, the system of categories that has emerged, how these have been defined, and if desired, the number of occurrences of each one of them. In addition, the results are often evidenced using verbatim quotations.

response 3：We have made significant revisions and have now followed the process of a systematic literature review. Thank you for your constructive feedback.

comment 4：The perspective of qualitative research is to be critical, to identify where research in this field converges, but above all to identify dilemmas.

response 4：We have conducted a comprehensive review of the literature, focusing on several key areas: the development of MTDTs, the validation of their reliability and validity, their representation of conceptual understanding, their implementation and application across different disciplines and educational stages, as well as the misconceptions revealed by MTDTs and the intervention suggestions proposed. In the discussion section, we have clearly identified the limitations within each focus area and provided recommendations for future development.

comment 5：Reading the article gives me the feeling that the positioning or conclusions were pre-established. There is no discussion, which is the key to this type of research. There is no detection of tensions. Nor are there any recommendations for the possible improvement of practice, and this is key because qualitative research aims to transform and improve what it analyses. 

response 5：We have revised the discussion section and strengthened the comparison with other literature. Additionally, we have included a new section on "Future Research Directions and Recommendations"

Reviewer 3 Report

Comments and Suggestions for Authors

The new version of the manuscript is improved. The authors have added clarifications and research questions in the Introduction section, as well as Table 1 and Figure 3 in the Results section. However, some issues remain. Therefore, I have several comments:

1. Consider removing the hypotheses and keeping only the research questions in the Introduction section. 
2. Figure 2 (Screening Process) appears unclear and illogical. The Social Science Citation Index (SSCI) includes journal articles but does not index conference proceedings. Therefore, conference reports should have been excluded in the first stage, not the third.
3. Consider merging the Results and Discussion sections, as the research questions were not addressed in the Results section. Alternatively, revise the research questions to align with the results.

Author Response

Thank you for your meticulous, thoughtful, and sincere review comments. We have made significant revisions to the manuscript based on your feedback. If there are any aspects you are not satisfied with, we are more than willing to make further adjustments.

Comments 1: Consider removing the hypotheses and keeping only the research questions in the Introduction section. 

Response 1:Thank you for your suggestion. We believe that the hypotheses play an important role in the research, as they not only help clarify the direction of the study but also provide a theoretical framework. The formulation of hypotheses allows us to predict potential outcomes based on existing theories and literature, thereby laying the foundation for subsequent data analysis and discussion. Therefore, we have decided to retain the hypotheses, as they will help readers better understand the background and objectives of the study. If the reviewer has any further suggestions, we would be happy to make additional revisions.

Comments 2: Figure 2 (Screening Process) appears unclear and illogical. The Social Science Citation Index (SSCI) includes journal articles but does not index conference proceedings. Therefore, conference reports should have been excluded in the first stage, not the third.

Response 2:Thank you for your careful and insightful comments. As you rightly pointed out, there was a lack of clarity and logical flow in Figure 2 (Screening Process). The Social Science Citation Index (SSCI) does indeed include journal articles but not conference proceedings. Therefore, conference reports should have been excluded in the first stage, not the third. We have revised Figure 2 to ensure that the screening process is clearer and more logical. The previous mistake was due to an oversight by a collaborator during the figure creation process, which led to this inaccurate representation. Thank you for your valuable feedback, and we have made the necessary adjustments in the revised manuscript.

Comments 3:Consider merging the Results and Discussion sections, as the research questions were not addressed in the Results section. Alternatively, revise the research questions to align with the results.

Response 3:We have revised the research questions and ensured they align with the results. Thank you once again for your valuable feedback. Should there be any further concerns, we are more than willing to make additional revisions.

Reviewer 4 Report

Comments and Suggestions for Authors

Abstrac.

The summary must incorporate the fundamental objective of the article.  Line 14

Introductión.

It is important to raise hypotheses, assumptions or research questions that are related to sustainability. It is recommended to look at the SDGs, especially those associated with education such as the fourth for example, as well as the use of technology; so that throughout the article but in a specific way in the objectives, the theoretical review, results and conclusions, important aspects about sustainability are presented as the central focus of the study in attention to the object of study associated with the evolution and application of misconception tier diagnostic technologies in science education.

Research Methodology

Incorporate the citation and author that supports the content analysis methodology

Indicate whether the methodology is your own, a modification or that of an author.

Results and Discussion

It is important that the key words of each section of the results can be specified and the authors of each article that refer to them can be cited, so that the categories or concepts of each section of the results that emerged from the content analysis of the results can be specified. texts or documents reviewed and studied.

Conclusión.

Since sustainability is the focus of this Journal, it is necessary and important that the conclusions are aimed at providing a contribution to it; it is recommended to incorporate sustainability in all sections of the article. It is also suggested to take a look at the SDGs, especially those associated with the quality of education and the use of ICT, in attention to the evolution and application of misconception tier diagnostic technologies in science education.

Finally, it is emphasized that in the revised manuscript, the word sustainability is not mentioned once, in this sense I consider it very important that the manuscript fits within the thematic objective of the journal.

 

Articles that talk about sustainability do not appear in mainstream literature, so it is considered imperative to incorporate concrete and specific aspects associated with sustainability in each component or section of the manuscript.

 

Comments on the Quality of English Language

The language must be improved to ensure perfect understanding of written speech

Author Response

Thank you for your thorough review and feedback. We have made every effort to implement the necessary changes.

Comments 1:The summary must incorporate the fundamental objective of the article.  Line 14
Response 1:We have completely rewritten the abstract to better reflect the key aspects of the study and ensure clarity.

Comments 2:It is important to raise hypotheses, assumptions or research questions that are related to            sustainability. It is recommended to look at the SDGs, especially those associated with education such as the fourth for example, as well as the use of technology; so that throughout the article but in a specific way in the objectives, the theoretical review, results and conclusions, important aspects about sustainability are presented as the central focus of the study in attention to the object of study associated with the evolution and application of misconception tier diagnostic technologies in science education.
Response 2:Thank you for your valuable suggestion. We have strengthened the connection to sustainability throughout the entire manuscript. These factors are now highlighted in the research background, discussion, results, conclusions, and directions for future research, with a clear emphasis on sustainability.

Comments 3:Incorporate the citation and author that supports the content analysis methodology. Indicate whether the methodology is your own, a modification or that of an author.

Response 3:Since this article has been classified as a systematic review, we have strictly followed the systematic review process and methodology. We referenced the following key studies in line with this approach:

• Miller, D. M.; Scott, C. E.; McTigue, E. M. "Writing in the Secondary-Level Disciplines: A Systematic Review of Context, Cognition, and Content." Educational Psychology Review, 2018, 30(1), 83-120. DOI: 10.1007/s10648-016-9393-z.
• Scott, C. E.; McTigue, E. M.; Miller, D. M.; Washburn, E. K. "The What, When, and How of Preservice Teachers and Literacy Across the Disciplines: A Systematic Literature Review of Nearly 50 Years of Research." Teaching and Teacher Education, 2018, 73, 1-13. DOI: https://doi.org/10.1016/j.tate.2018.03.010.
• Díaz-Burgos, A.; García-Sánchez, J.-N.; Álvarez-Fernández, M. L.; de Brito-Costa, S. M. "Psychological and Educational Factors of Digital Competence Optimization Interventions Pre- and Post-COVID-19 Lockdown: A Systematic Review." Sustainability, 2024, 16(1), 51.

Comments 4: It is important that the key words of each section of the results can be specified and the authors of each article that refer to them can be cited, so that the categories or concepts of each section of the results that emerged from the content analysis of the results can be specified. texts or documents reviewed and studied.
Response 4:We have rewritten the results section, focusing the literature review on different aspects. In the process of drafting the results, we closely linked the findings to the referenced literature. Thank you once again for your valuable feedback!

Comments 5 :Since sustainability is the focus of this Journal, it is necessary and important that the conclusions are aimed at providing a contribution to it; it is recommended to incorporate sustainability in all sections of the article. It is also suggested to take a look at the SDGs, especially those associated with the quality of education and the use of ICT, in attention to the evolution and application of misconception tier diagnostic technologies in science education.
Response 5: We have integrated the connection to sustainability in the conclusion and throughout other sections of the paper (you can easily locate references to "sustainability" throughout the document). We have focused on the relationship between the Education for Sustainable Development Goals and sustainable development in education. For example: "These efforts have strengthened public understanding of science and advanced the achievement of sustainable development goals, particularly in environmental protection and green technologies."

Reviewer 5 Report

Comments and Suggestions for Authors

Well written, but the authors did not explain why this study and this paper are important. What is important contribution to the field...The basic situation is important, but the rationale for this particular study is weak.

Author Response

Dear Reviewer,

Thank you for your constructive comments. Your feedback has been invaluable in enhancing the quality of this research. Below are our responses to your points:

Comment 1: The authors did not explain why this study and this paper are important. What is the significant contribution to the field?

Response 1: We have now addressed the importance and contribution of this study in the introduction, particularly in introduction 1.3, The Necessity of Conducting a Systematic Review on Multi-Tier Diagnostic Technologies. This section clarifies the significance of the research and outlines its contributions to the field.

Comment 2: The basic situation is important, but the rationale for this particular study is weak.

Response 2: We have strengthened the rationale for this study in introduction 1.1, The Core Role of Scientific Concepts in Scientific Literacy, and 1.2, Contributions of Existing Research in Revealing Misconceptions. These sections now provide a clearer theoretical foundation and context for the necessity of this study.

Wishing you a joyful Spring Festival and all the best in the year ahead!

Best regards,
Gaofeng Li

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

• The theoretical framework is much better, it focuses the study more concretely.
• The methodology remains unclear. In the abstract it appears that the steps specified in PRISMA are followed, but in the method, there is no reference to this issue and there is no reference to PRISMA having been followed.
• It still seems to me that the total number of papers selected is very small, and I think that either the search was not correct, or the exclusion criteria were inadequate.
• Study Participants and Locations  There are not participants, the title should be ‘Analysed documents and its locations’ for example.
• I don't know if these 28 articles selected for analysis are the result. I would not put this information here. It seems to me that it should be in the method, under ‘2.3 Types of Studies Reviewed’. The number of papers reviewed in a systematic literature review is the sample of a quantitative study. And that is not a result, that is part of the method.
• The method also still lacks two questions to be clarified:
  • How was the content analysis carried out? Which authors were used as references? What was the unit of analysis?
  • And above all, the categorical system that should be at the service of the questions or objectives is missing.
• The results section is not correct. It is much better than the previous one, now at least there is evidence related to the objectives and the documents analysed. However, instead of commenting on the data, instead of analysing them in depth, this section returns to theorising, which is not appropriate for this moment. This results in data being presented, but no analysis.
• My advice would be to revisit the research question: what do you want to know? Following the principle of parsimony, it would be advisable to be less ambitious and more concise in what you want to know in general and to operationalise this question, through the formulation of objectives, or through the formulation of more concrete questions. This will make the data section more economical and clearer. Perhaps a single table could be presented that summarises everything, and comments could be made, as well as going deeper into the specific operational objective/question, always avoiding references to theories/papers. If theory is included in this section, the data is not seen, it is not explored in depth, and it denotes that there is a position/look prior to the analysis, and this cannot be. Rigour in qualitative research is paramount.
• I encourage you to give it another look, because the study is interesting, but it still needs a lot of work.

Author Response

Dear Reviewer,

We sincerely apologize for not fully understanding your critical comments in the previous version, which resulted in our revisions not meeting your expectations. Thank you once again for patiently pointing out the issues in our research. In this version, we have carefully addressed all the concerns you raised, particularly making significant revisions to the results section. We hope this draft will meet your approval. If there are still any areas that require further adjustment, please feel free to contact us, and we will respond promptly. Once again, thank you for your constructive feedback (The places where we made changes in the manuscript are highlighted in red).

Comments 1: The methodology remains unclear. In the abstract it appears that the steps specified in PRISMA are followed, but in the method, there is no reference to this issue and there is no reference to PRISMA having been followed.

Response 1：Thank you for the valuable comments from the reviewer. We have explicitly addressed two PRISMA-related aspects in the revised Methods section. One clarifies that the study will be documented and visually presented following the PRISMA statement, while the other emphasizes that the PRISMA approach is used for quality assessment (highlighted in red in the original text: page 5, line 162, line 185).

Comments 2: It still seems to me that the total number of papers selected is very small, and I think that either the search was not correct, or the exclusion criteria were inadequate.

Response 2：We retained and excluded literature based on the selection criteria, ultimately identifying 28 SSCI papers. Additionally, while accurately identifying students' misconceptions is crucial, it seems that more researchers are now shifting their focus toward the transformation of misconceptions.

Comments 3: Study Participants and Locations. There are not participants, the title should be ‘Analysed documents and its locations’ for example.

Response 3：Thank you for pointing out the issue. We have revised the subheading to "Analysed Documents and Their Locations" based on your suggestion  (highlighted in red in the original text: page 8, line 254).

Comments 4: I don't know if these 28 articles selected for analysis are the result. I would not put this information here. It seems to me that it should be in the method, under ‘2.3 Types of Studies Reviewed’. The number of papers reviewed in a systematic literature review is the sample of a quantitative study. And that is not a result, that is part of the method.

Response 4：Thank you for your valuable comments. We have revised the manuscript according to your suggestion by moving the relevant content from the Results section to Section 2.4 Literature Screening Process in the Methods section to more accurately reflect the research process (highlighted in red in the original text: page 5, line 202) .

Comments 5: The method also still lacks two questions to be clarified: How was the content analysis carried out? Which authors were used as references? What was the unit of analysis?

Response 5：In the revised manuscript, we have added a new section, 2.5 Content Analysis Process, to elaborate on the specific implementation of content analysis. In this study, the unit of analysis is an individual article, with each article considered an independent analytical object. Additionally, to ensure coding consistency, we selected a 20% sample of the literature, which was independently recoded by three authors, referring to relevant studies by Risko, Bean, Nourie, and others (highlighted in red in the original text: page 6, line 212).

Comments 6: And above all, the categorical system that should be at the service of the questions or objectives is missing.

Response 6：In the revised manuscript, we have added “Table 1. Categorical Coding System Exemplifying Two-Tier Diagnostic Technology”, using Two-Tier Diagnostic Technology, one of the core categories, as an example to detail the structure of the categorical coding system (highlighted in red in the original text: page 7, line 246).

Comments 7: The results section is not correct. It is much better than the previous one, now at least there is evidence related to the objectives and the documents analysed. However, instead of commenting on the data, instead of analysing them in depth, this section returns to theorising, which is not appropriate for this moment. This results in data being presented, but no analysis.

Response 7：Thank you for your feedback! We highly value the critical issues you raised, and we have made substantial revisions in this section. We have classified the data in the table, incorporated quantitative data analysis, and examined the factors contributing to these results. Additionally, we compared different sets of data, explored the differences between them in depth, and analyzed the potential factors that could influence these data (Highlighted in red in the original text in the results section. Due to extensive revisions, specific locations cannot be individually identified).

 

 

Reviewer 3 Report

Comments and Suggestions for Authors

The manuscript has been significantly improved. The authors addressed my comments, revised the research questions, and updated Figure 2. Additionally, the Results section is now more appropriate.

Author Response

Thank you for your comments. If there are any further adjustments needed, please feel free to let us know, and we will respond as soon as possible.

Reviewer 4 Report

Comments and Suggestions for Authors

Based on the review, it is recommended:
< !--StartFragment -->

In Figure 3, it is important to place a legend that allows understanding the distribution of the graph in attention to the colors and what each color represents, this to facilitate its interpretation.

< !--EndFragment -->

Comments for author File: Comments.pdf

Comments on the Quality of English Language

It can be improved to make the text more understandable.

Author Response

comments 1: In Figure 3, it is important to place a legend that allows understanding the distribution of the graph in attention to the colors and what each color represents, this to facilitate its interpretation.

response 1：Thank you very much for the necessary feedback you provided. We have made the revisions and added detailed explanations after the title of Figure 3 (highlighted in green in the original text, page 21, line 448).

comments 2: It can be improved to make the text more understandable.

response 2：We deeply apologize for the language deficiencies. However, the manuscript is currently undergoing significant revisions, and the text is not yet stable. Once our text is more stable, we will promptly contact MDPI Author Services to improve the language.

 

Round 3

Reviewer 2 Report

Comments and Suggestions for Authors

The contribution presented by the authors is improved in the latest versions.

Consistency has been achieved between the abstract in which PRISMA is mentioned and the method in which it reappears, giving coherence to the discussion.

As for categorisation, an example appears in the latest version. However, this would not be sufficient. The categorisation to be accounted for should be the roadmap to follow for interpretation and understanding of the data. My suggestion is that work should continue in this section, and that all the tables shown in the results should be conceptualised, translated into a categorical system. Because these tables are the way in which it has been decided to organise the information. This issue must be made clear in the method, in a summarised, concrete and clear way. So that in the results, the reader does not have to infer how the texts have been organised.

In the results, the tables should not be presented as they are. That would be the raw data. In this section, the analysed data should be presented. It would be necessary to rethink it and look for another way. These tables can be left as annexes if you want, but not like this, because otherwise it is the reader who has to do the work.

On the other hand, I mentioned this in the previous review, but I would like to comment again. Bibliographical references should not be included in the results section. This section is exclusively for the presentation of the data, so if you follow the criteria of this author to analyse something, or if you test the way another author did it, you should not reference any work.

The fact that there are so many references indicates either that the theoretical framework is not sufficient and needs to be extended to locate the work that follows; or that the discussion and debate is being advanced. 

Author Response

Thank you for your thoughtful and constructive feedback on our manuscript. Your suggestions have significantly improved the quality of our work.

Comments 1：My suggestion is that work should continue in this section, and that all the tables shown in the results should be conceptualised, translated into a categorical system. Because these tables are the way in which it has been decided to organise the information. This issue must be made clear in the method, in a summarised, concrete and clear way. So that in the results, the reader does not have to infer how the texts have been organised.

Response 1：We transformed the original data tables in the results section into a categorical system, while the original tables containing raw data are provided as supplementary materials. To help readers better understand how the focus areas in the Results section were categorized and organized, the final paragraph of the Methods section clearly states that the tables presented were structured as a three-level classification system across five focal domains. This structure was guided by the overall categorical system and aligned with the research objectives.

Comments 2：In the results, the tables should not be presented as they are. That would be the raw data. In this section, the analysed data should be presented. It would be necessary to rethink it and look for another way. These tables can be left as annexes if you want, but not like this, because otherwise it is the reader who has to do the work. 

Response 2：The tables in the Results section have been processed to present analyzed and summarized data. The original tables containing raw data are provided as supplementary materials.

Comments 3：Bibliographical references should not be included in the results section. This section is exclusively for the presentation of the data, so if you follow the criteria of this author to analyse something, or if you test the way another author did it, you should not reference any work. 

Response 3：All references cited in the Results section are drawn exclusively from the 28 reviewed studies. They are used solely to present supporting evidence. This approach is consistent with practices adopted in previous some systematic reviews (1. Scott, C. E.; McTigue, E. M.; Miller, D. M.; Washburn, E. K. The what, when, and how of preservice teachers and literacy across the disciplines: A systematic literature review of nearly 50 years of research. Teaching and Teacher Education 2018, 73, 1-13. DOI: https://doi.org/10.1016/j.tate.2018.03.010. 2. Miller, D. M.; Scott, C. E.; McTigue, E. M. Writing in the Secondary-Level Disciplines: a Systematic Review of Context, Cognition, and Content. Educational Psychology Review 2018, 30 (1), 83-120. DOI: 10.1007/s10648-016-9393-z. Díaz-Burgos, A.; García-Sánchez, J.-N.; 3. Álvarez-Fernández, M. L.; de Brito-Costa, S. M. Psychological and Educational Factors of Digital Competence Optimization Interventions Pre- and Post-COVID-19 Lockdown: A Systematic Review. Sustainability 2024, 16 (1), 51).

Thank you again for your hard work. Wishing you success in everything and all the best!