Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper concerns an intervention in the teaching of VSEPR using AR on iPads. At least I think they were iPads, for some reason it is spelt Ipad throughout. The authors present an extensive introduction to the field cognitive load theory. I really enjoyed reading it but it seems somewhat disproportionate to the results presented. The whole article is somewhat wordy. It could be considerably shortened. It is also rather general to the field of education. For me the introduction is missing the insights of authors like Johnstone, and more recently Mahaffy and Taber into the unique challenges of chemistry. 

The research questions are clearly presented and well framed. However, the actual study is limited in scope and scale. The insights over-reach the evidence gathered but I do believe this will prove a valuable resource for future studies. I would like to be reassured about the ethical considerations given, which don't appear to be mentioned.

There are some peculiar technical issues with the manuscript that appear to relate to the referencing tools used. It is disappointing the authors appear not to have noticed. 

For me, the abstract contains too much content from the introduction.

 

Author Response

Thank you for taking the time to review the manuscript, please find responses to the points that you made uploaded into the attached word document. Kind regards.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

Abstract

Goal is not provided. Variables not clear. Explain how CL is addressed in the need you identified.

Data collection strategies and context not described.

How is achievement operationalized? Is it 36 or 42 students in total?

Also include implications.

Intro

Current knowledge on MR abilities and WM and AR and similar aids in chemistry education, previous empirical research, well presented. Achievement is not included in RQs, once again operationalize achievement. Clarify why it is important to explore effects of MR abilities, what do you mean unresolved, what is particularly missing and who can benefit from such result in what ways?

Remove main findings of your research.

MR abilities are not mentioned as outcome in Abstract.

Literature review

Report further on previous effective educational interventions and their characteristics (on visuospatial skills).

In general, more empirical evidence from relevant previous chemistry education research needs to be integrated.

The conflicting findings so far on use of AR technology in this context and how the goal of this study is addressing that needs to be clarified, therefore improvement of the study significance on final part of this section is recommended.

Method

What is an original group?

Further clarify the 2 x 3 experimental method adopted.

Provide more information on the context, the lessons (Structure and content).

Demographics on participants? How the school was selected. Variables and corresponding measures not clearly reported. Any relevant control variables?

Citation for post-test or did the authors develop those and how? Reliability-validity information on measures as well as data collection procedures is missing. Empirical studies or educational interventions that used the same app.

No information on data processing, types of analyses run on SPSS and on normality and reliability checks.

Results

Reporting of findings are not fully in line with APA.

Align your findings with RQs.

Discussion

In general, great improvement needed on comparing against earlier empirical results in chemistry education, compare/contrast also to research reported in Intro.

Improve theoretical/research implications and practical implications.

Comments on the Quality of English Language

Great improvement of scholarly reporting and a double-check against APA criteria are needed.

Students’ MR abilities (line 67)

Line 152 & 242 not complete.

One day & seven days on line 241.

Check for fonts and APA for Table1, Table 2.

Refine line 253.

Structure and fonts in Conclusion.

Author Response

Thank you for taking the time to review the manuscript, please find responses to the points that you made uploaded into the attached word document. Kind regards.

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors

The introduction is well written, and the article is well referenced. I think the topic will be of interest to the community. The primary concern I have is with the measurement of Cognitive Load (CL). The study uses a 9-point scale from too easy to too difficult. I am not convinced that this is an appropriate way to measure CL, and the article provides no references for why this was selected. The article also does not describe measures taken to determine validity and reliability of this survey as a means to establish CL. Because I am wary of the measurement of CL, it makes it difficult for me to trust any of the conclusions drawn in the paper. Along those lines, I would highly recommend adding a "Limitations" section to the article. 

Below I have included more specific comments:

Generally:

-              There are a lot of acronyms used. It may be worthwhile to keep some as full words for readability’s sake.

-              There seems to be some issue with referencing figures within the text (specific cases are given below)

 

Introduction:

-              Lines 42-51: You may want to consider referencing work done by Alex H. Johnstone (e.g., Johnstone’s triangle)

-              Line 152: Reference error

-              Line 242: Reference error

 

Methods:

-              It seems that the students were randomly placed in the Control and Intervention groups, but it would be helpful to see comparison statistics for the two groups (gender, race/ethnicity, GPA, etc.), especially since the introduction mentioned that these types of things can affect videospatial abilities.

-              Did students volunteer to participate in the study? Please provide details about recruitment, including IRB reference number. 

-              Were any instructions given to students other than those found within the provided booklet? If so, please describe that instruction, as well as how you ensured it was the same for both the Control and Intervention group.

-              When did the study take place? Was this part of a course? Were the control and treatment groups meeting at the same time, but just in different spaces?

-              Line 253: Reference error

-              Lines 250-257: Who developed the pre-test? It seems that it may have been developed by the researchers, but it is unclear that the pre-test and post-test have proper validity. I think more information is required on how the pre- and post-tests were designed and what methodology was used to support that design.

-              Please explain how the achievement test and CL test were scored.

-              Line 268: Reference error

-              Line 281: Reference error

 

Results:

-              It was unclear to me which statistical tests were run. Please clarify what specific tests were performed for data analysis. For instance, did the researchers control for pre-test scores when doing the post-test analysis? This feels like an important control given the higher pre-test achievement score for the Intervention group.

-              Did you look to see if the changes between T3 and T4 within a group were significant?

 

Discussion:

-              Lines 317-320: It seems that you are testing cognitive load in regard to the achievement test questions, not to the actual activity involving AR. As such, when students in the Intervention group report lower cognitive load on the post-tests, how do you know it is related to what they did with AR and not something else?

-               Lines 324-330: I do not believe the researchers have collected the appropriate data to make claims as to whether the intervention led to better WM capacity and transfer to LTM. I realize the word ‘suggest’ is used, but I still feel like these statements are hypothesizes that have not been tested in this study.

-              Lines 334-337: “The I group participants were however able to apply MR visualisations more freely with less obstruction. Those I group participants who experienced lower level of CL achieved improved learning results in part due to the greater availability of cognitive resources that could be assigned to the learning of new content material.” I am not sure the researchers have the proper evidence to support these claims. Group I participants may have reported lower cognitive load levels, but the study does not provide a means to say that it was the AR specifically. I feel like an open question at the end of T3 would have been helpful in getting student perspective on what made them respond to the CL test as they did. I would also be careful with the wording here – students report experiencing lower CL, as it has been shown that student perceptions may not always be accurate. This is self-reported data and should be referenced as such.

-              Lines 359-364: I feel like this statement is not well supported by the evidence collected in this study.

 

Conclusion:

-              Line 368 – It states iPads were used, but earlier it was stated that the program was downloaded onto student phones (there is also a reference to iPads in the abstract) Please check and correct.

-              Line 371: The study does not measure mental dexterity so I would recommend not making claims that the intervention benefited this construct.

Author Response

Thank you for taking the time to review the manuscript, please find responses to the points that you made uploaded into the attached word document. Kind regards.

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

I appreciate the efforts made by the authors to address my concerns. Many of my issues were addressed, but I feel the major issue of validation and reliability of the instrument used is still insufficient.

I agree that a Likert scale can be used to measure subjective constructs, but care must be taken when designing such surveys to ensure that they are indeed measuring the intended construct. This includes analyzing those instruments for reliability (Cronbach's alpha, test-retest) and validity (exploratory factor analysis, confirmatory factor analysis, review from outside experts).  

The authors cite Keller et al. (2021) and Klepsch et al. (2017) - both of these works use a validated survey that measures three types of cognitive load (intrinsic, extraneous, and germane). This instrument was carefully considered and revised based on detailed analysis. Based on the authors' response, it seems that no serious analysis was done to test the validity and reliability of the instrument used ("Additional prior pilot testing was also implemented to further ensure validity in the instruments designed. Reliability was considered by undertaking data collection within the familiar environment of the participants school during schooling hours, in addition to the prior pilot testing, to circumvent potential issues such as fatigue, nervousness, or misinterpretation of questions.

Author Response

Thank you for your recent feedback, please find our response to your comments below.

Comments	Response
I agree that a Likert scale can be used to measure subjective constructs, but care must be taken when designing such surveys to ensure that they are indeed measuring the intended construct.	We rebut the statements regarding reliability and validity in this context as the measure of cognitive load implemented is a standard, commonly used, and well known procedure within the field. This has been asserted and supported in section “2.2 Materials.”
This includes analyzing those instruments for reliability (Cronbach's alpha, test-retest) and validity (exploratory factor analysis, confirmatory factor analysis, review from outside experts).	The measure of load implemented is an established procedure within the field. We assert that it is an acceptable approach to tally the independent items without any presumption as to intercorrelations between the items tallied. A test such as Cronbach would not be applicable as each item/judgement is a discrete entity, whereas Cronbach assumes each item represents a trait as shared across items.
The authors cite Keller et al. (2021) and Klepsch et al. (2017) - both of these works use a validated survey that measures three types of cognitive load (intrinsic, extraneous, and germane). This instrument was carefully considered and revised based on detailed analysis. Based on the authors' response, it seems that no serious analysis was done to test the validity and reliability of the instrument used	We reiterate that we are employing respected procedures regarding the measurement of cognitive load. In this case there is no index against which to reference the scale as it stands for itself, rendering the concerns of validity/reliability in this respect irrelevant.