Procedural Learning in Mixed Reality: Assessing Cognitive Load and Performance
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe authors posit that this investigation constitutes the first examination of the split-attention principle in the context of a highly immersive medium. To the best of their knowledge, no studies have hitherto been conducted on the implementation of the split-attention principle in XR in the field of medicine (l. 58), the study however investigates the use of MR technology. This needs clarification.
As the authors emphasise in the abstract, the study provides a foundation for designing procedural simulation training that considers both learners' needs and cognitive processes in highly immersive environments.
However, the results did not demonstrate significant differences in performance or cognitive load between conditions during recall, which contradicts the initial hypotheses. The hypotheses have thus been left unverified by the study as the authors state. A question arises whether the specific sample has significance here, since as iti is mentioned in section 4.2 intrinsic motivation can interfere the results (ll. 476-485). This is also mentioned in section 4.3 - limitations.
The paper's structure and definitions are clear and reflect the latest knowledge in the field. It is recommended that the introduction be modified to present the notions that are important for the study first, with sections 1.2, 1.3 and 1.4 preceding all other clarifications, as they explain what is used throughout the paper from its very beginning.
It is evident that there is a recognition of the constraints imposed by the integration of MR tools within medical training. An overly technocentric approach, which prioritises technological aspects over the cognitive requirements of learners, frequently results in a discrepancy between expectations and actual experiences, occasionally culminating in the rejection of these tools. The Authors notice that the novelty effect of technologies like HoloLens 2 may hinder procedural learning.
The primary concern pertains to the selection of the research sample, which is modest in size, comprising only 25 individuals (as one was forged). This is a typical number for qualitative research but appears somewhat inadequate for quantitative research.The authors are requested to provide a justification for the binding nature of the conclusions derived from research with such a limited sample. Additionally, further information should be included regarding the characteristics of the surveyed participants. Were medical students deliberately excluded, or were there any medical students among the sample? The text mentions only psychology students, and thus the study participants are not sufficiently characterised, especially given the relatively small sample size. The authors adress the latter in section 4.3 - limitations, but it might be also explained as e.g. a contribution to further research projected by the same research group.
This fact significantly downgrades the overall merit of the article. However, I look forward to extensive explanations from the authors. The results might have been different had the sample consisted of first-year medical students (freshmen).
If medical students have been deliberately excluded from the study, this should be justified, as the results of the study are particularly binding for the education methods of future surgeons.
Moreover, Appendix E seems missing.
If the findings did not reveal a significant difference in learning performance shouldn't the authors be more critical of the intorduction of MR in medical training? This is an important question that should be explored in more depth with a larger and more representative sample, perhaps including only students undergoing medical training. It does not come out very cleat that the use of MR technology always holds promise for improving procedural training.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Comments and Suggestions for AuthorsThis study explores the application of the split-attention principle in mixed reality-based medical training, specifically in knot-tying procedures using Microsoft HoloLens 2, highlighting both its potential contributions and the challenges in aligning immersive technology with established learning theories. I read the manuscript with great interest, but I have some concerns that I describe in the following.
A major limitation of the study is the lack of statistical power due to the small sample size (N = 26). This affects the reliability and interpretability of the findings, necessitating caution when interpreting the findings. Given the complexity of the analysis, a larger sample would be required to detect meaningful effects. While the limitations section mentions a power analysis based on Ginns' meta-analysis and G*Power calculations, this should have been incorporated into the study design rather than as a post-hoc consideration. A clearer justification for the sample size selection is needed in the methodology.
The observed significant effect on learning time and perceived complexity should be interpreted with caution due to the small sample size. Confidence intervals and effect size reporting would help clarify the robustness of these findings. Similarly, the lack of significant differences in learning performance and cognitive load could be attributed to insufficient power rather than a true null effect. This should be acknowledged more explicitly in the discussion. The discussion should explore alternative explanations for the results, including the possibility of Type II errors due to the study's limited statistical power.
The statement about the "potential" of the split-attention principle in MR procedural simulations is speculative. Given the mixed findings and power limitations, stronger emphasis should be placed on refining the experimental design for future research. The authors should consider conducting a larger-scale replication study to confirm the validity of their findings.
The introduction, spanning 5 pages, could be more concise to enhance focus and readability.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
Comments and Suggestions for AuthorsDear author,
Thank you for submitting your manuscript. I believe your work addresses an interesting and relevant topic, undoubtedly enriching the discussion in this area of research. However, I would like to offer some observations and suggestions for improvement, particularly from a methodological perspective.
1. Introduction: While the introductory section is clear and well-structured, it appears too lengthy for the reader. Moreover, I have identified certain repeated ideas that could be synthesised. I recommend reducing the length of the introduction and carefully reviewing the citations to ensure they adhere to the journal's required format (APA 7th edition). Specifically, in Section 1.5, it would be advisable to explicitly include the study's objectives, research question, and hypotheses, as these elements are not clearly defined. The hypotheses, currently located in Subsection 2.4, should be relocated to this section.
2. Methodology: From a methodological perspective, Subsection 2.1 highlights significant limitations regarding the representativeness of the sample, which affects the study's reliability and robustness. It would be advisable to significantly expand the sample size to strengthen the validity of the results. Additionally, given the use of quantitative instruments, it is necessary to provide more detailed information about them, including descriptions, reliability coefficients (e.g., Cronbach's alpha), content analysis, and the number of items per dimension or factor. In Subsection 2.4, I suggest including only the data analyses employed in this study, avoiding redundancies.
3. Results: The analyses conducted are overly basic for the level of rigour expected by this journal. More advanced methods and deeper interpretations of the findings would be necessary.
4. Discussion: This section is clearly written and well-structured.
5. Conclusions: I recommend delineating the study’s findings more clearly and concisely to facilitate the reader’s understanding.
6. References: It is essential to adjust the references to comply with the journal's guidelines, which in this case adhere to APA 7th edition.
Given the limitations noted, particularly the lack of representativeness of the sample, I regret to inform you that your manuscript has not been accepted for publication in this journal. I hope the observations provided will be of use for future revisions and submissions.
Kind regards,
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Comments and Suggestions for AuthorsI have no further comments.
Reviewer 3 Report
Comments and Suggestions for AuthorsDear author,
Thank you for your time and for carefully considering the comments provided. After a thorough review of your second version, I find the manuscript to be clear and believe it represents a significant first step in the medical field.
Best regards.
