Enhancing Vocational Training in the Post-COVID Era through Mobile Mixed Reality

Round 1

Reviewer 1 Report

This is an interesting study presenting a novel mixed reality tool to enhance vocational training. The authors found that the mixed reality tool could be a valuable tool to supplement traditional vocational learning methods.

Overall, the study may have important implication. However, I have several major concerns that require the authors' attention:

1. First, the sample size of the current  study is too small. I don't think it is possible to infer the effectiveness of the tool with such as small sample. I strongly encourage the authors to include more sample in the current study. For now, the sample size is too small for publication.
2. Second, there is a need to conduct inferential statistics to evaluate the mean difference between the conditions. It is not appropriate to just compare the mean differences without any inference statistic.
3. More information about the sample characteristic is necessary. There is too little information for the readers and reviewers to evaluate the methodology of this study,

Author Response

Thank you for taking the time to read our paper and provide such detailed feedback. I have made the following changes to the manuscript thanks to your advice:

1. While we agree that a larger sample size would have provided more confidence in the tool’s effectiveness, we are unfortunately unable to conduct further experiments in the timescale available. We have now addressed this point in the discussion section of the paper (Lines 389-394, 416-420) and recommend expanding this study in future. The statistical analysis we conducted in response to your second point also addresses this to an extent.
2. We have now included a more thorough statistical investigation of the data in Lines 271-307, using a t-test to examine whether results bear statistical significance, discussed in Lines 396-407, 422-431.
3. We have included more information about participant demographics and background (Lines 186-190, Table 1).

The manuscript has been edited with tracked changes on, so changes in response to all three reviewers’ feedback may be easily identified. I hope this is clear and that all changes are to your satisfaction.

Reviewer 2 Report

This study present a novel MixedReality tool to simulate and guide learners through a simple fault diagnosis task of a three-phase powersupply. I consider that this study is perfectly constructed at a conceptual, methodological and critical-reflective level.

However, the authors need to make some minor modifications.

In the first place, it is very important to provide more information within the theoretical-conceptual framework in which the importance of the digital competence of teachers is specified in order to develop innovative methodologies related to socioconstructivist teaching (ICT, specially).  It is recommended to review the following papers:

Pozo-Sánchez, S., López-Belmonte, J., Fernández, M. F., & López, J. A. (2020). Análisis correlacional de los factores incidentes en el nivel de competencia digital del profesorado. Revista Electrónica Interuniversitaria de Formación del Profesorado, 23(1), 143-151. https://doi.org/10.6018/reifop.396741

Pozo-Sánchez, S., López-Belmonte, J., Rodríguez-García, A. M., & López-Núñez, J. A. (2020). Teachers’ digital competence in using and analytically managing information in flipped learning. Culture and Education, 32(2), 1-35. https://doi.org/10.1080/11356405.2020.1741876

Secondly, it is important that the authors explain in depth the prospects of the study, providing the added value of the research and how the results obtained contribute to improve the knowledge of the educational context and to expand the information available in the scientific field.

Consequently, with the aim of reinforcing a field of study of sustainability that currently needs pioneering publications.

Congratulations to the researchers.

Author Response

Thank you for taking the time to read our paper. We appreciate your kind words as well as your detailed and constructive feedback. I have made the following changes to the manuscript thanks to your advice:

1. I have included some more theoretical background to the learning theories underlying digital teaching methods as per your advice (Lines 87-93). Based on the findings of the papers you recommended, I have also added a note in the conclusions on the importance of having suitably skilled educators to support digital teaching methods such as this (Lines 396-408).
2. We have included more information about participant demographics and background in Lines 186-190, Table 1

The manuscript has been edited with tracked changes on, so changes in response to all three reviewers’ feedback may be easily identified. I hope this is clear and that all changes are to your satisfaction.

Reviewer 3 Report

Manuscript ID: sustainability-1217900

Enhancing Vocational Training in the post-COVID era through mobile Mixed Reality

The authors present a study investigating the MR based tool to simulate and guide learners through a simple fault diagnosis task of a three-phase power supply. Comparisons were made between expert and non-expert users (in the electrical engineering field) while using the tool. The results indicate the MR guidance proved useful and that the non-expert users outperformed the expert users in tasks provided.

All references are included in the manuscript.

The authors could consider the following suggestions for improvement.

1. The authors should provide a more detailed description of the study design, procedure and tasks. How exactly did the MR application look like (a web based MR app?) and how was it used in the experiment. Some information is provided in chapter 2, however it is unclear.
2. The authors should provide at least basic demographic participants’ information, e.g. average age and standard deviation of the study participants, how the participants were assessed on their professional background (to determine the expert and non-expert users), technology acceptance (e.g. previous experience using XR devices, …).
3. More detailed Nasa-TLX scores should be provided and briefly discussed. Only average score could be misleading.

Author Response

Thank you for taking the time to read our paper, and for your detailed and constructive feedback. I have made the following changes to the manuscript thanks to your advice:

1. A more thorough description of the app’s functionality is given in Lines 220-231, to supplement the explanation and screenshots of the app in Sections 2.1 and 2.2 (pgs 7-9, especially Tables 4 and 5) – I hope this is now sufficiently clear.
2. We have included more information about participant demographics and background in Lines 186-190, Table 1.
3. I have included a more detailed report of the NASA-TLX scores (Lines 293-305) including both a box and whisker plot to demonstrate the spread of results (Figure 9) and a breakdown of the components of the TLX score (Table 5). We have also now included a more thorough statistical investigation of the data in 271-307.

The manuscript has been edited with tracked changes on, so changes in response to all three reviewers’ feedback may be easily identified. I hope this is clear and that all changes are to your satisfaction.

Round 2

Reviewer 1 Report

I appreciate the authors' effort in revising the manuscript. However, I still have concerns. One major concern is that interpretation of the result. It seems that the results are not significant at .05. However, the authors interpreted it as significant (some marginal significant results should not be interpreted as significant too). This is very misleading and not appropriate for publication. The authors should rewrite the results section as well as revise their discussion to fit with their results.

Author Response

Thank you for your feedback.

• I have re-written these parts of the results section (Lines 285-302) to make clearer that these do not meet the threshold of p < 0.05.
• The discussion has been edited accordingly to match this (Lines 365-462).

As before, I have edited the document with tracked changes so you may easily see the edits. Some minor grammar errors are also corrected, and a few items moved around in the discussion for better flow. I hope these changes are to your satisfaction, and thank you again for taking the time to review our work.

Reviewer 3 Report

Manuscript ID: sustainability-1217900 review 1

Enhancing Vocational Training in the post-COVID era through mobile Mixed Reality

The authors have to a degree addressed the comments raised, however some issues remain, mainly for:

“I have included a more detailed report of the NASA-TLX scores (Lines 293-305) including both a box and whisker plot to demonstrate the spread of results (Figure 9) and a breakdown of the components of the TLX score (Table 5). We have also now included a more thorough statistical investigation of the data in 271-307.”

• Page 10, 11, While true, the statistical results indicate a difference it is not a statistically significant difference (usually a p value less than 0.05 is required). The authors should explain the results accordingly. Were any statistical tests made to check the normality of data?
• Which scores are represented in Figure 9?
• Table 5 is missing. NASA TLX scores and corresponding discussion are missing.

Author Response

Thank you for your feedback, please find revisions below:

Page 10, 11, While true, the statistical results indicate a difference it is not a statistically significant difference (usually a p value less than 0.05 is required).

• I have re-written these parts of the results section (Lines 285-302) to make clearer that these do not meet the threshold of p < 0.05.
• The discussion has been edited accordingly to match this (Lines 365-462).

Were any statistical tests made to check the normality of data?

• Visual inspection of QQ-Plots of dependent variables in Lines 272-283, Figure 7

Which scores are represented in Figure 9?

• Figure 10 (was Figure 9) represents the unweighted TLX scores, divided into ‘expert’ and ‘non-expert’ user groups – I have updated the figure caption (Line 329-330) to make clearer that this is the so-called ‘raw TLX’ score rather than the weighted version, and added in a bit more description of the figure in the main text (Lines 319-322)

Table 5 is missing.

• Tables 5-6 (was 6-7) are renumbered (Lines 334, 343)

NASA TLX scores and corresponding discussion are missing.

• NASA-TLX scores are summarise in Table 4 (Line 318) and Figure 10 (was 9) (Line 323)
• Discussion of NASA-TLX scores in Lines 391-40

As before, I have edited the document with tracked changes so you may easily see the edits. Some minor grammar errors are also corrected, and a few items moved around in the discussion for better flow. I hope these changes are to your satisfaction, and thank you again for taking the time to review our work.

Round 3

Reviewer 1 Report

The authors have sufficiently addressed my concerns

Reviewer 3 Report

The authors have to some extent addressed the issues raised.