Switching Learning Methods during the Pandemic: A Quasi-Experimental Study on a Master Course

Round 1

Reviewer 1 Report

This manuscript reporting a case study for a course at the master's level MSDG course, by teaching it entirely online during the Corona pandemic. A comparison was made between teaching through blended learning, and learning that is fully available online. The results showed that there were no statistical differences between the two academic years (i.e.,19/20 blended and 20/21 online academic years). The manuscript in general is good, but it needs substantial modifications, as detailed below:

- Despite the quality of the information mentioned, the manuscript needs a linguistic revision as it is difficult to understand what the authors mean (i.e., Moderate English changes required)

- In lines 47-51 The authors have stated that "it is necessary to change the way of teaching using technology .........." But they did not touch on the main competencies of teaching with technology, so it is better to write a paragraph dealing with the professionalism and readiness of the faculty members for online teaching using technology.

- In line 95 - 110 the authors addressed the topic of learning analytics (LA), with some studies and experiences monitoring this, but during the pandemic LA need to be further developed and so suggested addressing the topic of how to design cognitive and behavioral LA for supporting learners’  learner’s awareness, reflection, and learning process. Then determined which form has been applied in this research, since the current manuscript does not clarify the role or type of learning analytics on the objectives of learning analytics (i.e., awareness, reflection, and learning process).

- The research methodology needs further clarification (is it a case study, or a quasi-experimental research, and where is the control group) If so, please review the research methodology carefully and describe it well.

-Since the sample is small, the researchers resorted to non-parametric statistics (i.e., Wilcoxon-Mann-Whitney test) therefore, the criticism directed at this seems to be why some qualitative measures and analysis of interviews with students and faculty members were not used? Whereas, the statistical results did not show any difference between the results of the last year and the current one.

- In table 1 It was mentioned that there were tests that took place, despite that, the evaluation methods used, their dates, and the details of the measurement tools were not clarified.

- Figures 4 and 5 do not provide any new information and are considered a duplicate of Table 3.

- In the end, I would like to thank the authors for their effort. I believe that with more effort, this manuscript can be developed, and I am confident in their ability to do so.

Author Response

#1 - Despite the quality of the information mentioned, the manuscript needs a linguistic revision as it is difficult to understand what the authors mean (i.e., Moderate English changes required)

We gratefully welcome the reviewer's suggestion and have carried out a proof-reading by a native speaker to let the reading smoother.

#2 - In lines 47-51 The authors have stated that "it is necessary to change the way of teaching using technology .........." But they did not touch on the main competencies of teaching with technology, so it is better to write a paragraph dealing with the professionalism and readiness of the faculty members for online teaching using technology.

Thanks for this valuable comment aimed at increasing the quality of the paper. The board and its competencies were not actually described.

Accepting the suggestion of the reviewer we have inserted the following paragraph just before the description of the master.

MSDG involves a board of professionals who are experts in sustainability management and digital learning fields. Subject matter experts are responsible for ensuring high levels of teaching with constantly updated content. Digital learning experts are responsible for ensuring the best way of attending the online course (lessons, assessment, tutoring). Digital learning experts are highly skilled profiles from academia and business. They have skills in the field of digital teaching methodologies and technologies to support learning.

#3 In line 95 - 110 the authors addressed the topic of learning analytics (LA), with some studies and experiences monitoring this, but during the pandemic LA need to be further developed and so suggested addressing the topic of how to design cognitive and behavioral LA for supporting learners’  learner’s awareness, reflection, and learning process. Then determined which form has been applied in this research, since the current manuscript does not clarify the role or type of learning analytics on the objectives of learning analytics (i.e., awareness, reflection, and learning process).

We thank the reviewer for this valuable comment. We agree that it needs to discuss the learning analytics methodology used.

We added a new paragraph in the literature review section with a new reference (https://doi.org/10.1080/10494820.2021.2009881). It clarifies that we are going to use cognitive learning analytics in order to verify students’ behavior and analyze the data to evaluate the students’ performances.

#4 The research methodology needs further clarification (is it a case study, or a quasi-experimental research, and where is the control group) If so, please review the research methodology carefully and describe it well.

Thank you very much for underlying this important aspect. We have discussed the most proper research methodology, and we decided to adopt a quasi-experimental study using as control group students of the a.y. 19/20. We revised the article, results, and discussion accordingly.

#5 Since the sample is small, the researchers resorted to non-parametric statistics (i.e., Wilcoxon-Mann-Whitney test) therefore, the criticism directed at this seems to be why some qualitative measures and analysis of interviews with students and faculty members were not used? Whereas, the statistical results did not show any difference between the results of the last year and the current one.

Thank you very much for this important comment.

The aim of this research is to demonstrate that there is no difference between groups despite the change in methodology (from blended to full online). In this case, a non-parametric test demonstrates that there is no difference between groups by analyzing the performance of the users through the ongoing evaluations.

The paradigm shift was performed urgently and no comparable data on user satisfaction were collected in the two years presented.

#6 In table 1 It was mentioned that there were tests that took place, despite that, the evaluation methods used, their dates, and the details of the measurement tools were not clarified.

We thank the Reviewer for the comment. We did not have access to the evaluation methods used by each professor during the tests, thus we were not able to perform further analysis. We have added to the paper that all the students passed the assessments in the same session, thus a temporal analysis of the dates would not have given any additional information.

#7 Figures 4 and 5 do not provide any new information and are considered a duplicate of Table 3.

We thank the Reviewer for this comment. We agree that the images do not provide any additional information to the research, thus we deleted them.

#8 In the end, I would like to thank the authors for their effort. I believe that with more effort, this manuscript can be developed, and I am confident in their ability to do so.

We thank the Reviewer for the support, we worked hard to improve the quality of the article, following the precious comments. Many parts of the article have been totally rewritten.

Reviewer 2 Report

Dear authors, thank you for considering Applied Sciences for your submission and for conducting a study on learning education.

Allow me to share some doubts:

#1 The title should not have abbreviations, especially the ones that most of the readers do not know what they mean, such as MSDG. I recommend to use the extended name.

#2 The abstract should address also the methodology of assessment employed. I suggest to add it.

#3 I recommend to place the keywords by alphabetic order.

#4 The Introduction looks sound. However, I believe a relevant study has been missing from the lit review which is the DOI: 10.14744/eej.2019.43534 which is a study addressing the online education and the participants feedback of one of the first major online education activities in Dentistry. Some of those feedback are truly relevant and should be debated in the Introduction or Discussion sections (which presently is not available in the manuscript).

#5 The aim sentence is acceptable, but the last sentence of the Introduction is not needed and I recommend to be removed.

#6 The manuscript methodology is present, but due to the fact that the traditional, and the one recommended by Applied Sciences, is not present, makes the reading extremely difficult which makes hard to pass the message. Therefor I highly recommend the authors to re-structure the heading of the study in the traditional “Introduction, Material and Methods, Results, Discussion and Conclusions”.

#7 Since the Discussion section is not present, highly valuable information is missing such as the confrontation with previous knowledge, such as with studies such as the one mentioned in comment #4, and the debate of study limitations, strength or further studies perspectives.

Author Response

 #1 The title should not have abbreviations, especially the ones that most of the readers do not know what they mean, such as MSDG. I recommend to use the extended name.

Thank you for this valuable comment. We changed the title with a more precise definition of the study and a generic “master” definition.

#2 The abstract should address also the methodology of assessment employed. I suggest to add it.

Thank you very much for the point. We changed the methodology in the abstract

#3 I recommend to place the keywords by alphabetic order.

Thanks for this valuable comment aimed at increasing the quality of the paper. Now the keywords are ordered in alphabetical order.

e-learning, Learning analytics, learning effectiveness, sustainable development

#4 The Introduction looks sound. However, I believe a relevant study has been missing from the lit review which is the DOI: 10.14744/eej.2019.43534 which is a study addressing the online education and the participants feedback of one of the first major online education activities in Dentistry. Some of those feedback are truly relevant and should be debated in the Introduction or Discussion sections (which presently is not available in the manuscript).

We thank the reviewer for the valuable comment aimed at increasing the quality of the paper and the related works/discussion section. We added the suggested reference in the related works and we discussed it in the proper section.

#5 The aim sentence is acceptable, but the last sentence of the Introduction is not needed and I recommend to be removed

The last sentence of the introduction has been deleted

#6 The manuscript methodology is present, but due to the fact that the traditional, and the one recommended by Applied Sciences, is not present, makes the reading extremely difficult which makes hard to pass the message. Therefor I highly recommend the authors to re-structure the heading of the study in the traditional “Introduction, Material and Methods, Results, Discussion and Conclusions”.

We would like to thank the Reviewer for this comment. We found the proposed structure more suitable to the article’s content, thus we edited and restructured the sections accordingly.

#7 Since the Discussion section is not present, highly valuable information is missing such as the confrontation with previous knowledge, such as with studies such as the one mentioned in comment #4, and the debate of study limitations, strength or further studies perspectives.

Thank you for the valuable comment. We have improved the discussion and debated the limitations and strengths in the conclusions.

Round 2

Reviewer 1 Report

I thank the authors for this great effort in improving the quality of research and writing of the manuscript. I truly appreciate their work. Currently, I think this revised version is much better than the previous one, and I wish them all the best.

Reviewer 2 Report

Dear authors, i have no more concerns. Thank you.