MOOC Learners’ Engagement with Two Variants of Virtual Patients: A Randomised Trial
Department of Learning, Informatics, Management and Ethics, (LIME) Karolinska Institutet, 171 77 Stockholm, Sweden
Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore 247964, Singapore
Department of Bioinformatics and Telemedicine, Jagiellonian University Medical College, Kopernika 7e, 31-034 Kraków, Poland
Department of Clinical Science, Intervention and Technology (CLINTEC), 171 76 Stockholm, Sweden
Department of Education, Stockholm University, 106 91 Stockholm, Sweden
Author to whom correspondence should be addressed.
Educ. Sci. 2018, 8(2), 44; https://doi.org/10.3390/educsci8020044
Received: 6 March 2018 / Revised: 26 March 2018 / Accepted: 27 March 2018 / Published: 29 March 2018
Introduction: Massive Open Online Courses (MOOCs) are an increasingly popular form of education in health professional education. VPs have been introduced in MOOCs to increase interactivity. There is a lack of research in understanding the reasons behind high dropout rates in MOOCs. We explored how learners interact with VPs and compared the significance of different VP designs on dropout rates. Methods: RCT of 378 participants split into two groups to interact with two VPs using different design: branching and linear. Data on node progression and VP attempts was analysed using descriptive and quantitative analysis. Results: Eight groups of learner interaction patterns were identified. The majority of learners completed the VP in a linear path in one attempt. A significant number either completed the case in a loop path in one attempt, completed in a linear path in multiple attempts or dropped out without attempting the case. VP design has a significant effect on dropout rates of learners. There is a higher dropout rate from a branched VP compared to linear VP. Discussion: Prior research showed that branched VPs are more engaging and promote greater learning compared to linear VPs. However, our results indicate that branched VPs had greater dropout compared to VPs that require less time to be solved. Conclusions: We conclude that branching had a negative effect on completion of the VP activity in the MOOC. Moreover, we believe that more complex VPs require more effort on task and this might not be a design that facilitates the interaction in a MOOC audience, where the participants might wish to acquire the basic medical knowledge offered by the course.
Massive Open Online Courses (MOOCs) are an increasingly popular form of education in health professional education. In 2013, 98 MOOCs were offered related to health-care and medical education  and their numbers continue to increase. Still, there are challenges reported in relation to MOOCs, where lack of student interaction and engagement is a repeated concern [2,3,4] as well as low completion rates . Possible causes of the high dropouts include the voluntary nature of MOOCs, but also the limited interactivity that MOOCs offer to participants. Interactivity is seen, not only as an important element of learning process in elearning contexts , but also as a way to increase student motivation and completion in particular in MOOCs .
Virtual Patients (VPs) are “interactive computer simulations of real-life clinical scenarios for the purpose of training, education or assessment” . VPs have been previously introduced in MOOCs to increase levels of interactivity of learners with the MOOC content [9,10]. In one study, the results indicated that the VPs were perceived as a helpful exercise in the course by 83.1% of the responding participants (n = 442) . Other positive outcomes of MOOCs using VPs was seen in a Urology MOOC which had a higher overall student completion rate 10.4% compared to the reported 5% average completion in MOOCs .
In particular in medical education contexts, VPs are known to facilitate the training of clinical reasoning skills  but may also promote critical thinking and decision making, enable active and self-directed learning. As such, VPs provide students with interactive and dynamic problem-solving situations. Different VP designs can offer a variety of learning opportunities. For instance, branched VPs can provide learners alternative routes to reaching the desired outcome and potentially offer several different outcomes depending on the options .
Branching in virtual patients permits adding intrinsic (i.e., consequential) feedback to the scenarios. Feedback is known to be one of the most powerful instructional methods in simulation  and it forms a crucial feature of virtual patient design that determines user satisfaction and engagement . The wealth of different feedback forms  makes the choice of the right method difficult and as a scoping review has shown , there is not sufficient high-quality, evidence-based recommendations for feedback in medical education. By focusing on the influence of branching elements of virtual patients we would like contribute to the discussion on the right balance of instructional versus intrinsic feedback  in virtual patients. Intrinsic feedback allows an experiential approach to learning  but this may lead to confusion in novice learners and be contrasted with the wish of learners to get specific feedback  which is easier provided by corrective instructional feedback given in response to multiple-choice questions. The openness of MOOCs to different groups of learners makes it interesting to observe how branching will influence learners’ engagement in such a digital learning environments. Within a MOOC environment the right choice of feedback, may lead to positive effects on the dropout rates [11,12]. There is, so far, no firm evidence how this design feature may impact student completion of the VP activities and therefore impact the MOOC completion. Increased understanding of how the participants interact with different VP designs can inform the implementation process of the VPs and consequently make it more efficient.
The aim of this study is therefore to explore the influence of branching points on the learners’ engagement. In particular, we aimed to investigate: (i) How learners interact with VPs in MOOCs in terms of navigation patterns; and (ii) how branching points in a VP case may influence the dropout rate of learners within the VPs.
2.1. Study Context
Karolinska Institutet (KI) launched a Urology MOOC on the edX platform called an “Introduction to Urology”. This course was launched for the first time in autumn of 2015 and it was offered over a five week period .
The course is aligned to KI’s medical programme’s surgical course as part of the medical curriculum offered at one of the four main teaching hospitals in Stockholm, Sweden. The main learning outcome for this course was that students should be able to describe, argue and reason on the most common symptoms related to the urinary tract. The course consisted of video components, multiple choice questions, 3D models, glossary, discussion forums, a webinar, two VP cases and a final exam.
A VP focusing on bladder cancer was created in two different variants: one that offered many branching points but one end-state as the final outcome (VP1) and a second variant that replaced the branching points with multiple choice questions (VP2) that limited the navigation options of the participants to the main path. In particular, VP2 was created based on a main path of the VP1, including the same information about the patient diagnosis and treatment. Part of VP1 and VP2 cases are illustrated in Figure 1 where the nodes are represented by a number; in VP2 the multiple choice questions that replaced the branching nodes of VP1 are depicted. Nodes of VP1 and VP2 labeled with the same number imply that the nodes of the two cases consist of the same content.
The VPs were integrated in week 2 of the MOOC which had a focus on “Hematuria”. We implemented the VPs in the system OpenLabyrinth 3  which was hosted in a cloud infrastructure and embedded within the edX platform using the IMS LTI interface, a process we had tested previously . The users were automatically authenticated using credentials from the host system and had to declare the completion of the VP activity by inserting into the edX platform a four digits code that could be found by reaching the last node of the VP. By that, the students were acquiring course credits for the activity completion that was however not mandatory.
We modified the layout of the VP system to follow the graphical schema of the hosting MOOC platform and displayed it as a task in the weeks’ activity list, in order to create a smooth and seamless user integration. While solving the case, the learner interacted with Violet, a patient complaining for hematuria. Learners were required to take the patient history, order lab examinations and proceed to diagnostic and therapeutic decisions. The decisions were made either selecting one of a few options and exploring the consequential development of the scenario depending on the choice made (VP1) or selecting one of a few opinions in a MCQ and receiving instant corrective feedback (VP2). Additionally, the students were able to review the history of choice and return to one of the previous steps (looped behaviour).
2.2. Study Design
The study had a randomized design where the MOOC participants were randomly allocated to two different groups, facilitated by the A/B testing mechanism in the edX platform. Participants were allocated into either Group A where they received VP1 or Group B where they interacted with VP2. The participants were blinded to the intervention and for the purpose of avoiding contamination the discussion forums were disabled in the respective section of the VPs.
During autumn 2015, from 15 September to 27 October, 4925 participants enrolled to participate in the MOOC. The participants represented 172 countries; the three most represented ones were USA, Sweden and India. The median learner age was 30 years; 40.4% of the learners were females and 59% males. The remaining 0.6% did not report gender. In total, 514 participants completed the course. During week 2 of the course, 1349 participants logged in, and out of them 378 interacted with the VPs as either in Group A or Group B. In this study, we included the data for the 378 participants.
2.4. Data Collection
Data for the users’ VP pathway was collected from the cloud infrastructure where the OpenLabyrinth VP system was deployed. The data were collected and mapped to a user identification number to allow anonymization. The information collected from the OpenLabyrinth system included information regarding the time spent on VPs, node progression and number of attempts per VP case.
2.5. Data Analysis
To address the first research question, we conducted descriptive data analysis. We first classified the participants retrospectively depending on whether they had completed the case or dropped out. Within each group, we further classified them on the number of attempts; whether they attempted to solve the case more than one time. Subsequently, we classified the drop-out group based on whether they started the case, and the completed group based on their pattern of completion. To address the second research question, we analysed participant data for VP1 and VP2 separately, following the learner interaction patterns identified. We proposed a null hypothesis that a linear versus branched VP design has no influence on the dropout rate of learner. We examined the null hypothesis using the Pearson Chi-Square Test from IBM SPSS 64-bit at the 0.05 significance level for identifying the statistical significance between the two groups.
The research here falls outside of the vetting process of research in accordance with the Swedish Ethical Review Act (2003:460) as it does not handle sensitive personal data as defined in the Swedish Personal Data Act. Further, according to edX praxis and in compliance with ethical guidelines, at registration, all course participants gave their informed consent that their engagement and performance data, in unidentified form, could be used in coming research publications.
3.1. Learners’ Interaction Patterns with VPs in MOOCs
Figure 2 depicts results of our analysis on how learners interact with the VPs in the MOOC. The participants were first classified whether they dropped out or completed the VPs. There is a total of 71 dropout participants and 307 participants who completed the VPs.
Next, the participants were classified whether they attempted to solve the VPs once or more than once. Dropout participants were further classified based on whether they started or did not start the VP case. The participants that completed were further classified based on whether they completed the VP case in a linear or loop behaviour. A linear behaviour meant that participants did not go back to review previously visited nodes or completed questions. Loop behaviour refers to participants that went back to review previously visited nodes and completed questions. We mapped out each participant’s node progression and categorized them into eight different subcategories of interaction patterns.
In Figure 2 we label as:
- Distance between the black dots: length of VP case;
- Red dots: introduction page of VP case;
- Initiated line: user has loaded VP scenario;
- Straight line: linear behaviour in VP scenario;
- Loop line: loop behaviour in VP scenario;
- Multiple straight and loop lines: multiple linear and loop behaviours in VP scenario respectively;
- Incomplete line: dropout from VP scenario;
- Line reaching end dots (Straight or loop): completed VP scenario.
In Table 1 we label:
- Attempt: the act of loading the VP introduction page but not starting the VP scenario.
- Start: act of loading the VP scenario after the VP introduction page.
The majority of participants (n = 209) completed their VP case in 1 attempt following a linear path. A large group (n = 46) completed their VP case in 1 attempt in a loop path, meaning the participants returned to previous elements of the VP. Many participants (n = 36) also completed their VP case with more than 1 attempt in a linear path. It is also important to note that a big group (n = 51) dropped out in their first attempt without starting their VP case.
3.2. Influence of Branching Points in a VP Case on the Drop-Out Rate of Learners
To identify whether there is any influence of the branching points in VPs on the learners’ dropout rate from the assignment we analysed participant data for VP1 and VP2 separately, following the learner interaction patterns identified above. We excluded drop-out participants who did not initiate the VP case as their reasons for dropout was not influenced by the different VP designs or branching points in the VPs. We included participants who completed the VPs and participants who started the VP and dropped out (Table 2).
The calculated p-value is .000253 and the result is statistically significant at p < 0.05. The null hypothesis that VP branching points has no influence on the dropout rate of learner is therefore rejected and the results indicate that the VP1 versus the VP2 design has an influence on the dropout rate of learners. As the VP cases only differ in the added branching points of the VP1, it may be concluded that the VP1 design correlates with a higher dropout rate compared to the VP2.
This study was conducted in a MOOC setting, aiming to understand how the MOOC audience may behave when interacting with modalities such as VPs for the purpose of informing the learning design process. In particular, we aimed to understand how the learners interact with VPs in MOOCs and how a VP design with branching points influences the dropout rate of learners compared with a VP design that includes MCQs instead. To test these questions two VP forms were designed and the MOOC participants were allocated to engage with the two forms in a randomized trial setting.
The results demonstrate that the majority (81% of total) of participants completed the VP case of bladder cancer. This is in line with prior research results, which demonstrate high participants’ satisfaction when engaging with VPs in MOOCs . Nineteen percent (19%) of the total participants dropped out from the exercise, of which the bulk of these participants dropped out without starting the VP (76% of dropouts).
Prior research showed that branched VPs are more engaging and promote greater learning compared to linear VPs . However, our results indicate that branched VPs had greater dropout compared to VPs that require less effort to be solved. This could be due the fact that branched VPs have a higher level of complexity, and place a higher cognitive load on novice learners, but also, there could be an impact due to the difference in target audience. In the previous study  the linear cases were paper-based and the activity was part of a PBL setting for medical students when group processes could compensate the complexity of decision making.
We conclude that more complex VPs require more effort on task and this might not be a design that facilitates interaction in a MOOC audience, where the participants might wish to acquire the basic medical knowledge offered by the course.
One of the strengths of this study is the proposed classification of learner interaction patterns while progressing through a VP including factors as multiple attempts and looped progress. The study delivers quantitative data on how popular these behavioural models are in a MOOC audience. It is yet to be investigated whether belonging to any of those groups is sign of a particular learner characteristic (e.g., experienced vs novice learners) or results in superior knowledge. Further research is also required in order to identify any potential correlations between the satisfaction outcomes of the participants having different learning behaviours that could inform design of more adaptive VPs for the MOOCs in future. We plan to expand the current study by investigating differences in engagement in VPs having different characteristics than just the branching points.
One of the study’s limitations is that the results are based on the analysis of the learners’ interaction in only one case and therefore might not be representative. More research on the learning patterns in different VP designs is required. Baseline differences in the motivation or engagement of the participants as well as their professional background were not investigated.
The starting point of our research was to look into ways how to lower MOOC dropout rates. In the experimental setting at hand we have investigated VP drop-outs which are not identical with MOOC dropout (as the learner might switch to a different learning modality in the MOOC). Yet, we believe that loss of interest in one form of learning is likely to be correlated with disengagement in the whole course.
Further research on the topic may investigate the potential reasons of the higher dropout rates identified in the branched case in more depth by conducting a survey and examining the background correlation to the learning patterns and dropout behaviours. We plan also to look into the impact that different VP designs and features may have on the knowledge, skills and satisfaction outcomes.
This study’s results show differences in completion rates of the participants that interacted with different form of VPs. This could have an impact on the design of future MOOCs in health care educations settings. The study suggests that the integration of interactive elements in a MOOC setting has the potential to play a part in reducing the high drop-out rates previously observed in the MOOCs. From a design perspective, our results suggest that novices may benefit by first interacting with linear VPs before progressing to more challenging branched VPs.
N.S., and C.M.G. conceived and designed the experiments; N.S. and C.M.G. performed the experiments; N.S., M.L.S., N.Z. and C.M.G. analyzed the data; L.H. contributed materials and the V.P. design; all authors wrote the paper.
Conflicts of Interest
The authors declare no conflict of interest.
- Liyanagunawardena, T.; Adams, A.; Williams, S. MOOCs: A Systematic Study of the Published Literature 2008–2012. Int. Rev. Res. Open Distrib. Learn. 2013, 14, 202–227. [Google Scholar] [CrossRef]
- Isaac, C.; Andrew, H. HarvardX and MITx: Four Years of Open Online Courses—Fall 2012–Summer 2016. SSRN. 2016. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2889436 (accessed on 29 March 2018).
- Kearney, R.C.; Premaraj, S.; Smith, B.M.; Olson, G.W.; Williamson, A.E.; Romanos, G. Massive Open Online Courses in Dental Education: Two Viewpoints: Viewpoint 1: Massive Open Online Courses Offer Transformative Technology for Dental Education and Viewpoint 2: Massive Open Online Courses Are Not Ready for Primetime. J. Dent. Educ. 2016, 80, 121–127. [Google Scholar] [PubMed]
- Akyol, Z.; Vaughan, N.; Garrison, D.R. The impact of course duration on the development of a community of inquiry. Interact. Learn. Environ. 2011, 19, 231–246. [Google Scholar] [CrossRef]
- Ebben, M.; Murphy, J.S. Unpacking MOOC scholarly discourse: A review of nascent MOOC scholarship. Learn. Media Technol. 2014, 39, 328–345. [Google Scholar] [CrossRef]
- Laurillard, D. Multimedia and the changing experience of the learner. Br. J. Educ. Technol. 1995, 26, 179–189. [Google Scholar] [CrossRef]
- Khalil, H.; Ebner, M. MOOCs Completion Rates and Possible Methods to Improve Retention—A Literature Review. In Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunications, Tampere, Finland, 23 June 2014; pp. 1236–1244. [Google Scholar]
- Ellaway, R.; Candler, C.; Greene, P.; Smothers, V. An Architectural Model for MedBiquitous Virtual Patients; Technical Report; MedBiquitous: Baltimore, MD, USA, 2006. [Google Scholar]
- Stathakarou, N.; Zary, N.; Kononowicz, A.A. Virtual Patients in Massive Open Online Courses-Design Implications and Integration Strategies. Stud. Health Technol. Inform. 2014, 205, 793–797. [Google Scholar] [PubMed]
- Stathakarou, N.; Zary, N.; Kononowicz, A.A. Beyond xMOOCs in healthcare education: Study of the feasibility in integrating virtual patient systems and MOOC platforms. PeerJ 2014, 2, e672. [Google Scholar] [CrossRef] [PubMed]
- Berman, A.H.; Biguet, G.; Stathakarou, N.; Westin-Hägglöf, B.; Jeding, K.; McGrath, C.; Zary, N.; Kononowicz, A.A. Virtual Patients in a Behavioral Medicine Massive Open Online Course (MOOC): A Qualitative and Quantitative Analysis of Participants’ Perceptions. Acad. Psychiatry 2017, 41, 631–641. [Google Scholar] [CrossRef] [PubMed]
- Henningsohn, L.; Dastaviz, N.; Stathakarou, N.; McGrath, C. KIUrologyX: Urology as You Like It—A Massive Open Online Course for Medical Students, Professionals, Patients, and Laypeople Alike. Eur. Urol. 2017, 72, 321–322. [Google Scholar] [CrossRef] [PubMed]
- Cook, D.; Triola, M. Virtual patients: A critical review and proposed next steps. Med. Educ. 2009, 43, 303–311. [Google Scholar] [CrossRef] [PubMed]
- Ellaway, R.; Poulton, T.; Fors, U.; McGee, J.B.; Albright, S. Building a virtual patient commons. Med. Teach. 2008, 30, 170–174. [Google Scholar] [CrossRef] [PubMed]
- McGaghie, W.C.; Issenberg, S.B.; Petrusa, E.R.; Scalese, R.J. A critical review of simulation-based medical education research: 2003–2009. Med. Educ. 2010, 44, 50–63. [Google Scholar] [CrossRef] [PubMed]
- Cook, D.A.; Erwin, P.J.; Triola, M.M. Computerized virtual patients in health professions education: A systematic review and meta-analysis. Acad. Med. 2010, 85, 1589–1602. [Google Scholar] [CrossRef] [PubMed]
- Narciss, S. Feedback Strategies for Interactive Learning Tasks. In Handbook of Research on Educational Communications and Technology; Spector, J.M., Merrill, M.D., van Merriënboer, J., Driscoll, M.P., Eds.; Routledge: Abingdon, UK, 2007. [Google Scholar]
- Bing-You, R.; Hayes, V.; Varaklis, K.; Trowbridge, R.; Kemp, H.; McKelvy, D. Feedback for learners in medical education: What is known? A scoping review. Acad. Med. 2017, 92, 1346–1354. [Google Scholar] [CrossRef]
- Clark, R.C.; Mayer, R.E. Scenario-Based e-Learning: Evidence-Based Guidelines for Online Workforce Learning; John Wiley & Sons: Hoboken, NJ, USA, 2012. [Google Scholar]
- Kolb, D.A. Experiential Learning: Experience as the Source of Learning and Development; FT Press: Upper Saddle River, NJ, USA, 2014. [Google Scholar]
- Huwendiek, S.; Reichert, F.; Bosse, H.M.; De Leng, B.A.; Van Der Vleuten, C.P.; Haag, M.; Tönshoff, B. Design principles for virtual patients: A focus group study among students. Med. Educ. 2009, 43, 580–588. [Google Scholar] [CrossRef] [PubMed]
- OpenLabyrinth—Virtual Scenarios and Education Research. Available online: http://openlabyrinth.ca/ (accessed on 26 February 2018).
- Poulton, T.; Ellaway, R.H.; Round, J.; Jivram, T.; Kavia, S. Exploring the efficacy of replacing linear paper-based patient cases in problem-based learning with dynamic Web-based virtual patients: Randomized controlled trial. J. Med. Internet Res. 2014, 16, e240. [Google Scholar] [CrossRef] [PubMed]
Figure 1. VP1 and VP2 navigation designs (excerpt).
Figure 2. Typology of Learner Interaction Patterns of VP1 and VP2.
Table 1. Learner Interaction Patterns of VP1 and VP2.
|Learner Interaction Patterns||Total (n = 378)|
|VP dropout||1 Attempt||Does not Start||51|
|>1 Attempt||Does not Start||3|
|Completed||1 Attempt||Linear behaviour||209|
|>1 Attempt||Linear behaviour||36|
Table 2. Observed total dropout, (expected total dropout).
|VP1||16 (9)||149 (156)|
|VP2||1 (8)||158 (151)|
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).