A Mixed Methods Study on the Effect of Flipping the Undergraduate Medical Classroom

Abstract

The flipped classroom model is increasingly being adopted in healthcare education, despite the fact that recent systematic reviews in the nursing and medical education literature suggest that this method of instructional design is not inherently better or worse than the traditional classroom. In this study, we used a sequential, explanatory mixed methods design to assess the impact of flipping the hepatology classroom for preclinical medical students. Compared to students in the traditional classroom, students in the flipped classroom had significantly lower mean (SD) ratings of their learning experiences (3.48 (1.10) vs. 4.50 (0.72), p < 0.001, d = 1.10), but better performance on the hepatology content of the end-of-course examination (78.0% (11.7%) vs. 74.2 (15.1%), respectively, p < 0.01, d = 0.3). Based upon our qualitative data analyses, we propose that the flipped classroom induced a change in the learning process of students by requiring increased preparation for classroom learning and promoting greater learner autonomy, which resulted in better retention of learned material, but reduced enjoyment of the learning experience. This dissonance in outcomes is captured in the words of one flipped classroom student: “…I hated it while I was learning it, but boy did I remember it…”. Based upon our dissonant outcomes and the inconsistent findings in the literature, we feel that there is still equipoise regarding the effectiveness of the flipped classroom, and further studies are needed to describe ways of making the flipped classroom a more effective (±more enjoyable) learning experience.

1. Introduction

More than twenty years ago, Eric Mazur described an instructional design that he referred to as “peer instruction”, where the transfer of information was moved out of the classroom and converted to homework or preparatory learning, thus allowing the role of the classroom teacher to change from one of transmitting information to learners to one where the focus is on helping learners assimilate and apply their knowledge [1]. Since Mazur’s original description, adoption of this approach—which is now typically referred to as the “flipped classroom”—has increased in many areas of education, helped in part by advancing information technology and high profile advocates of this instructional design [2,3,4]. In healthcare education, the flipped classroom model was first adopted in nursing education, but there are now multiple descriptions of classroom flipping in medical education [5,6]. So, is the flipped classroom a better learning experience than the traditional classroom?

A recent systematic review of the early experience of the flipped classroom in medical education included a quantitative synthesis of nine empirical studies (eight of which were in an undergraduate curriculum) along with qualitative synthesis of 46 studies [6]. Using Kirkpatrick’s framework to describe their findings [7,8], the authors concluded that, in general, learners’ perceptions of the flipped classroom (level 1 outcome in the modified version of Kirkpatrick’s framework) were positive, with students reporting greater task value and enjoyment compared to traditional lectures and, in some cases, greater interest in the relevant discipline as a career [6]. Regarding changes in knowledge and skills (level 2b outcome), however, the median effect size for differences between the flipped and traditional classroom was 0.08 with a range of −0.27 to 1.21—implying that, based upon current data, the flipped classroom has not yet been shown to be superior to the traditional classroom at this outcome level. Interestingly, the outcomes reported in the medical education literature are somewhat at odds with those of the nursing education literature [9]. For example, studies reported by Missildine and colleagues from 2013 and Ratta from 2015 both found that nursing students in the flipped classroom gave lower ratings to the learning experience, but had improved scores for the outcomes of knowledge and skills [10,11]. The author of the systematic review of the nursing literature concluded that “…the flipped learning approach in nursing education needs further validation and study before it can be recommended as an effective teaching strategy [9]”.

The discrepant findings from the literature on the flipped classroom implies that this instructional design is not inherently better or worse than the traditional classroom, and that the outcomes of flipping may be conditional upon other factors, such as the context of the flipped classroom (including learners, quality of learning resources, and the skill of the teachers) and the mechanism of change in the process of learning (e.g., motivation, effort, and learning strategies) [12,13]. In this study we used a sequential, explanatory mixed methods design to assess the impact of the flipped classroom on the learning of preclinical medical students. In doing so, we adopted a realist perspective to address whether knowledge and skill outcomes differ between the flipped classroom vs. traditional didactic learning (quantitative analysis) and, if so, potential mechanisms for the differences in learning outcomes (qualitative analyses) [12,13,14].

2. Methods

2.1. Participants

Our participants were 158 students from the graduating class of 2017 at the Cumming School of Medicine, University of Calgary, who were taking their first pre-clerkship course. We have a three-year Clinical Presentation curriculum, comprising two pre-clerkship years followed by one year of clerkship. The pre-clerkship curriculum includes seven integrated, systems-based courses in which approximately two thirds of the curriculum is delivered as traditional lectures, with the remainder of the learning experiences being small group instruction, clinical skills training with standardized patients and human patient simulators, and bedside teaching. The first course taught in our medical school, Course 1, includes content related to infectious disease, gastroenterology, hepatology, general surgery, and hematology, and runs for 12 weeks. Prior to the class of 2017, hepatology content consisted of 15 h of lectures (primarily delivered by a single teacher, KWB), 3 small groups (6 h), 1 patient presentation (1 h) and a review session (1 h).

2.2. Materials

One investigator (KWB) flipped the hepatology content in the traditional lectures, replacing 15 h of lectures with 24 vodcasts (average 12 min each) and five hours of workshops. KWB and MP developed simulated hepatology cases (CARDS) using an online platform (http://cards.ucalgary.ca/deck/12). The content of the small groups, patient presentation, and review session was unchanged.

To assess the impact of the flipped classroom on learning we used the end-of-course summative examination, which included the same items for hepatology and non-hepatology gastroenterology content as the examination for the class of 2015 (n = 163). To assess students’ perception of the learning content, we used two sources of data. The first was data from a survey performed immediately after completion of the flipped classroom and at completion of the course (six weeks later). The survey immediately after the flipped classroom assessed students’ perceptions of various aspects of the learning experience via a combination of free-text comments and statements that students rate their level of agreement/disagreement with using a five-point Likert scale, while the end-of-course survey (which was also used for the class of 2015) asked students to rate each of the clinical presentations covered. The second data source was derived from post-course focus groups. For our focus groups we had a short list of a priori questions intended to capture student’s opinions on the learning content, the process of learning in the flipped classroom, their perceptions of learning outcomes (including preparation for the end-of-course examination), and their reaction to the learning experience as compared to the traditional didactic format.

2.3. Procedure

We obtained ethical approval from the Conjoint Health Research Ethics Board at the University of Calgary prior to implementing our flipped classroom. We used a sequential, mixed methods study design that included a quantitative comparison of examination performance of the flipped classroom cohort to a historical control group of students who were taught similar content by the same teacher in the same course two years previously using the traditional classroom format. We chose the class of 2015 rather than 2016 as our control cohort because, unlike the class of 2016, the 2015 cohort had the same end-of-course examination as our flipped classroom cohort (class of 2017). This quantitative analysis was supplemented by qualitative data from the flipped classroom cohort.

In the flipped classroom, students were given 5 h of independent study time (time previously scheduled for lectures) to view the vodcasts, and were encouraged to do this at their own pace at home. After reviewing the vodcasts, students worked together in full-class workshops and applied their knowledge to simulated hepatology cases (CARDS). Formative multiple-choice questions (MCQ) followed each case, with students receiving immediate feedback on performance and explanations of the answers. The instructors (KWB, along with another hepatologist, resident physician, and nurse practitioner) were present to answer questions, and a technology expert was present (MP) to provide instructors real-time statistics on MCQ performance. Where necessary, teaching points were clarified for the class in immediate “mini-lectures”. Following the workshops, the students could replay the CARDS multiple times until they achieved mastery of the material. Students completed a formative examination immediately following the flipped classroom, and end-of-course summative examination six weeks later.

After completion of the end-of-course examination and survey, students were invited via email to participate in a focus group interview as part of our quality improvement program to assess the impact of curricular change. One investigator (KM) completed the focus group interviews. Although there were a priori questions, the format of the interviews was semi-structured to allow students to bring up themes on their own. Informative student comments were recorded verbatim and repeated back to students for confirmation, and the interviews continued until students felt that they had discussed all of the relevant issues.

2.4. Analyses

To compare performance of the students from the class of 2017 (flipped classroom) and 2015 (traditional classroom) on the summative MCQ items, we used an independent sample t-test with Cohen’s d as our measure of effect size [15]. For our qualitative analyses, we used a grounded theory approach [16]. One investigator (KM) performed the initial constant comparative analysis of comments from focus group participants to identify codes related to learning in the flipped classroom. A second investigator (KWB) reviewed the codes, and consensus was achieved on identification and labeling of codes. We performed inductive coding of data after each focus group until the point of theoretical saturation was reached. We also coded survey data to assess for data triangulation. We sorted data codes into categories, and then proposed a theoretical relationship between categories and the outcomes of the flipped classroom.

3. Results

3.1. The Impact of the Flipped Classroom on Learning Outcomes

On the end-of-course survey, mean rating (SD) of the hepatology content of the course was significantly lower for students in the class of 2017 compared to students in the class of 2015 (3.48 (1.10) vs. 4.50 (0.72), p < 0.001, d = 1.10). There was no difference in the rating of non-hepatology content of the course (3.53 (0.41) vs. 3.62 (0.49) for class of 2017 and class of 2015, respectively, p = 0.6).

Students in the class of 2017 had significantly higher scores on the 15 hepatology MCQ exam items compared to students from the class of 2015 (78.0% (standard deviation 11.7%) vs. 74.2 (15.1%), respectively, p < 0.01, d = 0.3). By comparison, on the other 25 gastroenterology exam items there was no difference in scores between the class of 2017 and 2015 (86.7% (8.3%) vs. 86.6 (8.04%), respectively, p = 0.9).

3.2. Analysis of Focus Group Data

On qualitative analysis of comments from our first focus group of nine students, ten codes emerged. Each of these codes was identified again in our second focus group of six students, but no new codes emerged—suggesting that we had reached theoretical saturation. Below, we describe the ten codes that we identified, along with comments from focus group participants that we feel best illustrate these codes:

(1) Increased effort

Students reported that they worked harder in the flipped classroom as compared to the traditional classroom. While some students viewed this requirement negatively (“…lot of pressure…for some workshops there was not enough time to prepare…”), others actually found this to be motivating (“…ordinarily, it is difficult to motivate myself for a lecture, but if I know I have a workshop the following morning then I am more motivated to learn…”).

(2) Increased time spent preparing for class

Related to increased effort, students described having to spend more time preparing for classroom learning and felt that, despite our attempt to create a time-neutral transition to the flipped classroom, they did not always have time to prepare for workshops. Comments supporting this inference include: “…not enough time to prepare…” and “…needed more time to know it all before workshops…”.

(3) Reduced opportunity to ask questions

Compared to the traditional classroom where a faculty member is always in attendance to answer questions, students felt that a disadvantage of the flipped classroom was the lack of opportunity to have answers to questions in real time, despite the fact that students were encouraged after each vodcast to ask questions via Twitter (“…I can’t ask questions while I’m going through the podcast…”).

(4) Reduced engagement by the teacher

In addition to reduced opportunity to have immediate answers to questions discussed above, students also noted other negative effects of reduced exposure to their teacher, such as: “…I missed the mannerisms and visual cues…”, “…I would like to see the professor’s face…”, and “…I would like to have seen him lecturing…”.

(5) Reduced social interaction with other learners

Compared to the traditional classroom, students felt relatively deprived of interactions with fellow students in the flipped classroom, as illustrated by the following two comments: “…I would miss the classroom if everything was online…” and “…It would be nice to interact with students…”.

(6) Reduced enjoyment of the learning experience

Students in our focus groups reported reduced enjoyment during learning in the flipped classroom via terms such as: “…not fun to learn…”. In addition to the reasons discussed in the five previous codes, other possible reasons for lack of enjoyments include lack of familiarity with this format (“…I hated it a first…”) or disliking specific components of our model (“…I didn’t like Twitter…”).

(7) Increased learner autonomy

Compared to the traditional classroom, learners in the flipped classroom felt that they had greater control over their learning, including the pace of learning. Comments supporting this inference include: “…I can wait until I feel ready to learn…”, “…I liked reviewing the podcasts at home with coffee in my sweat pants…”, “…podcasts are less stressful. I can stop whenever…”, and “…needed to sit, pause, and make my own notes…”.

(8) Fewer distractions

Students felt that, compared to the traditional classroom, there are fewer distractions in the flipped classroom (“…in lecture, nobody is paying attention; they’re on email or shopping online…” ), and benefits of fewer distractions include improved note-taking (“…I had really good notes. Not like lectures where I can’t make notes because everyone is talking…”).

(9) More time spent applying knowledge

Students noted the change in emphasis for classroom learning in the flipped classroom versus traditional classroom, as illustrated by the comment that “…the process is very application-based…”.

(10) Better retention of learned material

Students felt that content learned in the flipped classroom was better retained. Two comments supporting this inference were: “…I knew the content very well and retained this for the final…” and “…I needed to review this less before the exam…”. However, some students recognized that there may be a price to pay for better retention, as illustrated by the following two quotes: “…I needed more time, but I knew it all…” and “…I hated it while I was learning it, but boy did I remember it…”.

3.3. Analysis of Survey Data

Of our 158 participants, 127 (80.4%) completed the survey following the flipped classroom. Of these, 125 (98.4%) attended each of the workshops and 102 (80.3%) viewed all of the pre-workshop podcasts. The majority of survey respondents agreed with the statements “I had to prepare for the workshops in order to be successful” (n = 105 (82.6%)) and “active student engagement was consistently encouraged by instructors” (n = 86 (67.7%)). Most students felt that the individual components of the flipped classroom were a “useful learning resource” (podcast 56.7%, workshops 64.6%, and CARDS 69.3%), but fewer than half of respondents agreed that “overall, the flipped classroom was a useful learning strategy” (n = 60 (47.2%)) and an even smaller proportion (n = 48 (37.8%) agreed with the statement “I would like to see more content presented via the flipped classroom in future.” The major theme in the free-text comments was the increased work requirement in the flipped classroom: “I feel this model took more time than going to class, even though this was supposed to be time neutral” and “the volume of material in the podcasts was unmanageable”. Despite this, however, students were still able to appreciate some of the positive attributes of the flipped classroom: “workshops were helpful and well done” and “the technique itself was very helpful in terms of learning, however it was considerably more work”.

Theoretical Relationship between the Learning in the Flipped Classroom and Learning Outcomes

After constant comparative analysis of our codes and then sorting and resorting our categories, we felt that the best description of our data was to consider categories as representing the learning process and the outcomes of the learning experience. In terms of the learning process, we postulated that the requirement for increased effort to prepare for classroom learning and greater learner autonomy captured all of the remaining codes. With regard to outcomes, we felt that students referred to two outcomes from the flipped classroom learning experience: greater retention of learned material and reduced enjoyment. Our proposed theoretical relationship between our codes and these categories is shown in Figure 1.

4. Discussion

From the available data in the nursing and medical education literature, we can conclude that learners’ perception of the flipped classroom (level 1 outcome) may be better or worse than the traditional classroom, and that flipping may or may not improve knowledge and skills (level 2b outcome) [5,6,9]. Given these inconsistencies, we used a mixed methods study design to explore and explain the impact of our version of the flipped classroom on both level 1 and 2b outcomes. On quantitative analysis, we found that, compared to a historical cohort, the flipped classroom cohort had significantly better level 2b outcomes. However, quantitative and qualitative analyses of students’ perceptions of the learning experience suggests that the reaction of our flipped classroom cohort to this instructional design was generally negative. Based upon the effect sizes, our versions of the flipped classroom was associated with a “small” increase in knowledge and a “large” decrease in students rating of the learning experience [14].

The seemingly incongruent outcomes of our version of the flipped classroom is consistent with the findings in the nursing education literature on the flipped classroom, where dissonance between level 1 and 2b outcomes is well described [9]. In terms of the explanation for the dissonance in our study outcomes, based upon the finding from our qualitative analyses, our proposed explanation is that the mechanism of change in the learning process of flipped classroom students—increased effort required to prepare for classroom learning and greater learner autonomy—resulted in both better retention of learned material and reduced enjoyment of the learning experience. This explanation is consistent with the concept of “desirable difficulty”, which Bjork and Bjork propose as the explanation for dissonance in outcomes in the assessment of several, effective instructional designs, such as varying the conditions of practice, interleaving learning content, and using testing to enhance learning [17]. This, of course, does not imply that the flipped classroom learning experience must be unpleasant in order for this to improve level 2b outcomes, but it does support the opinion of Benner and colleagues that “student satisfaction may not be a good indicator of learning [18].”

There are some important limitations of our study that we should highlight. First, our study was in a single context of first year students in a three-year undergraduate curriculum studying hepatology content. Thus, we cannot generalize our finding to other learning contexts. (The results of the systematic literature reviews on the outcomes of the flipped classroom would also suggest that outcomes of the flipped classroom are not generalizable [5,6,8]). Second, while our quantitative analysis was based upon comparison with a historical cohort, our qualitative study was not. Consequently, although the reaction to the flipped classroom was generally negative, we cannot present comparative, qualitative data on students’ reaction to the traditional classroom (beyond noting that the overall rating for the hepatology content was higher in the context of the traditional classroom). Finally, the results presented here are from the first experience of our teachers and learners with the flipped classroom model. As our learners and teachers gain experience with this method of instructional design, we may observe different outcomes from the flipped classroom.

5. Conclusions

From a pedagogical viewpoint, the notion of unburdening teachers of the task of transferring information and allowing them to focus their classroom time on helping learners assimilate and apply their knowledge is very attractive. Thus far, however, attractive pedagogy has not consistently translated into a positive reaction to the flipped classroom learning experience or improved knowledge or skill of learners. In our version of the flipped classroom, learners had a negative reaction to the learning experience but improved learning outcomes, which we believe is explained by the fact that learners in the flipped classroom work harder and are more autonomous that learners in the traditional classroom. Based upon our dissonant outcomes and the inconsistent findings in the literature, we feel that there is still equipoise regarding the effectiveness of the flipped classroom and further studies are needed to describe ways of making the flipped classroom a more effective (±more enjoyable) learning experience.