Efficacy of Multimedia Learning Modules as Preparation for Lecture-Based Tutorials in Electromagnetism

Abstract

We have investigated the efficacy of on-line, multimedia learning modules (MLMs) as preparation for in-class, lecture-based tutorials in electromagnetism in a physics course for natural science majors (biology and marine science). Specifically, we report the results of a multiple-group pre/post-test research design comparing two groups receiving different treatments with respect to activities preceding participation in Tutorials in Introductory Physics. The different pre-tutorial activities were as follows: (1) students were assigned reading from a traditional textbook, followed by a traditional lecture; and (2) students completed on-line MLMs developed by the Physics Education Research Group at the University of Illinois at Urbana Champaign (UIUC), and commercially known as FlipItPhysics. The MLM treatment group earned significantly higher mid-term examination scores and larger gains in content knowledge as measured by the Conceptual Survey of Electricity and Magnetism (CSEM). Student attitudes towards “reformed” instruction in the form of active-engagement tutorials were also improved. Specifically, post-course surveys showed that MLM-group students believed class time was more effective and the instructor was more clear than reported by non-MLM students, even though there was no significant difference between groups with respect to in-class activities and the same instructor taught both groups. MLM activities can be a highly effective tool for some student populations, especially when student preparation and buy-in are important for realizing significant gains.

1. Introduction

Students perceive little value in reading the traditional physics textbook before material is introduced in the classroom [1]. As a result, it is not surprising that few students prepare for class in this way, even when specifically required to do so by the instructor [2,3]. Effective reformed pedagogies in physics often require some degree of student preparation pre-class. In particular, the concept of the “flipped” classroom requires significant time-on-task outside traditional class meetings [4]. Care must be taken when implementing reformed pedagogies that rely on student preparation, specifically with respect to motivation, assessment of those activities, and assignment of value to individual methodologies.

The concept of replacing traditional textbook reading with on-line, multimedia learning modules (MLMs) for content has been discussed in the literature as a means to encourage student pre-class preparation. For example, improved student learning and attitudes towards instruction have been found when reading assignments have been replaced with on-line multimedia videos in physics classes across various institutions [5,6,7]. In particular, it has been shown that students arrive to class better prepared for learning when completing MLMs versus traditional reading assignments for large-enrollment lecture-based courses and hybrid on-line formats [5,8,9].

There has been recent interest in combining the concept of MLM pre-instruction activities with studio-style in-class pedagogies, such as Workshop Physics and/or Tutorials in Introductory Physics (TIP) [10,11,12,13,14,15]. In this study, we are particularly interested in the effectiveness of the inverted classroom, with MLMs replacing the lecture outside of class time and TIP dominating the face-to-face instructional time. This type of implementation would be particularly useful at small- to mid-sized institutions without graduate teaching assistants and no defined recitation sections.

In this article, we present a quasi-experiment based on non-equivalent control groups design. In particular, we compare student learning and affect across multiple introductory physics courses for natural-science majors that focuses on the topics of electricity and magnetism. All students participate in weekly TIP sessions, with the two groups different with respect to how they prepare for TIP sessions. Students prepare for the tutorials through either (1) a combination of textbook reading assignments and traditional lecture, or (2) MLM activities. Between these two groups, we measure differences in learning gains using the Conceptual Survey of Electricity and Magnetism (CSEM), performance on semester exams, and student perceptions of instruction.

It is hypothesized, based on the MLM literature, that students completing the MLM activities will achieve higher gains in learning, perform better on semester tests, and report a perception of greater instructor effectiveness [5,8,9]. However, successful implementation of new pedagogies requires caution, specifically because research shows that different populations can respond very differently to reformed instruction based on scientific reasoning ability, motivation, major, and general academic preparation [13,16,17,18]. In particular, non-science majors in college level courses may respond differently to science majors to the same instruction. Even different responses across science majors may be possible, with natural science majors (biology, marine science, etc.) responding to pedagogies in completely different ways to physical science majors. When combining two research-verified pedagogies to facilitate new goals (such as hybrid in-class/on-line courses, as one example), effectiveness as a whole may also be impacted either positively or negatively.

In this study, we are determining the efficacy of combining the research-verified on-line pedagogy of MLM with the studio-style, in-class pedagogy TIP. The general effectiveness of these teaching styles has been shown in several different contexts [5,8,9,10,11,12,13,14,15]. The contribution this study makes to the literature on both MLM and TIP is the measured efficacy of the combination of these two approaches, as well as reinforcement of the effectiveness of MLMs as a pedagogical tool across various groups.

2. Background

In this section, we discuss multimedia learning and the development and deployment of MLMs in instruction. Specifically, we describe the theoretical framework of cognitive load theory and working memory. We also discuss the specific physics-based MLMs used in the courses under study. The student population and course format is described, since different populations can respond very differently to reformed instruction, and factors such as the classroom environment and particular pedagogical approach can influence results of multiple-group studies.

2.1. Multimedia Learning Modules

MLMs as an instructional tool have been developed as a means towards reducing cognitive load in the learning process [19]. By mixing auditory and visual presentations, meaningful learning can take place by helping the learner make connections between multiple representations of the same content without taxing limited capacity memory channels [20]. Multiple representations can be introduced in both lectures and textbooks, with textbooks limited primarily to a single mode of delivery. With respect to textbook reading, there is also significant practical concern about whether or not students are using it as a learning tool at all in physics courses [2,3]. Furthermore, there is a long history of research findings that show the traditional physics lecture is ineffective, with more recently developed classroom-based active engagement reformed pedagogies demonstrating significant gains in student learning, possibly partially due to the increased focus on multiple representations incorporated within these pedagogies. Interestingly, research in educational psychology has shown that multimedia approaches to the presentation of content outside of the classroom can result in the same type of cognitive processes associated with these active learning approaches in the classroom [21]. This suggests great promise for MLMs as a pre-class preparation for deeper learning.

For this study, we have used on-line MLMs developed by the Physics Education Research Group at the University of Illinois at Urbana Champaign (UIUC), and commercially known as FlipItPhysics [22]. These MLMs are flash animations that are designed to introduce core concepts within a traditional physics course in relatively short presentations of approximately 15 min. The authors refer to these MLMs as “pre-lectures”, since their intended purpose is as activities preceding the classroom lecture period on the same subject. Each pre-lecture includes a narration with animations, resulting in a highly visual and auditory experience where certain concepts can “come alive” in a way not reproducible on the traditional whiteboard.

Figure 1 shows a screen capture of the student view for a FlipItPhysics MLM pre-lecture on electric current. The entire 15-min pre-lecture is split into multiple 1–2-min animated slides on smaller concepts, as shown on the right side of the figure. The student has access to a play/pause button for each section of the pre-lecture, as seen in the bottom-left corner of Figure 1; however, a student cannot proceed to the next section until after viewing the current section. After all sections have been viewed, the student is free to browse the content for review. Also, two to three multiple choice questions for each pre-lecture are embedded in-between some of the animated slides, which can be seen on the right side of Figure 1 as dark slides emblazoned with a large letter Q. In order for students to proceed to the next slide, they must correctly answer the question. Feedback is automatic, and if they incorrectly answer then the student is led through a short tutorial designed to lead them to the correct answer.

It should be pointed out that the design of the MLMs used in this study were informed by research in both physics education and multimedia learning. A more in-depth discussion of how the authors of the MLMs utilized the relevant literature in the design process is available in reference [8]. Furthermore, the efficacy of the MLMs has been verified in multiple other settings [5,6,7,8,9].

2.2. Course Information and Student Population

All studies were carried out at a mid-sized, primarily-undergraduate, comprehensive university in the south-east USA. During the summer of 2011, a second-semester calculus-based physics course (PHYS II) at the university was selected to pilot an implementation of MLMs for most of the major content within the course. The MLM-based courses discussed in this article served as a beta test site for the commercial smartPhysics product eventually put into production by W.H. Freeman. Since completion of the study, smartPhysics was purchased by MacMillan Learning and renamed FlipItPhysics [22]. During the fall of 2011, two sections of the same course also implemented MLMs. All courses implementing MLM content were taught by the same primary instructor. In this paper, we compare student learning and affect for sections implementing MLMs with sections previously taught by the same instructor without MLMs.

PHYS II was a calculus-based physics course covering content such as fluids, waves, thermodynamics, electricity and magnetism. The populations in all sections of the courses under study were predominantly composed of natural science majors from the Departments of Biology and Marine Science. All sections (both MLM and non-MLM groups) were taught in a lecture room designed for the Student Centered Activities for Large Enrollment University Physics (SCALE-UP) model [23]. A SCALE-UP course incorporates the high-impact practice of collaborative assignments and projects by fusing lecture, laboratory, and recitation into a single entity. Between two and three 24-seat sections were combined into one large classroom holding up to 72 students. Students worked at round tables that seated six students each. The course was led by a primary instructor and had one or two instructors serving in backup roles, with at least one undergraduate Learning Assistant [24].

During face-to-face class time, a lecture-based implementation of Tutorials in Introductory Physics was utilized for all groups, including the non-MLM group [15]. Similar to a traditional TIP implementation, students proceeded through TIP materials in groups, but with whole-class “checkouts” rather than instructor-intensive, individual group checkouts. We also intersperse short "micro-lectures" between TIP activities, as well as Peer Instruction activities [25]. This adaptation of TIP does stray in some significant ways from the intentions of the curriculum developers. However, this adaptation was necessary for logistical reasons. This type of implementation would be particularly useful at similar small- to mid-sized institutions without graduate teaching assistants and no defined recitation sections. For both the MLM and non-MLM courses, Just-In-Time teaching and on-line, instant feedback homework were utilized, which will be discussed in more detail in the next section.

Table 1. List of multimedia learning module (MLM) pre-lecture topics used during the course [22].

Electrostatics	DC Circuits	Magnetism
Coulomb’s Law	Conductors and Capacitors	Magnetism
Electric Fields	Electric Current	Biot-Savart Law
Electric Flux and Field Lines	Kirchhoff’s Rules	Ampere’s Law
Gauss’ Law		Motional EMF
Electric Potential Energy		Faraday’s Law
Electric Potential

shows a list of the MLM pre-lecture topics assigned to students during the courses utilizing MLMs.

Table 2. List of tutorials completed in-class that are associated with the pre-lectures in Table 1 Tutorials used correspond exactly to those found in Tutorial in Introductory Physics by McDermott, et al. [15].

Electrostatics	DC Circuits	Magnetism
Charge	A model for circuits Part 1	Magnets and magnetic fields
Electric field and flux	A model for circuits Part 2	Magnetic interactions
Gauss’ law		Lenz’ law
Electric potential difference		Faraday’s law and applications
Capacitance

lists the TIP activities completed in face-to-face class meetings for both the MLM and non-MLM groups. For this study, we confine our analysis to the topics of traditional electricity and magnetism, such as electrostatics, direct current (DC) electric circuits, and magnetism. Other topics were discussed in the course as described above; however, measures of efficacy were collected only for the topics listed in Table 1.

3. Methodology

In this section, we describe the methodology used in the reported study. In particular, we describe the study design and the treatments applied to the two groups. We also discuss threats to internal validity associated with this type of study, the efforts that were made to limit these threats, and the between-group differences before the start of the treatments and their potential influence on the results of the study.

3.1. Study Design

We have conducted a quasi-experiment with non-equivalent group design, which is structured like a multiple-group pre/post-test experiment without random assignment. Specifically, we compared two groups receiving different treatments with respect to activities preceding participation in TIP activities. Figure 2 summarizes the study design, where the different pre-tutorial activities were as follows: (1) students were assigned reading from a traditional textbook, followed by a traditional lecture; and (2) students completed on-line MLMs as described above. Four PHYS II sections served as our non-MLM group (n = 58) and three sections of PHYS II served as our MLM treatment group (n = 41). With respect to in-class activities, both groups had the same lead instructor and participated in the same lecture–tutorial activities.

An implementation of Just-in-Time Teaching (JiTT) was used with both groups [26]. JiTT is a strategy that uses feedback between classroom activities and work that students do at home in preparation for the face-to-face meeting. Specifically, students complete some outside class activity such as reading or MLMs and then answer short, concept-based questions on-line before the beginning of the class. The faculty member then uses the student responses to the questions to tailor the in-class materials to the expressed weaknesses of the particular class.

For the non-MLM group, we used JiTT concept questions from reference [26] that corresponded to the material. For the MLM group, JiTT-style “checkpoints” are automatically incorporated into the FlipItPhysics product that was being utilized [22]. There was variation in the JiTT/checkpoint questions asked of each group; however, both sets of questions are based on research in physics education. Also, since JiTT as a fundamental feature allows the instructor to tailor in-class course content for individual classes, there is the potential for some topics in one class to receive more focus than in another. This should be kept in mind when evaluating the results comparing the MLM and non-MLM groups. However, for the content topics discussed in this article, there was very little difference between the strengths and weaknesses of the two groups pre-tutorials, which led to little variation in in-class tutorial activity. Although it appeared that MLM students were better prepared than non-MLM students (we did not measure this directly), both groups generally expressed difficulties with the same content.

For the non-MLM group, an instructor-led passive lecture session lasting between 45–50 min was incorporated. This lecture served as a more traditional introduction to material and in many ways was similar to the pre-lectures viewed by the MLM group, with the main exceptions being the lack of multimedia content. This added lecture session resulted in greater time-on-task for students that did complete the reading within the non-MLM group compared to the MLM group. Less content outside of the area of electromagnetism was covered during the non-MLM semesters to compensate for the added instructional time.

An instant feedback homework system was used with both groups. For the non-MLM group, tutorial-style homework problems were assigned using the MasteringPhysics on-line homework system and the textbook “Physics for Scientists and Engineers” by Randall Knight [27]. For the MLM group, an on-line homework system was automatically incorporated into the FlipItPhysics product that was being utilized [22]. The style of the homework questions was similar for both groups, though the exact problems were different.

3.2. Threats to Internal Validity

The two groups under study were separated by different semesters, so they were not in the same class experiencing exactly the same in-class activities. This results in a history and maturation threat to the internal validity of the study. Furthermore, external factors may have influenced participants’ responses to experimental procedures, such as differences in the availability and quality of outside tutoring, as one example. Some variations in face-to-face instruction could have also occurred due to the increased experience of the instructor working with the population type during the course of the study. These threats to validity were kept to a minimum for the duration of the study; however, their presence must be considered when comparing the differences between the two groups.

The study design was based on practical logistics, and the necessity of relying on pre-study historical data. Although the study design is not optimal, we have employed triangulation to assess the effect of MLM treatment. Different methods were used in order to check whether different approaches to preparation for the course (MLM vs. non-MLM) led to different student achievement. In particular, cognitive achievement was measured by the CSEM and means of common exam grades. Furthermore, results for cognitive achievement are compared with measurements of student perceptions of instruction.

4. Results

The efficacy of MLMs was determined by comparing student groups performance on a validated assessment and exam scores. Student affect was measured via surveys. Specifically, learning gains on content in electricity and magnetism were measured via pre/post-test scores on the CSEM [28]. In-class summative assessments in the form of exams were held constant across both groups, and each group’s performance was compared. Finally, student course evaluations were administered at the end of each course for both groups and compared.

4.1. Student Learning Gains

The CSEM is a 32-item multiple choice assessment designed to test student understanding of electricity and magnetism concepts covered in the average introductory physics course. For all groups, we administered the CSEM both before instruction and after instruction. In particular, students completed the CSEM initially during the first class meeting of the semester, and then again during the last class meeting of the semester. No course credit was assigned for completing the assessment.

Table 3. Average pre- and post-test scores and normalized gain on the Conceptual Survey of Electricity and Magnetism (CSEM) for students in the four groups under study.

	Group	Pre-Score	Post-Score	ḡ	n
Fa10	non-MLM	19 ± 2	35 ± 2	19 ± 3	21
Sp11	non-MLM	22 ± 2	34 ± 3	16 ± 3	23
Su11	MLM	24 ± 3	50 ± 6	35 ± 7	15
Fa11	MLM	19 ± 2	47 ± 2	34 ± 3	18

shows the pre- and post-test scores on the CSEM for the two different student groups across four semesters of instruction. The non-MLM group is composed of students enrolled in the fall 2010 (Fa10) and spring 2011 (Sp11) semesters. The MLM group is composed of students enrolled in the summer 2011 (Su11) and fall 2011 (Fa11) semesters. There is no statistical difference between any of the groups for the pre-test, suggesting that there is no measurable selection bias with respect to initial content knowledge as measured by the CSEM.

Each student’s pre-test score $S_{pre}$ and post-test score $S_{post}$ were used along with the maximum possible score $S_{max}$ to calculate individual normalized learning gain g. Normalized gain is the ratio of the actual assessment score gain to the maximum possible gain, as follows [29]:

$$g=\frac{S_{post}-S_{pre}}{S_{max}-S_{pre}}.$$

(1)

To compare the results across groups, we averaged the individual normalized gains for members of the group. We only present data for students that completed both the pre- and post-test. Since the assessment was not a required and graded component of the course, we not surprisingly had fewer students complete both offerings compared to the numbers enrolled in the courses.

We have chosen to report normalized gain, since this metric is commonly used within the physics education research community. With respect to the literature on the CSEM, normalized gain, as opposed to effect size, has historically been used as the figure of merit when evaluating treatments, and we continue this tradition for comparison across studies. It should be noted that there are significant criticisms concerning the limitations of normalized gain as a metric. Specifically, Colletta and Phillips and Moore and Rubbo question the independence of gain to pre-score and scientific reasoning ability, respectively [16,30]. Miller et al. further highlight the inability of normalized gain to capture information about potential conceptual losses, since the measurement “implicitly assumes that losses are zero” [31]. These criticisms should be considered when evaluating a comparison of normalized gain.

Table 3 shows the average normalized gain (ḡ) on the CSEM for all four semesters of instruction. The non-MLM group was made up of students from the Fa10 and Sp11 sections of PHYS II, while the MLM group was composed of students in the Su11 and Fa11 sections. The average normalized gain on the CSEM for both groups is reported in

Table 4. Average normalized learning gains on the CSEM for the non-MLM (n = 44) and MLM (n = 33) groups.

ḡ, non-MLM (%)	ḡ, MLM (%)	p
18 ± 2	34 ± 3	<0.01

. Interestingly, the non-MLM group had an average normalized gain of $(18\pm 2)$%, which is lower than the national average gain observed for courses not utilizing active engagement pedagogies (23%) [28]. The MLM group had a significantly higher average normalized gain of $(35\pm 3)$%. A two-sample location t-test was used to determine whether or not the means of the two populations were equal. The MLM group had a significantly greater CSEM average learning gain compared to the non-MLM group valid at the p < 1% level. We found no statistical difference in average learning gains for the Fa10 and Sp11 groups, and the Su11 and Fa11 groups.

Initially, the below-average performance of the non-MLM group was surprising. In these two semesters, we were utilizing TIP and JiTT pedagogies, which are both research-verified methods that have been shown to produce normalized gains on the CSEM considerably above “traditional” pedagogy. As discussed, successful implementation of new pedagogies requires caution, specifically because research shows that different populations can respond very differently to reformed instruction [13,16,17]. However, we should caution the reader to first consider the limitations of normalized gain, specifically with respect to the relatively small population sizes within the groups under study. In addition to criticisms already mentioned, normalized gain also does not account for the size of the class or intra-class variations.

In education research outside of domain-specific physics education, it is more common to report Cohen’s d effect size, which normalizes the average raw gain for a population by the pooled standard deviation.

Table 5. Cohen’s d effect size on the CSEM for the non-MLM (n = 44) and MLM (n = 33) groups.

d, non-MLM	d, MLM
1.25	1.64

shows the effect size for both groups. Both the non-MLM and MLM groups demonstrate very large (>1.2) effect sizes, with the MLM treatment group demonstrating an effect size close to what Sawilowsky describes as “huge” [32]. A large effect size on the CSEM is not surprising for either group, since both the MLM and non-MLM groups used research-verified pedagogies. It should be pointed out that we have implemented an adaptation of TIP in both situations that strays in significant ways from the intentions of the curriculum designers. Therefore, the data should not be interpreted as condemnation or success of any particular pedagogy for any particular group. Our main point is certainly not that some specific pedagogy or collection of pedagogies fails or succeeds to impact learning beyond the national average. We report normalized gain and effect size here as a comparison metric between the non-MLM and MLM treatment groups to determine efficacy of pre-class MLM-based instruction.

4.2. Course Examinations

To further compare cognitive achievement across the two groups, we compared scores on common exams. During all four semesters under study, students completed five closed-book exams, with four two-hour exams on content within specific learning units, and one two-hour cumulative exam after the end of the semester. All exams were composed of five free-response questions. Two questions were analytical problems similar to assigned homework problems, and three questions were free-response concept questions either taken directly or slightly modified from the sample exam questions in the Instructor’s Guide to Tutorials in Introductory Physics [15].

One exam focused on concepts in electrostatics, and another exam focused on topics in electromagnetism and DC circuits, consistent with the content described in Table 1. These exams were the summative assessment of learning for these units, and for all four semesters studied (Fa10, Sp11, Su11, and Fa11), these two exams were the last two in-semester exams assigned. The same course instructor graded all of the exams. The exams for the Fa10 and Su11 semesters were identical. Likewise, the exams for the Sp11 and Fa11 semesters were identical, though different than the other two semesters. Reusing exams during non-consecutive semesters helped prevent old exams from being distributed while allowing for a common assessment across both groups.

Figure 3 shows the average exam grade for the MLM and non-MLM groups for exams on electrostatics and electromagnetism. For the electrostatics exam, the non-MLM group had an average of $(67\pm 2)\%$ while the MLM group had an average of $(75\pm 2)\%$. For the electromagnetism exam, the non-MLM group had an average of $(76\pm 3)\%$ while the MLM group had an average of $(85\pm 2)\%$. A one-tail un-paired t-test was used to determine whether or not the means of the two populations were equal. The MLM group had a significantly greater exam average for both exams compared to the non-MLM group valid at the p < 1% level. We found no statistical difference between exam grades when comparing between the two semesters of MLM and non-MLM instruction, which suggests the use of two different exams had little effect with respect to the average exam grade. There were also no measurable between-group differences with respect to content knowledge as measured by the CSEM (as discussed in the previous section).

On both exams, the MLM group scored between 8 and 9% higher than the non-MLM group. Both groups performed better on the electromagnetism exam than the electrostatics exam, which is consistent with observations from prior semesters. This is not well understood, but could be due to students’ poor performance with Gauss’ Law compared to surprisingly consistent success with applications of the right-hand-rule in magnetism. This is also evident in the CSEM scores, where students score approximately 5–8% higher on the magnetism questions compared to the electrostatic questions post-instruction (not shown).

The overall increase in exam scores with MLMs is consistent with the similarly observed increase in CSEM normalized learning gains and effect size. A similar increase in exam scores as a result of MLM use was observed for mechanics and electromagnetism content in other studies [6,9]. However, this is the first study of MLM efficacy showing a consistent link between increasing gains in content knowledge as measured by a nationally validated instrument and performance on in-class examinations.

4.3. Student Perceptions of the Instruction

A survey on student attitudes towards the instructor was administered at the end of the semester for all courses discussed in this study. The survey consisted of seven questions with all questions answered via a 7-point Likert scale ranging from “Strongly Disagree” to “Strongly Agree”. The survey used was administered during the last week of all classes and was part of the standardized faculty evaluations used in all courses at the university.

For this study, we were interested in the students’ attitudes with respect to the following areas: (1) instructor clarity; (2) class atmosphere; (3) effective use of class time, and (4) instructor effectiveness.

Table 6. Survey questions for the four topics of interest.

Clarity	The instructor presented material in a clear and understandable way.
Atmosphere	The instructor established a comfortable learning atmosphere in the classroom.
Class Time	The instructor made good use of class time.
Effectiveness	The instructor was an effective teacher.

shows the four questions on the survey used to elicit a response on these four areas. The other three questions asked students to assess the instructor on preparation for class, knowledge of subject matter, and enjoyment of teaching. These areas are not relevant to this study, and are therefore not discussed. It should be mentioned that the survey was designed as a student evaluation of the instructor, and not an assessment of the course. There were also no specific questions concerning the MLMs themselves. Student attitudes concerning specific course components, including MLMs, have been described elsewhere in the literature [9].

Responses to survey questions were scored on a 7-point scale and the average for each class was normalized to a percentage scale for comparison across groups. For example, a response of “Strongly Disagree” would be scored as a 1, and “Strongly Agree” would be scored as a 7. An average score of 5 would be normalized to 71.4%.

Table 7. Normalized scores on end-of-semester student evaluations. For the non-MLM group n = 42, and for the MLM group n = 37. Only student perceptions of effectiveness were not significant at the 1% level.

	non-MLM (%)	MLM (%)	p
Clarity	80 ± 3	91 ± 2	<0.01
Atmosphere	88 ± 3	99 ± 1	<0.01
Class time	77 ± 4	97 ± 2	<0.01
Effectiveness	84 ± 3	93 ± 2	0.012

shows the average normalized scores on end-of-semester evaluations for all categories for both the non-MLM and MLM groups. With respect to instructor clarity, an average score of $(80\pm 3)\%$ was reported by the non-MLM group, and a score of $(91\pm 2)\%$ was reported by the MLM group. For class atmosphere, an average score of $(88\pm 3)\%$ was reported by the non-MLM group, and a score of $(99\pm 1)\%$ was reported by the MLM group. For effective use of class time, an average score of $(77\pm 4)\%$ was reported by the non-MLM group, and a score of $(97\pm 1)\%$ was reported by the MLM group. For instructor effectiveness, an average score of $(84\pm 3)\%$ was reported by the non-MLM group, and a score of $(93\pm 2)\%$ was reported by the MLM group.

A one-tail unpaired t-test indicates a significant difference at the p < 1% level between the MLM and non-MLM groups with respect to students’ attitudes in all areas except instructor effectiveness ($p=0.012$). There was no significant difference in any area between the Fa10 and Sp11 non-MLM groups, or the Su11 and Fa11 MLM groups.

The largest improvement in student attitudes towards the instructor was in the area of the effective use of class time, where a 20% increase was observed. Also of interest is the large 11% increase in instructor clarity. In particular, improvements in student attitudes concerning the instructor are interesting considering all classes had the same instructor and there was no significant difference in face-to-face activities, with the exception being an instructor-led passive lecture session for the non-MLM group.

5. Discussion

The purpose of this study was to determine if MLMs could be effective for physics courses predominantly composed of natural science majors and that utilize a tutorials-style pedagogy for in-class meetings. The results suggest that MLMs were effective at improving student learning, as measured via a nationally validated instrument and examination scores. In particular, average student learning gains on the CSEM were 16% higher and exam grades were between 8 and 9% higher for students completing MLMs. We have also found that student attitudes towards the instructor of the course were improved. This result is non-intuitive, since the same instructor and similar face-to-face strategies were employed in all courses. In this section, we discuss both cognitive achievement improvements and student attitude improvements as a result of MLM incorporation.

5.1. Cognitive Achievement

Larger increases in cognitive achievement through the use of MLMs can be explained using cognitive load theory. In particular, the MLM modules utilized in this study were informed by research in cognitive load theory, working memory, and physics education research [8]. Therefore, the modules were developed as a means towards reducing cognitive load in the learning process and by mixing modes of presentation to help students make connections between multiple representations without taxing memory channel capacity.

Multiple modes of presentation can also be achieved in lecture, especially when the instructor follows lecture-based reformed pedagogical methods, such as Peer Instruction [25]. In this study, MLM modules replaced lecture by an experienced instructor with a foundation in physics education research. For students completing MLMs, the average learning gain on the CSEM was 16% higher and exam grades were between 8 and 9% higher. Even considering the threats to internal validity discussed previously, this is still a significant improvement in cognitive achievement that may not be completely explainable through cognitive load theory alone.

As we have discussed earlier in this paper, successful implementation of reformed pedagogies requires caution, specifically because research has shown that different populations can respond differently to reformed instruction based on scientific reasoning ability, motivation, major, general academic preparation, and student buy-in to the reformed methods [13,16,17]. In particular, responses across science majors may be possible, with natural science majors (biology, marine science, etc.) responding to pedagogies in completely different ways than physical science majors. Student attitudes towards the instructional methodologies can influence learning gains, where those students that consider the methodologies and/or instructor to be ineffective choose to expend little cognitive effort participating in those activities.

The students participating in this study were predominantly biology and marine science majors in their third or fourth year of university study, potentially carrying preconceived notions of what science instruction “should” be like, such as the traditional lecture followed by memorized constructs. When confronted with active lecture and group TIP-based instruction, they may have seen little value in participation. We are not claiming that this is the case, since we have not measured this. However, improved student buy-in to the course pedagogy through use of MLMs could explain the improvements in cognitive achievement. Hadzigeorgiou et al. report such an effect, where a novel “romantic understanding” through story-telling influenced student buy-in, leading to improved cognitive performance on topics in alternating current [33]. As discussed in the next section, we did also measure a significant difference between MLM and non-MLM groups with respect to their attitudes towards instruction.

5.2. Student Attitudes

Students in the MLM group believed that the instructor presented the material in a clear and understandable way 11% more than students in the non-MLM group. This result was initially surprising, since for the MLM group the in-class instructor/student interaction was almost completely in the context of TIP activities. These activities can be difficult and sometimes initially very confusing for students, exactly because they are designed to challenge previously held conceptions about physics content. For the non-MLM group, the instructor was spending considerably more time interacting with students and discussing the same content as the MLM group.

The improvement in student attitudes concerning instructor clarity may be attributable to the instructor shifting into the role of “clarifying agent” for the MLM group. For the non-MLM group, ideally the textbook is the student’s initial introduction to the content in the course. However, as shown in previous studies, most students do not complete reading assignments, even when rewarding schemes through grading are implemented [2]. For the non-MLM group, this means that the instructor serves as the students’ first introduction to course content during the lecture period.

Students have very poor pre-instruction understanding of the content in electromagnetism, especially compared to topics in mechanics [28]. They often arrive with very few conceptions about the topic at all, much less misconceptions. This means that for this set of topics, it is not surprising that their initial exposure to the material would be confusing. For the MLM group, a third-party was responsible for introducing the content in the form of MLMs. During the face-to-face class time, the instructor was able to serve as the clarifying agent. Although this cycle was also present with the non-MLM group, they lacked a third-party to “blame” for the initial confusion.

We did not measure student attitudes towards the MLMs themselves. However, students often complained about how confusing the MLMs were and how they learned more during class. This is consistent with studies showing MLMs ranking below lecture and interactive classroom activities in students’ views about the usefulness of various course elements [8]. The MLMs serve a critical role in preparing students for learning by introducing material and helping them understand what they do and do not understand. However, students do not necessarily realize this until after the course, or never at all.

The largest improvement in student attitudes towards the instructor was in the area of class time effectiveness. Students in the MLM group believed the instructor made good use of class time 20% more than students in the non-MLM group. Considering the MLM group spent less in-class time on the same amount of material, this is not necessarily surprising; however, the students in the MLM group did not necessarily know what students in previous classes had experienced. This improvement in student attitudes could be attributed to the same process described for instructor clarity. If students perceived the in-class activities to be clarifying agents, then they could have also perceived their time in class to be more useful than the non-MLM group. Although the non-MLM group worked on the same TIP activities in class, they also had to cope with much of their initial confusion in class during a lecture-based introduction to the content.

It should be noted that the MLM groups were taught nearly one year after the non-MLM groups. This could have also provided the instructor more experience working with the population of students enrolled at the university in natural science programs. This added experience could have resulted in improved micro-interactions with individual students that were not necessarily perceived consciously by the instructor. This should be taken into consideration when evaluating the results of this study, along with the variations in JiTT and homework activities. However, the reported improvements were achieved using less in-class time compared to the more traditional lecture-then-recitation approach to TIP employed with the non-MLM group. Even if learning gains had remained unchanged, the results would still suggest that MLMs are an effective replacement for the lecture in an inverted classroom strategy.

6. Conclusions

In summary, we have investigated the efficacy of on-line MLMs as preparation for in-class, lecture-based tutorials in electromagnetism in a physics course for natural science majors. It was hypothesized, based on the MLM literature, that students completing the MLM activities would achieve higher gains in learning, perform better on semester tests, and report a perception of greater instructor effectiveness. Students utilizing MLMs did demonstrate larger gains in learning, higher examination scores, and improved attitudes towards the instructor. The contribution this study makes to the literature on both MLM and TIP is the measured efficacy of the combination of these two approaches, as well as reinforcement of the effectiveness of MLMs as a pedagogical tool across various groups and as part of a flipped classroom strategy.