Investigation of the Effect of Badges in the Online Homework System for Undergraduate General Physics Course

Abstract

Badges in education are an increasingly popular phenomenon, and a variety of questions exists as to the abilities and effectiveness of badges. In this study, the effect of digital badges within a Moodle-based online homework system was studied for an undergraduate general physics course at a large research-based university in northeast Taiwan. One hundred and sixty-two participants from two General Physics sessions were involved in this study and divided into two groups through self-selected options. Sixty-eight students in the treatment group could use digital badges in the online homework system, being able to earn one badge per assignment for turning their assignments in earlier than the assignment deadline, while the other students in the control group had no digital badges in the online homework system. The results showed that students in the treatment group turned in their assignments earlier than students in the control group did, and this difference was statistically significant. Further analysis showed that students in the treatment group spaced out their assignment practice more than students in the control group did, and the difference was statistically significant. Additionally, students in the treatment group actively attempted to earn badges, as there was a statistically significant increase in the number of badges earned by students in the treatment group over those in the control group. Based on a questionnaire given to study participants towards the conclusion of the study, the study found that students’ perception of badges was positive. These findings corroborate earlier findings by other researchers that badges can be used to motivate specific behaviors in students whilst requiring minimal changes to the course structure. However, further corroborating earlier research is the finding that badges may not be particularly useful to motivate students towards challenging tasks. An earlier study of this course in a preceding academic year found that students are appreciative of the online homework system, and it appears from this study that the primary function of badges within the system is to enhance the experience of students, as well as to motivate timely engagement with assignments.

1. Introduction

Gamification is the usage of game elements or features in a non-game context to promote learners’ learning engagement, motivation, and performance, and has been proved to induce the learners’ change of behavior [1]. The gamification not only influences the structure of the social network but also impact learners’ learning success [2]. Among the gamified elements implemented in the educational setting, badges are a game element that is frequently used [3].

Badges in the information era come to form as digital badges, open badges, or educational badges, which have been seen as a visual symbol of achievement, accomplishment, and skills to act as rewards to motivate students’ learning within social communities [4], and a method to benefit learners with goal setting [5]. Digital badges, specifically in the world of academia, remain a relatively recent addition to the sphere of online artefacts [6]. Individuals are more likely to carry out activities that provide social validation that they perceive other users have achieved by earning the badges and would tend to engage in the learning activities with efforts in a more active way [7].

Nowadays, a growing use of digital badges as an innovative instruction and credentialing strategy could be seen in higher education, but digital badging studies have yielded equivocal findings. The involvement of digital badges serves as a stimulus to motivate students’ learning performance and engagement, but the means to maintain the effectiveness brought by digital badges are necessary. For example, the more time students spend studying, the more badges they would receive, which ultimately positively promotes their involvement, engagement, and learning experiences [8]. However, the effectiveness of using badge elements declined with the disappearance of novelty effects across the time [4]. Thus, more research is needed to identify effectiveness and optimal implementation tactics [9,10,11,12].

Understanding the aforementioned potential and issues of digital badges, the authors conducted this study based on a longitudinal study on implementation of a homework system (HWS) to deal with teaching and learning problems appearing in large-enrollment teaching in undergraduate Physics courses, such as homework tardiness, a perfunctory mentality or even cheating behaviors, heavy workload in homework correction, no immediate feedback to students, etc., at a research-based national university in Northern Taiwan [13]. The HWS was examined for its effectiveness by the target learners and instructors and the results indicated that the system received positive feedback from the users, including improving the homework efficiency and students’ learning and saving time on correcting homework, leading to more appropriate teaching interactions. In addition, after the implementation of the HWS, some suggestions for follow-up improvements, such as adding a badge feedback mechanism, were raised to further facilitate active and self-regulated learning. The longitudinal study by Young and Hung [13,14] of an earlier iteration of the same course found that students typically completed their assignments in the final three days before the assignment submission deadline. However, a majority of students self-reported earlier completion dates [13,14], suggesting a disparity between perceived (and/or desired) completion dates and actual completion dates. Additionally, the professor in charge of the homework system expressed a high interest in seeing if the earliness of assignment submissions could be improved. Thus, the purpose of this study was to design badges and explore the effectiveness of integrating the digital badges in terms of helping students’ timeliness of homework assignment submission and facilitating learning performance. The following research questions guided this study:

RQ 1.

What is the effect of digital badges on the timeliness of assignment submissions within an undergraduate physics course?

RQ 2.

Do students actively attempt to earn digital badges when given the opportunity?

RQ 3.

What is the relationship between digital badges and the distributed practice of assignments within an undergraduate physics course?

2. Literature Review

2.1. Teaching of Introductory Physics with Web-Based Homework System

Physics faculty in higher education institutions (HEIs) often rely on anecdotal experience to guide their teaching practice, and this unwillingness to adopt evidence-based practice is not due to a failure to communicate evidence-based practices on the part of physics education researchers [15,16]. The state of physics education is largely traditional lecture-based, i.e., “teacher-focused with passive students” [17].

Fraser et al. [17] state physics faculty are, more often than not, experts in evidence-based research, yet are often willing to discard evidence-based practices in favor of anecdotes, and further, they argue the adoption of evidence-based practices is an increasing need for Science, Technology, Engineering and Mathematics (STEM). There is great scope for improvement in teaching methods currently used in physics education. Given the increasing need for STEM skills, practice will need to catch up to research for physics education.

When learning is altered to include an online dimension, self-regulated learning could be an issue—a student has greater control of her/his learning. Five learning attributes that improve success in online learning were identified [18], including: motivation; experience with internet technology; time-management skills; study-environment skills; and help-seeking skills. Depending on the learners, different skills may require special attention. As an example, students at a leading research university in Taiwan—the study participants—are likely to have experience with internet technology.

An earlier work by Zimmerman [19] argues that self-regulated learning, rather than being seen as comprising skills, should be viewed as processes by which learners transform their mental abilities into academic skills. Following on from this, Zimmerman [19] identified three phases of processes learners go through to transform their mental abilities into academic skills. Forethought phase: There are two major classes of forethought phase processes—task analysis and self-motivation. Task analysis involves goal setting and strategic planning, while self-motivation involves self-efficacy, outcome expectations, intrinsic interest/value, and learning goal orientation. Performance phase: There are two major classes of performance phase processes—self-control and self-observation. Self-control involves the use of imagery, the abilities for self-instruction and attention focusing, and task strategies. Self-observation involves self-recording or self-examination to identify cause and effect. Self-reflection phase: There are two major classes of self-reflection phase processes—self-judgment and self-reaction. Self-judgment involves self-evaluation and causal attribution. Self-reaction involves self-satisfaction or affect and adaptive or defensive responses [19].

Young and Hung [13,14] studied self-regulated learning with a web-based homework system along the following dimensions: goal setting; environment structuring; task strategies; time management; help seeking; and self-evaluation. The results showed that students were cognizant and active across all the dimensions studied, with a majority of students giving markedly positive responses. The only question to which the students responded negatively was when asked if they used the teaching assistant as a means of help seeking; however, students responded positively when asked if they knew how to find the help they needed.

2.2. Digital Badges in Higher Education

With the advancement of technology, physical badges have gradually transformed into digital badges [4]. Digital badges help to represent skills and achievements of a person and can be used to visually symbolize a skill, an accomplishment, an educational qualification, an interest or a certification [9,10,11,12]. Thus, nowadays, digital badges can be found in a growing number of implementations, recognizing learner effort or mastery of learning. In addition to an increasing number of digital badge systems is also the belief that the primary value of digital badges is as a credentialing mechanism. However, badge skeptics have doubted the use of digital badges as credentials, usually with a variation of the question, “Why would a student want to earn a badge?” One common response to the skeptics from digital badge advocates is that future employers or educational organizations will provide employment or credit based on the digital badge [20].

Nevertheless, we have begun to use digital badges in higher education to encourage student persistence by motivating them, recognizing their generic skills, signaling their achievements, and capturing their learning paths. For instance, a badge-based achievement system was introduced into an online learning tool used by college students [21]. A randomized controlled experiment involving over 1000 students found evidence for the positive impact of badges on students’ levels of participation. Specifically, badges increased the quantity of students’ contributions and the length of time they engaged without decreasing the quality of their contributions. In a study on a university-level online computer science learning environment, badges were implemented to the treatment group and control group. The results show that achievement badges can be used to affect students’ behavior positively with the majority of the students reported being motivated by the badges [22]. In addition, digital badges were integrated into the peer evaluation system and introduced into university classrooms. This system was well received by students, and the digital badges also increased the participation of students in peer evaluation [23]. Moreover, uses of digital badges in the classrooms of Midwestern University and the results showed that the group using the digital badges had significantly better classroom performance than the unused group [24,25].

A number of works have found evidence for the positive impact of badges on students in higher education. However, badge uses in nursing students’ performance and participation in bioscience practical class led to declining results, with the disappearance of novelty effects across over time [4]. This indicates digital badging studies have yet yielded equivocal findings. Thus, more research is needed to identify effectiveness and optimal implementation tactics [9,10,11,12]. Nevertheless, digital badges have become a popular method of enriching academic experience at many universities. Nowadays, digital badges represent an accomplishment, which appear as icons or logos available online. Introducing digital badges into university courses offer an innovative way to deliver the learning objectives, as well as to motivate learning beyond the classroom [26]. General physics is the foundation of higher engineering education, and students’ low motivation to learn is the biggest problem encountered in general physics education and is of great concern. How to use digital badges to associate teaching methods to enhance student participation, motivation and engagement and enhance students’ interest in general physics courses has become an important issue of teaching and research [13,26,27,28,29]. Given the specific context of associating HWS with digital badges in this longitudinal study, we hope this study could shed light on the uses of digital badges, while the goal of this study focused on developing digital badge system and how such a system might impact on learning in general physics.

3. Materials and Methods

This section will cover information about the badge design and homework system, requirements to earn a badge, study context, sample size, data collection, management and analysis, etc.

3.1. Badge Design and Homework System

To meet the context of this study, a full badge system was designed and the look and feel of the assignment chapter badges were designed to reflect the assignment topic. Moreover, a formative feedback of badge design was conducted prior to the start of the semester using this feedback alongside advice from other studies on badges in education [21,28,30,31,32,33]. Students in the treatment group were always able to see the available badges at any point in time.

The homework system was hosted on the Moodle Learning Management System (LMS). The Moodle LMS Essential theme was used for the interface design. Each course section was administered in a Moodle LMS course of its own. The badges were made available to the treatment group using the badge feature in Moodle. Each assignment had about 6–9 questions and was assigned using the Quiz module. The questions were of the calculated question type with automated assessment—students knew whether they had passed or failed a question instantly. Each assignment question could be attempted an unlimited number of times without penalty.

There were 16 assignments, but they had shared submission deadlines. Assignments with shared deadlines in chronological order were: Chapters 5–9; Chapters 10–14 and 32; and Chapters 15–19.

Table 1. Weekly assignment: topics and badge images.

Assignment	Topic	Badge Image	Assignment	Topic	Badge Image
Chapter 5	Motion, Force and Newton’s Laws		Chapter 6	Work, Energy, and Power
Chapter 7	Conservation of Energy		Chapter 8	Gravity
Chapter 9	Systems of Particles		Chapter 10	Rotational Motion
Chapter 11	Rotational Motion		Chapter 12	Static Equilibrium and Oscillatory Motion
Chapter 13	Static Equilibrium & Oscillatory Motion		Chapter 14	Wave Motion, & Interference & Diffraction
Chapter 32	Wave Motion, & Interference & Diffraction		Chapter 15	Fluid Motion
Chapter 16	Thermo-dynamics		Chapter 17	Thermo-dynamics
Chapter 18	Thermo-dynamics		Chapter 19	Thermo-dynamics

shows the weekly assignment badges. Additionally, higher-level badges were created to motivate students to earn more badges and to strive for badge accumulation, as shown in

Table 2. Higher-level badges with requirements.

Chapters	Requirement (Name)	Image
Chapters 5–9	Obtain 4 badges from 5 assignments (Bronze Cup)
Chapters 10–14 & 32	Obtain 4 badges from 6 assignments (Silver Cup)
Chapters 15–19	Obtain 4 badges from 5 assignments (Gold Cup)
All Chapters	Obtain 3 badges from chapters 5–9, 3 badges from chapters 10–14 & 32, and 3 badges from chapters 15–19 (Star)
All Chapters	Obtain 4 badges from chapters 5–9, 4 badges from chapters 10–14 & 32, and 4 badges from chapters 15–19 (Einstein)

.

3.2. Requirements to Earn a Badge

For a student to earn a weekly assignment badge, s/he had two requirements:

• Score full marks on any of the assignment attempts—first, second, or third;
• The full-mark attempt occurred before a given date—typically one week after the assignment was made available—which was readily visible in the homework system.

This study adopted quasi-experimental methods. The samples self-selected themselves into the two course sections as is customary according to university regulations. Students in the treatment group were provided with an informational document in the system to inform them of these requirements. This full mark requirement was applied because a student’s maximum score of three attempts determined the effective score for the student on any given assignment. Without such a requirement, a student could turn in an assignment without taking care to respond to the questions in hopes of getting a badge. Such activity would come without penalty as only the attempt with the maximum grade contributed to the student’s effective assignment grade. While this could have placed an extra burden on students who were willing to achieve badges—correctness in addition to timeliness—an earlier study by Hung [14] showed that students routinely achieved the maximum assignment score for each assignment. Additionally, the individuals within the physics department who were responsible for the assignment questions in the homework system tried to make the questions relatively undemanding in an attempt to motivate the willingness of students to attempt questions [14]. This was because the students enrolled in the course were not physics majors.

3.3. Setting of This Study

The study was conducted at a research-oriented university in northern Taiwan. The course under study was the “General Physics B (I)”, which runs yearly in the fall semester. This course is typically offered to first-year undergraduate students drawn from a variety of colleges and departments other than the physics department. The text used for the course was Essential University Physics by Richard Wolfson (Second Edition). Classroom periods were held two times weekly. The course covers twenty chapters from the course text. The course curriculum treats the first four chapters as foundational, and students are expected to be familiar with the material from their pre-university study. The remaining sixteen chapters are primary to the course, and each one of these chapters has an assignment attached to it.

3.4. Sample Size

In total, one hundred and seventy-seven students had registered into both classes at the close of the course add/drop period. Seventy-nine students were in the treatment group, and ninety-eight students enrolled in the control group. Professors within the physics department developed a pre-test to measure students’ knowledge of the topics. This pre-test was administered to students before they began assignments. Of the 79 students enrolled in the treatment group, only 70 students were considered as part of the experiment, as nine students did not take part in the pre-test. Of the 98 students enrolled in the control group, 95 students were considered as part of the study; the other three students did not take the pre-test. Of the 70 students remaining in the treatment group, two students failed to take more than one exam and they were excluded from the analyzed treatment group, resulting in 68 students making up the treatment group. Of the 95 remaining in the control group, one student failed to take more than one exam resulting, in 94 students making up the control group. Thus, the analyzed sample size was 162 students.

The students in both groups came from markedly different colleges and departments within the university—a college contains multiple departments. Across both experiment and control groups, there was very little overlap in the colleges. The majority of students in the control group belonged to college G (n = 83, 88.30%), while no student in the experiment group belonged to this college. Students in the experiment group largely came from two colleges: A (n = 31, 45.59%) and E (n = 33, 48.53%). Additionally, almost every single student in both classes was a student in her or his first year of study. In the control group, there were 87 (92.55%) first year students, while there were 66 (97.06%) first year students in the experiment group.

3.5. Data Collection, Management and Analysis

This study employed a survey to obtain students’ perceptions of badges administered to students in the treatment group. The survey was a modified version of the survey developed by Haaranen et al. [31] for the same purposes. In addition to the original survey, an open-ended question was attached to each close-ended question to generate as much insight and unstructured feedback as possible from respondents. The survey was administered via the homework system.

Assignment data, alongside badge data, were stored on a server located in a graduate student laboratory at the university. Only the researcher and the research supervisor—the university professor—had the credentials required for access to the assignment data saved on the server. The assignment and badge data were retrieved from the study server using SQL scripts. Preparation of the data saved in the homework system for analysis was performed using the Ruby programming language. The output of the data preparation process was in narrow data presentation. All subsequent quantitative analyses were performed using the R programming language. Descriptive statistics were used to analyze timeliness across both groups over the assignments. The timeliness of assignment submissions was treated as panel data; students were the units of analysis, and the different assignment timeliness were the data along the time dimension. The timeliness data were fitted to the panel data regression model, with the study group as the independent variable and timeliness as the dependent variable.

4. Results and Discussion

The results are presented based on the data collected means of the survey via the homework system and data are interpreted below in reference to the reviewed literature to address each of the research questions raised earlier in the hope to shed light on the uses of badges along with the homework system to facilitate general physic learning.

4.1. Timeliness of Assignment Submissions

Timeliness of an assignment submission was determined using the positive difference between the time of the assignment submission and the time of the assignment deadline, measured in floating point days. The timeliness used for each assignment was the time at which a student achieved her or his maximum score of all the student’s attempts. If a student achieved this maximum score on more than one occasion, the earliest submission (maximum timeliness) was selected as the timeliness for the assignment.

Figure 1 shows the distribution of submission times of each assignment by study group; the dotted lines show the deadline to earn a badge for each assignment. As shown, the median is always higher in the treatment group than in the control group. A panel data regression model was used to analyze the effect of course selection on timeliness, with the study group as the independent variable and timeliness as the dependent variable. Additionally, the scaled pre-test scores of the students were added to the model as a predictor of timeliness.

The resulting panel data regression model was statistically significant (R-Squared = 0.078, Adj. R-Squared = 0.078, F-statistic = 100.49 on 2 and 2372 DF, p-value: <2.22 × 10¹⁶). As shown in

Table 3. Coefficients of independent variables in linear panel data model (treatment = 1, control = 0).

	Estimate (Days)	SE	t	p
Group	1.84	0.16	11.39	<2.22 × 1016 ***
Scaled pre-test grade	0.69	0.083	8.36	<2.22 × 1016 ***

Significance codes: *** p < 0.001.

, course selection had a statistically significant effect on timeliness; being in the treatment group decreased the average submission time by about 1.8 days.

Based on the modelling results, the question arises as to the meaningfulness of a 1.8-day average difference between both groups. Timeliness is a useful translational graduate attribute [34]—and an increase in timeliness as an end is worth it—and the modelling results show the ability of a badge system to positively influence this skill. Whether this improvement stays after badges are removed from a student’s digital environment is another question, one that is outside the confines of this study. Additionally, for a treatment that imposes minimal changes to the non-digital dimension of the course, badges appear to be a relatively useful tool.

In relation to the study’s first question: “What is the effect of digital badges on the timeliness of assignment submissions within an undergraduate physics course?”, the fact that students in both groups came from markedly different colleges (see Figure 1) weakened the internal validity of the results, and thus, our ability to assign cause and effect. Nevertheless, a statistically significant increase in timeliness was observed in the treatment group.

Reversing the treatment and control groups and repeating the experiment could resolve this issue. This would be dependent on the expectation that students in a set of colleges are typically attracted to the exact professors involved in both classes. If the results stay the same in terms of treatment and control group, then it is resolved that the outcomes found at the end of this study are not related to the colleges the students come from but are an effect of badges.

4.2. Attempt to Earn Digital Badges

In order to find out whether students in the treatment group actively attempted to earn badges, the criteria used to award badges to students in the treatment group were retrospectively applied to the control group to see how both groups fared in terms of acquiring badges.

Across all the periods in the semester, students in the treatment group never obtained fewer badges than students in the control group. To test whether this difference was statistically significant, a two-tailed Wilcoxon Rank Sum test with continuity correction was employed; the badge data were heavily skewed towards zero, as some majority students never earned a single badge.

As shown in

Table 4. Descriptive statistics and results of Wilcoxon rank sum tests on badge data.

Type of Badge	Group	Badge Count (Unique Earners)	M	SD	W	p
Sum: chapters 5–9 badges	Control	22 (12)	0.23	0.71	2375	0.00015 ***
	Treatment	67 (25)	0.99	1.54
Sum: chapters 10–14, 32 badges	Control	6 (6)	0.064	0.25	2759	0.0080 **
	Treatment	29 (13)	0.43	1.00
Sum: chapters 15–19 badges	Control	14 (10)	0.15	0.53	2806	0.035 *
	Treatment	31 (15)	0.46	1.01
Sum: all chapter badges	Control	42 (19)	0.45	1.09	2276	0.00013 ***
	Treatment	127 (31)	1.87	2.92
Bronze Cup	Control	1 (1)	0.011	0.10	2854	0.0035 **
	Treatment	8 (8)	0.12	0.33
Silver Cup	Control	0 (0)	0	0	3149	0.24
	Treatment	1 (1)	0.015	0.12
Gold Cup	Control	1 (1)	0.011	0.10	3136	0.38
	Treatment	2 (2)	0.029	0.17
Star	Control	0 (0)	0	0	3102	0.097
	Treatment	2 (2)	0.029	0.17

Significance codes: *** p < 0.001, ** p < 0.01, * p < 0.05.

, the mean of the badges earned by students in the treatment group across the semester is significantly higher than the mean of the badges gained by students in the control group. Thus, it appears that students actively attempt to earn badges when given the opportunity. However, there is no significant difference between both groups in terms of earning the higher-level badges except for the Bronze Cup. No student earned the Einstein badge.

4.3. Distributed Practice

Inter-session interval (ISI) was used as a measure of distributed practice [35]. To determine ISI in this study, the course assignments were grouped according to shared assignment deadlines: Chapters 5–9; Chapters 10–14 and 32; Chapters 15–19. The timeliness of assignment submissions within each assignment group for each student were sorted by size, and the positive difference between consecutive timeliness data (after sorting) was used to determine the ISI.

Table 5. Descriptive statistics for the ISI for each period of the semester.

Period in Semester	Group	n	Mean (sd)	Median	Min	Max	Range
1st period	Control	327	1.25 (3.01)	0.16	0	22.60	22.60
	Treatment	223	1.91 (2.82)	0.59	0	15.26	15.26
2nd period	Control	430	0.62 (1.37)	0.13	0.00013	18.99	18.99
	Treatment	321	1.12 (2.45)	0.24	0	19.02	19.02
3rd period	Control	354	0.84 (2.18)	0.080	6.94 × 105	19.01	19.01
	Treatment	259	1.42 (3.22)	0.10	1.16 × 105	21.67	21.67
All periods	Control	1111	0.87 (2.23)	0.12	0	22.60	22.60
	Treatment	803	1.44 (2.83)	0.20	0	21.67	21.67

presents descriptive statistics for the ISI for each period of the semester. The median and mean ISI are consistently higher in the treatment group as seen. However, the medians are all below one day, i.e., the median spacing between the completion of assignments in all periods of the semester are less than one day.

To determine whether the difference in ISI between both groups was statistically significant, a two-tailed Wilcoxon Rank Sum test (Ntreamtment = 803, Ncontrol = 1111) with continuity correction was employed due to the ISIs being highly skewed towards zero as seen in Figure 2. The results indicate a significant increase in ISI in the treatment group (M = 1.44, SD = 2.83) over the control group (M = 0.87, SD = 2.23), W = 399260, p < 0.001 (Table 5).

The analysis specifically answers one of the secondary questions of the study: “What is the relationship between digital badges and the distributed practice of assignments within an undergraduate physics course?” and the presence of badges is associated with an increase in distributed practice. Distributed practice is another translational graduate attribute—like timeliness—implying that an increase in itself is well worth it. Nevertheless, are the gains in distributed practice within the treatment group relative to the control group large enough to increase a student’s long-term recall in a complex subject such as university physics? While this is an empirical question which real-life time constraints did not permit this study to answer, related work by Grote [36] and Rohrer and Taylor [35] show that the ISI values have proven to be an effective determinant of long-term recall for complex learning tasks is in the magnitude of several days to weeks, not fractions of days [35,36]. For the treatment group, the mean ISI across the semester was 1.44 days and the median was 0.2 days. Following on from this, it appears a majority of students would have to have been obtaining badges or close to doing so on a weekly basis for inter-session intervals in the order of weeks to have been observed. However, the question remains open given the absence of evidence required to show what kind of link exists between badges and long-term recall.

4.4. Performance on Assignment and Exams

The performance of both groups on the assignments and exams is reported; however, there is nothing in the literature that suggests improved or reduced performance. The average assignment score was 94% in the control group and 96% in the treatment group. To test whether the difference in means of exam scores of both groups was statistically significant, a linear regression model with study group as the independent variable was used to predict each exam score. Only for the second exam is there a statistically significant increase in the average grade of a student in the treatment group (see

Table 6. Regression results using group (Control = 0; Treatment = 1) to determine exam scores.

Exam	Estimate	SE	t	p	Adjusted R-Squared
First Exam	2.969	2.824	1.051	0.295	0.00065
Second Exam	4.097	1.898	2.159	0.0323 *	0.02224
Third Exam	2.657	2.720	0.977	0.33	−0.00029

Significance codes: * p < 0.05.

).

5. Conclusions and Future Study

In this quasi-experiment, badges were added to a general physics homework system in an attempt to improve the timeliness of assignment submissions and distributed practice of assignments by students. Our findings show that badges can be used to motivate specific behaviors in students whilst requiring minimal changes to the course structure. These results corroborate results reported by Hakulinen et al. [32] and Denny [21]. However, as found by Denny [21], badges may not be particularly useful in motivating students towards difficult challenges—no student earned the demanding Einstein badge in our study and there was no statistically significant difference between both groups in terms of earning higher-level badges, except for the Bronze Cup (see Table 4). It is possible that badges are effective motivators for low hanging fruit—beneficial tasks that require little effort; further studies are needed to confirm this.

The overall significance of this study is that this is the first study to attempt to estimate the effect of badges on an outcome of interest using panel data modelling. This analysis allowed us to identify the 1.8-day difference in timeliness between students in both groups. While questions may exist as to the meaningfulness of a 1.8-day difference, this number is not to be considered in isolation. The end-of-semester survey on badges revealed that students felt badges were a useful element within their homework system—an element they would like to see return. Moreover, for some students, earning badges was an “honor”. The reported benefits in terms of learning outcomes are mixed when an online homework system is introduced into the teaching of general physics.

Despite this, badges were not able to influence exam performance of students in this study [37]. The reported benefits in terms of learning outcomes are mixed when an online homework system is introduced into the teaching of general physics [38,39]. However, major gains are to be found in improved attitudinal stances towards the course under study [38,39], whilst studying an earlier iteration of the same course studied in this quasi-experiment found that students greatly appreciated the homework system. It appears that badges primarily serve to enhance this experience and motivate their efforts.

Additionally, it might be worth exploring the effects of badges using the switched replication design. This would allow researchers to see whether students retain the behavioral changes they made in the presence of badges once badges are removed.