An Evaluation of an Educational Video Game on Mathematics Achievement in First Grade Students

Abstract

Development of early math skills is linked to future success in mathematics and other academics. Educational video games have been shown to promote academic achievement; however, few rigorous studies have evaluated the use of educational video games in supporting math development, especially in early primary education. In the current study, an open-label randomized controlled trial was conducted involving 134 first grade students to determine, using standardized assessments, the impact of the educational mathematics tablet-based video game, Knowledge Battle, on math scores and self-competency. Overall, Knowledge Battle did improve math skills in participants who played the game. Among those with lower pre-game math skills, the Knowledge Battle group’s mean math score increased more than the control group’s mean math score (9.7 vs. 6.0; p = 0.02). There was no association between perceived sense of self-competency and total math score (p = 0.8141). However, players who had a higher sense of self-competency were more likely to enjoy playing the game. In conclusion, our findings suggest that Knowledge Battle was an acceptable and enjoyable educational mathematical video game for first grade students, and may be most impactful for those with low math skills.

1. Introduction

Development of early academic skills is essential for children’s later success in school. In particular, research suggests that early mathematics skills are the most important predictors not only for later math achievement but also for achievement in other content areas and grade retention [1]. General math achievement measured around kindergarten entry has been found to be highly predictive of subsequent mathematics achievement, measured around the time of third grade [2,3]. Additionally, those children who score highly in mathematics in kindergarten continue to score highly, while those who score low continue to score low throughout their educational careers [1,4,5]. Mathematics scores in the United States have barely moved in the last 30 years [6], and the Common Core State Standards Initiative [7] has been trying to address this problem.

Educational video games are thought to be effective tools for teaching because they (a) use action instead of explanation; (b) create personal motivation and satisfaction; (c) accommodate multiple learning styles and skills,; (d) reinforce mastery skills; and (e) provide interactive and decision making context [8,9]. In addition, there is evidence that educational video games promote student learning of important skills, positive motivation, persistence, curiosity, attention, and attitude toward learning [10,11,12,13]. A meta-analysis that examined 39 studies published between 1990 and 2012 reporting on the use of video games in an educational context showed that video-game-based approaches were significantly more effective than typical instructional methods in improving learning [14].

Educational video games have been used to promote children’s mathematics achievement in various domains including problem-solving and algebra skills [15], strategic and reasoning abilities [16], geometry skills [17], and arithmetic procedures [18]. However, there are a limited number of rigorous studies exploring the effects of the use of video games on mathematics skills development and empirical findings on the effects of educational video game use on student achievement are inconsistent [19]. To date, only a few studies have evaluated the use of educational video games in supporting math development in early primary education [20,21,22].

The primary purpose of this study was to determine if the educational video game, Knowledge Battle, compared to a series of attention/time control games, increased math scores in first grade students. The secondary objective was to determine if Knowledge Battle, compared to a series of attention/time control games, increased math self-competence in students. In addition, data was collected on participants’ gameplay satisfaction and experience of playing Knowledge Battle.

2. Materials and Methods

To determine the impact of the educational math video game, Knowledge Battle, on math scores and self-competency standardized assessments, we conducted an open-label randomized controlled trial with a cohort of first grade students.

2.1. Educational Video Game Description

Knowledge Battle is a tablet-based educational math video game designed and developed by the educational mobile game company, Yogome, Inc. (Mexico City, Mexico). Designed for first grade students, Knowledge Battle extends beyond most conventional math video games in that it does not focus solely on rote memorization of math facts. The game consists of 21 mini-games nestled inside a larger battle-style game with the goal of encouraging the player to exercise and master skills that are built upon Common Core Standards for Mathematics [7] (Figure 1). In the game, the player is able to take on the role of one of eight Yogotar characters, each of whom has strengths and weaknesses. As their chosen character, the player must master each mini-game through practice and repetitive play in order to advance in the game, with the added bonus of earning Power Cubes along the way. Power cubes strengthen the player’s Yogotar and allow them to battle increasingly stronger enemy characters as a reward for successfully completing mini-game levels. Knowledge Battle scaffolds the player by providing immediate feedback and explanation for incorrect responses. Once a particular mathematics skill has been mastered, a new mini-game (or new level within previously mastered mini-game) is unlocked for the player. To increase player engagement, Yogotar characters and storylines are incorporated into the overall gameplay.

2.2. Participants

Eligible participants were boys and girls enrolled in the first grade and were recruited from several school and community-based after-school programs.

2.3. Instruments

2.3.1. KeyMath-3 Diagnostic Assessment

Mathematics skills were assessed using the KeyMath-3 Diagnostic Assessment (KeyMath-3 DA), a comprehensive norm-referenced measure of essential mathematical concepts and skills [23]. The KeyMath-3 DA item content reflects the content and process standards described in the National Council of Teachers of Mathematics (NCTM) Principles and Standards for School Mathematics, a widely used publication that outlines the essential components of a school mathematics program. The standardized assessment contains two parallel forms (Form A and Form B) that were developed concurrently and are matched statistically and by content to reduce item familiarity and allow for a reliable measure of a student’s progress over time. The KeyMath-3DA includes 10 subtests that represent three general math content areas: Basic Concepts (conceptual knowledge), Operations (computational skills), and Applications (problem solving) and can be used with individuals aged 4 years 6 months through 21 years. The Key Math 3-DA uses basal and ceiling rules and start points, which allows for an accurate measure of an individual’s math performance. The standardized assessment is administered and scored according to uniform procedures to allow for direct comparison of a student’s results to other students of the same grade level or age and has been accepted as an appropriate instrument to assess students’ general mathematical ability, especially strengths and weaknesses in different mathematics content areas [24]. For the purpose of this research study, we collected data using the following six relevant and developmentally appropriate subsets:

• Numeration: early number awareness; place value and number sense; magnitude of numbers; fractions; decimals; percentages; exponents; integers; multiples; factors
• Measurement: early awareness of measurement; standard units; time; money
• Data Analysis and Probability: early awareness of data and probability; charts, tables, and graphs; graphical representation; statistics; probability
• Mental Computation and Estimation: early awareness of mental computation; mental computation chains, mental computation with whole numbers; mental computation with whole numbers; estimation and whole numbers; estimation and rational numbers
• Addition and Subtraction: algorithms to add and subtract whole numbers; algorithms to add and subtract rational numbers; integers; algebra
• Foundations of Problem Solving: analysis of problems; word problems; problem-solving strategies

2.3.2. Perceived Math Competence Scale

Math self-competence was assessed using a six-question, four-point “structure alternate format” Likert scale survey designed for young children adapted from the 28-item Perceived Competence Scale in Children [25]. We defined self-competence as “a child’s sense of cognitive competence”. Each item was scored from one to four using smiley-faces to represent responses of no, only a little, mostly, and yes (Figure 2). Examples of questions included, “Can you figure out math problems on your own?” and “Are you good at learning new things in math?”

2.3.3. Gameplay Satisfaction and Player Experience

Gameplay satisfaction and player experience was assessed using a six-question, four-point Likert scale survey developed by the research team, which included corresponding smiley-faces to represent possible responses, including no, only a little, mostly, and yes. Examples of questions from the survey included, “Did you enjoy playing Knowledge Battle?” and “Did you learn new things from playing Knowledge Battle?”

2.4. Procedure

The Yale School of Medicine Human Investigation Committee approved the research. Research and after-school program staff discussed the study with interested parents and children and screened for eligibility. If the child was eligible and interested, informed written consent was obtained from the parents. Demographic information and baseline measures of math skills and math self-competency were collected in a one-on-one interview with the child and a research staff member. Participants then played Knowledge Battle or a series of attention/time control games that did not include any math content on an iPad for one hour, two to three times per week for four weeks to accumulate 8–10 hours of gameplay. Measures of math skills and math self-competency data were collected again immediately at the end of the gameplay sessions (approximately four weeks after baseline). For the experimental group, data on gameplay satisfaction was collected.

2.5. Data Analysis

Baseline descriptive statistics (means (sd), median (interquartile range), and proportions/number/percent) were tabulated. Total scores as well as sub-scores from the study assessment tools were used for comparisons among groups, using parametric (t-test; analysis of variance (ANOVA) or non-parametric (Wilcoxon Rank Sum; Kruska–Wallis) tests as deemed appropriate. Two-tailed alpha level of 0.05 was used to declare statistical significance. Analysis was carried out using SAS v. 9.3, SAS Institute Inc., Cary, NC, USA.

Baseline assessments stratified the participants into two groups based on the median score of the cohort, those with more pre-game math knowledge (n = 67) and those with less pre-game math knowledge (n = 67).

3. Results

Participants were 134 first grade students (49% boys) with a mean age of 6.6 years. Seventy-two (54%) participants were black, 33 (25%) were white, 10 (8%) were Asian, 11 (8%) were other races, and the race of 8 (6%) was unknown. Twenty-nine (22%) were Hispanic, 98 (73%) were non-Hispanic, and 7 (5%) were reported as unknown.

All participants played approximately eight sessions over a four-week period. The Knowledge Battle group played a mean of 395 s (6.6 h) with a range of 96–800 s; 1.6–13.3 h). The control group played a mean of 398 s (6.6 h) with a range of 120–700 s; 2.0–11.7 h).

3.1. KeyMath-3 Diagnostic Assessment Scores

3.1.1. All Players

Among all players, there were no statistically significant differences in any of the baseline scores. After gameplay, the Knowledge Battle group’s mean math score increased more than the control group’s mean math score (7.4 vs. 5.9; p = 0.22), but this change was not statistically significant (

Table 1. Mean math scores for experimental and control groups.

GROUP	N	Mean	Std Dev	Std Err	Minimum	Maximum
Control	67	5.8955	5.7922	0.7076	−17.0000	23.0000
YoGoMe	66	7.3788	7.9358	0.9768	−17.0000	36.0000
Diff (1-2)	-	−1.4833	6.9391	1.2034	-	-

).

For the Numeration sub-score, after gameplay, the Knowledge Battle group’s mean Numeration sub-score increased more than the control group’s mean Numeration sub-score (2.3 vs. 1.5; p = 0.049). For all other sub-scores, there was not a statistically significant change in mean math scores.

3.1.2. Players with Lower Pre-Game Math Skills

For the group with lower pre-game math skills, there were no statistically significant differences in any of the baseline scores. Among those with lower pre-game math skills, the Knowledge Battle group’s mean math score increased more than the control group’s mean math score (9.7 vs. 6.0; p = 0.02).

For the Numeration sub-score, after gameplay, the Knowledge Battle group’s mean Numeration sub-score increased more than the control group’s mean Numeration sub-score (2.8 vs. 1.3; p = 0.008). For all other sub-scores, there was not a statistically significant change in mean math scores.

3.1.3. Perceived Math Competence Scores

Analysis of data from all players indicated that there was no association between perceived sense of self-competency and total math score (p = 0.8141).

For participants who played Knowledge Battle, there was evidence of differences in the mean post-pre game change in total self-competency score by responses to two gameplay satisfaction questions, “Would you tell your friends to play Knowledge Battle? (p = 0.0295) and “Did you enjoy playing Knowledge Battle?”; there was a strong association, though not significant (p = 0.0624). In both cases, those who responded “Yes/Mostly” for these two questions had a higher change in score.

3.1.4. Gameplay Satisfaction and Player Experience

Post-gameplay satisfaction data was collected from 65 (98%) participants who played the Knowledge Battle game. Of these participants, 83% reported that they enjoyed playing the game; 72% reported they would play it again; 82% reported they liked the way it looked; 62% reported that they liked the Yogotar characters in the game; 71% reported that they would tell their friends to play it; and 74% reported that they learned new things from playing the game.

There was evidence to suggest that performance level was not independent of gameplay satisfaction. There was an association between better performance and gameplay satisfaction as assessed by the questions, “Did you like the characters (Yogotars) in Knowledge Battle?” (p = 0.0501), and “Did you learn new things from playing Knowledge Battle?” (p = 0.0525). Similarly, for participants who played Knowledge Battle, those who responded “Yes/Mostly’ to the question “Did you like the characters (Yogotars) in Knowledge Battle?” had a higher change (post-pre game) in total math score than participants who responded “No/Only a little” (p = 0.0539).

4. Discussion

Overall, Knowledge Battle did improve math skills in participants that played the game. Specifically, there was a difference between the Knowledge Battle and control groups’ Numeration subtest mean scores. This finding is important, given that the Numeration subtest measures the most instructionally important math concepts and skills in the early elementary mathematics curricula. A child’s understanding of the concepts introduced in the Numeration subtest serves as a foundation for important math concepts involving estimation and computation, measurement, data interpretation, and problem solving [23], which are addressed throughout the Knowledge Battle game.

Another important finding in this study was that among participants with less pre-game math skills, the Knowledge Battle group’s mean overall math score increased more than the control group’s mean math. Consistent with prior literature that suggests that educational video games are most effective for children who start out with relatively poor skills [26], our findings indicate that participants with a lower level of first grade math skills may have benefited the most from playing Knowledge Battle, with a specific focus on increased first grade math skills related to the Numeration subtest. Additionally, participants with lower pre-game math skills had more room for gains, which was demonstrated in our study. Given that conventional educational instruction usually happens within the limitations of the classroom, the use of instructional games may provide the opportunity of allowing students to practice and master math skills outside of school, with the greatest benefit to those with less pre-game math skills. The use of instructional games at home or as part of an after-school program may provide these students in particular the opportunity to catch up to their peers.

While mathematics is often viewed as a difficult and tedious subject to learn [27], students’ confidence or sense of self-competence of their math ability has been shown to greatly influence performance and academic achievement in mathematics [28]. In contrast, in our study, participants’ sense of math self-competency was not associated with their total overall math score. This may be important to consider for the future development of math games, especially for those that are uniquely tailored to a child’s skill level. Even if a child has a high math skill level for their age or grade level, their perception of their sense of self-competency in math may in fact be low. Therefore, games should consider providing reward systems such as points or badges equally across all skill levels and not just to players that may need more support. Well-designed tools that not only optimize learning but also serve to increase self-competence in mathematics are important for all skill levels.

The majority of participants assigned to the experimental group were very positive about Knowledge Battle. Those participants who liked the characters, compared with those who did not, had a greater change in their math scores. This finding suggests that including likable characters in a game like Knowledge Battle may help to increase the player’s engagement. Players who had a higher sense of self-competency were more likely to enjoy playing the game, suggesting that players with greater confidence in their ability to succeed in math may find a greater interest in mathematics overall, which was reflected in their enjoyment of playing the game.

There were several limitations to this study. First, this study was conducted in after-school programs and findings from this study may not be generalizable within organizational settings or to other regions of the country. Second, we did not strictly control for participants’ exposure to other instruction in the classroom. However, given that each after-school study site had participants who were either in the control group or the experimental group, in general all students were exposed to the same educational instruction outside the gameplay, thus controlling for other potential exposure. In addition, the study was undertaken during a brief period of four weeks, decreasing the likelihood of significant exposure to additional content outside the game. Finally, because the video game was developed based on Common Core Mathematics Standards for first grade students, participants with a higher level of pre-game math skills at the start of the program may have reached a ceiling in their ability to learn new skills, thus preventing the video game from having an effect on their math scores at the end of the study.

Future research should include conducting a subsequent comparative effectiveness study to directly compare the Knowledge Battle game to other conventional methods of educational instruction as an appropriate next step. We also suggest future studies to include both higher and lower game levels as part of their game-based program that extend beyond only first grade instruction with the goal of creating a more tailored gameplay experience for each student. For instance, students with lower levels of pre-game math skills at the start of the program could begin at a Kindergarten or lower level in the game, while students with higher level pre-game math skills could begin at a second grade or higher level, thus providing greater opportunities to gain and master math skills.

5. Conclusions

In summary, Knowledge Battle is both an acceptable and enjoyable game-based math program with a high potential for repeated and continued gameplay. It had the greatest impact on students with lower math skills and may be most appealing to players that have a high sense of math self-competency.