Active Learning via Problem-Based Collaborative Games in a Large Mathematics University Course in Hong Kong

Abstract

Research has demonstrated the positive impact of active learning on students’ learning outcomes, particularly in science, technology, engineering and mathematics (STEM) fields. However, few studies have explored the impact of active learning via problem-based collaborative games in large mathematics classes in the context of Asian tertiary education. This study assesses the effects of active learning on students’ learning outcomes using class test scores and the calculus concept inventory (CCI) to test the conceptual understanding of the basic principles of differential calculus, in a first year calculus course in Hong Kong. Three hypotheses were tested to determine the effects of game-based collaborative learning on learning mathematics among students. Active learning through a problem-based collaborative learning methodology was employed, using Kahoot!, a game-based learning platform. Results supported all three hypotheses, demonstrating a statistically significant increase in students’ conceptual understanding and examination performance, based upon their individual perceptions of active engagement and time spent in active learning. Our results indicated that active learning “levels the playing field”, in the sense that students with less pre-requisite background knowledge, using a problem-based collaborative learning methodology, were relatively more inclined to catch up or even exceed the performance of students with a stronger prerequisite background knowledge, at the end of the course.

1. Introduction

1.1. Active Learning

The basic premise of active learning involves focusing on reinforcing higher-order thinking skills and instructional techniques, requiring learners to actively participate in their learning process [1,2]. However, the term active learning lacks a concise definition, even though it is used frequently in educational literature and educational research. Moreover, a major obstacle is the lack of universally accepted definitions and measurements as different researchers from different fields, such as education, social psychology, healthcare and engineering disciplines, provide different definitions of the term. Active learning, as defined by Prince [3], is “any instructional method that engages students in the learning process.” Instructional methods such as the flip classroom pedagogy promote engaged learning and complex thinking skills in students [4]. When active learning is employed, both the instructional strategies faculty employ to promote active learning and the strategies the students themselves use to actively engage in the learning process come into effect.

Specifically, a key essential element of active learning is to actively engage students in deeper learning by fostering their ability to create new knowledge and apply the acquired knowledge and skills by demonstrating well-developed judgement and responsibility as learners [5,6]. Moreover, active learning strategies motivate student learning in the classroom, subsequently promoting students’ initiative to take ownership of their own learning as stakeholders in their own learning trajectory [7]. Furthermore, active learning strategies are designed to shift traditional slide-based lectures to facilitated, problem-solving and collaborative learning activities that actively engage learners and result in positive academic outcomes (for example, the development of life-long learning skills and the retention of classroom-based experiences in the form of knowledge, skills, and abilities) [8].

One of the key goals of active learning is to enable students to use higher levels of cognitive functioning through cognitively deeper and richer learning experiences. Learners are able to combine prior knowledge and engage with abstract concepts that require problem-solving, collaborative discourse, critical thinking and reasoning skills [9,10]. Moreover, mobile learning tools support compelling learning experiences in a learner centric setting that is dynamic and engaging and meets the individual needs of learners, while simultaneously imparting them with the skills required to be lifelong learners [11]. With the rapid development of advances in mobile technology and game-based learning approaches, clickers or game-based student response systems such as Kahoot! or CloudClassRoom (CCR), are often used in conjunction with active pedagogies such as problem-based and inquiry-based learning, think-pair-share, peer instruction, and flipped classroom instruction [12,13,14]. Furthermore, only a limited number of studies have attempted to explore the impact of active learning via problem-based collaborative games in large mathematics classes in the context of Asian tertiary education. Hence, the objective of this study is to examine the effects of active learning via problem-based collaborative games in large mathematics classes in the context of Asian tertiary education.

1.2. Different Approaches to Active Learning in STEM Education

An active learning context refers to the various learning approaches and instructional methods such as experiential learning, collaborative learning, cooperative learning, case-based, inquiry-based, problem-based, team-based and game-based learning [15,16,17,18]. These different models cover the subset of active learning—hence, active learning is an umbrella concept that encompasses the different learning approaches and instructional methods of learning [4]. Over the years, active learning as a model of instruction in science, technology, engineering and mathematics (STEM) education has evolved, placing emphasis on different instructional approaches and the context in which learning occurs. Problem solving, inquiry-based learning and interactive learning activities all play significant roles in STEM education. For example, a problem-solving based active learning approach in a mathematics course enables students to view problems with a deeper perspective, thereby experiencing deeper learning, undertaking critical thinking and utilizing an analytic reasoning process. More specifically, research on active learning approaches in STEM education has shown to improve student performance by increasing cognitive stimulation [1,19,20]. Moreover, a quantitative meta-analysis conducted by Freeman et al., [21] found strong evidence that student performance in undergraduate STEM courses was positively influenced by the inclusion of student-centered methodologies in the classroom.

1.3. Conceptions of Teaching and Learning Within the Hong Kong Context

Despite extensive evidence-based research on the benefits of active learning strategies on student achievement, retention and performance in STEM related subjects, a large number of STEM instructors in Hong Kong do not put these teaching strategies into practice due to a lack of incentives or support from their tertiary institutions [21,22]. A key issue to consider is that the Hong Kong tertiary education system is depicted as highly competitive, examination-oriented, besides being characterized by classes with a large number of students and expository teaching, based on lecture-based delivery [23]. Hence, undue emphasis is placed on abstract and declarative knowledge in classrooms. As a result, students tend to find learning tedious, boring and/or irrelevant to their daily lives, leading to a lack of attention, passivity and ’off-task’ attitudes that have become commonplace in Hong Kong classrooms. Consequently, university level instructors in Hong Kong are more likely to resist active learning teaching practices, since they contrast sharply with their intrinsic deeply held beliefs and their own background education in passive learning and teaching [24,25]. Advocacy for STEM education in Hong Kong has only emerged in recent years, and such advocacy is frequently centered on the secondary or high school education sector, rather than on tertiary institutions [26].

2. Literature Review and Research Hypotheses

2.1. Active Learning in Mathematics in Tertiary Education in Asia

With respect to prior research on active learning in Asia, extensive research has been conducted on the topic, motivated by various aspects of active learning in STEM subjects, such as flipped classrooms in medicine [27], role playing and experiential learning in computer science [28], or the use of student response systems in physics [29]. Consequently, there are many more studies on active learning with respect to STEM education in Asia. However, up to now, only a limited number of studies have been conducted to assess the impact and effectiveness of active learning in university mathematics in Asia. For example, Kaur [30] discussed a broad overview of the challenges of implementing active learning in math education within primary and secondary schools in Asia. Moreover, Chen and Chiu [31] studied design-based learning (DBL) via computerized multi-touch collaborative scripts and its effects on elementary school students’ performance in Taiwan. However, the authors acknowledged there were limited studies that focused on active learning in university mathematics in an Asian context [31]. Furthermore, Li, Zheng and Yang [32] examined the effect of flipped methods on 120 Master of Business Administration (MBA) students of a national university in China and concluded that the flipped classroom approach supported cooperative learning and subsequently enhanced students’ learning achievements, course satisfaction and cooperative learning attitudes in science education. This provides compelling evidence that increasing active learning in STEM education for students in Hong Kong universities would be a worthy pursuit that warrants further investigation.

2.2. Student Performance and Active learning in Large University Math Classes

Research has shown that active learning increases student performance in the fields of science, technology, engineering and mathematics (STEM) [19,20,21,33]. Though numerous studies have been conducted on active learning and student performance in large STEM tertiary classes in non-mathematics subjects, there have only been a few studies on creating sustainable teaching and learning strategies for large mathematics classes [34,35,36,37]. One study examined student performance in small tertiary math classes (< 30) but did not observe any statistically significant improvement in the group with more student engagement [38]. We may put forth the argument that Freeman’s [21] meta-analysis does provide some evidence that active learning improves student performance in large mathematics classes, but his review of articles does not take into consideration large mathematics classes. However, investigations exist for methods to improve student performance via active learning pedagogies in large classes in other STEM disciplines, like team-based learning in introductory Biology [39] or project-based learning in Engineering [40]. Other studies on student achievement, performance and other outcomes have tenuous connections to active learning [20,22,41,42]. For example, Isbell and Cote [43] provide evidence to suggest that sending a simple personalized email to students expressing concern about their performance and providing information about course resources improved student performance.

2.3. Active Learning Through a Problem-Based Learning Methodology in Mathematics Education

It is commonly recognized that there are various limitations in the traditional lecture style approach to the teaching of mathematics. Mathematics is often taught purely using the lecture format, which encourages passivity and subsequently, diminishes creativity in students. However, one of the most well-known approaches to active learning in mathematics is a problem-based learning methodology. In the context of mathematics, problem-based learning approaches engage students in the thought-processes instrumental in solving mathematical problems and discovering multiple approaches and/or solutions [44]. Moreover, instead of attempting to find a single correct solution or answer, students clarify problems, discern possible solutions and assess viable options. Consequently, mathematics educators are beginning to employ problem-based learning approaches in teaching mathematics at the tertiary level, including applying several innovative and creative methods, such as the use of digital learning tools to provide interactive contexts for exploring mathematical problems [45,46].

2.4. Applying Technology-Enabled Active Learning in Mathematics Education

The application of digital learning tools and its capabilities creates environments where students create knowledge by being actively engaged in learning activities rather than being passive recipients of information [47]. Accessible learning technologies and digital learning content provide interactive and authentic contexts for exploring mathematical concepts and solving problems. By purposefully applying technology-enabled learning contexts in mathematics education, students are able to develop the necessary higher order cognition skills and subsequently, take responsibility for their own learning. Hence, active learning strategies, such as problem-based learning, can be presented through a variety of technology-enabled learning contexts, such as virtual environments, simulations, social networking and digital gaming technologies, within the classroom [48]. Research has shown that students have a proclivity towards active, engaging and technology-rich learning experiences, based on instructional strategies and technological capability [12,48,49]. Furthermore, prior studies support active learning and the integration of mobile learning technologies into mathematics course design as an effective pedagogy to enhance student engagement and collaboration [50,51,52]. Hence, mathematics educators must be able to implement innovative active learning strategies by leveraging these game-based learning and gamification technologies to make learning, in the context of mathematics, more engaging, interactive and collaborative [46].

2.5. Game-Based Learning in Mathematics in Tertiary Education in Asia

One active learning methodology that is emerging in its application is the use of game-based learning. In recent years, games have been used in traditional classrooms setting to augment active learning strategies for cognitively diverse students, by providing a context for problem solving and inquiry [53,54]. Game-based learning is an interactive learning methodology and instructional design strategy that integrates educational content and gaming elements, by delivering interactive, game-like formats of instruction to learners [55]. Moreover, game-based learning integrates aspects of experiential learning and intrinsic motivation with game applications that have explicit learning goals, thereby allowing learners to engage in complex, problem-solving tasks and activities that mirror real-world, authentic situations [56,57]. For example, results from a study conducted by Snow et al., [58] demonstrated that a computer game simulation of Newton’s laws of motion was actively effective in helping non-physics students understand key concepts of force. In addition, studies have demonstrated that game-based learning may enhance student achievement in reading skills [59], self-efficacy [60] and student performance [61,62].

A significant amount of research in game-based learning has focused on examining student performance and how game-based learning is applied, in order to make the learning process more interactive and engaging for learners [63,64,65]. For the specific subject of mathematics, game-based learning assists students to visualize graphical representations of complex mathematical concepts in a particularly engaging way [66]. However, reviews of the effects of game-based learning research on student performance are typically limited to secondary school students in North America and Europe. For example, a research study conducted by Bai et., al., [67] demonstrated that 3-D games help eight graders learn algebra more effectively, thus attaining higher scores on their tests. Moreover, various additional research studies have investigated the relationship between game-based learning and mathematics achievement and motivation [68,69].

2.6. Embedding Formative Assessment into Game-Based Learning

Formative assessment comprises of learning activities in which learners engage with learning materials as they progress towards achieving specific learning outcomes [70]. The primary purpose of formative assessment is to improve learning and facilitate the learning process. Formative assessment can be embedded into a game-based learning context, whereby learners actively engage with the technology, by responding to questions and receiving frequent, timely and relevant feedback on their progress and performance [71]. Prior studies have shown that game-based learning supports learner engagement and provides effective feedback [72,73]. Ismail and Mohammad [74] in their study demonstrated that students perceived Kahoot!, a game-based student response system, as a potential formative assessment tool to provide feedback for learning. By embedding formative assessment to a game-based learning context, learners are provided with feedback on their learning progress during the course, thus giving the instructor a way to assess learners’ understanding and also highlight concepts that require additional clarification [75]. Game-based formative assessment allows learners to monitor their progress and optimize the overall learning process. Moreover, embedding formative assessment into game-based learning in a pedagogically sound way allows for the precise assessment of learners’ progress and performance in terms of achievement of learning outcomes [70]. Furthermore, game-based learning technologies for formative assessment create active learning environments, ensuring learners are moving toward content mastery, thereby resulting in more successful learning outcomes.

To conclude this section, analysis of game-based learning in mathematics in Asia has been carried out for primary and secondary school students [66,76,77]. However, as per the authors’ present knowledge, to date, there has been only one study on game-based learning in mathematics in tertiary mathematics education in large classes in Asia [78]. The study was conducted with a sample of 326 Taiwan students on three-hour classes for 16 weeks using game-based instruction. The results of the study demonstrated that game-based instruction had a significant influence on learning achievements, whereas learning motivation had significantly positive effects on learning achievements [78]. In addition, the authors of this study did not examine the effect of game-based learning on student performance directly. Since only a limited number of studies have attempted to explore the impact of active learning via problem-based collaborative games in large mathematics classes in the context of Asian tertiary education, this research study has long-term significance for students, instructors and institutions at large. The use of technology has the potential to change the nature of learning environments and the ways in which we design activities to support active learning. Hence, an examination of the effects of active learning via problem-based collaborative games in large mathematics classes in the context of Asian tertiary education should play a pivotal role in furthering this research.

2.7. Research Hypotheses

Specifically, for the purpose of this study, the authors aim to better understand the effects of active learning via problem-based collaborative games in large mathematics classes in the context of Asian tertiary education. Consistent with related literature, this study tested the following hypotheses:

H1: 

A positive correlation exists between students’ perceptions of their “level of active engagement” and “time spent in active learning” and their academic performance (i.e., midterm test scores).

H2: 

Students’ perceptions of “time spent in active learning” is a significant predictor of their level of conceptual understanding of differential calculus.

H3: 

Students’ perceptions of their “level of active engagement” is a significant predictor of their level of academic performance.

3. Research Methodology

3.1. Research Setting and Activity

The broad purpose of this research was to examine active learning via problem-based collaborative games in a large mathematics university course in Hong Kong. A convenience sampling methodology was used as it is considered appropriate in exploratory research of this type [79]. The purpose of using a convenience sampling methodology was to facilitate a large sample size and to collect data. A total of 1017 (N = 1017) undergraduate students enrolled in a first year one semester 13-week “Applied Mathematics” course offered at the Hong Kong Polytechnic University constituted a sufficient pool of subjects, who fit well within the intent and objectives of this study. A power test was conducted in providing the appropriate sample size necessary for the study [80]. A power test is an appropriate method to dispel suspicions that a sample is too small and is thus, calculated to assess the sample size required [81,82]. Ethical clearance for the study was obtained from the ethical review board of the institution. Furthermore, informed consent, which included information about the purpose of the study, voluntary participation, risks and benefits, confidentiality, and participants’ freedom to withdraw from the study at any time, was obtained prior to participation in the research.

The course comprised of two hours of lectures and one hour of tutorials, where the medium of instruction was English. The first 6.5 weeks covered differential calculus topics, while the latter 6.5 weeks covered probability and statistics. In this study, we only considered the differential calculus part, covering topics such as functions, limits, continuity and differentiation. There were approximately 900 students in the six lecture sections labelled elephant, dog, cat, mouse, lion and tiger, taught by four different instructors. The main difference across the six sections was the delivery of the teaching methods (i.e., the independent variable) in the lectures (i.e., traditional instructor-led lecture-based method versus a game-based active learning method via Kahoot!). Instructors of the sections tiger and dog would teach the class with the game-based active learning method via Kahoot! In the sections lion, mouse, cat and elephant, the instructor used the traditional instructor-led lecture-based method. Students who enrolled in the course were randomly assigned to one of the six sections. It should be noted that students in the study were subjected to either one of the teaching methods (i.e., the independent variable) during the course. Moreover, for equivalency, the assessment tasks and one hour tutorial sessions were kept the same among the six sections to reduce the influence of mediating factors or any potential confounds.

A table of demographic variables of the students in all six sections is shown below (see

Table 1. Distribution of all students in the courses (N = 1017).

Section	Male	Female	Bachelor Level	Higher Diploma	Total Number of Enrolment	N	Test Score Mean	SD
Elephant	144	34	83	95	178	168	69.9583	24.08427
	80.90%	19.10%	46.60%	53.40%	100.00%
Dog	120	29	143	6	149	143	74.7937	19.99545
	80.50%	19.50%	96.00%	4.00%	100.00%
Cat	111	56	109	58	167	158	60.9241	24.62574
	66.50%	33.50%	65.30%	34.70%	100.00%
Mouse	130	21	31	120	151	136	58.6544	24.90995
	86.10%	13.90%	20.50%	79.50%	100.00%
Lion	128	47	174	1	175	170	72.3941	21.7977
	73.10%	26.90%	99.40%	0.60%	100.00%
Tiger	121	76	93	104	197	184	57.5761	21.91837
	61.40%	38.60%	47.20%	52.80%	100.00%
Total	754	263	633	384	1017	959	65.6439	23.86943
	74.10%	25.90%	62.20%	37.80%	100.00%

), with respect to gender, and their enrollment in various programs, with valid midterm test scores (with mean and standard deviation) in differential calculus. Each of the variables was examined for outliers and distributional properties. This was done in order to ensure that basic statistical assumptions were met.

3.2. Pre and Post Concept and Midterm Tests

The calculus concept inventory (CCI) is a measure of students’ conceptual understanding of the principles of differential calculus, with little to no computation required [83]. The CCI was developed by Epstein and Yang [83] and has undergone extensive development and evaluation including item specification, pilot testing and analysis, field testing and analysis and post-examination analysis [84]. The CCI contains 20 multiple choice questions in English. Pre and post CCI tests were administered in weeks 1 and 10, respectively. The CCI test was administered in the tutorials and students were given 20 minutes to complete the test. It should be noted that all students had finished the calculus portion of the course in week 8. A midterm test covering the calculus part of the course was conducted in week 9. The CCI was administered via a hardcopy question sheet and a one-page hardcopy bubble sheet for answers. In addition to administering the CCI, we also asked the following two questions at the pre-test assessment in respect to students’ background knowledge of calculus: (1) have you taken calculus previously? (2) Have you taken a pre-calculus course previously (functions, trigonometry, and algebra, module M1 (Calculus and Statistics) or module M2 (Algebra and Calculus))? Both questions had the following choice of answers coded as follows: 1 = no, 2 = yes, in high school/college, 3 = yes, in university.

For the post-test, we asked questions involving students’ perceptions of their active learning in class. The first question we asked was as follows: “If an active classroom is one in which students actively work on underlying concepts and problems during the class and receive feedback from the instructor or other students on their work in class, how would you describe your class this semester?” The response scale was a five-point scale coded as 1: very active; 2: active; 3: somewhat active; 4; a little active; 5: not active. Hence, the lower the value, the higher the level of engagement in active learning in classroom. The second question we asked was their approximation of the percentage of class time that they were active. The second question we asked was as follows: “On an average, about what percent of your time in class would you say was spent with active learning working on problems and receiving feedback from your instructor and/or your classmates, e.g., student response (clickers) questions, “question and answer”, etc.?” The response scale was a five-point scale coded as 1 = 76%–100%; 2 = 51%–75%; 3 = 26%–50%; 4 = 1%–25%; 5 = 0%. Hence, the lower the value, the more time was spent by students in active learning. It should be noted that, in a study conducted by Epstein [83], these same questions were asked of students in North America to measure their perceptions of active learning in their classes, except a four-point scale for “level of engagement in active learning” was used. For calculation of learning gain, we modeled our concept test normalization procedures from previous work on normalizing gains in STEM learning [85].

3.3. Active Learning Procedures and Interventions in Large Classes

In this subsection, we describe the active learning procedures and interventions applied in some lecture sections. We classify the active teaching methods according to the ICAP (interactive, constructive, active, and passive) framework for active learning [42]. In a majority of “traditional” lecture sections, students take notes - according to the ICAP framework for active learning, note-taking is the most “passive” form of active learning. In two to three of the lecture sections, a wide range of active learning pedagogies were employed. It is important to note that the most prominent active learning pedagogy used in the lecture sections was cooperative and team-based learning with games, as described earlier. In yet another lecture section, students were asked to try and solve math problems posed in class individually and the instructor provided support by checking students work and understanding. We classify the pedagogical interventions in increasing order, based on the ICAP framework proposed by Chi [42,86] (see

Table 2. Pedagogical interventions featured under the interactive, constructive, active, and passive (ICAP) framework.

Passive (Receiving)	Active (Manipulating)	Constructive (Creating)	Interactive (Social Exchange)
Sitting still and listening to explanations	Summarizing throughout lectures; highlighting sentences	Building concept maps	Cooperative groups—facilitation of student involvement
Sitting and reading slides silently	Activating existing knowledge	Asking questions	Student-student discussions
Sitting and watching a video	Assimilating, encoding or storing new information	Explaining concepts and integrating new information with existing knowledge	Small group activities + feedback
Information is stored in encapsulated form without embedding it in a relevant schema	Applying knowledge to similar but non-identical contexts (i.e., similar problems or concepts that need to be explained	Comparing and contrasting to prior knowledge or other materials	Cooperative/team-based and collaborative/group learning;
Minimal understanding	Selected information activates prior knowledge and schema	Justifying or providing rationales; deep understanding	Group interaction—communication between group members, including cooperative problem solving exercises

below).

3.3.1. Question and Answer

For large class sizes, which we consider to be courses with over 100 students enrolled in each lecture section, it is challenging to use the active pedagogy “question and answer (orally),” which would correspond to an “interactive” degree, according to the ICAP framework. Numerous research studies have demonstrated that students are reluctant to ask questions because of fear of possible embarrassment in front of other students [87,88,89]. Moreover, as passive learners, they have been trained culturally not to “speak up and ask questions” because of the fear of being ridiculed or because their questions or comments may be perceived as unintelligent or of no value [87,88,90]. To solve this problem, the instructor in this study, introduced “TodaysMeet”, an online “backchannel” used for conversations that functions concurrently with a primary activity, presentation or discussion. It was found that the instructor received more questions in the large class than one would normally have, on average, when compared to a traditional class format. Some sample student questions are as follows: “Are there ways to check if a function is 1 to 1 without a graph?” or “For d/dx 2^x can we simply apply the power rule?”

3.3.2. Student Response Systems

Student response systems or computer-based interaction systems or clickers were also used in lectures. There have been numerous studies which explore the benefits of student response systems in STEM classes [91,92]. Studies have shown that the use of student response systems in large classes is particularly useful, since students typically do not experience an opportunity to give their opinions, comments and post answers [93,94]. A student response system called “uReply”, developed by a local university, was used for challenging multiple-choice questions and for peer instruction. An example of a challenging question using a student response system in a lecture would be the following:

The error of using the tangent line y = f (a)+ f ^, (a)(x − a) to approximate y = f (x) is

E(x)= f(x) − [f(a)+ f’(a)(x − a)]. Then the limit $\underset{x\to a}{\lim }E\left(x\right)$ is (a) 0; (b) does not exist; (c) depends on the value of a; (d) none of the above”.

The “uReply” student response system was also used for the following “active” pedagogies (ICAP): “brain dump/free write” (in which students are expected to elaborate on all that they know about a specific topic) and “muddiest point” (in which students write a topic most confusing to them). “uReply” was used at least once per lecture. For example, an open-ended question would be, “What was the most confusing thing you learned in the “L’Hospital Rule section?”

3.3.3. Cooperative Problem-Based Learning with Games

In a small- to-medium-sized mathematics class of approximately 30 students, a cooperative problem-based learning method enables students to form groups of three to six to solve mathematical problems posed by the instructors. Consequently, by providing scaffolding, students are able to discuss, brainstorm and solve the problem in their respective groups and then report their answers, ideas and findings to the whole class. However, a cooperative problem-based learning method is not conducive for large class settings (over 100 students) which are typically held in large lecture style theatres. Subsequently, it is very difficult to replicate the above scenario because it would be problematic for students to form groups due to the nature of the seating layout. Moreover, there is no easy mechanism or channel for each group to communicate their solutions to the instructor, who is typically positioned at the front of the lecture theatre.

Instructors have to continually contend with the challenge of dealing with large classes [95]. In order to resolve this problem, the instructor employed an online web-based student response system. The Kahoot! game-based digital learning platform was selected to supplement this study. Primarily, the course provided a rich opportunity for applying Kahoot! in the context of a classroom. Kahoot! has shown to be a valuable learning tool for instructors to use in a classroom setting as it engages students in the active learning process through use of problem-solving strategies, activities that evoke interest as well as activities that encourage feedback. Kahoot! offered the instructor an effective and efficient method to create and generate quizzes including polling and voting functionalities. Game-based learning activities in the form of quizzes, discussions and surveys, using Kahoot!, were carefully structured into the design and organization of the course.

To begin with, the instructor created four to six Kahoot! questions per lecture, based on mathematical concepts and problems that were reviewed in the lecture. After each topic or section in a lecture was completed, the instructor would ask a Kahoot! question, based on the topic or section just covered. When playing Kahoot!, the instructor would first launch a Kahoot! game session, which in turn generated a unique game pin for each session. The students were required to go to Kahoot! (https://kahoot.it/) and enter the game pin to log into the game session on their mobile device (tablets, smartphones, laptops). Once logged in, the objective of the students (individual or team-based) is to attempt to answer a multiple-choice question correctly, and in the shortest amount of time to score the highest number of points. Firstly, the instructor posted a question, which was displayed on a screen, together with several optional answers shown in various colors and corresponding graphical symbols. Secondly, students attempted to answer the question, by selecting the correct color and corresponding symbol associated with the correct answer. Some examples of a Kahoot! multiple-choice question are as follows. On the topic of intermediate value theorem at the end of section, we asked the following:

“Claim: If the function f (x) is an odd degree polynomial, then f (x) has a root. The possible four answers are: (A) Claim is true; (B) Claim is false; (C) If the claim is true depends on the odd number; (D) None of the above.”

On the topic of limits, we asked the following question (see Figure 1 below).

In between each question, a distribution was presented by means of a scoreboard, displayed on the screen, posting the performance of the students by revealing the team’s names or individual player’s nicknames and ranked scores of the top five players. It should be noted that the instructor augmented student motivation to participate by offering a prize to the winning team at the end of each Kahoot! game. For example, gifts would be under US$7.00 on an average, and were typically in the form of a coupon or gift voucher.

4. Results and Analyses

In this section, we report the results of our study. Inferential statistics on scores of the pre-tests, post-tests and in-class exams were analyzed using Statistical Package for the Social Sciences Version 24.0 (SPSS, Inc., Chicago, IL, USA) and visually reviewed for potential data entry errors. A comparison between class sections was conducted using the independent sample t-test, whereas the performance of the same group of students in the pre- and post-tests was analyzed using the paired sample t-test [96]. There were 750 students (N = 750) who took the pre-CCI test and 470 students (N = 470) who took the post-CCI test. In the end, there were 425 students who were formally matched who took both the pre- and post-test. However, observing the scores more closely, we noted that there were 17 and 43 “missing marks” in the pre- or post-CCI, respectively. Missing marks, in this case, implied that students only submitted an answer sheet with their student ID but did not answer any of the questions. In the end, we had 365 valid and matched pre- and post-test scores, i.e., matched scores of students who had attempted to answer at least one question in each of the pre- and post-tests.

Table 3. Demographic information of students who completed both pre- and post-calculus concept inventory (CCI) test (N = 365).

Section	N	Male (%)	Female (%)	Level B*(%)	HD** (%)	Pre-CCI Mean	SD	Post-CCI Mean	SD	N	Test Mean	SD
Elephant	62	50	12	34	28	7.18	3.85	8.73	3.53	61	79.41	19.65
	100%	80.60%	19.40%	54.80%	45.20%
Dog	49	41	8	49	0	9.14	3.46	11.37	4.54	48	77.2	20.97
	100%	83.70%	16.30%	100.00%	0.00%
Cat	50	36	14	36	14	6.32	3.65	8.48	4.06	50	71.37	21.45
	100%	72.00%	28.00%	72.00%	28.00%
Mouse	59	47	12	10	49	5.98	2.58	7.47	3.57	58	58.88	23.86
	100%	79.70%	20.30%	16.90%	83.10%
Lion	81	59	22	81	0	7.14	3.54	11.99	4.35	79	76.47	21.44
	100%	72.80%	27.20%	100.00%	0.00%
Tiger	64	26	38	33	31	4.33	2.52	11.56	6.02	64	56.96	20.34
	100%	40.60%	59.40%	51.60%	48.40%
Total	365	259	106	243	122	6.62	3.57	10.07	4.76	360	70.05	23.01
	100%	71.00%	29.00%	66.60%	33.40%

B* = Bachelor; HD** = Higher Diploma.

below shows the demographic information (in terms of gender and level of study) and pre, post and midterm test scores of each of the six lecture sections.

4.1. Pre-CCI Test and Pre-Calculus Knowledge

As mentioned, there were 750 students (N = 750) who took the pre-CCI test, of which the mean, median and standard deviation CCI score (out of 20) was 6.27, 6 and 3.35, respectively. Overall, 34.6% of the students had taken calculus courses before the start of the course and only 54.9% had taken some pre-calculus courses (see Section 3.2 for coding of these variables). A correlational analysis was done (

Table 4. Correlation between pre-CCI scores and backgrounds in mathematics (N = 750).

Calculus Background	1	2	3
Pre CCI-Score Taken calculus before the course	–0.21**	–
Taken pre-calculus before the course	0.32**	0.45**	-

Note: **indicates p < 0.01.

), and not surprisingly, positive correlations were found between calculus and pre-calculus knowledge and pre-CCI scores, with correlations r = 0.21 and r = 0.32, respectively. In other words, those who had taken any calculus or pre-calculus before the class were more likely to perform better in the pre-CCI test.

4.2. Post-CCI Test and Students’ Perceptions of Active Learning

As indicated, there were 470 (N = 470) students who took the post-CCI test, of which the mean, median and standard deviation CCI scores (out of 20) were 10.9, 9 and 4.873, respectively. It is clear that the mean scores of the post-CCI were higher than the mean score of the pre-CCI, i.e., 10.9 > 6.27.

Table 5. Students’ perceptions of levels of active engagement.

	Likert Scale	Value	Frequency	Percent (%)	Valid Percent	Cumulative Percent
Valid	Very active	1	49	9.2	14.2	14.2
	Active	2	117	21.9	33.9	48.1
	Somewhat active	3	127	23.8	36.8	84.9
	A little active	4	41	7.7	11.9	96.8
	Not active	5	11	2.1	3.2	100
	Total Count		345	64.6	100
Missing			189	35.4
Total			534	100

Note: The mean and standard deviation of “students’ perceptions of their level of engagement in active learning” are µ = 2.56 and s = 0.98, respectively.

shows students’ perceptions of their active learning—84.9% students rated the courses from “somewhat active” to “very active”; and half of the students in the course rated the course “active” and “very active”.

Approximately 67.1% of students spent at least 51% of time in active learning activities in the courses (see

Table 6. Students’ perceptions of time spent in active learning.

Percentage of Time	Value	Frequency	Percent	Cumulative Percent
76–100%	1	79	23.2	23.2
51–75%	2	149	43.8	67.1
26–50%	3	77	22.6	89.7
1–25%	4	31	9.1	98.8
0%	5	4	1.2	100
Total		340	100

Note: Mean = 2.21 and SD = 0.94.

).

Correlation analysis was performed using the Pearson’s and Spearman’s correlation tests (r = correlation coefficient) to calculate the correlation between the two variables [97,98]. A p value of < 0.05 was considered statistically significant. Most importantly, correlations were found between the “level of active engagement” and “time spent in active learning” and post CCI scores, with correlations of r = −0.30 and r = −0.22, respectively (see

Table 7. Correlation between post-CCI scores and engagement in active learning (N = 470).

	1	2
1. Post-CCI Score
2. Level of active engagement	−0.30**
3. Time spent in active learning	−0.22**	0.67**

Note: **indicates p < 0.01.

). The correlation coefficient was 0.30 (p < 0.01) between the post-CCI Score and “level of active engagement” and 0.67 (p < 0.01) between “level of active engagement” and “time spent in active learning.” Note that the negative correlational values imply more involvement of active learning, which in turn, implies better post-CCI results, attributable to the coding of active learning in Section 3.2.

4.3. The Normalized Gain

The normalized gain procedure developed by Hestenes, Wells, and Swackhamer’s [99] was used to measure the gain in the mean between the pre- and post-CCI tests. Hestenes, Wells, and Swackhamer’s [99] Normalized Gain procedure is appropriate when comparing across different sections [100]. Hestenes et al.’s [99] Normalized Gain (g) is predicated upon students receiving a larger postscore than prescore, and is defined as the change in score divided by the maximum possible increase. Among those 365 students who completed both the pre- and post-CCI tests, we defined the individual student normalized gain, as

$$g=\frac{Post CCI Score-Pre CCI Score}{20-Pre CCI Score}$$

(1)

We define the students’ perceptions of active learning to be dependent on the following two variables: levels of active engagement and time spent in active learning activities. As such, correlations were found between normalized gain and student’s perceptions of engagement in active learning. More precisely, correlations of normalized gain and levels of active engagement and the time spent in active learning activities were r = -0.18 and r = -0.19, respectively (see

Table 8. Correlation among normalized gain, engagement in active learning and math background (N = 365).

	1	2	3	4	5
1. Normalized Gain	-
2. Level of active engagement	−0.18**	-
3. Time spent in active learning	−0.19**	0.64**	-
4. Taken calculus before the course	−0.06	0.01	−0.05	-
5. Taken pre-calculus before the course	−0.1	−0.02	−0.11	0.43**	-

Note: **indicates p < 0.01.

). Therefore, this suggests that the main factors contributing to an increase in normalized gain are student’s perceptions of engagement in active learning.

We also see from the above table that there is no correlation found between whether students had taken calculus or pre-calculus courses before normalized gain. Hierarchical regression analysis method was performed to test which variables were significant predictors of the normalized gain [101,102]. Step 1 included students’ learning background, i.e., students who have “taken calculus” and “taken pre-calculus” before the course. Step 2 included students’ learning background but also their participation in active learning, i.e., level of active engagement”, “time spent in active learning” and “interest to take any online preparatory course.” Using a hierarchical regression analysis method, the variables defined by students who have “taken calculus” and “taken pre-calculus”, the “level of active engagement” and “time spent in active learning” were all independent random variables. Most importantly, we found that only the “time spent in active learning” was a significant predictor (p = 0.018) of the normalized gain contributing to a 22.1% change of the normalized gain (see

Table 9. Hierarchical regression analysis of predictors of normalized gain (N = 220).

Predictor Variables	R2	∆R2	b	F	p
Step 1	0.01			1.42	0.25
Taken calculus before the course			−0.02
Taken pre-calculus before the course			−0.11
Step 2	0.08	0.06		3.19**	0.009
Taken calculus before the course			−0.02
Taken pre-calculus before the course			−0.14
Level of active engagement			−0.03
Time spent in active learning			−0.22*
Interest to take any online preparatory course			−0.02

Note: *indicates p < 0.05; **indicates p < 0.01.

).

Both correctional table and regression analysis provide evidence of the fact that engagement in active learning would affect the normalized gain, i.e., more engagement in active learning will lead to a higher normalized gain. However, the low value of R² indicated that there could be other latent factors to normalized gain and hence, future studies could examine other variables that could affect normalized gain with a higher influence.

4.4. Midterm Test Scores

The midterm test covers all the differential calculus material and was given in week 9. It is reasonable to assume that the midterm test score may be a means to quantify “student performance” in the course. However, it is not much of a surprise that we have found a significant correlation between midterm test scores and the post-CCI score (r = 0.24, p < 0.001 (see

Table 10. Correlation between midterm test scores, post-CCI scores, level of active engagement and time spent in active learning (N = 470).

	1	2	3	4
1. Midterm test scores	–
2. Post CCI score	0.24**	–
3. Level of active engagement	−0.25**	−0.27**	–
4. Time spent in active learning	−0.19**	−0.20**	0.64**	–

Note: **indicates p < 0.01.

).

In addition, Table 10 suggests that the midterm test scores were found to be correlated again with the “level of active engagement” and “time spent in active learning”, r = −0.25 and r = −0.19, respectively. To test the inference of whether an active classroom enhances student performance (in terms of midterm test scores), we use stepwise regression analysis. After eliminating the influence of student background knowledge of calculus, and using stepwise regression analysis, we found that the level of active engagement was a significant predictor (p = 0.037) of the midterm test scores, contributing to a 16.4% change of the midterm test scores (see

Table 11. Hierarchical regression analysis of predictors of midterm test scores (N = 220).

Predictor Variables	R2	DR2	b	F	p
Step 1	0.26				25.24**	< 0.001
Pre CCI Score			0.22**
Taken calculus before the course			0.14*
Taken pre-calculus before the course			0.31**
Step 2	0.3	0.04			18.30**	< 0.001
Pre CCI Score			0.21**
Taken calculus before the course			0.15*
Taken pre-calculus before the course			0.30**
Level of active engagement			−0.16*
Time spent in active learning			−0.05

Note: *indicates p < 0.05; **indicates p < 0.01.

).

4.5. Active Learning Groups and Normalized Gain

As mentioned, there were six lecture sections taught by four instructors. To minimize the difference across the four instructors over the 6 lecture sections, for consistency and equivalency, all instructors used the same set of teaching materials and assessments. This was done to minimize any potential confounds and/or threats to validity. To ensure anonymity of the six sections, we randomly ordered and designated the sections using names of animals: section tiger, section lion, section dog, section cat, section mouse and section elephant. Statistical analysis was performed using a one-way analysis of variance (ANOVA) to test the difference in means across different lecture sections for each one of the variables [80,103]. Using one-way ANOVA, we found a statistically significant difference between the lecture sections in terms of the variables of pre and post CCI test scores, normalized gain, and midterm test scores. However, it can be stated cautiously that all six lecture sections were equivalent on the variables of level of active engagement and time spent in active learning, i.e., there was no significant difference between each lecture section in terms of student’s perceptions of their active learning (see

Table 12. Multiple comparisons across different lecture sections using ANOVA (N = 365).

Variables	F	p
Pre CCI scores	13.13**	<0.001
Post CCI scores	11.73**	<0.001
Normalized gain	10.37**	<0.001
Midterm test scores	12.92**	<0.001
Level of active engagement	1.28	0.28
Time spent in active learning	0.83	0.53

Note: **indicates p < 0.01.

).

Based on Table 3 above, students in sections elephant, mouse and lion had higher scores in the midterm test compared to the students in other sections. Moreover, students in sections tiger, lion and mouse had higher scores in the post CCI results. Finally, students in sections tiger, lion and mouse had the highest normalized gain. For the normalized gain of each student, it was already established using the ANOVA test that differences were found between the lecture sections.

Additional Tukey and Scheffe post hoc comparison tests [104,105] were conducted and three subsets were formed in both tests (see

Table 13. Post-hoc analysis of normalized gain among six lecture sections (N = 365).

Sections	n	Mean	SD	Tukey 1	2	3	Scheffe 1	2	3
Elephant	62	0.074	0.36	0.07			0.07
Mouse	59	0.1002	0.25	0.1			0.1
Cat	50	0.1612	0.21	0.16	0.16		0.16	0.16
Dog	49	0.2217	0.33	0.22	0.22		0.22	0.22	0.22
Lion	81	0.3386	0.35		0.34	0.34		0.34	0.34
Tiger	64	0.4234	0.46			0.42			0.42
			Sig	0.18	0.06	0.08	0.36	0.16	0.07

). In the Tukey post hoc comparison test (α = 0.05), sections lion and tiger were in the highest normalized gain group. Furthermore, in the Scheffe post hoc comparison group (α = 0.05), sections lion, tiger and dog were in the highest normalized gain group. Based on both comparisons, sections lion and tiger were shown to have the highest normalized gain.

A further in depth exploration of the differences in normalized gain among the groups can be performed by calculating the “group normalized gain,” defined as

$$g=\frac{Post CCI Average of Group-Pre CCI Average of Group}{20-Pre CCI Average of Group}$$

(2)

Note that this is different from the individual normalized gain defined in Section 4.3. We summarize the group-normalized gain for the six lecture sections in

Table 14. Group normalized gain (N = 365).

Section	N	Pre-CCI Mean	SD	Post-CCI Mean	SD	<g>
Tiger	64	4.33	2.52	11.56	6.02	0.4614
Lion	81	7.14	3.54	11.99	4.36	0.3771
Mouse	59	5.98	2.58	7.47	3.57	0.1063
Cat	50	6.32	3.65	8.48	4.06	0.1579
Dog	49	9.14	3.46	11.37	4.54	0.2053
Elephant	62	7.18	3.85	8.73	3.53	0.1209
Total	365	6.62	3.57	10.07	4.76	0.2382

.

As shown in Table 13, we found that there were three subsets formed. By Scheffe, sections tiger, lion and dog had the highest normalized gain. Based on one-way ANOVA tests, post hoc comparison (Tukey and Scheffe) tests, and observations of the group normalized gain < g >, sections tiger, lion and dog can be classified as “medium to high normalized gain group”, and sections dog, cat and elephant “low to medium normalized gain group.” Moreover, two out of the three lecture sections employed the active learning pedagogy of cooperative and team-based learning with games described in Section 3.3.3. More precisely, sections tiger and dog had the same lecturer who used Kahoot! (team-based, problem-based learning via games) the most in his lectures. Furthermore, the lecture section lion, which had the second highest average group normalized gain < g > = 0.3771, also employed an active pedagogy of problem-based learning (without technology, collaboration and games). The instructor would pose a mathematical problem in front of the large class (e.g., a limit question) and then proceed to move around the class to support and assist any students who encountered difficulties in trying to solve the problem.

4.6. Hypotheses Testing

Overall, all three of the proposed hypotheses were supported by the data.

H1: 

A positive correlation exists between students’ perceptions of their “level of active engagement” and “time spent in active learning” and their academic performance (i.e., midterm test scores).

Data indicates that those students who perceived they were more “actively engaged” or spent more “time in active learning” performed better in the post CCI test. Since there is a correlation between post CCI test and midterm test performance, it is reasonable to conclude that our data indicates that there is a positive correlation between students’ perceptions of their “level of active engagement” and “time spent in active learning” and their academic performance.

H2: 

Students’ perceptions of “time spent in active learning” is a significant predictor of their level of conceptual understanding of differential calculus.

Another meaningful correlation discovered was the correlation between students’ perceptions of their “level of classroom engagement in active learning” and the “normalized gain” (see Section 4.3). Stepwise regression analysis indicated that time spent in active learning contributed to a +22.1% change of the normalized gain, and hence, it is reasonable to suggest that students’ perceptions of time spent in active learning is a significant predictor of their level of conceptual understanding of differential calculus.

H3: 

Students’ perceptions of their “level of active engagement” is a significant predictor of their level of academic performance.

Moreover, our results indicated a direct correlation between students’ perceptions of their “level of active engagement” and students’ academic performance. Using stepwise regression analysis again, it was demonstrated in Section 4.4 that students’ perceptions of their “level of active engagement” contributed to a +16.4% change in midterm test scores, and therefore, it is reasonable to suggest that students’ perceptions of their level of active engagement is a significant predictor of their level of academic performance.

Furthermore, this study found direct correlations (r = 0.21 and 0.32, p < 0.01) existed between students’ backgrounds in calculus and pre-calculus before the class, as these students tended to perform better in the pre-CCI test, as discussed in Section 4.1. This relationship is consistent with our expectation, as students who already possess prior pre-requisite calculus knowledge will clearly have an advantage on the pre CCI test, as opposed to those students who do not have a background in calculus. In the same light, there was also a direct correlation (r = 0.24, p < 0.01) between post-CCI scores and the midterm test scores. This reinforces the fact that the CCI test developed is a valid and reliable instrument to measure students’ conceptual understanding of differential calculus [84].

4.7. Comparison of North American Versus Asian CCI Scores

In this section, we compare the results of CCI scores and students’ perceptions of active learning from North America [83] to those in Asia (Hong Kong). We list the first three results of CCI scores for the Hong Kong Polytechnic University and compare them to the CCI scores (indicated below in parentheses) of the study conducted by Epstein [83] at the University of Michigan (U-M).

• The average gain over all six sections was 0.23815 (0.35).
• Two sections have a gain of 0.37 to 0.46 (0.4 to 0.44).
• The range of gain scores was 0.12 to 0.46 (0.21 to 0.44).

The first and most noticeable difference is in item 1, which demonstrates that the average normalized gain of 0.238 at the Hong Kong Polytechnic University was much lower than the normalized gain of 0.35 at the University of Michigan (U-M). Items 2 and 3 are almost the same, however, it was observed that the Hong Kong Polytechnic University had a wider range of normalized gain; and the highest normalized gain of 0.46 at the Hong Kong Polytechnic University was greater than the highest normalized gain of 0.44 at U-M. Although 0.46 is not significantly higher than 0.44, this result is consistent with our current (research in progress) meta-analysis findings that Asian students may benefit more from active learning than North American students.

5. Discussion

The results of the study supported all three hypotheses, demonstrating a statistically significant increase in students’ conceptual understanding and exam performance, based on their perceptions of “time spent” and “level of active engagement” in active learning. In this study, we report normalized gain to characterize the change in scores from pre-test to post-test. Furthermore, our study found a direct correlation between post-CCI test scores and midterm test scores. However, there was no correlation found between students who had taken pre-calculus or calculus courses before and after normalized gain. These two observations suggests that if we informally equate the “medium to high normalized gain group” with an “active learning group” (see end of Section 4.5), then those students who have a “weaker background” and who are taught actively can “catch up” to the students with a “stronger background,” and are in fact, at an equivalent (or sometimes exceeding) level of competency at the end of the course. This can be identified by referencing Table 13, where section tiger’s mean pre-CCI score was the lowest among all six sections (4.33), but section tiger’s mean post-CCI score was the second highest (11.56) among the six lecture sections. Hence, active learning may “level the playing field” in the sense that if students with less background knowledge when coming into a course are taught actively, they tend to catch up or exceed the performance of students with a stronger background knowledge at the end of the course.

Furthermore, it should be noted that the relationship between CCI scores and active learning have not been studied in sufficient detail for similar results and conclusions to be drawn. The low correlation may be explained by the fact that the post-CCI test scores may have been subjected to some external factors apart from “level of active learning” and “time spent on active learning,” such as learners’ academic background, learning style, motivational disposition during or before the start of the course and previous learning experience. Hence, low correlations may be explained by the weak link between post CCI score and active learning. Moreover, higher correlations could be expected with more heterogeneous samples in respect to subjects’ academic background and prior learning experience. We also perceived that the seriousness of the subjects in taking the CCI test may have an effect on the results of this study.

6. Conclusions

With the support from the Hong Kong Education Bureau, Hong Kong’s higher education system is beginning to promote and apply active learning methods and approaches in STEM-related disciplines. Because of the passive nature of learners’ involvement in traditional teacher-centered mathematics courses, active learning methods such as problem-based, discovery-based and inquiry-based learning, for example, have typically not been applied and, hence, examples of powerful instructional methods and active learning pedagogies remain relatively rare. Another reason is that instructors teaching in the tertiary education sector may not have formal teacher training or teacher experience. Therefore, they may lack the knowledge and experience in designing classes that are rich in engagement and active learning.

This study is a preliminary effort to examine active learning via problem-based collaborative games in a large mathematics university course in Hong Kong. The results of this study could encourage a growing number of tertiary instructors to adopt active learning methods and pedagogies into their teaching repertoire. One question that needs to be raised is the extent to which other active learning pedagogies such as peer learning, team-based learning and flipped-classroom have improved positive effects on student performance in large mathematics classes. In this study, the CCI was used to test the conceptual understanding of the basic principles of differential calculus and is based on the premise that developing an understanding of calculus requires an interactive process, which offers an opportunity for game-based interactivity and engagement, by employing an online student response system that incorporates different gaming elements.

We believe that the work described here can contribute to this effort by demonstrating the numerous methods by which insights into research on teaching and learning in mathematics, have been drawn upon to revise pedagogical and instructional approaches and thereby, to improve student learning outcomes. This study is a careful first attempt towards providing statistically significant evidence that active learning in the form of collaborative game-based problem solving increases students’ conceptual understanding and exam performance in a first year calculus class in Hong Kong.