Is Classroom Gamification Opposed to Performance?

Abstract

Learning through digital media is currently considered as a mixture of methodologies that aim to approach the reality of students to motivate them. Gamification emerges as one of the pedagogical methodologies on the rise in digital classrooms, but is it really practical? What are its effects? A gamification longitudinal case-study experience was carried out in the subject: complements for disciplinary training in technology of the master’s degree of secondary school teachers (technology specialty) that is taught at the Universitat Politècnica de Catalunya, where gamified activities and alternatives to traditional master teaching have been introduced in a progressive way throughout three academic years (2016/17 to 2018/19). The results show that there is a negative correlation between the numerical scores of the different components of the evaluation and the marks obtained in the activities of gamification. It is surprising that the group less involved in the gamification obtained better academic results, although gamification improved the motivation and the valuations of the subject, due to the inclusion of more games in the course. This raises doubts on whether the positive effects of gamification on the climate of the classroom and on motivation are opposed to academic results.

1. Introduction

Under the term “gamification” are strategies and methodologies outlined in scientific literature that encompass the approaches and possibilities of applying different types of games to teaching–learning (TL) processes [1,2,3]. With the use of tests, game rules, temporal limitations, punctuation and many other elements of the game, teachers transform their classrooms into playful environments to stimulate the motivation, action and positive feedback of students [4,5,6,7,8].

The introduction of games into the educational system is not new, since Quinto Horacio Flaco, in the first century before Christ, in his Ars Poetica (Epistula ad Pisones), suggested the idea that teaching can (and should) entertain or even amuse, creating the Latin cliché of “Prodesse et delectare”. Although Horacio refers to poetics, his advice goes further and can be very useful when proposing the gamification of our classes: he encourages us to make our narrative useful for life, to keep our advice brief (eliminating the superfluous), since the fictions that we invent are plausible [9]. These three Horacian tips are essential for both good gamification and evaluation. They also touch on two arguments used by many teachers for not incorporating gamification in their classes: the lack of time and the unreality of gamification (distance from the curriculum or knowledge).

Gamification seeks to directly impact the stimulation, attention, and retention of knowledge by an increasingly dispersed student body (in part due to the interaction with technology). Although more studies and empirical evidence are still required regarding the adequacy, results and viability of gamification [1,3,10,11,12,13,14], it offers students the opportunity to get involved in an experience that, with fun as a means and as a goal, allows them to control their TL process and improve their skills, without detriment to the acquisition of new knowledge and skills of the digital era. Teachers, on the other hand, can also provide digital devices with new uses linked to learning.

Although games improve certain cognitive abilities [4,15], it is debated whether education is worth taking on its hypothetical cost (both temporary and economic investment) that is implied in order to carry out a good development of classroom gamification [16,17] considering that other skills such as socialization in the case of video games, which have been analyzed outside of the educational context [18,19], or the hypothetical one, could worsen promotion of competitiveness and violence [1,20] in critical approaches that are often dichotomous and reductionist [18].

The lack of rigorous studies makes it difficult to establish whether gamification really improves student learning, beyond increasing fun and motivation in the classroom (which is already a relevant achievement). If the gamification neither improves nor worsens the learning, but it does improve the classroom climate and the motivation of the students, what does seem to be observed in empirical studies [21,22], although with differences depending on whether the motivation is intrinsic or extrinsic [23], is that it is obvious that an objective would have already been achieved, especially in classrooms where demotivation is a serious problem. For example, a good part of the studies included in the review on the assessment [13] are perhaps good teaching practices that do not suppose empirical evidence of quality to consider the gamification as a better methodology than others, although the teachers who applied them found they worked in their classrooms.

However, the strategy of mixing methodologies in the classroom allows focusing the class towards different types of students. Gamification emerges then as a methodology that is mixed with other pedagogical tools (project-based learning, flipped classroom, master class, problems, challenge-based learning...), with it being a complementary ingredient of a mixed classroom full of varied methodologies that enables teachers to work with the diverse competences of students [24,25,26].

To adapt to this changing, mobile and hyperconnected context, teacher training now incorporates learning experiences based on elements, mechanisms, techniques and innovative and attractive dynamics for students [27], and thus provides educational alternatives to mobile devices. On the other hand, the bring your own device (BYOD) wave and advancement in learning technologies pave the way to new pedagogical methodologies and promote engagement in the classroom [28]. In fact, a diversified class with games, technical projects, problems, challenges, oral presentations, artistic or musical creations and exams will offer enriched and diverse evaluation mechanisms [29].

1.1. Gamification Levels

From a theoretical point of view, it has recently been suggested [28,29,30] that partial gamification may be a more suitable approach than full gamification of a course [27]. Incorporating gamified elements throughout the course allows, in the case of teacher training, the learning of various techniques that future teachers can later extend to their classes [30].

To begin with, although it is true that every tool requires an initial investment in time, there are many tools that allow you to save time later, especially in the corrections of large groups. To give a simple example, if you take multiple-choice exams, pass them to real-time response platforms, such as the well-known “Kahoot!” or “Socrative”, it will involve little more than a “cut–paste” action, and in return they will obtain their results scored instantly and in an Excel-like spreadsheet, which is operated easily, and they will also give immediate feedback to their students [31]. The level of gamification that is carried out (see

Table 1. Different levels of gamification possible in a course, with examples and general assessment considerations, following [30].

Gamification Level	Definition and Scope	Assessment	Example
0% Specific gamification	Teachers play a specific game in very specific activities. Ideal for teachers who want to start in gamification, with little time or whose evaluation is governed by external evaluations that do not depend directly on them.	The marks of the games are qualifications of class activities, within the continuous evaluation and with a small weight in the final mark of the subject, generally not greater than 10%, since there are other projects, works or exams that weigh more.	Specific “Kahoot!” or “Socrative” tests to review the assimilation of content or specific activities. Challenges in external contexts outside the conventional classroom.
Partial gamification	Teachers decide that a part of the subject is taught through games, but not all, combining games with other mechanisms and pedagogical methods. Intermediate, flexible system, in which traditional non-gamified elements are not completely renounced, combining games with conventional projects, exams and activities.	A significant percentage of the subject (20–80%) is evaluated with games, which are one more ingredient within the methodologies used. Depending on the percentage that the game supposes in the assessment, we will get closer to the total gamification and the students can already have the feeling of playing quite or a lot.	Assessment of all the course readings through gamified tests. Gamified oral presentations.
Total gamification 100%	The whole subject is a game, which can be composed of different games with which to evaluate all the competences. Recommended only for teachers already quite experienced in the specific and partial gamification, or when the characteristics of the course are appropriate (seminars and practical courses).	The scores and merits of the games are 100% of the mark, so they must be ‘translated’ into the official marks at the end of the course. Ideal for courses evaluated through projects or in which students can choose between a final exam and the mark for the course’s gamification.	Challenge Based Learning. Final exams validated by the continuous evaluation of gamified activities. The final exam remains as recovery for those who do not pass the game.

) will mark the temporary investment necessary for teachers [30].

On the other hand, playing in the classroom does not imply moving away from the “reality” of the theoretical contents of the subject. Playing is not a useless approach, but quite the opposite, as long as the fictions developed (avatars, tasks, parallel or fantastic universes...) do not flee excessively from verisimilitude and get lost in the narrative [9] or, worse, result in addiction to the screen (our students do not need extra screen hours) students will value the proximity between the gamification fictions and the tangibility of their experience, which is especially relevant in science.

The main idea is that teachers consider games as one more ingredient in their classes, within a hybrid teaching methodology, so that the specific or partial gamification of their subjects is very simple [27,30].

Turning to full gamification is a more complex step that can be progressively reached, once the teacher feels safe and comfortable with this new approach to the subject, when partial gamification transcends specific games, and when it is clear that you can easily evaluate the impact [30]. Teachers must give their evaluation narrative sufficient credibility and, of course, adapt to the reality of their classroom and their school context [16,25,27,30]. In other words, it is the teachers who must assess whether in widely dispersed class-groups it is appropriate to introduce a total gamification or it is better to stay in a partial or merely specific gamification (Table 1), or even not play [30].

1.2. Gamification: Preparing the Course

To start gamifying a course, you should especially consider [30]:

(i)

The level of gamification that you want to introduce (specific, partial, or total), and be consistent in the percentage weight that is given to the games in the evaluation (%).

(ii)

The main activities and assessable evidence to be collected from students, both gamified and non-gamified (if gamification is not total). Consider covering all the curricular and cross-curricular competencies of the course with the activities.

(iii)

The narrative of the gamified course in which the students will be immersed.

The narrative of the gamified course is crucial to its success, especially when the degree of gamification is increased. In punctual gamification, the narrative will not be relevant, since gamified activities will be proposed in a discretionary way. However, ideally, in partial gamification it is necessary to find a common thread that links the different activities that are introduced in the classroom and that, in the case of total gamification, is essential. A very simple way to do this is to take advantage of existing narratives that are in the minds of our students, either because they know them or because they, at least, sound familiar.

Regarding the activities and their evaluation, an easy-to-understand course is an easy-to-explain course, that is, it is a course from which we can make a simple outline of both the most important activities and their percentage weight in the final evaluation, and if the games are part of the course, they should be integrated into the evaluation proportionally.

1.3. Tools for Gamification

One of the most common criticisms of gamification is that it “generates competitiveness” among students, by proposing classifications or rankings. However, do the grades of the students not usually do so throughout their entire school life? We want to defend here that attention to diversity involves including varied methodologies and activities in the classroom, which is perfectly compatible with the gamification of part of a course and its evaluation (partial gamification, see [30]).

We are in a transition period in which we can have students who have been working in the same way for many years, they have adapted to the system and are performing well. Evaluating them just by playing could harm them—are they to blame or is it due to the fruits of the educational system they have received? If gamified activities are carried out as part of the continuous formative evaluation, while submitting work for those who are not satisfied with the points obtained in the games, there will be few complaints as they will be able to choose. The deliveries must be spaced and distributed across the course so that the most hardworking students do not become overloaded, so that they will add points, and they will go to the next level and free themselves from tasks, as we will see.

Critical reviews of gamification have shown that many of the experimental studies are difficult to replicate, undermining some of the evidence on gamification [13]. Something fundamental and still controversial is how to evaluate whether or not a methodology [32] or teaching practice [33] has been effective. The methodologies can be aimed at obtaining some knowledge and skills of the students, so if in the end we evaluate a methodology under its parameters, it is likely that we will have good results, while if we compare between different methodologies but we evaluate only under the paradigm for one of them, we are likely to introduce an evaluative bias at the end that makes one of the methodologies lose out in our analysis [30]. The latter could happen in some cases both “in favor” and “against” gamification [13] or project-based learning (PBL) [34].

That is, if, for example, direct instruction is compared with learning through discovery [35], with mobile learning [10] or PBL [34], but in the end it is evaluated with an exam or test, learning by direct instruction is sure to give better results. However, what if we asked students who only received theoretical classes to build a prototype and evaluate the construction of the prototype? The same can happen with games and gamification [36] if it is compared with direct instruction and is evaluated only under the parameters of the latter.

In the same way, attending to the diversity of the classroom leads us to play in a different way, without giving up exams or projects. How to discern what effect each item has on the evaluation, separately? Well-designed field studies are needed in this regard [13,37], to which teachers are invited, but while educational science advances and provides evidence, with this holistic approach to gamification, we will be ensuring a varied approach to the subject, or at least an approach that goes beyond the traditional tests and exams that only contemplate one type of student.

1.4. Evaluating the Flipped Classroom with “Kahoot!” and “Socrative”

In the context of mobile learning, mobile devices, especially cell phones and tablets, have become authentic personal response devices (PRD) or “Student response systems” that allow students to answer questions and receive feedback immediately, thanks to online platforms such as “Kahoot!” or “Socrative” improving interaction with games in the classroom. Actually, the use of these experiences involves and motivates students, and even allows the continuous evaluation in a gamified manner. Thus, the implementation of games in the classroom implies involving the students and teaching them that they can learn through these technologies that have traditionally been forbidden as they are considered leisure elements [16].

There is a consensus on the changes in cognitive habits brought about by the irruption of the internet, as Marshall McLuhan already warned [38], and on the benefits and costs of the use of technologies in education [39]. For some time now, the teachers have confirmed that it is very difficult for students to face long readings, especially if they are technical. In this sense, a simple way to encourage reading is by undertaking flipped classes, in which students must first read the texts of the subject, accompanied by an explanatory video summary (in which there is not all the information from the text, it is a summary) and, after solving the doubts raised in class, a small test is passed with “Kahoot!” or “Socrative”. These two “Student response” platforms have been chosen, in their free versions, to the detriment of others that may offer interesting dynamics and tests, but which are quite restricted if not paid for. Both distribute points easily and create a spreadsheet in an exportable and easily actionable Excel format to configure the general classification of the week, for example.

In general, given the limitation of participants in the free version of “Socrative”, and due to its motivational characteristics, it is recommended to use this platform for team games (sharing a device), while “Kahoot!” is used for individual tests, although it can also be played in teams. Socrative is a somewhat more relaxed platform, as there is no background music (“Kahoot!” can also be silenced, if necessary) and it does not generate so much pressure due to the time factor—it is recommended in lively groups for which “Kahoot!” can be an excessive mess. The free version of “Socrative” allows students to answer short answer questions, which may also be important, depending on the assessment mechanism we want to implement. Working as a team generates a collaborative environment, which is less competitive than when playing with “Kahoot!” individually (see

Table 2. General comparison between the main features of the free versions of “Kahoot!” and “Socrative”, in May 2020 [30]. Both platforms have become closer, especially in their payment options.

Features	“Kahoot!”	“Socrative”
Point assignment	It depends on the time and correct answer	It depends on correct answer, not time
Time control	It is marked for each question, with options between 20 s and 4 min	The teacher controls the time and marks the end of the test
Teacher/ Classroom screen need	Yes, the questions appear on the screen, the mobile device acts as a ‘clicker’	No, the questions appear on the student’s mobile device. The screen can act as a competitive element (example: space race)
Maximum number of players	For teachers, up to 1000	50 (20 in space race mode)
Kind of questions	True/False (T/F), multiple answer up to 4 options	Multiple answer (more than 4 can be added), T/F and short answer (allows writing)
Question feedback	No	Yes, it is displayed after the answer
Question randomization	Yes, and you can also randomize the order of the answers	Yes, you can select, and also randomize the answers
Music	Yes, melodies can be chosen	No
Accessories	Open image banks, possibility to insert videos	Images and accessories are added manually
Results files	Excel with scores and response times, general and per question	Excel with general scores of answers and of each question; pdf of each participant, pdf of correct results

, [30]).

1.5. Challenge-Based Learning

Challenge-based Learning can be described as extended problem-based learning, but it also contains some components from experiential and project-based learning perspectives [40]. Project-based and problem-based activities are usually focused on a driving question or problem. In challenge-based learning, the question or the problem is replaced by a challenge. The challenges to be solved might include ways to develop, design and implement solutions for problems related to scientific phenomena. A meaningful learning activity consistent with challenge-based learning is to present learners with a challenge scenario and to ask them to think about a number of possible solutions using a variety of interactive tools. Such an activity serves to center thinking around meaningful problems and is typically effective in facilitating small-group collaboration [41].

Gamifying the subject and its evaluation must go beyond “Kahoot!” and “Socrative” or other student response devices [2,14,17,28,30]. Students can become accustomed and demotivated by feeling that both platforms have become the weekly “exam”, especially if they are not accompanied by the gamification of a good narrative [25]. In this sense, the challenges are an ideal complement to break this activity–test dynamic that is generated if we only use “Student response” platforms. The score we give to the challenges can be all/nothing or progressive. In general, in an all/nothing challenge, only the first to solve it gets the score, while in progressives, we can give points as the students advance in solving the challenge or achieve minimums [30].

A key factor is that you opt for points above the maximum: that is, if 10 is obtained with 100,000 points, you should opt for a higher amount, considering that it is impossible, for example, to respond instantly to the tests of “Kahoot!”, or that if there are all/nothing challenges, there will be students who will not get those points [30].

1.6. Evaluation of Gamified Activities

One of the key competences of students is communication and, in particular, oral communication is essential. Oral presentations can go beyond the student facing the audience of his/her class and, especially, the teacher (whom the student usually looks at almost exclusively, since the student knows that the teacher is the one who evaluates). Oral presentations can be gamified by organizing courts in which students will be the ones who, in a guided and reasoned way (for example by providing rubrics or having them create them), evaluate their classmates (co-evaluation), who can also evaluate themselves (self-assessment). Regarding this type of peer-assessment, it is advisable to review the extensive literature on the matter ([40] and the references included there), and a recent meta-analysis [41]. See

Table 3. Example of possible basic scores assigned to different components of oral communication with complementary material (slides, videos...), out of 10,000 points. The rubric should specify how the scores for each evaluable item are assigned [30].

Element to Evaluate	Adequacy of Supplementary Material (Images, Videos, Slides, ...)	Verbal Communication		Non-verbal Communication		Time Control
		Diction	Content	Movements	Voice Tone
Possible score	2000	2000	2000	1000	1000	2000

.

Exams are still perceived today as the opposite of gamification. They allow students to assess skills and knowledge through written expression, assess their creativity, reasoning and argumentation ability, and many other components, directly through questions that can go beyond the multiple-choice test or rote writing. In this sense, the exams must continue to be part of the assessment, or a diversified assessment, and can of course be part of a gamified course [30].

Another element of gamification is to include challenges in the exams that award points in the test and that could “free” some students from written theoretical questions. It can also be decided that the challenge is mandatory, depending on the characteristics of the course. Another possibility, to conclude, is that the final exams are validated by the continuous evaluation that gamification implies. The final exam is then a recovery test for those who have not passed the game [30].

1.7. Research Questions and Objectives

In this investigation, we ask ourselves the following questions about the real possibilities of gamification as another learning methodology:

-

Can the game and the academic results be opposed?

-

What would happen if motivation is shown to improve in a gamified classroom, but the results are worse?

-

Should we sacrifice play and fun for academic achievement?

-

Can a gamification experience in classrooms be a sufficient complement to motivate students?

This research aims to provide new evidence and to debate the real possibilities of implementing a gamification experience [5,36] as one more methodology of learning [30]. To do this, several gamified and alternative activities were proposed to traditional master teaching within the subject complements for disciplinary training in technology (TEC) of the master’s degree of secondary school teachers (technology specialty) taught at the Universitat Politècnica de Catalunya. This paper analyzes the last three academic years (2016/17 to 2018/19) of this master’s degree in which a gamification experience was applied, which entails equipping students with the tools to improve their own training, following a novel theoretical approach to partial gamification [30].

2. Material and Methods

In this study, we have applied the longitudinal case-study methodology [32] throughout various academic courses, in order to verify the influence of these gamification techniques on academic results.

The TEC subject in which the experience is carried out has 7 European Credit Transfer and Accumulation System (ECTS), for a total of 175 h divided into 42 classroom hours. The rest of the autonomous and blended learning activities (133 h) were carried out through a virtual platform (“Moodle”).

From the 2016/17 academic year, some gamified activities were introduced in a progressive way in the classroom. In the 2016/17 academic year, two teachers taught the TEC subject to 52 students and some tests were carried out with the “Kahoot!” tool as a gamification experience. In the 2017/18 academic year, the subject was taught by a single teacher to 42 students and the experience of the gamification was carried out in a flipped classroom activity and affected 10% of the final mark.

In the 2018/19 academic year, a full gamified experience was introduced. There were 60 enrolled students taking the subject. Three people were discarded in the 2018/19 academic year who finally retired for various reasons. The students were divided into two groups whose acronym names came from the subject name (TEC1 and TEC2) of 28 and 29 students. The same teacher gave the two groups the face-to-face part of the subject on Thursday afternoons from 3:00 p.m. to 6:00 p.m. (TEC1) and from 6:00 p.m. to 9:00 p.m. (TEC2). In the TEC1 group, there were 12 women and 16 men and in the TEC2 group, there were 8 women and 21 men.

2.1. Study Variables

The study variables consist of two types: on the one hand, there are the elements of the assessment to be considered (following the teaching guide of the subject) and, on the other hand, there are the statistical variables collected for each student, on which it was investigated whether there were correlations between them, as it will be seen in Section 3.1, combining quantitative and qualitative variables, as usual in research on education [32].

In this way, firstly we will analyze the results of the qualifications of the two groups (TEC1 and TEC2) of the 2018/19 academic year. The assessment was undertaken according to the following evidence: an individual piece of work on the history of technology in which an infographic is designed and the pedagogical material for secondary school students (20%); the development of a technology project (in a group) in the technology classroom consisting of a prototype, a memory and an oral presentation (25%); active attendance to class (10%) and delivery in the virtual platform of various activities (continuous evaluation, 25%). Finally, a final exam (20%) that consists of two parts: a text commentary where the students must perform some compulsory readings related to the syllabus of the subject and, finally, a set of questions related to the course syllabus.

The final mark is as follows:

FM = 0.2 · IW + 0.25 · PROJ + 0.1 · AT + 0.25 · CE + 0.2 · FE

(1)

where:

• FM = final mark;
• IW = individual work;
• PROJ = technology project (in groups);
• AT = attendance and active participation in class;
• CE = class activities (continuous evaluation);
• FE = final exam.

Secondly, to analyze relevant aspects of the experience (different components of the evaluation and the points obtained in the activities of the gamification), some statistical variables have been considered, following previous theoretical approaches [30], and combining quantitative and qualitative variables [32], where:

• Points: points obtained in the gamification;
• TEC group: group binary variable (TEC1 or TEC2);
• Gender: male or female;
• Exam: binary variable to take the final exam;
• CE: mark of class activities (continuous evaluation);
• IW: individual work mark;
• PROJ: mark on the technology project (in groups);
• FE: final exam mark;
• AT: mark of attendance and active participation in classes;
• FM: weighted final mark of the subject.

With the exception of the binary variables belonging to the group (TEC group), the gender (M or F) and the realization or not of the examination (exam), in the rest of the cases the variables are numerical quantitative variables from 0 to 10 points, except in the case of the points of the gamification (points, 0 to 123,000). Binary variables, on the other hand, can be easily quantified to establish cross-correlations.

Contingency tables and Pearson cross-correlation linear tests were developed to analyze cross-references of variables and significant correlations. The chi-squared test was applied to each question to determine the corresponding p values. High p values indicate that no differences exist between the variables. On the other hand, p < 0.05 values indicate that statistically significant differences between both variables do exist for that question. The statistical program used was IBM SPSS v19 Solutions for Education^®.

Finally, we will compare the marks obtained by all the groups (TEC1 and TEC2) of the three academic years (2016/17 to 2018/19) to analyze the variations of the academic results with the introduction of the gamified experiences.

2.2. Games and Scores

The full gamified experience was developed throughout the 2018/19 academic year. In it, individual and group games were interspersed throughout twelve of the fourteen sessions of the subject (the initial presentation session and the final evaluation were excluded). The games allowed the students to stop doing any of the parts, or both parts, of the final exam based on the points obtained.

In all the gamified activities, points that could be exchanged at the end of the course for the final exam grade were obtained, so that upon reaching 50,000 points (first level), a five was obtained in the final exam grade (out of 10) and it was possible to choose to not solve part of the exam question (although it was obligatory to make the text comment). From 50,000 points (and up to 100,000), the students were not required to make the text comment. The maximum final scores were 123,000 points, impossible to reach, as, for example, achieving the maximum score in the “Student Response” questionnaires (“Kahoot!” and “Socrative”) meant answering correctly and immediately, which was impossible. Therefore, to allow the student the option to obtain a 10 (the highest mark), the maximum score could be 100,000 points.

The planned games consisted of three different gamified experiences: flipped classroom, challenges and gamified courts (see

Table 4. Summary of gamified experiences of the 2018/19 academic year.

Type of Gamified Experience	Games	Number of Lectures	Grouping	Maximum Score
Flipped Classroom	“Kahoot!”/“Socrative” readings	4	Individual/Group	40,000
Challenged Based Learning	Class Challenges	2	Individual/Group	30,000
	Optional activities	3	Individual	30,000
Gamified Courts	Oral defense	3	Individual	20,000
	Court participation		Individual/Group	3000

). Table 4 shows a summary of the different types of play experience, the games in each experience, the number of lectures in the academic year, the grouping and the maximum score that could be obtained. Figure 1 complements the information shown in Table 4. The main idea is to allow a diversified assessment that considers the different skills of our students [30].

In the flipped classroom activities, four questionnaires of ten questions each were carried out on the recommended course readings: two through the mobile learning platform “Kahoot!” and two with “Socrative”, individually and in groups, so that the students had to develop collaboration strategies. The texts of the readings were provided at the beginning of the course and they were previously notified of the class in which it would be done, following the methodology of the flipped classroom [42]. Whether the activities were individual or in group was a surprise factor. In the initial presentation session, a “Kahoot!” was performed to obtain data from the sample of the students (studies of origin, age, gender...), and it also served as training and example.

In the challenged-based learning, it was possible to obtain 15,000 points per challenge (up to 10 different challenges/games were considered). Some of the challenges were: playing cards related to the teaching of technology [43], inventing activities on simulators, creating an infographic on a topic, discovering the operation of electronic circuits with Lego, determining which was the incognito material of some samples presented or building paper mechanisms in a limited time, among other challenges—always following the usual dynamics of teaching based on the challenges applied to education [44]. Through the activities of mobile learning, technology, and art, 5000 points were awarded and up to 20,000 more could be obtained for developing activity cards or playing video games, which could potentially represent up to 30,000 more points in total.

The gamified courts consisted of a role play with three possible characters: presenter (student presenting the work), public (observers) and evaluating court. Thus, up to 20,000 points were awarded according to the grade obtained in the development of the oral presentation of the work (presenter role) that the court evaluated. The courts received 1000 more points for each day in which they evaluated and the public could also receive 1000 points per day according to their interventions, with a maximum total of 23,000 points (3000 as court and public and 20,000 for the presentation). The formative intention of this activity, without taking into account its ludic and motivational component, is for the students to face one of the main tasks that they must undertake in the future: the evaluation.

3. Results

3.1. Groups of the 2018/19 Academic Year

From the analysis of the marks obtained from the two class groups of the 2018/19 academic year, the TEC2 group was the most involved in the gamification activities and, consequently, had less participation in the final exam; the opposite of the TEC1 group, which completed the exam mainly because the ratings of the gamification were somewhat lower. The TEC1 group showed greater inclination to try to improve their marks (out of 28 students, only two students resigned to take the final exam, keeping note of the gamification and showing the teacher external burdens that had prevented them from studying), while the TEC2 group, who were much more involved in the games, mostly declined to take the exam (the test was only undertaken by six students, four men and two women, one of whom had not been able to follow the course, nor play regularly). Failure to take the exam, mostly observed in the TEC2 group, influenced the final grades of the subject, where this group scored significantly lower than TEC1.

Figure 2 shows the comparative ratings of the two groups of the 2018/19 academic year (in two different graphs, with bars with standard deviation and radial with values relative to a central point). The figures show the means of the marks of the gamification, the continuous evaluation (CE, marks of class activities), the mark of the individual work (IW), the mark of the project of the group (PROJ), the final exam (FE), the attendance (AT) and the weighted final mark of the subject (FM).

The diverse involvement in the game, according to the group, had an impact on the results of the gamification that were better in the TEC2 group (average of 6.0 ± 0.6 points) than in the TEC1 group (5.4 ± 0.6 points), this being the only exception in the course ratings, as results were generally better in the TEC1 group. The final scores in the gamification were somewhat inferior to the other evaluation activities, which could be attributed either to the design of the gamification or to the lack of custom of the students in participating in gamified activities.

The Pearson correlation study between variables showed a weak positive correlation between the attendance score (AT) and the points obtained in the gamification (points) (p = 0.3) so that the score in the gamification decreased if the students did not attend. There were also positive correlations between the weighted final mark (FM) and the activity score (CE) (p = 0.69), the individual work (IW) (p = 0.68), the group project (PROJ) (p = 0.57), the attendance (AT) (p = 0.42) and the final exam mark (FE) (p = 0.67), which was expected given the calculation formula of the final grade that weighted these elements.

Taking the exam correlates with belonging to the group TEC1 (p = −0.72), which has slightly better marks compared to the TEC2 group in the activities of the continuous assessment (CE) (p = −0.30), in the group project (PROJ) (p = −0.30), in the final exam (FE) (considering that they could choose to replace part of the note with the gamification) (p = −0.64) and, therefore, in the final weighted mark of the subject (FM) (p = −0.50). The TEC2 group was only slightly better in the points obtained in the gamification (points) (p = 0.18), although in the individual work both groups behaved in an almost identical way, which resembles the personal academic homogeneity that they had in the beginning. The students who obtained the best marks in the individual work (IW) also did better in the activities of continuous evaluation (CE) (p = 0.36), in the exam (p = 0.24), in the project in the group (PROJ) (p = 0.18) and they attended class a little more (AT) (p = 0.19).

The analysis showed no significant correlations between the score of the gamification and the final grade (p = 0.02). There is a weak correlation between the examination and the gamification (p = 0.27), which means that the students with the lowest scores tended to undertake the exam slightly more.

Table 5. Cross-correlated Pearson correlations between the various study variables of the 2018/19 academic year (with a p < 0.001).

Variables	Points	TEC Group	Gender	Exam	CE	IW	PROJ	FE	AT	FM
Points	1.00
TEC Group	0.18	1.00
Gender	0.17	0.09	1.00
Exam	−0.27	−0.72	0.17	1.00
CE	−0.03	−0.30	−0.08	0.04	1.00
IW	0.13	0.00	−0.18	−0.12	0.36	1.00
PROJ	−0.11	−0.39	0.13	0.48	0.11	0.18	1.00
FE	−0.06	−0.64	0.00	0.68	0.26	0.24	0.38	1.00
AT	0.30	−0.19	−0.05	−0.20	0.43	0.31	0.04	−0.04	1.00
FM	0.02	−0.50	−0.06	0.35	0.69	0.68	0.57	0.67	0.42	1.00

shows the Pearson cross-linear correlations between the various study variables defined in Section 2.1. The negative sign of the correlations is due to the binarization of the variables in the case of belonging to the group (TEC1 or TEC2), gender (M or F) and the realization or not of the examination, so that a negative correlation in “TEC group” indicates that there is a correlation with the TEC1 group, while in the case of “Gender”, it indicates correlation with respect to the masculine gender, and in the case of “Exam” towards the students who did not take the exam.

As an interesting result, no significant gender-related differences in involvement or gamification results have been found, that is, men and women statistically play in a similar way. Therefore, there were no notable differences with respect to gender, which did not practically influence any of the partial notes (IW, CE, PROJ) nor the gamification (points), nor the final mark (FM).

To analyze the degree of satisfaction of the gamified experience and to evaluate positive and negative aspects of the experience, at the end of the 2018/19 academic year, an anonymous survey was conducted in the two groups (TEC1 and TEC2) with the mobile tool “Kahoot!” (

Table 6. Results of the internal surveys of the subject of the 2018/19 academic year.

Questions	Answers	Quantitative Responses of the Two Groups
		TEC1	TEC2
Q1: My age is	Under 25 years old	2	5
	Between 25 and 35 years old	8	7
	Between 35 and 45 years old	10	7
	More than 45 years old	5	3
Q2: I consider myself	Male	14	14
	Female	9	7
	Other identities	1	1
Q3: The gamification in this course has been satisfactory	Strongly agree	2	6
	Agree	17	12
	Disagree	4	4
	Strongly disagree	0	0
Q4: In my group I have worked correctly	Strongly agree	11	12
	Agree	4	7
	Disagree	4	2
	Strongly disagree	3	1
Q5: Positive aspects of the subject I would highlight	Class activities	12	13
	The teacher	3	7
	Other	9	1
	I would not highlight anything positive	1	0
Q6: Negative aspects, or to improve, of the subject I would highlight	The teacher	1	1
	Class activities	12	6
	Other	5	4
	I would not highlight anything negative	5	11
Q7: I have participated in optional activities	I have participated in all of them	0	19
	I have participated in some of them	11	1
	I have not participated	14	2
Q8: About the final mark of the subject I think that	It is remarkable or superior	13	7
	I have passed	6	9
	It is low with respect to work	1	4
	I hesitate to pass	0	1
Students	(They took the test/total group)	25/28	22/29

). In the survey, questions were asked about aspects related to the profile of the participant, the evaluation of the gamification experience and the positive and negative aspects of the subject, among other questions. In all the questions, the students chose from different possible answers (for example: strongly agree, agree, disagree, strongly disagree). The people included in the survey gave their consent to participate.

This satisfaction survey was answered by 25 students from TEC1 and 22 from TEC2. Furthermore, 37 of 47 students (78.7% of those who responded to the survey) rated the gamification positively or very positively (question Q3). From them, 7.1% of the TEC1 group and 20.7% of the TEC2 group rated the experience as excellent and would have preferred a wider gamification. Of this last percentage, all those who responded were men.

A greater involvement in the game of the TEC2 group was verified as a result of one of the role playing games of the course; a good part of the TEC2 group decided to participate in optional activities (question Q7), whereas the other group mainly decided not to involve themselves. These types of actions are an indirect way of measuring the degree of students’ engagement.

In the TEC1 group, four students out of 25 (two men and two women, 16%) acknowledged not having been involved in the game, as they generally recognized that they thought more about the conversion of score-points than fun and play. The same thing happened to 13.8% of the TEC2 group (three men and one woman, four out of 22, 18.18%). Table 6 summarizes the different questions and their possible answers together with the quantitative results discriminated from the two class groups.

3.2. Final Marks of the Three Academic Years (2016/17, 2017/18 and 2018/19)

Figure 3 compares the final marks of the three academic years in which the gamified activities were progressively carried out.

It can be seen that the final marks for both groups (TEC1 and TEC2) in the 2018/19 academic year, in which the gamification experience was higher, decreased compared to the previous two academic years. It is observed that the difference between the final marks of both groups (TEC1 and TEC2) in the 2018/19 academic year is lower than the previous years analyzed.

3.3. Valuations of Teaching Performance and Subject in Official Surveys of the Three Academic Years (2016/17, 2017/18 and 2018/19)

In addition, at the end of each academic year, the university itself conducts an anonymous survey in which the teacher and the subject are evaluated externally. In these surveys, students rate using a Likert scale of one to five, (where one is completely disagree and five completely agree). We analyzed the results of the 2016/17, 2017/18 and 2018/19 academic years. The people included in both surveys gave their consent to participate. The participation of the students in the surveys for the 2016/17 course was 52.4% (22 of 42 students), 63.5% in the 2017/18 course (33 of 52) and 70.3% in the 2018/19 course (40 of 57). The higher the degree of gamification of the subject, the greater response to official surveys.

The related questions on the teaching performance regard: “The teacher is accessible for the consultation on the subject” (Q1), “I think the teacher is a good teacher” (Q2) and “The teacher helps me understand the subject” (Q3). See Figure 4.

The questions related to the subject were the following: “The contents of the subject I found interesting” (Q1), “The evaluation corresponds to the contents and the level of the subject” (Q2), “Overall I am satisfied with the subject” (Q3) and” The coordination between the different activities of the subject is satisfactory” (Q4). See Figure 5.

The results were very positive both in the assessment of the teacher (Figure 4) and in the assessment of the subject (Figure 5). As can be seen in Figure 4, with regard to the teacher, in these three years the good records of the previous two years to gamification were maintained, in which the gaming had been less, but with an improvement in the students’ feeling about the teacher’s accessibility in the 2018/19 course. As can be seen in Figure 5, the subject is better evaluated in the surveys by increasing the gamification in the 2018/19 course.

4. Discussion

In general, in the context of this master’s degree, the high involvement of students is such that only students who could not keep up with the pace of work have failed. Specifically, it was the case of one student in the 2016/17 academic year, one student in the 2017/18 academic year and three students in the 2018/19 academic year. Everyone passed the subject in the last course 2018/19. The increase in students who did not follow the course can be attributed more to conjunctural effects than to an hypothetical increase in their difficulty.

It is perceived that the increase in the gamification in the subject improves the valuations of the student body both in the subject and in the teaching performance (Figure 4 and Figure 5). The context of this master’s degree is ideal for proposing experiences of methodological diversification in which different ingredients are exposed so that each student can focus on the profession. Internet and other digital media are in this case good allies of the gamification, for example in the class invested with online systems of type “Student Response” (“Kahoot!” and “Socrative”) [30].

In the TEC2 group, a more playful environment was generated in the last courses (the teacher and the contents were the same) than in the TEC1 group, although the latter group obtained better academic results. This fact questions whether the gamification contrasts with academic performance and other indicators (Table 6). The approach of the experience of gamification by the same teacher, in two similar groups, has given similar quantitative results, although there are obvious differences between types of players [45,46].

The differences point to the fact that the involvement and the results in the gamification was somewhat better in the TEC2, which obtained worse academic results at the end of the course by not choosing to take the exam (or part of it) compared to their TEC1 colleagues, which in general would have meant better grades. Did you forget to get more involved in the game of the mark knowing that the approved was not in danger? Did the most aggressive types of players [47] in the TEC2 group coincide? The fact that official surveys are anonymous avoid being able to make considerations on how the two groups (TEC1 and TEC2) have responded to them.

When the teacher proposes gamified activities throughout the course, students choose whether they accept the marks of the gamifications and want to substitute the exam, totally or in part, for the points they obtain, or perform the final exam in a traditional way (it represents 20% of the mark). The text commentary, the second part of the final exam, becomes part of the evaluation that the students decide whether to do or not, and the teacher of the subject allows him/her to verify in person (unlike work done at home) the students’ level of written expression and ability to analyze and synthesize. After completing these activities, it is surprising that the group less involved in the gamification has obtained better academic results, which opens the debate to whether the positive effects of the gamification in the climate of the classroom and in motivation are opposed to the academic results. It is surprising that the group less involved in the gamification has obtained better academic results, which opens the debate to whether the positive effects of the gamification in the climate of the classroom and in the motivation are opposed to the academic results.

Regarding the evolution of the final marks in the previous academic years (with less gamification) compared to 2018/19, it is observed that the final marks are higher in the previous academic years (Figure 3). This is contrary to the results of the evaluation of the students of the teaching staff and of the subject, which improve as more gamification is introduced (Figure 4 and Figure 5).

It can be seen (Figure 3) that for each of the academic years the final marks of the TEC1 group are higher than those of the TEC2 group. On the other hand, the difference between the final marks of the students in the TEC1 and TEC2 groups is less in the 2018/19 academic year than in the previous academic years with less gamification (Figure 3).

A clear case of negative evidence is observed in the results of gamification activities and those obtained in academic performance. The evidence indicates that gamification improved the classroom climate but did not improve the grades, although the results are not so conclusive as to affirm that they worsen. Students with lower historical scores improved with gamification, while students with good historical scores either stayed the same or worsened. This statement is logical, since they have been studying in the same way for years (a way that has given them good results), while for those who do not study too well in the traditional way, any change can be a stimulus to improve.

The discussion about improving students’ cognitive abilities when gamified activities are implemented still continues, as evidenced at the beginning of this research. The lack of rigorous studies makes it difficult to establish whether gamification really improves student learning, beyond increasing fun and motivation in the classroom. A non-negligible goal is achieved, considering that demotivation in classrooms is a serious problem [23].

Assessment (an important aspect of the teaching and learning process) is an essential part of the teaching function. A good evaluation method is one that involves students, helps consolidate and assimilate content, and considers different evaluation models to adapt to the diversity of the class group. In this research study, we consider assessment as a combination of traditional educational practices and gamified methods.

According to Lister [21], gamification neither improves nor worsens learning, but it does improve the classroom climate and student motivation, which does seem to be verified in some relevant empirical studies [22], although more evidence is needed on experiences and results with a large sample of the positive effects on learning by applying the gamification methodology [1,11].

Study Biases and Limitations

Although the statistical analysis is rigorous, the effect size (ES) values that detail the importance of the þ-values obtained using the Pearson correlation test have not been taken into account and these could be significant data to validate the results and the conclusions of the study. Although the groups are homogeneous, the number of examples is low in both groups (both TEC1 and TEC2).

On the other hand, the studies collected in the review on gamification of Dicheva and Dichev [12] are good teaching practices that, although the teachers who applied them found they worked for them in their classrooms, do not yet suppose quality empirical evidence such as to consider gamification as a better methodology than others.

Certain differences between the two groups are attributed to several biases that should be analyzed in future work and that again pose open hypotheses, including:

-

Tiredness bias: the students of the TEC2 group finish later (9:00 p.m.) compared to the TEC1 (which ends at 8:00 p.m.). TEC2 students prefer to start classes later due to external burdens (work, family...). The TEC1 group receives the less tired teacher, first thing in the afternoon, while the TEC2 group receives the teacher after three hours of class, so fatigue could slightly affect the performance of the TEC2 group, as documented [48,49].

-

Schedule bias: related to the previous bias; teaching the class later implies being further away from the time of lunch and closer to dinner, which affects productivity in the educational context [50]. Thus, perhaps a more relaxed atmosphere was generated in the classroom that favors the gamification but worsens the academic results of the TEC2 group.

-

Gender bias: in the TEC2 group, there is a greater proportion of men than women (different from TEC1). Our results discard significant differences in the gamification due to gender, but it should be studied in depth how gender influences the involvement of students in gamification [51].

5. Conclusions and Future Works

An amalgam of possibilities for a diversified gamification that goes beyond simple testing using “Kahoot!” or “Socrative” has been presented in this paper. It has not been intended to give a single recipe but quite the opposite: discover that the possibilities of gamification depend on your creativity and that its evaluation is not a problem [25,27,30,44]. Throughout this work, in addition, many other alternatives have been presented that could be part of a gamified course, to account for the diversity of the classroom, beyond the types of players. In any case, gamification must be implemented without neglecting their assessment which, in any case, will involve making a weighted conversion of their scoring systems to the official marks that, by legal imperative, must be included in the student records.

Finally, while pedagogy research discusses the validity (or not) of the empirical evidence of gamification [12,13,37], a varied gamification must account for the crucial aspects of evaluation according to the context, taking into account [30]:

-

The diversity of students;

-

The evaluation of the entire program and the competencies involved;

-

The correct alignment of the learning results obtained and the tasks set;

-

The specification of the students’ workload (gamified or not).

This experience is framed in the context of a gamification experience of a subject of the master’s degree of secondary school teachers (technology specialty) and supposes more evidence in favor of the increase in the motivation and satisfaction of students through gamification [21,37], integrating digital learning media with traditional elements of the game.

With the implementation of the gamification experience, no improvement was found in the academic marks, although the results are not so conclusive as to affirm that they worsen.

For the next academic years, it is proposed to introduce in the final mark an option that does not discriminate against students who are harmed by gamification, introducing a maximum between the mark obtained with gamification activities and the mark of traditional teaching.

No gender bias has been found in the game [51]. At a qualitative level, the students very positively receive all the methodologies and practical strategies that will be potentially applicable in the classroom.

We must, nevertheless, investigate more deeply the objective effectiveness of gamification, and its digital pedagogical possibilities that are effective and that are useful for the integral formation of the students of the 21st century. On the other hand, in the current context of the educational system, it is essential for future teachers to integrate another key element in their training: creativity [52,53]. In this sense, teaching is as important for creativity as creativity is for teaching [53]. Games are not opposed to content, but rather, from Horacio [9], they are a creative way of presenting content that, of course, does not exclude other methods and that allows a diversified assessment in the classroom [30].

Thus, the following open questions are proposed for future work:

-

Is the environment of the gamification generated, for example, by a few games in the classroom, enough to consider a course completely gamified?

-

Is a wide gamification necessary to notice an improvement in student motivation?

-

Does the degree of gamification of a course influence the results and conclusions of these studies?

-

How to assess creativity, within the current educational system, in a context of partial or total gamification?

Answering these questions implies the deepening and exploration of other possible experiences within the framework of the master’s degree of secondary school teachers (technology specialty), such as gamification through digital media or approaching conventional videogames and mobile learning strategies. It will, of course, be the experience that will allows them over time to improve the validity and reliability of the gamified assessment, and to select the most appropriate assessment methods for their subject through the flexible design of assessment tasks.