Analysis of Three Methodological Approaches in the Use of Gamification in Vocational Training

Abstract

A reduced interest and low motivation in learning amongst vocational students has become a challenge with many traditional strategies not capable of providing a solution to motivation and encouraging participation in learning. The use of elements of games in non-recreational environments (gamification) may be a possible solution, since research indicates an improvement in user experience and engagement, with possibilities of improved motivation and behavioral results. However not all studies obtain positive results, the success of gamification is influenced by the design, the sample, and the context. This study analyzes a gamification design with the most common elements in three methodological approaches (teacher-centered, student-centered, and mixed) in three different periods throughout a program of study with vocational training students. The results indicate that the mixed approach performed worse than the other two. Carrying out a greater number of tasks did not imply a better result in the subject marks, but rather paying more attention to each task influences the result positively.

1. Introduction

A major problem schools and educators often encounter today is that many students lack the motivation and interest to learn. Furthermore, if given the choice, many would rather play video games than read a book or complete a task [1]. Taking this scenario into account, different pedagogical innovations have emerged that incorporate the logic of games, such as gamification and game-based learning.

Aligned with this, the lack of motivation and commitment is a particular problem for students taking courses at universities or schools [2]. According to the findings of many studies [3,4], traditional strategies cannot provide a solution to the lack of motivation of students. Nor can they generate participation in learning [5,6].

Hattie conducted meta-analytical reviews that take into account about 1000 meta-analyzes of factors affecting student achievement and include approximately 60,000 studies, some of the studies are based on up to five million students and the overall meta-analysis covers a total of 245 million students [7,8]. These studies reveal that the individual characteristics of the student, and particularly their intellectual capacity and motivation, are among the variables that contribute most to academic performance. The loss of student motivation is one of the most pressing problems that threatens the sustainability of educational systems today [9].

Gamification

Gamification is defined as the use of game design elements and techniques in contexts outside the game. Gamification includes a number of game elements such as points, badges, levels, leaderboards, status, trophies, rewards, and progress bars [10,11]. These elements are incorporated into tasks to engage, motivate, and reward users to learn new skills or change behaviors [1,12]. The general objective is to align intrinsic and extrinsic motivation activating the commitment and motivation of students to actively participate [13]. Numerous studies highlighted positive learning outcomes as a result of gamification [14], although research presents mixed results or even negative effects [15].

In recent years, gamification has gained great popularity thanks to its ability to influence participant behaviors in applying its methods in the most diverse contexts [16,17]. Interest grew exponentially as gamification was introduced in various aspects of life, such as health management, work, education, and training habits among various areas [18,19,20,21,22,23,24]. Little by little, consumers have come to expect that most of the systems they use are gamified in one way or another [20,25].

The use of gamification in educational settings and contexts is constantly increasing and encourages a greater frequency of research in this field of knowledge [26]. Gamification as a teaching medium makes learning processes more enjoyable, while ensuring that students are receptive to the information received [27,28]. In fact, there is a high level of interest among education experts to make learning more interesting for students [29]. Although the results of the implementation have not always been positive [30,31].

Gamification has a very important role in the field of education, it enables learning process to be a more motivating and enjoyable experience. In this way, greater receptivity to the information provided to students is achieved [27,28]. Greater motivation improves learning outcomes and encourages students to continue delving into a specific topic [32,33]. If the material provided does not stimulate students, then learning may not be effective [34]. Motivation stimulates the desire to learn and facilitates learning activities [35]. Overall, the results of empirical studies show that gamification improves user experience and engagement, motivation, and behavioral outcomes [10,11,36,37,38]. A gamified system can bring benefits such as broader participation, long-term commitment, and academic success [39]. The main reasons for implementing gamification are due to its potential to motivate and enhance positive behavior [18].

2. Previous Studies

Gamification emerges as a relevant approach to motivate content development and student participation in the classroom [36,40]. Gamification studies have reported improvements not only in student motivation and engagement, but also in their learning achievement. The principle of “challenge” in a gamified system makes a significant contribution to positive learning achievements [41]. According to Ardilla-Muñoz [42], gamification in education brings benefits such as: greater control and monitoring of the actions carried out by students; evaluative activities lose their punitive character; the teaching-learning relationship is characterized by competitiveness and cooperation; and promoting problem-based and discovery learning.

Some studies have indicated that motivation is an important predictor of student academic performance and influences the effort and time a student dedicates to learning [41,43,44,45,46,47,48,49]. Other studies have reported mixed results, some positive effects with different effect sizes [50,51], other adverse effects on student test scores [52], and others reported no effects at all [53]. Several studies have shown that the addition of game mechanics (such as badges, levels, and leader boards) has positive effects on student engagement. However, critics have argued that this increased participation is due to extrinsic motivation, not intrinsic motivation; where students complete an assignment simply to earn a badge and not for the satisfaction of gaining new knowledge and skills [30,31].

Hamari et al. [18] found that most studies implementing gamification in learning contexts resulted in positive changes in engagement. The gamification process can have different effects on different participants including schoolchildren, university students, doctoral students, or others, indicating that its effects on different samples should be investigated [54]. The effects of gamification are linked to the target audience and context [11,15,20,38,55] and gamification results vary according to the topic and the field of application [38,41].

Dicheva et al. [36] conducted a mapping study of gamification in education that investigated current empirical research on its implementation. They found promising results but most empirical studies did not provide a rigorous evaluation, therefore making it difficult to understand the reasons behind the positive or negative results. Consequently, they suggest that more empirical studies are needed to investigate the motivating effects of using single-game elements in different educational contexts and for specific types of students. Researchers generally agree on the need for stronger empirical results [15,20,21,38,56,57]. A general review of the publications studying the use of gamification in higher education show that most studies published were activities of short duration, no more than 3 weeks [15,58,59]. Dichev and Dicheva [55] reached a similar conclusion in their review where they argue that there is not enough evidence to support the benefits of long-term gamification in educational contexts.

This study aims to complete this need and contribute to the field of knowledge regarding gamification. The project studied a sample of vocational training students with a specific gamification design for a full year analyzing a methodological approach each quarter (teacher-centered, student-centered, and mixed). The research question was based on which methodological approach of the three proposed, together with gamification, achieves better participation and better learning results amongst the participants.

3. Method

Throughout a full academic year, a gamified experience was developed with dual vocational students; the idea was to compare how different methodologies affected student learning in a gamified environment.

3.1. Sample

The students in the sample were between 18 and 22 years old, except for one student over the age of 30. All the students were taking their first year of dual vocational training in a cycle of the computer science branch, specifically in administration of computer systems in network or in multiplatforms and web development. These students came from different fields: from high school, from intermediate vocational training, from the university and from the labor market. This type of dual vocational training in Madrid consisted in 1 year at the school complemented with 9 months in a company. In this study the students expended their time entirely in the high school.

The course began with 60 students enrolled, of whom 44 attended the first week. After several dropouts throughout the course, the experience finally ended with 37 students. The main cause of the initial dropouts was the lack of understanding of what the learning program would be like, and typically dropouts that occurred throughout the course were due to offers of employment and not being able to continue teaching face-to-face. The reasons for the dropouts were personally alleged by the students to the tutor, in this case one of the researchers. Of the 37 students who finished the course, four were women and 33 were men. Although the number of women was small, all of them completed the course.

3.2. Design

The ‘learning program experience’ was made up of three time periods of 3 months duration. The first 3 month section was based on teaching content about hardware with a methodology focused on the teacher (where the teacher explained the content and the tasks consisted of reading documentation and answering theoretical questions). During the second 3 months basic knowledge of operating systems was taught, using a methodology that combined theoretical sessions and practical exercises (where the teacher explained the content and the tasks consisted of doing small practical exercises). During the final 3 month period a project-based learning (PBL) [60] methodology was used to deliver content on networking and administration of operating systems focusing mainly on servers (where the students had to carry out a project configuring a series of network elements and servers).

The students belonged to two different class groups, they had the same teacher and exactly the same tasks in the first and second 3 month section of the ‘learning program experience’. In the third and final 3 month period, due to the obligation to cover a defined agenda in each group, the two groups performed the same types of tasks, but about different contents. The projects for the third 3 month period were slightly different. A total of 17 students focused on the configuration of network elements for their project, while the other 20 students focused on the configuration of network servers. In both groups, the instructor was the same person and one of the researchers.

The gamification designed for this research study used the GameMo plug-in [61] that allows expanding the possibilities in Moodle. The following elements were used as explained below:

• Badges: They were given when a certain task was completed, a certain task list was completed, one of the sections of the subject was passed, and when the entire course was completed.
• Points: For each completed task, a certain number of points were awarded, the amount of points per task depended on the difficulty and the estimated time to complete it. As the course progressed, the points awarded for the tasks increased.
• Levels: A total of 20 levels were created, you started at level 1 and after achieving a certain number of points you went to the next level. The difference in points for leveling up was greater as you were leveling up.
• Leaderboard: It showed the points of all the students, the level, the profile image, the name, and the progress bar of the current level. You could consult the daily, weekly, monthly, or total rank.
• Blocked content: To unblock access to certain tasks, it was essential to have completed one or more previous tasks.
• Time limit: Specific dates were established to finish the course assignments, after the expiration date access to the task was closed.
• Feedback: When completing the tasks, the students received an automatic message informing about the event.
• Progress bar: Students could see the progress bar of their current level and the score they had, in addition to showing the percentage completed in the overall course.

3.3. Methodology

Each user activity on the Moodle platform was stored with the date, user, and action. These records were used to track student navigation; they served to detect on what hours and on what days students accessed the resources, thus helping to compare the effect on the activity according to the methodology adopted.

The Moodle activity recorded detailed interactions but not if the students were in the same session or if they were paying attention to the content. In order to evaluate the time spent in a student session, we considered the time between access and the last click of the session, establishing 60 min as the maximum time between clicks on the web within the same session. This time was chosen because normally if a student left the session it took at least several hours until they reconnected. It may seem like a very long time, but the values did not change in allocating less minutes. Consequently the time spent per student could be counted approximately.

Other data collected included the number of tasks completed and the grade of each task. At the end of the ‘learning program experience’ all necessary records were downloaded and analyzed in detail. Additionally, one of the authors was the professor of the subject, so there was also a direct observation in the face-to-face classes.

4. Results

The learning program experience was carried out over a full year in vocational education training which was divided into three periods: T1 (64 days) in which mainly theoretical classes and theoretical tasks were used, T2 (102 days) in which theoretical and practical tasks were mixed and T3 (55 days) in which they carried out only practical tasks using the PBL methodology.

4.1. Activity Generated

Figure 1 shows the activity that the students generated during the three periods. This activity reflects the number of interactions that took place in the virtual classroom used during the course. We can see that in general, more records were produced in the first period, in the second they decreased slightly and in the third there was a more extensive decrease. However, the evolution of the activity generated by the four females who participated in the learning program experience increased every period. Due to the small number of female participants, this trend was not evidenced in the global average.

What was previously demonstrated did not fully reflect the activity, since T1 was 64 days long, T2 timeframe was 102 days and the third period only 55 days. In Figure 2 we analyze the average time per day in each time period. In this case, the first period T1 reflected a greater activity per day in men and women, in the second period it reflected a significant decrease in activity per day and in the third period there was a rebound in activity.

The activity of the students regardless of gender was carried out mainly on weekdays, with Monday and Thursday being the busiest days in all periods. The change of activities did not produce any alteration in the habits of the students. What stands out is that on Fridays and weekends the students reduced their activity significantly. Figure 3 shows the average activity in each period by day of the week.

As in the activity generated by day of the week, students recorded similar peaks per hour of the day regardless of the period. Figure 4 shows the daily activity per hour generated in each period. We can see that the main activity was recorded in the afternoon which coincided with the class attendance schedule at the institute, which ran from 3:00 p.m. to 9:00 p.m. In the mornings between 9:00 a.m. and 1:00 p.m. there was a certain activity of the students, although it was much lower than that generated in the afternoon.

4.2. Tasks

To analyze in more depth the work carried out by the students, the percentage of activities completed, the time dedicated per task, the grade obtained in the final exam and the average grade of the tasks were obtained (scale 1 to 10).

Table 1. Tasks, dedication, and qualifications in each period.

	% Completed Tasks	Avg Tasks Grade	Avg Exam Grade	Time per Task (min)
T1	78.62	86.96	6.15	77.00
T2	79.17	82.53	4.51	23.75
T3	86.92	89.95	6.17	76.17

shows the values.

The percentage of tasks completed by students hardly varied between the first and second period, although in the third period it was slightly higher. If we observe the grades obtained in the tasks, the exam grades, and the dedication of time per task, we see that all the values were lower in the second period and similar between the first and third period.

Table 2. t-test of completed tasks, dedication, and grades.

	% Completed Tasks	Avg Tasks Grade	Avg Exam Grade	Time per Task (min)
T1 vs. T2	0.90	0.11	0.002	>0.001
T1 vs. T3	0.10	0.28	0.960	0.94
T2 vs. T3	0.11	0.02	0.002	>0.001

shows the p-value obtained in the t-test when comparing the different periods.

Considering that a p-value less than 0.05 in the t-test indicates significant differences, we note that there was no significant difference between the periods in the percentage of tasks completed. Between the first and the third period there was no significant difference in the values. It stands out that the second period had a significant difference with the other periods in the grade, the average grade of the exam and the average dedication to each task. There was also a significant difference between the second and third period in the average grade of the tasks.

Table 3. Pearson correlation coefficient between tasks grade, exam, and dedication.

	Tasks Grade vs. Exam Grade	Exam Grade vs. Dedication	Tasks Grade vs. Dedication
T1	−0.53	−0.01	0.26
T2	−0.13	0.28	−0.09
T3	0.10	0.39	−0.21

shows the Pearson correlation coefficient between the grade of the tasks and the grade of the exam, between the grade of the exam and the average dedication per task and between the average rating of the tasks and dedication.

We observe that in the first period the grade of the tasks was inversely related to that of the exam, the grade of the exam was not correlated with dedication and that the grade of the tasks was positively related to dedication. In the second period, the exam grade was positively correlated with dedication, the grade of the tasks and dedication were hardly related, as was the grade of the tasks with the exam. In the third period, the exam grade was related to dedication, the task grade was inversely related to dedication, and the task grade was almost unrelated to the exam grade.

5. Analysis

The main student activity took place between the weekdays, specifically during school hours, while during the weekends participation decreased significantly. It should be noted that the students had classes from Monday to Friday, so this could explain the decrease in activity at the weekend. In the first two periods, the highest activity on Monday may be because after a few days of little activity the students carry out the tasks accumulated since the previous Friday. After a peak of work on Monday and Tuesday, on Wednesday the students relax and therefore on Thursday they have accumulated work again. This behavior cannot be related to the design of the ‘learning program experience’, but it could be due to the workload of other subjects throughout their course. In the third trimester with the ABP methodology there is a peak of work on Mondays that decreases throughout the week, possibly because on Monday the students consult the tasks to be carried out and they plan their work until Friday, it does not mean that they work less, rather they navigate less on the platform. Therefore, the implemented gamification design did not work for students to work more in their free time and it did not work for students to work consistently every day.

In the first period, a high volume of participation was detected with a teacher-centered methodology, it is possible that part of these good results are due to the novel effect of the course. In this period, a percentage of tasks completed is recorded at 78.62%, very similar to that of the second period, 79.17%. However, the dedication per task with 77 min contrasts with the dedication of 23.75 min per task in the second period. Due to the fact that the grade in the tasks carried out does not have significant differences between the first and second period, the only possible explanation for the great difference in the final grade of the exam for both periods is the dedication time per task.

The third period with a student-centered methodology obtains values in terms of activity, dedication per task, percentage of tasks completed, grade per task and final grade similar to those of the first period. This reaffirms that the difference in the final exam grade with the second period is due to not paying enough attention to the tasks, although a direct relationship could not be found using the Pearson correlation coefficient.

Observing the correlations in Table 3, the second period with a theoretical-practical methodology was always found between the other two periods. In the first theoretical period, the dedication in the virtual classroom was not related to the exam grade, while in the more practical methodologies they are related in a positive way. Greater dedication was positively related to the rating of tasks in the first period and neutrally in the second and negatively in the third. Therefore, in our case a greater dedication has implied better marks in the exam under a practical methodology, but it is not related to the qualification in the theoretical methodology. Greater dedication reflected better grades in theoretical tasks, but worse in practical tasks, it is possible that a student who spends a lot of time on practical tasks is due to lower skill and therefore grades suffer.

6. Discussion

Although the literature has evidenced that gamification has an important position in education [14] there is still little effective guidance on how to combine different gamification functions in different educational contexts to improve learning performance [11,20,38]. In reviews on gamification applied to education [36] it is suggested that more empirical studies are needed on the use of game elements in contexts and relative to specific student type. For this reason, we studied the activity of a sample of vocational students throughout a full year applying three different methodological approaches.

The study evaluated students during a period of three months duration in order to complement previous research that did not exceed 3 weeks in duration [15,58,59]. Apart from analyzing different teaching methods, it sought to support the benefits of the use of gamification in a long-term educational contexts, an area still to be studied according to Dichev and Dicheva [55].

The results of the first and third 3 month time period were positive corresponding to other research [41,43,44,45,46,47,48,49]. The second 3 month period did not obtain positive results, similar to the work of Buckley and Doyle [15] and De Marco et al. [52]. These data reinforce the idea that the effects of gamification are undeniably and significantly linked to the educational context [11,15,20,38,55].

These findings are of interest to an area of growing interest in education-adaptive teaching. Adaptive teaching is an educational method that uses computer algorithms and artificial intelligence to orchestrate the interaction with the student, and offer personalized resources and learning activities to address the unique needs of each student [62]. The findings in this paper between the dedication of the students and the grade obtained in practical or theoretical tasks will be useful for the investigation of adaptive teaching by establishing certain parameters for the decision-making of its algorithms.

Limitations

This study originates in the discussion about the benefits of the use of gamification but focuses on finding out which methodological approach of the three proposed works best in a design of a specific gamified course in vocational training students. As mentioned in other studies, the effects of gamification are linked to the target audience and context [11,15,20,38,55] consequently results obtained may vary if the sample of students or the application context is changed. In long-term experiences, it must be considered that the context may undergo changes. The number of female participants in the sample is very small, so the information on Figure 1 and Figure 2 should be verified in future studies. As for future work, the study can be replicated with students at post-primary or university level, or with students in another subject discipline. Evaluating different gamification designs with different approaches may be more advisable, along with evaluating which should be avoided in specific student contexts and samples.

Additionally, to confirm that the improvement in the third period is due to the combination of gamification and PBL, it would be necessary to replicate the study with non-gamified students. In our case, the data of all the students in the third period have been shown together despite the fact that the final task was a little different, it could be checked whether by changing the content had any adverse effect.

7. Conclusions

This study was concerned with the analysis of student behavior in a gamified learning experience using different approaches to learning. The results obtained in the activity generated by the participants in this study demonstrate that the greatest activity occurred during school hours. Most of the accesses occurred on Mondays and Thursdays, highlighting a very low activity on weekends. The implemented gamification design did not motivate the students to work more in their free time, and it did not work for students to work consistently every day.

The first theoretical period began with good participation data on the platform, possibly due to the novel effect. In the second theoretical practical period, the number of interactions per day was significantly reduced and in the third mainly practical period, part of the participation lost in the second period was recovered, without reaching the data obtained at the beginning of the course. Therefore, it was determined that the PBL methodology produces good results in terms of student participation in the proposed gamification design.

We note in this gamified design that the percentage of tasks completed is not related to the grades of the tasks or the final learning result. The grades in the first and third period in which the dedication per task was higher than in the second period, as well as the grades in the tasks and in the final exam, after applying the Pearson correlation coefficient with the data obtained, we cannot affirm that there is such a relationship. Therefore, future work related to this study will be conducted with a larger sample and with a control group to analyze if such a relationship really exists, verifying the methodological approaches studied in different samples of students and in other gamification designs.

Although the objective of this study was the analysis of behavior in a gamified ‘learning program experience’ using different approaches, the correlations between dedication and grades provide a new way of studying the prediction of students’ grades related t their behavior and methodology used.