Audience Response Software as a Learning Tool in University Courses

Abstract

The use of information and communication technologies (ICTs) has become a fundamental tool in all areas of today’s society, including higher education. Lessons cannot be envisaged without the use of tools such as computers, tablets or mobile devices. Many lecturers use audience response systems (ARS) to keep their classes engaged. ARS software allows teachers to interact with students via polls, text responses, or multiple-choice questions displayed via their mobile devices. A new example of the use of this type of devices in education is gamification, a technique that uses a set of activities with ludic character as a learning methodology in order to facilitate the acquisition of knowledge and competences. One of the most used gamification tools is Kahoot!, a free learning application based on a mixture of game and creativity, which encourages attention and participation of students through questions and answers formulated by the teacher and designed in a way that students respond via their mobile phones. This paper examines the use of Kahoot! in a subject belonging to the chemistry area. In order to assess the benefits of this tool, it was tested in a group of students to review the knowledge and skills acquired during the theoretical lessons prior to the exams, and the academic results were compared with those of a control group of students who did not use the tool. The results demonstrate that the use of Kahoot! led to an improvement in the teaching–learning process of the students and a noteworthy rise in their marks, and that its positive effects rise with increasing the frequency of use of this didactic tool.

1. Introduction

Taking into account the relevance of information and communication technologies (ICTs) in all areas of our current society, lessons cannot be envisaged without the use of technological tools [1]. Therefore, one of the leading focuses in the educational area is to define how ICTs can contribute to the current requirements of education. Several researchers have reported that ICTs can improve the quality and effectiveness of education by increasing student motivation, via making the achievement of basic skills simpler and through improving the teacher training [2].

On the other hand, the question that arises is how to promote student interest and participation in the classroom in order to develop an active role in the learning process, mainly because students do not recognize the application of knowledge to their field of study. Previous studies [3] demonstrated an effective learning process when the students played an active role. Collaboration, cooperation and student engagement as well as problem-based learning are essential for knowledge acquisition. Although it is not feasible to address all the problems listed above, it is possible to try to solve some of them in order to minimize their influence on the student’s academic results. In particular, the didactic contents can be reviewed with tools that are attractive to the student.

Lately, most university students attend the lectures with tablets, smartphones or laptops. In particular, mobile devices are used to encourage interaction between students and promote self-learning (M-learning). Mobiles are proposed as a complementary tool for the assimilation of concepts by students. Nonetheless, it is essential to show scholars how to the use mobile devices for educational aims [4]. A recent example of the usage of mobiles in education is gamification, which attracts student encouraging action, promoting learning and solving problems [5]. Clearly, current technologies have collaborated to the development of gamification: games can be performed in a dynamic way and enable the incorporation of typical devices such as smartphones, thus opening up novel possibilities for attracting students’ attention [6].

Gamified activities based on the use of audience response systems (ARS) are a common practice nowadays. They incorporate different game elements such as goals, rules, competitions, timing, points, or feedback [5]. They use clickers, small movable devices that transmit and record student´s answers to questions displayed in the classroom. Once the educator asks a question, students press buttons on their clickers to reply. Owed to the real-time feedback offered by this tool, students can evaluate their level of comprehension of the material taught [7]. This type of technology promotes students’ active collaborative learning, encourages interaction and engagement, enhances classroom attendance and attention [8]. However, clickers can be stressful due to technological issues, such as when remote devices do not function properly, and can be costly for most educational institutions.

Amongst the most commonly used online game-based learning platforms are Kahoot!, Quizziz, Quizlet Live and TalentLMS. All are useful, interactive, and fun ways to learn and/or review concepts, each with its own advantages as well as drawbacks. Kahoot! was one of the original online games launched in August of 2013. It is a free application based on a mixture of entertainment and creativity that has increased attention among educators owed to its easiness and its capacity to develop dynamic activities. Questionnaires can be launched at the beginning of a course or a thematic unit, to check the former knowledge of the students on the topic, during the class as an activity to keep the attention of the students or at the end of a lesson or even the whole subject to assess the learning acquired [6].

The teacher chooses the questions and defines the right answer as well as the response time. Typically, the queries are projected in the class and the scholars respond to them by means of their mobiles (Figure 1). Points are provided for the fastest right answers, while incorrect answers get no points. Applicants can enter nicknames, and their points are matched after every question. The design style is attractive and very simple.

The major benefits arising from this approach are: (a) Students feel it to be an entertaining activity. (b) It makes them actively participate in the learning process. (c) It is a shared experience (everyone responds at the same time). (d) Teachers can get immediate and fair information about how students have assimilated the concepts described in the lessons, can identify their weak and strong points, detect specific problems in the learning process with a certain topic and take the corresponding actions. (e) Students obtain instantaneous constructive feedback on their learning without risk of shame [9]. (f) It can be used for the initial, formative and final evaluation of the scholars.

This tool is more active than Quizziz and Quizlet Live, owed to its intense colors onscreen and the animated music, which boosts students to reply each question quickly. There is a leaderboard of the players depending on speed and accuracy. It allows the preparation of online questionnaires, discussions or exams [10]. Moreover, it only requires an internet connection, and does not need former software or knowledge, since questions can be created quickly and in a very simple way. It also enables one to record the answers of all the students in an Excel file, and to create multiple choice question images, videos and diagrams [11]. However, it also shows some drawbacks: students are persuaded to answer quickly, hence speed tends to prevail over correctness. Further, it is easy to copy the responses of other students.

Previous studies [12,13] have compared the results of the same questionnaire answered using three different methodologies: printed form, a remote control system and Kahoot!, and increased motivation and participation in the classroom were found for the latter methodology. Former researchers on Kahoot! agree that the application of this tool enhances both the learning process and the students’ participation, as well as the relationship amongst classmates. Considering this tool as a game enables one to share the comments among the students in a relaxed manner, and also promotes the participation of the whole class, facilitating the development of social skills [14]. Further, in view of the relevance that the formative and continuous assessment has nowadays in our educational system, this strategy is very appropriate, since it can also allow the design of test exams about specific concepts.

Quizizz is very similar to Kahoot!, except that the questions and possible answers are displayed individually on each student device. The game is not as lively and musical as Kahoot!, though it creates a well set starting line for students to see their upgrading. The major advantages of this platform are that it is student-paced, hence no one gets disappointed because their device did not load the game fast enough to compete. Besides, educators can exhibit a student progress dashboard on the projector and immediately realize how many queries the class replied right/wrong. It is not necessary to complete a game live in-person. Homework mode can be used to assign it to be completed by a deadline. Nonetheless, most of the excitement is lost, since everyone answers the different questions at different times.

Quizlet Live is more focused on vocabulary compared to the other gaming platforms. Students are randomly divided into small groups and are encouraged to move next to their partners, where they have the opportunity to interact and communicate with other students to win the game. This mixes up their environment and encourages physical movement, which boosts cognitive function. It is a true team play: each team player is provided with a set of words to match to the definition, and all teams are excited trying to be the first to answer all the given words. However, Quizlet Live is not as useful for grammar or other types of questions that are not vocabulary-based. In addition, at least six students are needed to play a game (at least two teams of three students) and at least six cards in a flashcard set.

TalentLMS is an award-winning, cloud-based learning management solution designed specifically for usability and easy course creation: the system enables you to create courses by re-using any material you already possess or by directly integrating content from the web. The main benefits of this platform are its ease of use, great integration, and numerous add-ons. It has a minimalistic interface that allows users to focus on the content and it is easily customizable. It offers embedded content tools for video, audio, SCORM, xAPI and flash content, along with search functionality to find and insert YouTube videos, SlideShare presentations, Wikipedia articles, etc. It can be used on multiple devices including desktop, smartphone, tablet, and phablet. Further, the platform is easy to navigate and does not require much training. Additionally, it provides rewards to the winners such as discounts, special offers and so forth. Levels can be assumed as a hierarchy that the user can move up on (such as receiving a promotion at work or completing a course at school). By default, all users start on Level 1. Moving up a level enables users to unlock courses, and TalentLMS administrators have the choice to arrange the minimum level required for unlocking any particular course. On the other hand, it is very complex to make any changes to the interface or to add items to the portal, and the tool can be expensive for the institutions.

The following is a case study, carried out by a group of students representing about 37% of the total enrolled scholars in a subject of chemistry at the University of Alcalá (Spain) during the academic years 2018–2019 and 2019–2020. In order to be able to assess the possible benefits of the application of ARS software, a game-based learning platform such as Kahoot!, on the students’ academic progress, results were compared with a control group of students who did not use the tool, and also with those obtained for previous years in the same subject. Although a large number of studies dealing with Kahoot! have been published in recent years, very few of them have accomplished this comparative approach [15]. In particular, to the best of our knowledge, it is the first time that this type of comparison is applied to a subject in the field of chemistry. Further, the methodology has been implemented in two consecutive academic courses, in order to check for reproducibility, and the influence of Kahoot! questionnaires frequency on the scholars’ academic performance has also been evaluated.

2. Gamification as a Didactic Tool

Different methods related to education and gamification have been developed, such as game-based learning (GBL), and two main education gamification categories have been proposed [16]: structural and content. According to former investigations, the success of gamification in the educational field depends on several factors. Firstly, the adaptation of game design elements is very complex. Secondly, it is conditioned by the learning styles. Thirdly, it is highly influenced by the motivational response of the students to the game elements. Finally, the learning environment is also of significant importance.

2.1. Levels of Gamification Modification in Education

The adaptation of gamification in education can take place in several levels: In the first one, a single application is employed to enhance the student´s activity; nonetheless, the course conforms to the conventional teaching way. Such applications are commonly interactive response systems (IRS), such as Kahoot!, Socrative, Mentimeter, Quizlet Live, Poll Everywhere or Plickers [17]. These tools offer numerous question–answer choices, quizzes and short answers to select from, and frequently use points and scoreboards. Students using IRS have been reported to be more involved, interested and concentrated compared with paper forms or non-game-based student responses [18].

The second level consists of the incorporation of game design elements to the course to recompense certain actions. A broad variety of elements can be selected to boost a change in the student’s behavior including points, badges and leaderboards [19]. Points are the simplest way to reward a competitor for completing an action. This technique is useful to motivate people who like to feel progress. The gathered points are frequently displayed on a leaderboard, which makes the performance of every participant accessible to the others. Badges are a visual plot of some achievement and are used to boost and distinguish certain behaviors. In addition, they can be used to incentivize reaching a milestone. Two types of badges can be set: invisible and visible. Participants typically know what kind of behavior or action they have to take to attain the second ones, while the former would arise as a surprise; hence only those who spontaneously have a certain behavior are rewarded.

In the third level, game design elements are incorporated to attain improved results when using gamification. They offer processes and factors such as the Hook Model [20], which considers the four phases of a typical habit-forming product. Other models are the Fogg’s Behavior that identifies three factors that should converge so that a behavior takes place [21]: motivation, ability and trigger. Besides, the self-determination theory (SDT) [22] outlines things people find motivating, based on three main ideas: autonomy, competence, and relatedness.

The final and most complex level takes into account the different learning styles and the diverse scholar types. To prevent masking the possible positive consequences of gamification, proper design of the learning environments is required, considering students’ differences and motivations [23].

2.2. Students’ Types, Learning Styles and Environments

GBL has been applied in all educational levels, from elementary education [24] to university courses [25]. However, it is important to notice different learning styles when GBL is used: an Index of Learning Styles (ILS) has been proposed for classifying learners, and four main types have been identified: active/reflective, sensitive/intuitive, visual/verbal and sequential/global. Further models have been developed in which learning styles are based on a four-stage cycle [26]. Other researchers recognized typologies of game competitors and their features such as competitiveness, openness and investigative skills [27]. The Bartle taxonomy of player types identifies four characters: achievers, explorers, socializers and killers. Several studies have also been devoted to exploring the student classification; though, there is an absence of works analyzing the influence of gamified elements on the learning environment. It appears that player types can be correlated with the students’ activities and their participation patterns.

In recent years, gamified learning environments have grown substantially to achieve the needs of both teachers and students. Applications and platforms with different forms have been developed to fulfill the needs from primary school (e.g., ClassDojo, Socrative, Schoooools.com) to higher education (e.g., ClassFlow, Google School) [25]. In particular, you can push lessons out to a smartphone if your students have one and collect assessment data through the ClassFlow lesson (Figure 2).

Gamification in higher education is effective if a digital learning management system such as Moodle or Blackboard is applied. Students gain access to various materials, communicate with each other and teachers and can independently test their knowledge online. Teachers can provide materials and setup reviews, as well as evaluate and supervise students’ activities. The importance of gamification has been corroborated by the EU project calls in the Horizon 2020 funding scheme (ICT-21-2014 and ICT-24-2016) [28].

3. Purpose, Research Questions, and Methodology

3.1. Aims of the Study and Research Design

The general purpose of this article is to assess the extent to which students’ knowledge can be improved via using a game-based learning platform, in particular Kahoot! as a learning tool, its possible benefits on the students’ academic performance and how the efficiency of this methodology depends on its frequency of application. In order to evaluate its beneficial effects, it was implemented only in a group representing around 37% of the total enrolled scholars, both during the academic courses of 2018–2019 and 2019–2020, and the outcomes were compared with a control group of scholars who did not use the tool, and also with those attained for previous academic courses. The frequency of use of the tool during the period 2019–2020 was twice that of the academic course 2018–2019. The questionnaires did not imply extra work, and the two groups received the same teaching hours. Further, there was no correlation between the different groups and the student’s abilities.

The idea was to find out how well they recalled the information during the exam, considering that they had already answered some of the questions during sessions on Kahoot!. The use of the tool was optional, since its compulsory character would have turned it into a duty, whereas activities in higher education should not be mandatory. Its voluntary nature promotes the students’ autonomy and management of the learning process, which is crucial from an educational perspective. Therefore, the questionnaires made with the tool were not included within the assessment criteria in the course guide, although they had a positive effect on the final mark of the students.

The specific aims of the current study are listed below:

• To assess the potential of gamification within the framework of higher education.
• Implement the use of novel ICTs within an educational context.
• To determine whether there are differences and whether improved learning results are attained when using an ARS software compared to conventional teaching methods.
• To encourage the practice of new technologies and digital games in teaching–learning processes via ARS software.
• Increase the motivation of the students for learning the course contents through an attractive environment, which will result in a better academic performance.
• To improve the understanding of the gamification tool in order to use it as a means of evaluation in subjects belonging to the field of chemistry.
• Boost students’ capacity for discussion, critical logic, choice, and making decisions.
• Facilitate a game-based learning platform as an educational technology.

Considering the differences between GBL and the learning models most commonly used in higher education, the following questions arose: (1) Can gamification tools like Kahoot! be used to promote student learning and engagement, that is, to support the active participation of the students in the matter? (2) Can this tool aid in memorizing concepts, thus facilitating the students’ learning process and consequently improving their academic progress? (3) Can it expand students’ skills by reviewing concepts in an amusing and motivating way? (4) Can it help to identify the weak and strong points of the students, detect specific problems in the learning process and take the corresponding actions? (5) Does the effect of this methodology on the students’ learning process depend on its frequency of use?

3.2. Research Methodology

The research methodology chosen was action research, given that it uses a cyclic or spiral process which alternates between action and critical reflection suitable for an educational context [29]. This methodology includes five consecutive steps: problem identification, action plan, data acquisition, data analysis and interpretation, and plan for future action.

Following the abovementioned steps, firstly, different questionnaires were designed using the web https://kahoot.com, which is free and very easy to deal with. Afterwards, the teachers explained to the students the aims of using the tool and the work dynamics. Once the ARS software was implemented, the students voluntarily participated in the different launched questionnaires. All the students to whom it was proposed agreed to participate in the activity, although it did not have weight in the final grade.

Each questionnaire consisted of 15–20 questions and provided four different answers with a single correct option; depending on the difficulty, the students had between 30 and 60 s to answer each question. Some were short questions referring to theoretical concepts while others required small numerical calculations. Additionally, many of them included an image or short explanatory video. As the test progressed, and the questions were answered, the teacher promoted a process of collaborative reflection that aided reviewing concepts. Thus, a debate was generated that had a very positive impact on the students’ learning process.

The teacher provided “feedback” after each questionnaire, encouraging students to think about their issues. Students’ participation and scores were monitored via “reports” option, which enables one to identify the most difficult questions that need reteaching, identify if some learners need follow up or struggled with completing the challenge. Further, the “Questions tab” allowed teachers to obtain a detailed breakdown of correct and incorrect answers, assess class performance and identify gaps in knowledge. Reflection and analysis on the gamification activity results were explored in successive seminar discussions. Finally, the teachers reflected on the results and proposed future actions.

To verify whether the ARS software contributed positively to learning outcomes, questions similar to those used in the questionnaires were included in the final exam of both the control group and the group using it as a learning tool. Thus, it was possible to corroborate if the ratio of positive answers increased for students that have reviewed the concepts with this gamification means.

4. Design and Implementation of the ARS Software

The easiest way to create the gamification tests is using a computer. The address of the application is: https://kahoot.com. Once inside, you have to identify yourself as a user or create an account if it is the first time you use the tool (Figure 3).

There are two options in each user’s space: create a new Kahoot! or modify a previous one. The creator tool allows designing four different types of tests: (1) questionnaires (quiz) to select the correct option from multiple answers, (2) puzzles (jumble) to order a sequence of stages, (3) debate (discussion), where open questions are formulated, and (4) survey to make an inquiry about students’ opinions on a specific topic. The tool allows you to add details such as a title, a depiction, the idiom, the intended audience and a cover picture. In addition, it allows you to include a link to a video to be played in the background when the scholars are joining the activity.

Once the type of exercise has been selected, the questions and up to four possible answers for each one are written (Figure 4). The teacher marks the correct option for each question and sets the time the students will have to answer the questions. This time can vary from one to another, being typically between 30 and 60 s. In the case of questions such as debate or survey, up to four possible answers can also be formulated, but the correct one cannot be established. An interesting option of this tool is the preview, which allows you to view the test as the students will see it in the classroom. In this option, the computer screen is divided into two parts: On the left is what the student will see projected in the classroom and on the right what he will see on his mobile phone.

Before starting the activity, the teacher imparted a seminar in order to explain how to handle the tool to the students. Thus, all of them understood how the application works as well as the purpose and dynamics of the game.

The stages to develop the gamification activities are:

(a)

The teacher launches a questionnaire, and the students join the game through their mobile phones. They need to enter the game PIN and a name (or nickname). The use of a pseudonym gave them more freedom when answering the questions. These data are recorded on the teacher’s computer and will be displayed on the classroom screen along with the updated total number of participants. This allows students to verify if they have successfully registered. In case the teacher recognizes any inappropriate nicknames, the participant can be kicked out. A tune can be played during the game, to provide a suitable environment.

(b)

The game begins when the professor presses on the “Start” button. The game session moves over three stages. First, only the question is shown on the screen, then students can read the question and the answers. Each response is framed in squares of different colors. They have to choose the correct option among four possible answers by pressing on their mobile phones the color that contains the response they consider correct (Figure 5).

(c)

Once the response time has expired or all students have answered the question, students will be able to check on their mobile whether they have succeeded or failed. During this second stage, the screen will indicate the correct option and a bar chart showing the number of students who have answered each option.

(d)

The third stage is the scoreboard, showing the top five participants (Figure 6). During this phase, each individual screen will display the scholars’ score for the last question, the total score and his/her position in the student’s list.

(e)

All questionnaires will go through these three stages until all questions have been displayed. At the end, the scholars can rate the activity with one to five stars, indicate whether they learned anything and rate their satisfaction level.

Once the test is finished, students receive on their mobiles the number of correct and incorrect answers and the position they occupy compared to their peers in a scoreboard. This information is very valuable, since it allows students to improve self-knowledge and individual learning. Starting from the idea of error as part of the learning process, and following the game, students can evaluate themselves and, therefore, identify the aspects in which they should focus as well as raise their doubts about the subject.

At the end of the game, results can be downloaded into an Excel table with the statistics of the successes and failures by questions, which provide an overview of the level of knowledge of the group. From the teacher’s perspective, this tool enables them to compare the progress of individual and group work through checking the contributions of each student, paying attention to the class diversity and the different learning rates. This facilitates the formative evaluation of the group and a more in-depth and constructive feedback. In addition, knowing the answers with the highest failure rate allows the teacher to reflect on the aspects with less understanding and make decisions about the best strategies to address them.

5. Results and Discussion

The statistics of the students who participated in this gamification activity during the academic course 2018–2019 and their relationship with the final grade obtained in the course are shown in Figure 7.

As shown in Figure 7a, the gamification activity was undertaken by 37% of the total enrolled scholars. More importantly, 85% of the students who reviewed their knowledge with the questionnaires passed the course in the first call (Figure 7b). In total, 31.5% of the scholars who passed the course correspond to students who reviewed the concepts before the evaluation exams with this gamification activity (Figure 7c). Likewise, 36% of the students who presented and passed the final exam for the course correspond to students who used this gamification tool (Figure 7d). Therefore, statistics show that this methodology is very beneficial to improve students’ knowledge acquisition. A similar trend was found for the academic year 2019–2020 (Figure 8), in which the questionnaires were carried out by a group representing 36% of the total enrolled students.

The comparison of Figure 7 and Figure 8 indicates that upon increasing the frequency of use of the tool, the percentage of students that passed the course in the first call increased. Further, the percentage of scholars using the tool who attended and passed the final exam also improved. This can be explained considering that this tool creates a good learning environment, increases student motivation and reduces their anxiety (as corroborated by the educators). Further, it results in greater student confidence in their knowledge and skills. Thus, the more frequent the questionnaires, the better the learning outcomes. These findings also suggest that students take formative assessment via this ARS software as an amusing learning activity. Students are prone to be more interested in the course if it is entertaining, attractive, and enjoyable. Consequently, the higher the number of questionnaires, the greater the students’ interest in the subject. It is worth noting that students respond positively to learning activities that allow them to interact with their teachers and receive immediate feedback.

To obtain more information about the beneficial effect of the ARS software on the students’ learning process, the statistics of the students that passed the final exam on the first call were compared to those of scholars of control groups who did not use the gamification tool, as well as to those of previous years of the same subject, and the results are shown in Figure 9. It should be noted that in all cases the exam structure was very similar, with the only exception that in the last two academic courses a few questions were similar to those previously formulated in the questionnaires, in order to ascertain the contribution made by this gamification tool.

Based on the gathered data, it is clear that the percentage of students who participated in the questionnaires and passed the exam is considerably higher compared to those of the control groups of students who had not undergone the activity. This could arise from several reasons. Firstly, it was easier for students that took questionnaires to recall the information since they had reviewed most of it before. Furthermore, students who participated in the gamification tool tended to mark less incorrect answers in the exam, even if they did not answer correctly during the activity. These conclusions can be drawn considering both the control groups and students from previous years. Therefore, participating in the gamification tool raised the efficiency of the students’ learning process, the improvement being more significant upon raising the game frequency.

Further information about the effect of the ARS software on the students’ academic progress was obtained by comparing the statistics of the final subject grading for the control group and those who used the gamification tool (Figure 10 and Figure 11 for academic courses 2018–2019 and 2019–2020, respectively).

Clearly, the marks of the group participating in the activity are significantly better: for the academic course 2018–2019, only 2% of the students that participated in the questionnaires got a grade lower than 3, while for the control group this percentage increased to 9%. Thus, just 15% of the scholars that underwent the gamification activity failed the course, whereas for the control group it increased to 32%. On the other hand, 29% of the students that used the ARS software got a final score higher than 9, while for the control group this percentage was only 1%.

Even better results were obtained for the academic course 2019–2020 (Figure 11), in which the differences between the control group and the students using the didactic tool were larger. In this case, 32% of the students that used the tool attained a final score higher than 9, while none of the students of the control group got this grading. For the group that participated in the gamification activities, just 1% of the scholars got a mark lower than 3 and 5.5% of them failed, whereas for the control group the corresponding percentages were 8% and 35%, respectively. Overall, results demonstrate that the use of the ARS tool led to a considerable rise in the final grade and in the number of students passing the subject. The application of a simple gamification tool has been proven to be very helpful for improving the students’ academic performance; the beneficial effects are more pronounced the greater the frequency of questionnaires. More questionnaires worked on likely imply more cognitive learning, since they involve a more participatory and applied learning than that of a conventional theoretical class. Further, it promotes knowledge retention, since it aids with memorizing key concepts. It was also noted that participants were more likely to remember the questions they answered wrong, given that they had to think about the reason for the wrong answer and seek the correct one. In addition, it triggers positive attention and focus, which would be reflected in reduced study time.

Regarding the academic course 2018–2019, it was found that the marks for the control group follow a normal distribution (Figure 10) with mean 5.62 and standard deviation 1.81 (

Table 1. Statistical comparison of the final marks of the group that used the gamification tool (1) and the control group (2).

Group	Academic Course	Mean Mark	SD	RSD (%)	Median	Mode
1	2018–2019	7.59	1.96	25.82	8.00	8.25
2	2018–2019	5.62	1.81	32.21	5.50	5.50
1	2019–2020	7.95	1.38	17.35	8.50	8.50
2	2019–2020	5.21	1.49	28.59	5.00	5.00

SD: standard deviation of the mean; RSD: relative standard deviation.

). The marks obtained for the following academic course display a very similar statistical distribution as shown in Figure 11, with mean 5.21 and standard deviation 1.49. In both cases the distribution is approximately symmetrical around the mean, which is within the interval 5 to 6, hence the mean, median and mode are almost coincident. However, the mentioned figures reveal that the marks for the group that used the didactic tool do not follow a statistical normal distribution, but a right skewed distribution; consequently, the mode is higher than both the median and the mean, which take values of 7.59 and 7.95 for the two aforementioned academic courses. All these statistical parameters have been included in Table 1. Thus, the percentage of students with an average mark higher than 5–6 is 82% and 88% for 2018–2019 and 2019–2020 courses, respectively, and the median (7.59 and 7.95) is more representative of the obtained data than the mean. A potential cause of this skewness could be start-up effects, since a tool that has a lot of success during a long start-up period is known to create a positive skew on the data [30]. Nonetheless, results obtained for both academic years are quite similar, which accounts for the reproducibility of the data. On the other hand, for both courses, the relative standard deviation (RSD) of the mean final marks obtained applying the ARS software are smaller than those of the control group, indicating better precision and repeatability.

To further assess the effect of the questionnaires on the students’ knowledge and exam preparation, scholars were provided with a self-assessment rubric (see

Table 2. Rubric used to evaluate the effectiveness of the questionnaires.

	Exemplary	Effective	Minimal	Unsatisfactory
Questions’ content	Questions contain relevant information related to the main concepts learned in the course. Critical thinking is necessary to answer the questions.	Questions contain some information related to the concepts learned in the course. Do not require too much knowledge to be answered.	Questions contain information not directly related to the concepts learned in the course. Do not require specific knowledge to be answered.	Questions do not relate to the concepts studied during the course.
Formulation and Correctness	Always uses the specific vocabulary. Perfect spelling and syntax. Clearly expresses what the question is asking.	Usually uses the precise vocabulary. Correctness in spelling and syntax. Generally expresses clearly what the question is asking.	Seldom uses the specific vocabulary. Poor grammar, mistakes in spelling and syntax. Difficult to understand what the question is asking.	Never makes use of the specific vocabulary. Incorrectness in spelling and syntax. Questions are confusing.
Answer Options	Each question has four thoughtful answers connected to the contents that promote critical thinking. Best possible incorrect answers.	Each question has four answers generally connected to the contents.	Correct answer is included but does not fit the question perfectly. Answers seem random.	The correct answer is not included.
Presentation and Creativity	Quiz is very well constructed and creative. It has pictures and music to promote student´s participation.	Quiz is very well constructed and creative. It has a few pictures and/or music.	Quiz was not well constructed. It has very few pictures,	Quiz has missing information. There are no pictures or music.

). The students had to evaluate the gamification tool according to the following criteria: (a) questions’ content (to what extent they contain relevant information related to the main concepts learned throughout the course); (b) formulation and correctness (correct use of the specific vocabulary of the course, accuracy of the contents, spelling and syntactic correction, easiness to understand the meaning and so forth); (c) answer options (response choices reflect a logical flow, degree of difficulty, etc.); (d) presentation and creativity (degree of creativeness, number of pictures or videos included/music). An anonymous survey was conducted so that the students were able to value the new methodology with total freedom, indicating its weaknesses and strengths, and to express their opinions.

Most of the students graded the gamification tool as “effective”, meaning that they believed it had a positive effect on their learning process, although there were still some aspects that could be improved. Some of them claimed that the number of questionnaires should be increased, may be one after each thematic lesson. Consequently, it is essential for teachers to carefully structure lectures in order that Kahoot! time is properly allocated. Others indicated that more numerical questions (including problems) should be included in the questionnaires, and that more time should be provided for each question. In addition, some indicated that it would be highly desirable to offer students with more and timely feedback.

Overall, the students believe that they have managed to self-assess their learning process, which has become more active and practical, and that they had the opportunity to demonstrate what they have learnt, which is highly remarkable from the viewpoint of the metacognitive process. Additionally, other adjustments of the use of the tool were mentioned, for instance to allow scholars to make questions that can be replied by the other colleagues, as a review.

To get a deeper insight on the effect of the gamification tool on the students’ academic performance, the average percentage of correct answers obtained in the questionnaires were correlated with the final grading in the subject, as depicted in Figure 12 and Figure 13.

A direct correlation can be observed between the number of correct answers in the questionnaires and the final score in the subject exam. Thus, during the academic year 2018–2019, students with a final mark lower than 3 only got an average of 5% correct questions in the questionnaires, while those with a score higher than 9 reached about 87%. Those with a grade between 5 and 6 had about 50% right responses.

A very similar trend was found for the academic year 2019–2020 (Figure 13). Scholars with a final grade inferior than 3 just attained 5% correct questions, those with a mark in the range of 5–6 got around 55% of the answers right whereas scholars with a score higher than 9 got almost 90% correct responses. Thus, those students who obtained the highest number of hits in the questionnaires did better on the full exam. This result cannot be fully attributed to the gamification tool, since it has been found that those students were more hardworking.

Overall, five different patterns could be distinguished among the students according to their average percentage of correct answers, which were in agreement with their final score:

• Type 1: Students who systematically reviewed the concepts after each lesson and had developed a critical thinking skill, who replied correctly to most of the questions, hence successfully passed the course in the first call with a final grading higher than 9.
• Type 2: Students that frequently reviewed the learned concepts and have gained a general knowledge of the subject, who answered correctly more than 60% of the questions and got a final grading higher than 6.
• Type 3: Students who randomly reviewed the concepts and worked just enough to pass the subject, who only replied correctly to about 50% of the questions and attained a score between 5 and 6.
• Type 4: Students who seldom reviewed the concepts and had not developed a critical thinking skill, who only replied properly to 15–35% of the questions and got a final grading between 3 and 5.
• Type 5: Students who never reviewed the concepts and got less than 10% correct questions, who are not interested in the subject and their final mark was lower than 3.

6. Limitations on the Application of Gamification Activities Based on an ARS Software

Despite the huge potential of gamification activities as a learning tool, a number of limitations are also envisaged, which promote further investigation in the field in order to solve certain issues. An odd concept characteristic of this type of activities is that to gain more points, one should be the first in answering the question, or faster than the other participants. This game factor communicates that speediness overcomes precise knowledge of the course material and comprehension of the question. The questionnaires are also frequently a base for extra credit in lessons, hence students became more concerned on participating in the game for extra credit and do not retain the responses to the different questions. When a novel question is raised, the student has just a short time to read it before the answers are revealed along with it. Besides, due to the time constraints and the tool format, it is difficult to ask challenging and complex queries that will measure the students’ knowledge on the course material more accurately.

Although questionnaires are not used to systematically teach material, the approach is strongly dependent on the time provided to answer the question and the teacher debate regarding erroneous answers. For instance, if several students respond to a question incorrectly, depending on whether or not the educator is willing to explain the reason, or just continue with the activity, the scholars will not attain a clarification to why they answered the question in the wrong way, nor will the educator comprehend why the students selected a certain response over the correct one. For such purpose, teachers could diminish the length of game-based sessions albeit devote more time to the post-test discussion of the answers and the problem-solving strategies taken to achieve the correct answers. Thus, the productivity and outcomes of the tool are profoundly dependent on the teacher who is leading the activity; hence it might be better not to use it as the foremost didactic tool. Teachers should also reach a balance between testing students on novel versus recently acquired content to preserve their attraction and maximize the efficiency of the gamification activity as a learning tool.

Another disadvantage is that it is hard to track student progress. Students have nicknames they can create and during the game points are tallied based on how fast and how accurate the given answers are. In order to monitor student progress, the teacher should match the nicknames to the students’ names and then record the number of answers each student got right in every questionnaire that was performed, and this is a very tedious task.

Another limitation is that there is no function to measure open response questions. The software is limited to multiple choice or true and false. It will be more useful if it could include more answering formats such as text answers. On the other hand, because of multiple participants connected to the same platform, this tool requires a strong WiFi connection. Further, sometimes the background music can be distracting and stressful, which ultimately leads to the deviation of the student from its goal.

7. Conclusions and Future Perspectives

The use of gamification activities based on ARS software, in particular Kahoot! tool, has been found to be helpful for improving the students’ academic performance in a subject of chemistry. Noticeably, all the students that got a high percentage of correct answers in the questionnaires attained a good mark in the subject exam and passed the course in the first call. The gamification tool increases the motivation of the students for learning the course contents via an attractive environment, which results in better academic progress, and promotes memorizing concepts, thus facilitating the students’ learning process. Further, it encourages self-learning and self-evaluation, and allows the scholars to better understand those points on which they should focus and deepen their study. It has been an amusing, appealing, dynamic and collaborative activity that has encouraged the participation of all the students. It has been effective in reducing the frustration level of students and the stress about the fear of formative assessments in the traditional method. Likewise, the opinion of the students towards the developed tool has been very positive, and during the activity, the students showed considerable attention and interest. Further, improved learning results were attained when using the gamification tool compared to the control group that used conventional teaching methods, and its positive effects were found to be more pronounced with increasing the frequency of use of this didactic tool. Overall, it is demonstrated that the integration of mobile devices in the educational system favors the teaching–learning process. The applied tool provides teachers the facility to carry out a formative assessment of the whole class at one time without putting scholars on the spot.

Future work will include a larger number of students from other institutions in order to check for data reproducibility. In addition, a comparative study between different subjects would be interesting, considering more cognitive aspects, as well as to apply the tool to more students, matters and courses (even from different departments) to better assess its effectiveness and to generalize the outcomes of this research to other matters, especially in the fields of Science. On the other hand, a comparison between the user experience of Kahoot! and other tools such as Socrative, Edpuzzle, Slido, Gimkit, Quizziz, Quizlet and TalentMS could also be highly valuable. Despite there still being some hindrances in the path of Kahoot! to become a successful leader in the educational world, it can be considered a very efficient and interactive learning tool for the students. Nonetheless, a balance between testing students on new versus recently acquired tools should be attained, in order to keep their attention, and maximize the efficiency of the gamification activity as a didactic means.