In recent years, there have been many changes in society, most of which are related to the digital phase that has invaded all areas of life. This is especially marked in the younger sectors of the population, resulting, in some cases, in new forms of fun and entertainment. In particular, video games are a popular type of entertainment software that incorporates principles from multimedia and gameplay, which are being used with the additional purpose of improving certain areas of knowledge and skills. Specifically, in education, serious games are becoming important instructional tools and are increasingly being integrated into courses in many different areas to help individuals concentrate on the subject matter and enjoy learning [1
]. Serious games offer new learning opportunities, providing challenging tasks that promote more autonomous and exciting instruction. Their flexibility in use and scalability makes them an attractive pedagogic instrument, but it is necessary to check the effectiveness of serious games compared to more traditional formats. With this aim, in Dankbaar et al. [2
], a comparison with e-modules was performed, concluding that video-lectures (in a game) and text-based lectures (in an e-module) can be equally effective, although serious games are more motivating in the long term.
Numerous efforts have been made to evaluate the effectiveness of serious educational games and a large number of educational advantages has been revealed [3
]. However, as the demand for this type of game increases, game developers face the challenge of creating games that are not only effective but also attractive to play. Since there are different opinions on the essential elements that constitute a “good game” [8
], it is not easy to determine which criteria should be analyzed to assess the suitability of a video game and, especially, of a serious educational game. It is exceedingly hard to adequately describe and measure a gaming experience because there is a great variety of games and the kinds of experiences that users will have when playing these games will differ considerably [9
As a consequence, in order to adequately judge the impact of the design of the game on the user experience, traditional usability metrics must be redefined to incorporate the game elements in an overlay model.
The systematic review of the literature, described in the next section of this work, shows that, although many creators of video games and in particular serious educational games have conducted evaluations with significant user populations, there is currently no standard questionnaire to extract the opinion of the users about their experiences during a game session. Existing questionnaires are partial, dependent on the area of application, and are often too complex. To fill this gap, we propose a very simple questionnaire (Gameplay-Scale) that can be used by children and which has been used to evaluate a serious game for comprehensive reading in an educational experience carried out with 237 students between 10 and 12 years old.
Regardless of the effectiveness of the game (which will require evaluation methods specific for the purpose of the game, in our case pre-tests and post-tests of comprehensive reading), the opinion questionnaire about a serious educational game should consider several factors, such as: The difficulty perceived by the users in learning to play the game, the ease and pleasantness of the game environment, the satisfaction produced in the player about each element of the game, etc. With this aim, a questionnaire (Gameplay-Scale) to discover the opinion of the users of an educational game in terms of learning about the game, ease of use, satisfaction about graphics, sounds, interaction mechanisms, story and narrative, characters, challenges, rewards and empathy is presented, applied, and validated in the rest of this paper. Additionally, two questions were included in the evaluation experience to analyze the player immersion understood as the degree to which the serious component of the game, that is, the learning process, is implicit, and therefore, goes unnoticed by the player.
2. Related Work
We conducted a systematic search in the main collection of the Web of Science to identify the efforts made in the last ten years (2008 to 2018) in relation to the evaluation of video games and serious games. Finally, our interest will focus on serious educational games, but this broader vision will help. Concretely, the search for studies comprising a “questionnaire”, “poll”, or “survey” to discover the opinion of the players of the games retrieved over 621 results (within the categories Science Technology and Social Sciences).
This search was later refined to select studies including some of the following topics: a) “Human computer interaction” (or “human–computer interaction”), b) “validation” and/or “evaluation” of a proposal, c) quality attributes of the video game, such as: “Usability”, “ease of use”, “learnability”, “effectiveness”, user “satisfaction”, “playability”, “empathy”, and “awareness of learning”, and d) elements of the game, such as: “Graphics”, “sound”, “story”, “characters”, “avatar”, “challenges” and “scoring” (or “score”). The number of studies taking into account some of these topics (or a combination of them) was reduced from 621 to 252 papers. The title of each paper in this set of 252 papers was analyzed to determine its suitability to be part of our review. This filtering process allowed us to discard all those studies where, through the title, it seemed clear that the contribution of the paper was not concerned with evaluating a specific game nor a set of them nor even the concept of a video game itself.
Studies focused on the analysis of physical or psychological aspects of the player were also ignored. As a result, 84 papers were selected and their abstracts were read. During this systematic search of the scientific literature, some topics unrelated to our object were excluded: Video game addictions, violence, health, teachers’ opinion about the educational use of the game, etc. Exactly 34 papers were selected based on the abstract. Finally, after a complete reading, 30 studies were found that deal with the evaluation of a video game or serious game from the perspective of discovering the opinion of its players after the experience of playing the game.
Of the 30 reviewed studies, more than half (17 papers) refer to serious educational games. Nevertheless, all proposals of evaluation questionnaires for video games and serious games were analyzed to get a broader view. During our study, it was detected that there is a great diversity of proposals to evaluate the experience of using a game, but there is no standard. Since there is no widely accepted evaluation test, we tried to classify the different approaches found. The aim of this effort of classification is to identify common points among the existing proposals and to facilitate the work of future researchers/educators who need to face this task.
and Table 2
summarize the performed classification, focusing only on the proposals of evaluation for serious educational games. Table 1
describes the proposals of evaluation questionnaires that put the emphasis on the learning generated as a result of the use of the game. Table 2
describes the proposals of evaluation questionnaires that deal very closely with the user experience during the game. In both cases, the following information is included: Main objective of the proposed evaluation questionnaire, type of users who have answered the questionnaire (population), attributes considered in the items of the questionnaire, and attributes used in the questionnaire to evaluate the usability of the game (Table 1
and Table 2
During the complete review of the 30 papers, we found some previous models that can be useful to define certain aspects of an evaluation questionnaire for video games (or serious games). For example, in Agarwal and Karahanna [18
], an interesting, multidimensional conceptualization of cognitive absorption about information technologies usage is presented. This construct is exhibited through five dimensions: Temporal dissociation, immersion, enjoyment (pleasurable aspects of the interaction), control over the interaction, and curiosity. In addition, a nomological net is defined, trying to capture relations among the cognitive absorption and other factors, such as personal innovativeness, playfulness, self-efficiency, perceived usefulness, perceived ease of use, and behavioral intention of use.
This model has been used (with adaptations) in Wang et al. [1
], however, the multidimensional construct is focused on cognitive absorption and is not specific to video games. In a similar line, in Malhotra and Galletta [15
] a multidimensional commitment model of volitional systems adoption and usage behavior is presented (also used in Wang et al. [1
]). In this model, the commitment to system use appears related to perceived usefulness, perceived ease of use, and attitudes toward using and behavioral intent. Because voluntary systems require users’ volitional behavior, this model may be applicable to video games. However, again this proposal does not cover the particularities of this type of ludic system. A complementary approach is the presence questionnaire proposed in Witmer and Singer [24
] to measure the subjective experience of being in one virtual environment. This questionnaire has been adopted in Lorenzini et al. [23
] and its items are organized into six subscales: Sensory exploration, involvement, interface awareness, control responsiveness, reality/fidelity, and adjustment/adaptation. The questionnaire addresses the sensation of presence during the user experience in the virtual environment in a comprehensive manner, but does not include the rest of the elements of a video game: Story, challenges, characters, etc.
Additionally, a few existing models to evaluate usability were found during the review. For example, TURF [28
] is a unified framework of EHR (Electronic Health Records) usability. Under TURF, usability is understood through three dimensions: Usefulness, usability, and satisfaction. This approach, followed in Johnsen et al. [19
], is structured as a set of representative measures: Across-model domain function saturation, within-model domain function saturation, learnability, efficiency, error prevention and recovery, and user impression of how useful, usable, and likable the system is. Another interesting proposal in this line is the System Usability Scale (SUS) in Brooke [25
], which is followed in Lorenzini et al. [23
], Boletsis and McCallum [29
], and Vallejo et al. [30
SUS is a simple, Likert ten-item scale giving a global view of subjective assessments of usability. Evaluated aspects in SUS are: Easy to learn to use, easy to use, integration of functions and consistency, like using, and confidence. Again, none of these two approaches is specific to video games, so they need to be adapted to take into consideration the characteristics of this type of software.
Regarding models specific for video games, very few contributions can be mentioned. For example, in Norman [31
], a review of two very interesting questionnaires is presented: Game Experience Questionnaire and Game Engagement Questionnaire. The Game Experience Questionnaire (GEQ) [9
] involves usability, flow, and immersion and consists of three modules: Experiences during gameplay, social presence, and experiences once a player has stopped playing the game. Questions in each module deal with how the user feels (successful, bored, frustrated, challenged, etc.) and the only element of the game explicitly referenced is the story. The In-Game Experience Questionnaire (iGEQ) is the shorter in-game version of the Game Experience Questionnaire and contains 14-items, rated on a five-point intensity scale distributed between seven dimensions: Immersion, flow, competence, tension, challenge, negative affect, and positive affect [28
]. For its part, the Game Engagement Questionnaire [32
] aims to provide a measure of an individual’s potential for becoming engaged in video gameplay at differing levels, taking into account presence, flow, absorption, and dissociation.
In conclusion, during the systematic search, we reviewed very different proposals to evaluate the user experience during the utilization of a video game and also of a serious game (proposals for serious educational games are summarized in Table 1
and Table 2
). Although we found a few already accepted models that can be used to define certain dimensions of the required questionnaires, there is no standard model or test for this evaluation task. In addition, most of the proposals found during the performed systematic review process are partial and are not specific to video games (much less for serious educational games). One of the most complete revised questionnaires is GUESS (Game User Experience Satisfaction Scale) [8
]. GUESS includes attributes to evaluate: Usability/playability, narratives (characters, story, etc.), play engrossment, enjoyment, creative freedom, audio aesthetics, personal gratification, social connectivity, and visual aesthetic. In addition, this proposal defines usability centered on the game (for example, it includes ease of navigation, the manual to play the game, etc.). However, GUESS is not specific to educational games and consists of 55 items with nine subscales, so it does not seem appropriate for children. More precisely, only three papers were included in the systematic map focus on children: Canals et al. [27
], putting the emphasis on aspects of the game [33
] putting the emphasis on enjoyment and Harrington and O’Connell [34
] with the focus on empathy and prosocial behavior.
Accordingly, a new questionnaire to evaluate the user experience during the game, Gameplay-Scale, is proposed in this paper. This proposal takes into account the specific elements of the game (story, characters, challenges, etc.) and the player’s perception during their interaction with the game, including how he/she identifies with their own representation within it (that is, the avatar). In addition, the level of awareness of the user about their own educational process (awareness of learning during the game) was consulted to the players to analyze the ludic–educational balance of the game. The questionnaire has been specifically designed to be suitable to be answered by children.
Thus, the literature shows the need to work with new tests in order to quantify the player’s experience during the game, knowing the degree of gameplay of video games and, accordingly, improving the attitude towards them. We aimed to perform confirmatory analysis (CFA) to confirm the factor structure obtained via exploratory factor analysis (EFA), and so have an assessment instrument in which validity is tested. Consequently, the main purpose of this study was to develop a valid and reliable Gameplay-Scale for child players.
5. Discussion and Conclusions
In this study, which had as its main objective the analysis of the psychometric properties and structure of the Gameplay-Scale, a new questionnaire to evaluate the player’s experience during the utilization of a serious educational game was adapted and applied. The adaptation of the Gameplay-Scale has been adequate, obtaining figures that show a very good factorial adjustment, good stability, and excellent internal consistency.
In addition, these data indicated the validity of the instrument to establish and analyze the gameplay in students. Based on a sample of 237 children using an educational video game, Cronbach’s alphas were all satisfactory. This is important because the period of childhood is considered to be essential for human development and behavior as patterns of behavior followed during this stage are often repeated in adulthood. It is, therefore, a vital period for developing adaptive types of motivation towards video games [40
]. Moreover, weekend video gaming was positively associated with the academic performance [41
]. Children and adolescents with a greater acceptance of new technologies are more likely to experience repeated satisfaction and pleasure from their engagement and will continue to engage in the future.
Additionally, this item was used previously to demonstrate the analysis of play in people who practice sports non-professionally [35
]. Video games are elements that promote intrinsic and extrinsic factors in children. In particular, they promote motivation and the strengthening of relationships within the family. Thus, the findings suggest that exergames (i.e., active-play video games) could be a potentially viable option for families, especially during bad weather in winter. Intervention efforts could benefit from promoting the social aspects of family exergaming and attitudinal factors [42
]. Others have discussed how the computer game-based methods that have been analyzed made it possible to distinguish the most significant features that can promote the development of executive functions at the preschool age [43
The three dimensions (usability, satisfaction, and empathy) load properly in the three factors of the Gameplay-Scale. The structural model developed to discover the relationships between the factors of the scale revealed positive and direct relationships between all dimensions.
The greatest strength of relationship was found between satisfaction and empathy in the same way as found by Harrington and O’Connell [34
] or Hainey, Connolly, Boyle, Wilson, and Razak [44
], who show how positive moods and the feeling of well-being produced by satisfying a need are associated with affective participation closer to other people’s realities. In the same way, both empathy and satisfaction were directly associated with usability. It seems that ease of use and learning in the game directly influence satisfaction and wellbeing, where it is essential to consider the level of the player to avoid negative experiences with the serious educational game in order to promote adherence to it, the degree of learning, and to avoid premature abandonment due to demotivation [34
Consequently, this study makes contributions to the literature. In this way, it is important to indicate the promotion of the level of playability of a video game in the learning and practice of it. The satisfaction that videogames bring together with autonomy and personal improvement must be the objective of educational interventions.
This is linked to what Chen, Sun, Harrington, and O’Connell [46
] and Thompson [47
] propose with the acquisition of adherence to videogames to generate changes in behavior through the promotion of patterns of motivation in learning processes.
The study is not without its limitations. The sample was self-selecting and the data that were utilized were collected from Spanish children. Consequently, the findings are not representative of Spanish children and may not be generalizable to other countries. For validation studies, representativeness is not a major issue, and arguably, over-sampling excessive video game players is more important. However, the present authors were unable to find a way of over-sampling, such a group in the way permission was sought to recruit participants, so this is another limitation that needs to be taken into account.
Furthermore, the scale was designed to be used in epidemiological studies rather than in a clinical context, but given the nature of the instrument, in-depth exploration using items in the scale could be used in a clinical context to get a detailed picture of the psychosocial impact of video game playing on children’s lives.
Future investigations should examine other groups. It would also be interesting to try to establish links between students who are video game players and athletes. It would also be interesting to analyze it together with educational, emotional, or psychosocial variables.
In summary, the Gameplay-Scale is a valid questionnaire for the analysis of the user experiences during the game in Spanish schoolchildren and the psychometric properties of the same are a valid instrument for children in different contexts (cultural or socioeconomic). The validated scale is an effective instrument to assess some basic dimensions in educational video games for school-age children, such as usability, satisfaction, and empathy. Specifically, it shows how ease of use and learning in the game directly influence satisfaction and wellbeing, where it is essential to consider the level of the player to avoid negative experiences with the serious game in order to favour adherence, as well as to build affective participation closer to other people’s realities.