Games with a Purpose for Part-of-Speech Tagging and the Impact of the Applied Game Design Elements on Player Enjoyment and Games with a Purpose Preference

Abstract

Linguistic tasks such as Part-of-Speech (PoS) tagging can be tedious, but are crucial for the development of Natural Language Processing (NLP) tools. Games With A Purpose (GWAPs) aim to reduce the monotony of the task for native speakers and non-experts who contribute to crowdsourcing projects. This study focuses on revising and correcting PoS tags in the Corpus Oral y Sonoro del Español Rural (COSER), the largest collection of oral data in the Spanish-speaking world, to create a parsed corpus of European Spanish dialects. It also examines how game design elements (GDEs) affect players’ enjoyment. Three games—Agentes, Tesoros, and Anotatlón—were developed, incorporating different GDEs, such as rewards and challenges. The results show two levels of enjoyment: at the concept level with Anotatlón, and at the level of individual GDEs with Tesoros. This suggests that certain GDEs influence player enjoyment and, consequently, their preference for certain games. However, the study also shows the complexity of evaluating triggers for player enjoyment in games with more than one implemented GDE.

1. Introduction

Natural Language Processing (NLP) tools used for Part-of-Speech (PoS) tagging have been trained with samples from written standard language, due to the lack of specific linguistic resources for oral corpora or varieties. This is even the case for languages that are spoken worldwide, such as Spanish. Consequently, the PoS tagging accuracy tends to drop on text types other than those the PoS tagger has been trained on, necessitating manual correction of the tags. It goes without saying that this manual correction is time-consuming and costly, even when several experts are involved. As a result, Games With A Purpose (GWAPs) implemented for this purpose have shown to be a promising alternative approach for creating large annotated corpora, compared to high-cost and time-consuming manual data annotation or other data collection methods (e.g., Mechanical Turk) [1]. However, designing a GWAP for linguistics is still a challenge, as many linguistic tasks are complex, specialized, and tedious. Even the most straightforward tasks, such as assigning a PoS tag (e.g., verb, noun, adjective, pronoun) to each word in an input text, can be problematic, given that the terminology may be unfamiliar to players.

Additionally, the corpus itself might present additional difficulties. To illustrate, let us consider the COSER corpus (Corpus Oral y Sonoro del Español Rural, ‘Audible Corpus of Spoken Rural Spanish’ [2]), on which the current research is based. COSER is the most extensive collection of oral data in the Spanish-speaking world and consists of transcripts of semi-directed interviews with rural elderly male and female informants who have little to no education and who have experienced low mobility [3]. As the corpus contains dialectal data, players from other geographic areas or different age groups might experience difficulty in identifying the words’ meanings and, therefore, the correct PoS tag.

Given these difficulties, this study aimed to investigate whether certain design elements can increase the engagement of players, in order to ensure the completion of the linguistics tasks and so that corpora can be successfully created and annotated. To engage players, we explore enjoyment, defined as the positive experience players undergo throughout a gaming session [4]. From this point of view, the design should allow players to understand the task at hand but, above all, significantly strengthen the attractiveness of games.

The existing body of research on GWAPs has presented various games to address specific linguistic tasks and build diverse linguistic resources. However, whether the GWAPs and their design are enjoyable for the players has not been addressed. Verbosity [5] was designed to collect commonsense knowledge about words, with that knowledge built up of facts that many people recognize as true. Zombilingo [6] focuses on dependency syntax annotation, in which two words are related, one the governor and the other the dependent. WordClicker [7] focuses on collecting judgments about PoS tags. Other games are centered on semantic tasks, such as Phrase Detectives [8], TileAttack [9], and Wordrobe [10]. More concretely, Phrase Detectives aims to identify the referent in an anaphora relation, whereby the anaphora remits semantically to an entity already introduced in the discourse. TileAttack targets a similar task called a co-reference annotation, in which the task is to find the relationship between multiple terms with a common referent. Meanwhile, Wordrobe focuses on different levels of linguistic annotation such as PoS tagging, named entity tagging (locate and classify named entities mentioned in a text), co-reference resolution, and word sense disambiguation (identifying which sense of a word is meant in a sentence).

As mentioned above, Wordrobe and Wordclicker have already explored the game-based approach for PoS tagging, but in two different directions, the former using Gamification and the second GWAPs. While Gamification uses GDEs in non-game contexts [11] to engage users with applications, the approach of GWAPs attempts to design video games that entertain and generate data as a side effect [12]. In that regard, in Wordrobe, researchers make use of ludic components, such as betting and awards, making it appear more like a gamified application. On the contrary, in WordClicker, the researchers attemped to increase the enjoyability of the GWAPs using a particular game mechanic, which consists of clicking [7]. If participants click on the correct PoS tag, they automatically produce and sell virtual cakes and earn virtual money that they can use to upgrade the bakery by buying new ingredients and equipment. Another difference in their approaches is that Wordrobe focuses on obtaining high-precision annotations, and WordClicker explores metrics to assess the enjoyability of the mechanics. However, neither of them analyzed or evaluated how the different GDEs used in their game design influence the user’s experience in terms of enjoyment, which is argued to be the main incentive in GWAPs [13]. Additionally, enjoyment has been identified as a critical factor of games, which not only motivates players [14] but helps to maintain their interest in the game [15]. Therefore, this study aimed to draw conclusions on these game characteristics in the context of GWAPs for linguistics by answering the following research questions:

• RQ1: Do the implemented GDEs influence players’ enjoyment and, consequently, their preference for games? Based on the type and number of implemented GDEs (see Figure 1), the order of preference is hypothesized as follows: (1) Tesoros, (2) Anotatlón, and (3) Agentes.
• RQ2: Are certain GDEs associated with higher values of player enjoyment?

Since variations in the design will obviously affect how participants perceive each of the games, answering RQ1 may reveal whether the integration of different GDEs increases player enjoyment and crowd participation. On the other hand, answering RQ2 will help to identify which GDEs are specifically responsible for increasing player enjoyment.

2. Materials and Methods

The study examined the implemented GDEs and their influence on the players’ preference for any of the three game concepts. Participants were split into three groups to evaluate their preference between two of three games that they played. They were separated this way to avoid biases due to the fatigue of performing more than two sessions of playing and evaluating. Participants played and evaluated their perceptions of the GWAPs by answering three questionnaires. Details of the experiment are described in the following sections.

2.1. Participants

As this study aimed to assess the GWAPs in terms of player enjoyment, the researchers’ networks were used to recruit potential players that met the inclusion criteria: (1) participants are native speakers or highly proficient in Spanish, with or without experience of PoS tagging; and (2) they are of legal age in the country where they have been recruited. Players did not need prior knowledge of PoS tagging, as the GWAPs included a training session to introduce or remind them of the different PoS tags. Fifty-four participants were randomly assigned to one of three groups (n = 18 in each group A, B, or C). Each participant played and evaluated two games. As a result, each game was played by 36 participants. In total, 30 male and 24 female participants took part in the study, 42 of these were native Spanish speakers, and 12 proficient.

2.2. GWAPs and the Implemented GDEs

In the context of this research, three game concepts to verify or correct pre-tagged PoS were developed: Agentes, Tesoros, and Anotatlón. Agentes is a roulette-style game where players categorize a word highlighted in a sentence by dragging or clicking on the correct PoS tag. Tesoros is a platform game in which players help an avatar named Gummy over obstacles, collect coins, and reach a treasure chest. Gummy moves automatically after the player assigns the PoS tag to the highlighted word. Anotatlón is a racing game where players navigate a car to avoid obstacles and then select the correct PoS tag for a highlighted word at the end of the race. Figure 2 shows the three game concepts. They include the same set of sentences extracted from the COSER corpus. As the COSER corpus comprises more than 4 million tokens, a sample of 12,402 sentences was extracted. These were automatically tokenized, lemmatized, and morpho-syntactically labeled using the spaCy library [16]. Additionally, the sentences were randomly presented to the players in each game.

For each GWAP, a selection of GDEs was implemented to enhance the player enjoyment, as shown in Figure 1. The design details are explained in [17,18].

2.3. Measurement

To investigate whether the implementation of GDEs influenced player enjoyment and whether there was a preference for certain games, both standardized questionnaires, such as the Intrinsic Motivation Inventory (IMI), and custom-designed ones were included. The IMI assists in measuring need satisfaction and evaluating intrinsic motivational characteristics, such as Interest/Enjoyment, Tension/Pressure, Competence, and Effort/Importance [19]. Two ad hoc questionnaires were designed in the context of this research: a post-game and a comparative questionnaire. The post-game questionnaire aimed to record players’ impressions of the game they had just played, and the comparative questionnaire sought to compare the two games played and their GDEs. The latter was assessed at two levels: the implementation, and preference/favorite.

The study also included an assessment of personality traits using the Ten-Item Personality Inventory (TIPI). However, the results are reported in [20] and not here because they fall outside the scope of this paper. Nevertheless, it is worth mentioning, because the TIPI was also collected during the experimental design.

2.4. Procedure

The study was conducted between May and July 2021 after being approved by the Social and Societal Ethics Committee (SMEC) of UHasselt (approval number REC/SMEC/VRAI/201/115, granted on 25 March 2021).

The experiment was carried out online for three reasons. Firstly, the setting was prepared to keep distance due to the COVID-19 pandemic. Secondly, as the research was developed in a non-Spanish speaking country, the online format allowed more participants from Spanish speaking countries. Thirdly, conducting the study online was a natural choice, as the GWAPs will be released on a crowdsourcing platform. Figure 3 provides an overview of the entire experimental procedure.

Participants were recruited via convenience sampling using the researchers’ networks. To those interested, the facilitator sent an official invite, including the informed consent form, the scheduled appointment, and the link to connect to the meeting. Participants who attended the appointment were first welcomed and then instructed about the study. They were asked to share their screen during the gaming session and stop sharing if they did not want to show their responses to the facilitator while answering the online questionnaires. After that, to formally initiate the playing and evaluation sessions, the facilitator sent a link that allocated the participant into one of the three experimental design groups. The first step was to accept the informed consent, and the second step consisted in completing a personality questionnaire. Then, the participants played the first game, which was randomly assigned, to ensure that both games were played the same number of times within the group, n times as the first game and n times as the second. This approach ensured that the evaluation had an equal weight for each game, avoiding the experience of playing the first or second game affecting the results. Once the game session was finished (minimum 10 min to maximum 20 min), participants completed the IMI and post-game questionnaire, which were the fourth and fifth steps of the study, respectively. Afterwards, participants played the remaining game, evaluated it by answering the questionnaires (steps 6–8). Finally, in the ninth step, participants completed the comparative questionnaire to identify differences and characteristics of the played games. At the end of the study, the facilitator acknowledged the players’ participation and concluded the study. Figure 3 also shows the multiple data sources used to collect the data, such as the questionnaires, observations of game sessions, and annotations (PoS tagging) obtained from the participants while they played the games.

3. Results

The following shows the results of the statistical analysis of the standardized questionnaires and the close-ended questions in the custom-designed questionnaires.

3.1. Game Concept Preference

The descriptive statistics, one-way analysis of variance (ANOVA) of each variable of the IMI, and the correlations between the IMI dimensions and the implemented GDEs in the games helped to answer the first research question (RQ1). For RQ1, it was hypothesized that, based on the number of implemented GDEs, the game preference order would be (1) Tesoros, (2) Anotatlón, and (3) Agentes. However, when participants were asked which game they would play again, the results showed a different order of preference: Anotatlón came out as the preferred game for 24 of the 36 participants who had played it. In second place, we find Tesoros (15/36 participants), and thus in third place was Agentes, with only 5/36 participants. In order to explain the participants’ reasoning for choosing one game concept over another, the dimensions of IMI were analyzed. First, the descriptive statistics in

Table 1. Mean scores and standard deviations for participant perceptions in each game regarding IMI constructs (1—Not at all true, 7—Very true).

IMI	Agentes		Tesoros		Anotatlón
	MEAN	SD	MEAN	SD	MEAN	SD
Interest/Enjoyment	4.49	1.57	5.18	1.30	5.46	1.15
Competence	5.37	1.15	5.79	0.89	5.07	1.22
Effort/Importance	4.08	1.36	3.74	1.26	4.21	1.30
Tension/Pressure	2.80	0.91	2.62	0.81	2.70	1.06

show that the Interest/Enjoyment and Effort/Importance levels were higher for the game Anotatlón. For Tesoros, Competence was higher, and the Tension/Pressure was lower. One-way analyses of variances (ANOVA) revealed significant differences for Interest/Enjoyment and Competence. Post hoc comparisons using Tukey’s test were conducted to investigate which pair of games presented these differences. The perceptions of Interest/Enjoyment ($F_{2,105}=4.827$, p < 0.001) showed a significant difference between Anotatlón and Agentes. Similarly, perceived Competence was also statistically significant ($F_{2,105}=3.941$, p < 0.01) between Tesoros and Anotatlón. Effort/Importance and Tension/Pressure did not show significant differences.

Additionally, a logistic regression analysis was conducted to determine whether the IMI factors influenced the concept preference between the two games played by the players. As shown in

Table 2. Logistic regression analysis of participants playing Anotatlón. * indicates p < 0.05, ** indicates p < 0.01.

	Estimate	Std. Error	z Value	Pr(>$\left|\mathit{z}\right|$)		OR
(Intercept)	−14.8777	5.6344	−2.6400	0.0083	**	0.0000
Interest/Enjoyment	0.9423	0.6939	1.3580	0.1745		2.5660
Competence	1.3923	0.6153	2.2630	0.0237	*	4.0243
Effort/Importance	0.4474	0.5094	0.8780	0.3799		1.5642
Tension/Pressure	0.8707	0.7750	1.1230	0.2612		2.3886

, perceived Competence was a significant predictor for preferring Anotatlón and, in fact, the only construct with an effect on game preference. Thus, the results in Table 1 and Table 2 show that, to some extent, the Competence and Interest/Enjoyment influenced the concept preference. However, to discover whether those IMI levels resulted from implementing the GDEs, correlations among IMI constructs and the GDEs were analyzed to determine the differences in respondents’ preferences. The following section shows these results.

3.2. Game Design Elements

The analysis of correlations between the IMI dimensions and the GDEs implemented in the games helped to a certain degree to answer RQ2, given that some GDEs were positively associated with high values of player enjoyment. Concretely, leaderboards, challenges, and levels were evaluated as the best implemented GDEs, with challenges, adaptation, and theme as the favorite GDEs. The results in this section are presented following the two factors that were evaluated, i.e., Implementation and Preference/Favorite.

3.2.1. Implementation

As shown in

Table 3. Descriptive statistics and correlation matrix between the GDE implementation and the IMI. Bold numbers show the highest-rated GDE implementations. INT.ENJ = Interest/Enjoyment, COMP = Competence, EFF.IMP = Effort/Importance, TEN.PRES = Tension/Pressure. * indicates $p<0.05$, ** indicates $p<0.01$, *** indicates $p<0.001$.

GDE	Agentes				Tesoros				Anotatlón
	Mean	SD	INT.ENJ	COMP	Mean	SD	INT.ENJ	EFF.IMP	Mean	SD	INT.ENJ	COMP	EFF.IMP	TEN.PRES
Adaptation	6.97	2.34		0.50 *	7.61	2.37	0.49 **	0.39 *	7.53	2.20
Avatar	2.74	2.49			6.72	2.95		0.42 *	6.05	3.55
Challenges	7.43	2.13	0.36 *		7.31	2.24	0.57 ***	0.60 ***	7.91	2.08	0.60 ***		0.49 **
Customization	3.29	2.88			5.00	3.56	0.46 *		8.03	2.99	0.41 *			−0.39 *
Leaderboards	7.91	1.84			7.47	2.57		0.46 **	7.75	2.41	0.37 *	0.41 *
Levels	7.15	2.58	0.45 **		7.31	2.84	0.60 ***	0.38 *	7.40	2.78	0.36 *		0.40 *
Progress	5.88	3.09			7.57	2.68			6.50	3.06
Rewards	6.89	2.76			8.69	1.72	0.36 *	0.38 *	8.39	1.68
Storyline	5.58	2.88			5.11	2.59			4.53	2.84
Theme	6.47	2.55	0.42 *		7.69	2.10	0.56 ***		7.29	2.97

, the GDEs in Tesoros presented more correlations with Interest/Enjoyment and Effort/Importance in comparison to Anotatlón, which was the best-rated game. One-way ANOVA was performed to compare the GDE implementations, but no significant difference was found between the games. This table also provides the correlations with the other dimensions of IMI, in which only two GDEs (i.e., adaptation and customization) were correlated with Competence and Tension/Pressure.

3.2.2. Preference/Favorite

As concerns the preferences for the implemented GDEs, challenges and adaptation were consistently the top preference for the three games, as can be seen in

Table 4. Descriptive statistics and correlation matrix between the GDE preferences and the IMI. Bold numbers show the best rated GDEs. INT.ENJ = Interest/Enjoyment, COMP = Competence, TEN.PRES = Tension/Pressure. * indicates $p<0.05$.

GDE	Agentes			Tesoros					Anotatlón
	Mean	SD	TEN.PRES	Mean	SD	INT.ENJ	COMP	TEN.PRES	Mean	SD	INT.ENJ	COMP	TEN.PRES
Adaptation	3.44	2.16		4.06	2.46				3.89	2.26
Avatar	8.05	1.99		5.69	2.88			−0.37 *	6.10	3.16
Challenges	3.40	1.82		3.28	1.88		0.35 *	0.38 *	3.03	2.16			0.36 *
Customization	7.10	2.30		7.45	2.54				5.83	2.76
Leaderboards	5.23	2.35		6.22	2.40	0.40 *			5.89	2.14	0.38 *	0.35 *
Levels	4.59	2.30	0.38 *	4.81	2.18				4.60	1.97			0.38 *
Progress	5.76	1.69		5.43	2.17				6.15	2.02
Rewards	5.37	2.28		4.42	2.32				4.00	2.26
Storyline	5.24	3.39		7.41	2.44				7.79	2.70			−0.49 *
Theme	4.03	2.97		4.00	2.86				4.57	2.84

. However, this result for the latter GDE was a rather unexpected outcome, given that adaptation had only been implemented in Tesoros. One-way ANOVA followed by Tukey post hoc comparisons were used to identify the differences between all games. These comparisons revealed a significant difference for rewards ($F_{2,104}=3.351$, p < 0.05). Post hoc tests indicated significant between-group differences for Agentes and Anotatlón. Similarly, there was a significant difference in storyline ($F_{2,78}=6.257$, p < 0.05). Again, post hoc tests indicated significant between-group differences for Tesoros-Agentes and Anotatlón-Agentes.

4. Discussion

This study was designed to evaluate three game concepts and obtain insights into concept preference and the effect of GDEs on player enjoyment. Analysis of the experimental results yielded the following findings.

4.1. Game Concept Preference

Regarding the game concept preference (RQ1), participants preferred the Anotatlón game. This rejected our hypothesis that Tesoros would be the preferred game. The high level of Interest/Enjoyment shows that this construct was a factor in the selection of this game. The finding is supported by the players’ comments (Their comments are in Spanish, but translations are ours) on why they would play this game again. One participant argued that ‘…is more enjoyable, the concept is the same, but the part before the classification of the word makes it less tedious and more interactive (…es mas divertido, el concepto es el mismo pero la parte previa a la clasificacion de la palabra lo hace menos tedioso y mas interactivo), P2476 (A unique code is used to identify research participants and keep them anonymous)’. Another factor in choosing Anotatlón as a game to replay in the future was the high level of perceived competence. This is confirmed by the comments of participants, such as ‘…I felt challenged while playing by avoiding the obstacles, it motivated me to play again. (…me sentí retado mientras jugaba al esquivar los obstáculos, me motivaba a jugar de nuevo.), P1103’.

The second most preferred game was Tesoros, which also showed a high value of perceived Competence. This is corroborated by the qualitative results, which demonstrated that Tesoros was the game that provided a feeling of achievement for various participants. For example, one participant mentioned the following: ‘perform perfect rounds and win jumps (Hacer rondas perfectas, y ganar saltos), P3957’; another commented ‘try to solve everything correctly to gain lives and coins, and thus position myself in the top 5 (intentar resolver todo de manera correcta para ganar vidas y monedas, y así posicionarme en el top 5), P4162’. That Tesoros came in second could be explained by the fact that five implemented GDEs (i.e., adaptation, challenges, levels, rewards, and theme) were found related to the enjoyment in this game. The previous fact was corroborated by the participants’ comments, such as ‘it is more enjoyable because there is a goal: to collect gold (es más divertido, porque hay un objetivo: colectar oro), P3530’.

Finally, although Agentes did not get rated as an enjoyable game, the study offered some interesting results on it. One of these relates to the text and phrases presented in the game: participants pointed out that they liked to see a variety of words and sentences. To illustrate this with concrete comments, ‘I liked discovering the words that are used… (Me gusto descubrir las palabras que se usan…), P3772’ and ‘the infinite sentences that it shows, haha they are nice, never repeat the same ones (las infinitas oraciones que muestra, jaja son muy padres nunca repiten las mismas), P0603’. Additionally, participants considered that the game facilitates learning, e.g., ‘ They help you a lot to strengthen basic knowledge of the Spanish language that can be forgotten over time (Te ayudan bastante a fortalecer conocimientos basicos del idioma Español que pueden olvidarse con el tiempo), P0239’. These findings for Agentes might be the result of being exposed to a more significant number of words and phrases compared to Tesoros and Anotatlón. Agentes shows ten phrases per round, Tesoros five phrases (which gradually increase as the game progresses), and Anotatlón only three.

Regarding annotations, Agentes produced 54% (3742/6893) of the total annotations compared with 33% (2306/6893) coming from Tesoros, and only 12% (845/6893) from Anotatlón [18]. Designers can benefit from our experience with the three GWAPs we have developed. Agentes demonstrated a more effective game mechanic for collecting PoS tags. In contrast, Anotatlón saw a reduction in data collection due to its additional task—the car race—which was designed to increase player enjoyment. This effect is consistent with the recommendations formulated in Tuite (2014) [21], which suggested only including game mechanics that contribute to the main goal (e.g., the PoS tagging), because additional game mechanics (e.g., car races) can negatively affect players’ ability to complete the main task. Although the tasks in Anotatlón are not performed simultaneously, the time and skills required for the driving task certainly affected the amount of data collected with this GWAP. In terms of balance between annotation collection efficiency and user enjoyment, Tesoros can be positioned between Agentes and Anotatlón. This GWAP received fewer annotations than Agentes, but overall it was rated as more enjoyable. Compared to Anotatlón, Tesoros was less enjoyable but collected more annotations.

The following section discusses whether the implemented GDEs influenced the levels of the dimensions of the IMI to answer RQ2.

4.2. Game Design Elements

To answer RQ2, the present study showed correlations between Interest/Enjoyment and seven GDEs, namely, adaptation, challenges, customization, leaderboards, levels, rewards, and theme. This is consistent with previous studies that also demonstrated relationships between these GDEs and enjoyment. However, precise result comparisons cannot be made with our study, given that most previous research assessed individual GDEs or game concepts are too different.

Rewards was rated as the best implemented GDE in Tesoros and Anotatlón, and, even when it was not the most preferred, its implementation translated into a significant difference in GDE preference between Agentes and Anotatlón. Further, for Tesoros, rewards showed a high level of Interest/Enjoyment. A possible explanation for these results might be that the implementation of rewards is similar for Anotatlón and Tesoros, but not for Agentes. For Agentes, the rewards consist of points that eventually become badges. In contrast, in Tesoros and Anotatlón, the rewards are coins that players can use to buy items, e.g., unlock new categories or power-ups (i.e., jumps and shields) that help them to continue playing. In that regard, participants mentioned ‘the second [Anotatlón] seems better to me with a little more entertainment, better rewards which you can use in the same game as the shields. (el segundo [Anotatlón] me parece mejor con un poco mas de entretenimiento, mejores recompensas las cuales puedes usar en el mismo juego como los escudos.), P3772’. The results on the use of points in Agentes are thus consistent with those from previous studies, which suggests that rather than simply using a scoring system, designers should contextualize this reward to the main purpose of the game, helping to connect and better motivate players in the game [22]. Our findings are also in line with the observations by Johnson et al. [23], who showed that a higher number and diversity of rewards (as in Tesoros and Anotatlón) lead to more enjoyment.

Participants consistently rated challenges as the preferred GDE for the three games. A correlation between Interest/Enjoyment and the preference of this GDE was not observed, but a correlation between Interest/Enjoyment and its implementation for the three games was found. The qualitative analysis also corroborated these quantitative results, as participants stated that challenges made the game more enjoyable and motivating, e.g., ‘It has obstacles and that makes it more fun. (Tiene obstáculos y eso lo hace más divertido), P2274’, and ‘…there is a challenge and it is more dynamic (…hay un reto y es más dinámico), P3015’. Additionally, Tesoros and Anotatlón presented a higher correlation value between challenges and Interest/Enjoyment compared to Agentes, which might be explained by the fact that there were more challenging elements that the players could encounter, such as complete the race or perform perfect rounds. These results align with [24], who indicated that optimal levels of challenges result in better values of competence and flow. In contrast, in Agentes, the goal of obtaining more than 100 points to be validated as a human agent seemed easy, so it did not represent a challenge for the participants.

Adaptation was also rated as one of the preferred GDEs. The reason for this may be the inherent variability of the sentences that were presented throughout the games, leading participants to believe that all three games adapt their difficulty to the player as they play. However, this GDE was only implemented explicitly in Tesoros based on player performance, which led to higher values of Interest/Enjoyment in Tesoros and a positive correlation between adaptation and Interest/Enjoyment. These results are consistent with findings from [25,26], who observed increased flow state and scores when playing a game that adapted to players’ performance.

Other GDEs that had a particular effect on the game concept preference were the avatar and storyline. For Tesoros, the avatar implementation showed a correlation to Effort/Importance, and avatar preference correlated to Tension/Pressure. These results might have been due to people’s empathy for the avatar, so they tried a little harder to help him, e.g., ‘one feels enthusiastic helping the Monitor (uno se siente entusiasmado ayudando el Monito), P0343’ and ‘Seeing the avatar move may motivate to keep playing… (El ir viendo cómo se mueve el avatar quizá motiva a seguir jugando…), P2918’. This aligns with the study of Birk et al. [27], which claimed that greater identification leads to an increase in intrinsic motivation. Storyline presented a significant difference in GDE preference between Tesoros-Agentes and Anotatlón-Agentes, and a negative correlation with Tension/Pressure for Anotatlón. The first result can be explained because the storyline in Agentes was developed explicitly, including the participant as an agent inside the story. Participants were aware of this and rated it positively compared to Tesoros and Anotatlón, in which no storyline explained why they have to collect coins or win a race. The second result could be because participants who experienced less Tension/Pressure rated the storyline as the less preferred GDE. In contrast to previous studies [28,29], the implementation of this GDE in Agentes did not increase enjoyment as much as desired.

Levels, leaderboards, and customization also provided positive outcomes in the different constructs. On the contrary, progress did not yield results. This may have been because players did not perceive progress, or they could associate it with other GDEs, such as levels. Taken together, the results suggest that there is indeed an association between GDE implementation and concept preference. Figure 4 shows a visual summary of the results. The quadrants represent each of the constructs of the IMI. The circles represent the game concepts, and their size indicates the values of the IMI constructs for each game. The squares show the GDEs that had an impact on each of the constructs of the IMI, arranged in order from the highest to the lowest level of influence, as indicated by the colored triangle. As seen, Anotatlón obtained the highest Interest/Enjoyment and Effort/Importance scores. Tesoros showed the highest feeling of Competence, and Agentes presented the highest level of Tension/Pressure. The arrow from Competence indicates that this construct was the only one that predicted a player’s decision to play again, in this case, Anotatlón. Concerning the relationships between the GDEs and the IMI constructs, Tesoros showed the highest number of relationships with Effort/Importance and Interest/Enjoyment. In second place was Anotatlón, with GDEs mainly related to Interest/Enjoyment. These results suggest that preference for a game is influenced by the overall game concept and the perception of the different GDEs implemented. Furthermore, the GDEs demonstrated their ability to influence positive or negative outcomes in the satisfaction of basic needs.

4.3. Limitations and Future Work

Due to COVID-19, the experiment was conducted remotely to maintain social distancing. The games were designed to be played online, so a remote study was a natural extension. However, some observations during playing sessions may have been missed, which could have provided additional insights into participants’ experiences. Another limitation is a possible bias in the data due to the differences in time played, which could have affected the results. Some players spent 10 min, while others spent 20 min, mainly because the sessions were supervised or unsupervised. In the unsupervised session, it could not be determined whether players explored all game features and played effectively for 10 min. The analysis between time and enjoyment showed a tendency to increase enjoyment as the participant played longer. However, as the time variable was not controlled, it would be interesting for future experiments to control it and explore whether it is a factor in increasing enjoyment.

Follow-up work included improving the games based on the study results and releasing them to the crowd. Once they were available and freely accessible, another study was conducted. This study was less controlled than the one presented in this paper, as it only required informed consent, and the questionnaires were presented at two points (i.e., at the end of training and the end of one level). The goal was to monitor long-term player enjoyment and to continue collecting PoS tags. From a linguistic point of view, additional data analysis evaluated the accuracy of the annotations collected, and these results are reported in [30]. The study highlighted the effectiveness of using GWAPs to verify PoS tags and improve accuracy rates. It also showed that factors such as the education level, field of study, and geographical background of the GWAP players had a significant impact on the accuracy of PoS tagging. Obviously, a linguistically proficient annotator would provide more PoS tags per time unit than a GWAP player (with or without linguistic knowledge). However, in the long term, the scale of the corpus may make the tasks expensive. On the other hand, linguistic tasks such as PoS tagging can be tedious and may not appeal to participants unless they incorporate gamification. For these reasons, GWAP approaches based on the principle of crowdsourcing are promising, as they focus on scalability and enjoyment rather than the efficiency of one particular contributor.

5. Conclusions

This study discussed the evaluation of three games—Agentes, Tesoros, and Anotatlón—that focus on the linguistic task of revising and correcting the pre-tagged PoSs of a corpus. Although many GWAPs have been designed before, including games for PoS tagging, this work aimed to ground the design decisions to explore the effects of GDEs on enjoyment and concept preference. The evaluation results showed that, in linguistic task annotations, the best rated games in terms of enjoyment were Anotatlón and Tesoros. The former was the most enjoyable game as a whole concept, and the second showed enjoyment through its different implemented GDEs, such as levels, challenges, and rewards. The findings reveal that the GDEs contributed to positive outcomes for enjoyment. Therefore, we encourage game designers to be inspired by the findings of our paper and to explore similar or alternative GDEs in the context of their GWAP or other types of serious games. At the same time, we want to encourage designers and researchers to thoroughly explore the effect of these game concepts and GDEs in different contexts, to prevent generalizations that lack demonstrated benefits.