Engaging Children in Story Co-Creation for Effective Serious Games

Abstract

Despite a growing interest in player-centred methods for serious games, little is known on how to achieve this goal in practice when prospective users are children. Foundational questions remain unanswered, such as to which design dimensions children should contribute, and how and when they should be engaged. This paper presents the methods and results of two studies that inspired Skies of Manawak, a game for developmental dyslexia remediation. The first study engaged 60 children (age 8–13) in 15 ideation workshops to highlight the story and aesthetics of the game. The second study engaged 258 children (age 8–11) in the comparative evaluation of the game demo with a commercial cognitive training system. The results proved the importance and complexity of the early involvement of children in design. Children strongly appreciated the demo, particularly the story their peers contributed to shaping. However, this story deviated from their desires in several critical dimensions. It had to reconcile gender stereotypes and the violence embedded in their narratives with the game’s purpose. An apparent conflict between designers and children’s values emerged, supporting the idea that children’s engagement in serious game design requires effective mediation to avoid compromising the purposes they intend to achieve.

1. Introduction

The encounter between human–computer interaction (HCI) and game research [1,2] has brought to the foreground a growing interest in a range of player-centred design approaches for serious games [3]. Several of these studies involved children in research projects that led to informative user requirements, exciting proof of concepts and profound methodological considerations [4,5,6]. Nevertheless, these studies also exposed significant difficulties concerning the real contribution children can provide to serious game design [6,7]. It emerged that many ideas contributed by the children were irrelevant to the game’s functional objective or were very general, difficult to implement and limited in creativity. Consistently, a meta-analysis comparing the effect of different levels of user involvement on serious games’ effectiveness did not support the benefit of user participation in design [8]. On the contrary, the most effective games involved the player only in the summative evaluation of finished products.

Another significant limitation of player-centred research in HCI is the lack of product development to test methodological assumptions. Most of the studies discussed in the literature stopped at the conceptual design phase [4,5,6] or presented formative evaluations of prototypes with small user samples [7,9]. Consequently, they addressed the aesthetics of the serious game, but they could not test its effectiveness in reaching the functional goal for which serious games are designed nor in maintaining engagement. From this perspective, this paper presents a unique possibility for reflecting upon a successful case study moving from requirements to successful development. The case study was motivated by the need to maintain engagement in cognitive training for developmental dyslexia remediation and led to the design of Skies of Manawak (SoM). Developmental dyslexia (DD) is a neurological characteristic that makes reading or writing tasks complex. At the time of writing this paper, SoM consisted of 12-h gameplay that had undergone extensive longitudinal evaluation in schools. It demonstrated robust clinical validity and strong engagement over a two-month period [10].

We attribute the success of SoM to its design approach that engaged several stakeholders, including children, psychologists, game designers and developers, over five years. The paper documents the methods and discusses the results of two studies that inspired and improved the design of the game story and its aesthetics. The first study engaged 60 children (age 8–13) in 15 ideation workshops. The second study engaged 258 children (age 8–11) in the formative evaluation of the game demo. The results identified preferences and desires of children for game design, exposed significant conflicts between designers’ and children’s values, and supported methodological reflections on how to involve children in serious game design. The paper is structured in six sections. Section 2 presents a literature review on player-centred game design and developmental dyslexia as the basis for the research. Section 3 introduces the design case study framing a player-centred focus in the multi-dimensional space of game design. Section 4 and Section 5 present the studies and discusses their results. Finally, Section 6 closes the article by providing directions for future research.

2. Related Work

The term “serious games” refers to a wide variety of systems. They encompass numerous application domains, such as the military [11], healthcare [12], emergency management [13] and education [14]. Despite their resounding success (see [2] for a review in HCI and [15] in education), serious games have proved challenging to define, and many interpretations have emerged through the years [15,16,17,18]. To reconcile them, Marsh described serious games “along a continuum from games for purpose at one end, through to experiential environments with minimal or no gaming characteristics for experience at the other end.” [15] (p. 63). Games for purpose, such as SoM presented in this article, emphasize the entertaining and functional components of the experience at the same time.

The effectiveness of serious games has been proved in several contexts [8,17] and explained by different disciplinary perspectives. Game research has explained that games can teach, inform, persuade and entertain because they possess the power of procedural rhetoric [19]. This form of rhetoric describes the ability to make arguments “not through the construction of words or images, but through the authorship of rules of behavior, the construction of dynamic models” (p. 29). Rule-based representations and interactions give video games a unique persuasive power that has shown considerable potential in politics, advertising and education. The psychological perspective has argued that some game mechanics, such as the dynamic adaptation of game elements and in-game rewards, have the potential to increase engagement and performance by supporting intrinsic motivation while satisfying the needs for autonomy, competence and relatedness [20,21,22,23]. Finally, the educational perspective has linked gameplay to experiential learning [24,25]. While playing, people experience fictional worlds with all their senses and act and reflect on them; these processes promote meaningful learning [24].

2.1. User-Centred Game Design

Several classifications attempted to define the different roles a child can play in ICT design. Among them, this paper builds upon the proposal by Alison Druin [26], which differentiated among roles based on how adults relate to the child, the design phase, and the researcher’s goals. According to this view, children can be users when they use a product designed by adults, while the latter observe, ask for comments or test for skills and employ this data for improving their understanding of the role of technology on child behavior. Children can also be testers when they use a prototype, while adults observe, ask for comments or test for skills and modify the prototype accordingly. At a higher level of involvement, children can be informants, when they take part in different phases of the design process based on the needs of adults who are committed to incorporate their opinion in the artifact. Finally, children can be co-designers when they are equal partners in the design.

Human–computer interaction research on game design has encompassed different demographics in a variety of contexts and within a range of active user’s involvement. For example, related work included young children in the kindergarten [27], primary school children [5], teenagers in pupil referral units [7], and young wheelchair users [6]. The majority of these studies applied constrained forms of participation where children played the role of testers or informants [26], providing feedback to ideas created by designers or inspirations to their creativity [3,4,28,29]. Attempts to involve the children as co-designers [26], have proved difficult and required long term engagement with tools and techniques suitable to the child developmental stage and the context of the study [5,6,27]. Recurrent problems included low creativity, unfocussed or irrelevant ideas and violent content, which contrasted with the educational objective of the game. The problems were attributed to the lack of domain knowledge and the limited game-design literacy of children [3].

Research on game design with children can be roughly differentiated into two positions. One claims that children should be constrained to the role of informants [26], which the authors imagined as mythological muses [3,29]. Consequently, their preferences and desires should inspire the designers, but children should not be involved in design decisions. The other position attempted to put children in the role of designers [26], fostering their literacy with structured methodologies [5,6,7]. These two approaches reflected different concerns. The former was design-oriented, emphasizing the hedonic quality of the game; the latter focused more heavily on their educational outcome. Following the recommendation by [30] that children should be involved in the design of educational technologies as informants rather than co-designers [3], this paper belongs to the first stream of research. Its design-oriented focus suited the space of a game for purpose, and there is evidence that leaner user engagement produced more effective games for healthy lifestyle promotion [8].

2.2. Developmental Dyslexia

Globally, developmental dyslexia (DD) is the most common neurodevelopmental disorder, affecting almost 1 out of 10 children [31]. Children with DD struggle to read fluently or correctly despite having an IQ that is in line with their peers, having a typical education, and not having a history of neurological or psychological disorders. Dyslexia impacts students’ academic achievements and can affect several adverse life outcomes [32,33]. However, early intervention can mitigate these outcomes, and a number of phonological and orthographic programs are available, including some game-based proposals [34,35].

SoM has moved out of the traditional symptoms-based approach and entered the domain of cognitive training of executive functions (EFs), which is a collection of general-purpose mechanisms that direct various cognitive abilities, such as working memory, cognitive flexibility and inhibitory control [36,37]. Its foundation is grounded in leading-edge research on clinical and experimental psychology. In recent years, evidence has emerged that there is a connection between dyslexia and some deficits in the executive domains [38,39,40,41]. Due to brain plasticity, the performance of several EFs can be improved by the systematic repetition of standardized exercises. For example, working memory capacity can be trained by telling the player a list of words and asking them to repeat the ones that shared some characteristics (e.g., they belong to the same category). Likewise, attentional control can be trained through the selection of task-relevant information and simultaneous suppression of task-irrelevant information [42,43,44].

In adults as well as children, cognitive training has been found to improve working memory, attention skills, inhibition skills, problem-solving abilities, and cognitive flexibility [45,46,47]; for a recent review, see [48]. More importantly, it has provided encouraging results on dyslexia remediation with a sample of Italian-speaking children (8–14 years old) [49]. After regular training over five weeks with 30 min of daily exercises, these children showed significant improvements in attention and working memory, inhibition ability, visual–motor integration skills, cognitive flexibility, and fluid intelligence. In relation to the literacy of the participants, significant improvements also resulted in the accuracy of reading.

Despite its many potential advantages, the provision of cognitive training requires a strong commitment to the user. Dropout [50] is a significant concern because existing tools are not particularly engaging [51]. SoM aimed to decrease dropout by providing an adequate level of motivation [21,52] and increasing resistance to frustrations to enable broader improvements [53]. Furthermore, games can provide challenging experiences that demand players to be continuously committed while providing appropriate feedback to create a sense of competition [54,55]. For example, a recent review suggested that digital games can be used effectively as gameful approaches to language learning [56]. There is also evidence that commercial off-the-shelf action video games (but not other types of games) can improve reading efficiency, even if they do not include any direct orthographic or phonological stimulation [57,58,59]. The effect was related to the mechanics of action video games, such as presenting multiple, unexpected and rapidly moving targets, constant speed, and high sensory–motor demand, which stimulated attention and related EFs. In addition, Franceschini and colleagues recently confirmed the causal link between play and cognitive enhancement [60]. In fact, two crossover randomized controlled trials revealed that funnier and more engaging games (i.e., games with action-based mechanics) improved visual perception and reduced reading disorders. Moreover, the cognitive and reading improvements correlated with fun in the played game. This is in line with previous research on adults, which investigated the impact of fun on cognitive enhancements in video gameplay [61].

3. Skies of Manawak

The purpose of the design case study was to develop a training program to mitigate dyslexia that enhanced the emotional and motivational elements of cognitive treatment [62] at the same time as maintaining clinical validity. The design philosophy was strongly player-centred acknowledging that games are inherently different media when considered from the player or the designer’s point of view. According to the MDA (mechanics, dynamics, aesthetics) framework [63,64], the player prioritizes the Aesthetics, which are the emotional responses induced by gameplay (e.g., in Super Mario Bros, the timed missions generate challenges). The designer prioritizes the Mechanics, which define the rules through data and algorithms (in this case, the timer that shows how long a level has left). The two perspectives are mediated by the Dynamics, the emerging rules established at run time (the timer generates pressure). These rules are affected by the adoption of a lusory attitude [65] from the player, who spontaneously decides to overcome unnecessary obstacles set up by the mechanics.

The project was developed around an extension of the elemental tetrad proposed by Schell [66] to describe the basic elements of a video game a designer should consider. The tetrad is composed of story, aesthetics, mechanics and technology. In a video game, the story provides the basic elements for understanding the objectives and describing the events (e.g., in Super Mario Bros, Princess Peach has been kidnapped and Mario must rescue her by fighting through castles). The aesthetics are the visual and acoustic elements that the player perceives (pixel graphics and 8-bit sound effects). The mechanics are the procedures and the rules of the game (the player can move horizontally and jump, wins if the character reaches the end of the level and loses if touched by an enemy). The technology concerns the apparatus with which the player interacts (e.g., the Nintendo Entertainment System and a television in the first release of Super Mario Bros). Serious games add an extra dimension to this intricate design space because they have a purpose and require a pentad to be described (see Figure 1; [67]). The five dimensions are strongly interrelated, and serious game design involves delicate negotiations in which each decision affects the others and ultimately the game’s outcome.

The project unfolded over 5 years and involved designers, domain experts and players. In this article, we focus on the two initial years, since our main aim is to reflect on how children can be involved in the design of an effective serious game. All the different stakeholders contributed to the process, but with different levels of involvement [26] and focus on different game dimensions (Figure 2). Particularly, the story and aesthetics were elaborated in collaboration with the players. In contrast, the purpose, the mechanics and the technology were developed in collaboration with the domain experts. Figure 3 illustrates a schematic representation of the design process which took place following an iterative process [68]. The development team was composed of two PhD students in Computer Science and a visual artist, alongside a PhD student in Cognitive Science with clinical expertise on dyslexia. A steering committee of four senior researchers in HCI and psychology supervised the project. A group of 16 clinical psychologists contributed requirements, formative and summative evaluation feedback. Children were involved at three stages in the design: ideation workshops, formative evaluation and summative evaluation. This paper focuses on the two initial stages, which provided the design foundation in terms of children requirements. Despite being relatively old, the studies addressed preferences and desires of children, which are unlikely to have substantially changed in time. In addition, the more recent summative evaluation of the game demonstrated robust clinical validity and sustained engagement [10], showing how the method reported in this paper is conducive to successful design.

The project was granted ethical approval by the University of Trento (Italy). The experimentation with dyslexic children was conducted as part of the clinical activities of the [omitted for blind review]. Following the principles of the Declaration of Helsinki, all activities included the previous collection of written informed consent by parents or guardians. The consent form clarified the purpose of the research, the fact that audio and video recordings will occur, and the possibility of leaving at any time. Moreover, before each study, the researchers explained their structure and goals, asking the participants to provide verbal assent of their willingness to participate. Finally, the researchers monitored children’s behavior for any sign of distress during the studies.

Stages involving designers and domain experts are in blue; stages involving designers and players are in gray; stages involving only the design team are in white.

4. Ideation Workshops

At the beginning of the project, the team envisioned a high-level game scenario with multiple mini-games built into it. They represented gamified versions of the cognitive training exercises. Based on this vision, the player engaged meaningfully with the mini-games while following a compelling story. This structure allowed the designer to maintain full control over the cognitive stimulation induced by the mini-games, ensuring clinical compliance while maintaining an engaging experience. Ideation workshops were used to capture anticipated player experiences that could inspire the story and the aesthetics of the game [4,69].

4.1. Participants

Fifteen workshops were organized involving 60 children (8–13 years old, 37 M and 23 F) as informants (

Table 1. Participant’s characteristics.

Date	Participants	Groups	Age	Location
6-May-2015	4 (4 M)	1 (pilot)	11	university
11-May-2015	9 (6 M, 3 F)	2	8–10	clinic
12-May-2015	24 (11 M, 13 F)	6	9–10	university
13-May-2015	11 (8 M, 3 F)	3	11–13	clinic
18-May-2015	12 (8 M, 4 F)	3	11–13	clinic

). With the aim of supporting inclusive design, this sample included children with dyslexia and typically developing children [70]. Participants were recruited from three classes of a school and the ODFlab clinic. They worked in groups composed of three to five peers and at least one researcher. Groups were formed by a therapist who knew the children or by their teacher before the study.

Parents’ and children’s consent were sought, following the procedure previously detailed. The workshops were hosted in different rooms of the Department of Computer Science and the clinic. All spaces were set up as a child-friendly environment, which created a pleasant atmosphere composed of informal elements (friendly attitudes of researchers), familiar elements (playful theme) and professional elements (empowerment of the child to the role of designers). The procedure was inspired by the studies of Moser and colleagues [4,69] adapted to the design space and refined in a pilot session (n = 4).

4.2. Procedure

Each workshop lasted two hours. The session began with a 10 min presentation of the members of the team and the goals of the activity. A strong emphasis was put on valuing the children’s work, which was presented as the conceptual design of a video game that could be enjoyed by their peers. Any information about the functional goal of the game was omitted since the focus was on the entertaining aspect of the game. Next, the researchers outlined the schedule of the activities, taking care not to anticipate the prototyping phase. The first phase consisted of the creation of a design document (40 min). Differently from Moser [4] and in line with the constraints of our design space, the researchers provided children with a semi-structured outline. It included three elements: the character, the obstacles and the goal. Consequently, the children were invited to develop their stories around the following questions: (1) who? (the protagonist/s of the story); (2) where? (the spatial setting of the events); (3) when? (the temporal setting of the events); (4) what? (will the protagonist/s run into someone or something, e.g., enemies and companions?); and (5) why? (the reason for the events).

Following the methodology presented in [71], ideation consisted of an individual phase followed by group work. Initially, the children outlined their ideas independently in the form of text and drawings on several A4 papers. Then, each of them presented one idea to the group while a researcher summarized it on an A1 poster, using the framing questions as the basic structure (Figure 4). Finally, the researcher used the poster to facilitate a group discussion to scope a single idea and eliminate any dysfunctional element in the game. The idea consolidation process followed different strategies entrusted to the participants. This new proposal developed—in some instances—from combining previous ideas or expanding an already intriguing idea; in other cases, it evolved into a wholly new idea only marginally inspired by previous ones.

The second phase consisted of low-fidelity prototyping (30 min). Each group received a set of materials (modelling clay, Lego^®, cardboard and PlayMais^®) to enact a scene of their game (Figure 5). After a short break (10 min), children started film shooting (30 min). Videotaping was performed by the researcher as each group simulated a few minutes of gameplay. The children were free to choose the prototyping material, as well as how to structure and simulate the scene, while the researcher provided technical support.

4.3. Results

The posters reporting the final ideas were transcribed; notes collected during field observations and video analysis were used to enrich their content. The data were analyzed through several iterations of inductive and deductive thematic analysis in order to identify patterns of meanings [72]. Deductive analysis was supported by a selection of the lenses of Schell [66], including competitive and cooperative mechanics (Lenses #36 #37, #38), the type of challenge (#31) and reward systems (#40). These lenses were selected following the initial phase of familiarization with the data. Double coding was performed on the entire database, and divergences were reconciled by group discussion.

Almost all videos (87%) featured stories that incorporated elements of the hero’s journey, a path of personal growth that eventually leads to victory. All of them had complicated plots, involving a series of characters and complex missions, and often mixing real and fantasy elements. With only two exceptions (e.g., cooking games), the story concerned fighting enemies (almost always for defense rather than attack). The fighter took different shapes, although, in the majority of the cases, it was a strong and smart adult (always male), and only in three cases a male child of their age. The hero most often played the role of a policeman or a soldier, a powerful king or a scientist who had to fight a pandemic—and even Obama, who rode a centaur.

The violence in these games was well-intended, and the hero was tough but fair. In three cases, he started the game as a thug or a gangster but finally redeemed during the game. Only in one case, the main character was a criminal, and the final goal of the game was to steal the Freedom Statue, which contained a large treasure. Female characters were rare and could play the role of supportive allies (who fell in love with the hero), cruel enemies (who fought the hero) or trophies to be saved and finally married. For example, in one game proposal, the children envisioned two female characters who, alongside a giant, helped the smart male final boss (the enemy to be defeated in the decisive battle in order to win the game).

In almost all cases, the environments included fantasy elements (i.e., magical portals, enchanted castles) or (92%), and they were usually set in a distant future (45%). Interestingly, when envisioning real worlds, the children often made reference to large American metropolises, in sharp contrast to the medium-sized city surrounded by mountains where they lived. Several games included allies and companions (39%), which took an array of shapes ranging from mythological creatures (dragoons, centaurs or giant eagles) to robots, and human friends with specific skills which were useful for progressing in the adventure (e.g., a policeman had an engineer friend who built a space-time machine). Real animals, specifically dogs, were also popular.

Some game mechanics were mentioned and described at a high level of abstraction. Recurrent elements were the use of crafting systems, upgrades and rewards, mainly inspired by Minecraft (37%). Obtaining items, armors, and weapons would allow the hero to advance the adventure. In addition, shooting was frequent, although most targets were not humans (e.g., aliens and zombies) or were dangerous enemies. Almost all the games were competitive, the opponents were non-player characters controlled by the computer, and most games incorporated a final boss. Three multi-user games were proposed. Only one of them incorporated some cooperative dynamics, envisioning groups of users who would fight together against non-player characters.

An age-related difference emerged. Older children (11–13 years old) created games with more detail in comparison to younger children attending elementary school (8–10 years old). Similarly, the violence and realism of the game tended to increase as a function of age. Regardless of age, children frequently used existing video games as boundary objects to define a common ground that could support co-creativity. Overall, the children were very engaged throughout the workshop, although they demonstrated increasing enthusiasm in the prototyping and film shooting phase than in the initial phase aimed to create the design document. Participation was good, and almost all children took an active role in the study. Equal turn-taking was ensured by the procedure of alternating individual and collective activities and the moderators.

4.4. Discussion

These results share important similarities with related work, thus supporting the consolidation of knowledge on children’s preferences and desires for game design. Similarly to the study by Gerling and collaborators [6], children mainly focused on the game story. Their narratives contained general themes of interest with only marginal connection to the other constituent game dimensions [66]. Most stories were described at a high level of generality, hinting at complex ideas with partially elaborated and often confused goals. These stories almost entirely disregarded the procedures and rules to achieve the goals. Therefore, their implementation as games would require a substantial translation effort to create the right mechanics underlying the expected experience [63].

The game genre (hero’s journey), the presence of companions, and the use of crafting systems were important elements of the proposals and had a very similar pattern of occurrence as compared to the result reported by [4]. Similarly, the same age divide was confirmed. Many of the ideas generated by the younger children (8–10) dealt with fantasy worlds, whereas older children (12–15) tended to be more realistic. Violence was a common theme [73] across age, but the act of killing became more explicit in the ideas generated by the older children [4]. This divide is coherent with the game classification proposed by the Entertainment Software Self-Regulation Body USK, which progressively disclose violence and realism to children as they grow.

In line with [4], children’s storytelling contained important elements of fantasy and adventure. However, the relevant emphasis on sci-fi was unique to our study, whereas in [4], design ideas focused on contemporary cities or famous locations. This inconsistency may be due to the framing of the previous study, which required children to think of physical spaces and foreign visitors. Moreover, children in our workshop may have been influenced by the recent release of a popular sci-fi movie. Another commonality with previous work is the frequent occurrence of design fixation [7], which often resulted in stereotypical rather than creative thinking. An illustrative example of the influence of the prototyping tool on children emerged during the pilot study. In the ideation phase, children had imagined an original and articulated mythological creature as the final boss. However, during the prototyping phase, this creature was immediately transformed into a dragon due to the affordance of a Lego figure available in the prototyping kit. Consequently, for the final study, we provided only low-structure objects which tend to stimulate fantasy play from early childhood [74]. Children’s ideas were constrained within the game examples they had encountered before, and novelty in terms of game design was generally low [3]. Consistently with previous work, the enactment of violent behavior [3,7] and strong gender stereotypes in storytelling [75] were common. Most of the heroes were tough male fighters, and the few female characters were relegated to loving companions or naughty witches. The gender stereotype affecting the video game world was also evident in the recruitment of the sample. The majority of the participants who chose to enrol in the game through the clinic were male (69%), a selection bias already emerged in other studies [3].

5. Formative Evaluation

The results of the ideation phase were used to inspire the development of a playable 30-min demo, corresponding to one-day training in a typical treatment. It contained an introduction to the mechanics, three scenarios with different interactions and slight variations of the aesthetics, and four mini-games (to train working memory, divided attention, inhibitory control and attentional control). Given the age divide identified in the ideation workshop, the target sample was restricted to primary school pupils between the age of 8 and 10 years old, and their preferences were given increased attention. The game was inspired by the sci-fi and hero’s journey genre. The plot was set on Manawak, a planet in another solar system under attack by outer space invaders, and an engaging world was carefully elaborated through literary, symbolic and visual languages (Appendix A). To save the planet, the protagonist had to face a set of challenging missions (i.e., the cognitive exercises alternating between action and puzzles). He/she (the player could choose between a male and a female avatar) moved by flying on giant creatures (inspired by the many dragons and mythological creatures enacted by the children) in a surreal world in which continents floated hundreds of meters above sea level. The two-dimensional graphic rendering generated an aesthetic that resembled that of an illustrated book. The embodiment of the protagonist was carefully designed to avoid any gender and racial stereotypes (see Figure 6 for an example). Similarly, although the main aim was a fight against the invaders, it was decided to avoid any form of direct violent behavior, and instead favored strategic action.

The study took place during the Dyslexia Awareness Week 2015 [76]. Its purpose was to evaluate whether the players appreciated the game and to collect feedback on possible improvements. The emphasis was on entertainment and the game’s potential to engage the users in cognitive training. The evaluation of the functional goal (i.e., EFs’ reinforcement) would have required a much longer gameplay time [77], which could not be supported by the status of the product. BrainHQ, a leading computing tool supported by extensive neurocognitive research (for a review, see [78]) was used as a baseline. BrainHQ offers 29 exercises that train attention, processing speed, working memory, social skills, navigation and fluid intelligence. Each exercise is enriched with a few gamification elements, such as badges and points. Moreover, each exercise adapted its difficulty so that each user could always train at the optimum level where they are most likely to see improvement.

5.1. Participants

The evaluation was conducted in five primary schools across different cities of the region. A total of 258 pupils (127 M, 131 F) of 12 classes (grade 3 and grade 4) participated in the study, mainly as testers and partially as informants. Therefore, the sample included a range of abilities, skills and social backgrounds. Children were aged between 8 and 11 years old (see

Table 2. Participant’s information of the evaluation at the primary schools.

Date	Participants	Classes	Age
05-Sept-2015	87 (41 M, 46 F)	4	8–11
06-Sept-2015	43 (19 M, 24 F)	2	8–11
07-Sept-2015	45 (22 M, 23 F)	2	8–11
08-Sept-2015	44 (26 M, 18 F)	2	8–11
09-Sept-2015	39 (19 M, 20 F)	2	8–11

).

5.2. Procedure

Each session lasted an hour and a half and was divided into two parts: (1) gameplay (60 min) followed by a questionnaire (15 min) in the computer lab of the school; (2) focus group in the classroom (15 min). Approximately half of the children used both SoM and BrainHQ (N = 142); the rest of the sample used only SoM. The presentation order was counterbalanced across classes. The questionnaire was administered immediately after each gameplay session and filled in by each child individually. A minimum of two researchers and one or more teachers attended each activity.

The questionnaire (Appendix B) included an adaptation of the Short Feedback Questionnaire (eSFQ) created for the rapid assessment of game experiences [79]. Two reasons led to the selection of this instrument. A first advantage is that it was designed specifically for children, as opposed to other questionnaires such as the Game Experience Questionnaire [80], which would have been too complex for the audience. In addition, the eSFQ is relatively short, which was considered a positive factor considering the short time available for the evaluation and the attentional demand required to the participants to fill it. The questionnaire consisted of 11 items. They included the “funometer”, a captivating Likert scale originally proposed by [81] and frequently used in research with children; multiple selection items where children would tick the adjectives that better described their experience; and a Likert-scale with two questions related to Flow, two to Challenge, two to Curiosity, and one to Aesthetics [80].

During the focus group, the children played the role of informants and reflected on the game and their experiences. The discussion started with the presentation of a large poster containing four questions: what did you dislike? What did you like? What would you change? What would you add? Each participant had a block of Post-it^® notes that were used for about ten minutes to write personal answers. These notes were finally clustered in the poster during a group discussion moderated by the researchers.

5.3. Results

All post-it notes were transcribed and analyzed using a selection of the lenses of game design [66] as a reference coding. Field observations and questionnaire data were used to enrich the analysis. Overall, the observational data demonstrated a high level of sustained engagement with both systems, which was clearly reflected in the questionnaire scores. Regarding the comparison between SoM and BrainHQ, the results were generally in favor of the former. The funometer recorded a mean score of 4.57 out of 5 (SD = 1.15) for SoM against a score of 3.99 (SD = 1.42) for BrainHQ. Positive adjectives were selected more often to describe SoM than to describe BrainHQ and the opposite for negative adjectives (Figure 7). However, three adjectives displayed a reverse trend, namely easy and intuitive, among the positive items and difficult among the negative.

The Likert-scale questions confirmed the general preference for SoM: the mean on the seven items was 4.07 out of 5 for SoM (SD = 1.23) and 3.45 for BrainHQ (SD = 1.55). Separate 2 × 2 MANOVA with the item type and group as within-subject factors were run. For Curiosity, a significant main effect of group was found (F(2,140) = 13.739, d = 0.47, p < 0.05) indicating higher curiosity and interest for the development of the story for SoM compared to BrainHQ. These results were confirmed by separate univariate analysis on the single items: “I would like to know more about the story” (p < 0.05), “While I was playing, I was curious about what could happen next” (p < 0.05). Interestingly, a significant tendency was also found for Flow (F(2,140) = 14.001, d = 0.24, p = 0.06) highlighting a higher level of immersion when playing SoM in comparison to BrainHQ: “I was focused playing the game” (p = 0.07); “When I was playing, I stopped paying attention to my surroundings” (p = 0.06). Both tools were considered equally challenging (MANOVA for Challenge: F(2,140) = 9.842, d = 0.09, p = 0.10): “I had to work hard to play with it” (p = 0.189); “Playing this game was a nice challenge” (p < 0.05). No statistically significant differences were found between the two systems in children’s evaluation of the Aesthetics (Item: “I like the aesthetics” (p = 0.07).

According to the focus group comments and behavioral observations, the game difficulty was a critical determinant in SoM. It was observed that some players explored the tutorial without paying attention or understanding the information related to the game controls [73]. In particular, some children had difficulty in understanding what keys to use to control the game (e.g., where the enter key was located on a keyboard). As a result, they were unable to advance in the game without the intervention of the researchers. Most of the comments from the focus groups concerned themes similar to the ideation phase. The majority of the players (78%) desired more action elements, such as large quantities of weapons and enemies with diverse aesthetics and skills. Approximately half of them (43%) proposed to add a final boss to the story. They also expressed a desire for greater customization, such as a wider range of characters (86%) and the addition of new features based on crafting, upgrades and rewards. However, these mechanics were generally described at a high level of abstraction, with little consideration on how they should fit in the game plot. Some players also expressed the desire to modify the embodiment of the protagonist, adding more masculine/feminine features (42%). Almost the same amount of people proposed the inclusion of explicit violent content in the game (46%). Finally, around a third of them (35%) asked for the addition of an animal companion of the main character.

5.4. Discussion

Overall, the demo of SoM proved capable of sustaining attention for 30 min, which corresponds to the daily amount of cognitive training a child should undergo. In general, despite its demo status, the game was preferred to a commercial product as regards entertainment and fun. The main difference between the two systems regarded the Curiosity dimension of the feedback questionnaire [80]. While BrainHQ consisted of separate mini-games which were gamified with badges and points, SoM was designed as a game with a unifying and meaningful story that proposed adaptive game modules linked through a core action-based game. Indeed, the integration of gamification into a narrative context contextualized activities and characters in the game and gave them meaning beyond simple point-scoring and achievement-seeking [21,53]. In this regard, several studies have suggested a crucial role of a meaningful narrative to provide engagement, subsequent experiences of autonomy in regard to task meaningfulness, and ultimately to drive the player to keep coming back to train [82,83,84].

The main limitation of SoM regarded the game difficulty. Some children found the system too difficult to use. This problem may be due to the specific challenges of cognitive training (the D of the MDA), the personal characteristics of the player (the A of the MDA), and the design of SoM (the M of the MDA). On a task level, cognitive training implies a certain element of difficulty. As a matter of fact, the option “difficult” was selected almost with the same frequency also with respect to BrainHQ (−5%). Nevertheless, a larger sample of pupils (+10%) selected the item “easy” to describe BrainHQ than to describe SoM, probably due to the increased complexity introduced by the hero’s journey genre as compared to self-contained gamified exercises. On a personal level, interindividual variability has to be expected in cognitive training [54] and can be counteracted with careful automatic balancing. A personalization module was already implemented in the game demo but proved ineffective due to the brevity of the evaluation session. On a design level, the most critical factor affecting SoM was the tutorial, which several children skipped too quickly as they were eager to engage in action. However, the tutorial contained fundamental information about how to use the game controls, which had to be given verbally by the researchers. Speech output was available, but it could not be used due to the lack of headphones and the group-based setting of the evaluation. Due to a lack of interest in written information [73], it was extremely difficult to manage parts of the game that required a thorough understanding of the rules.

Information obtained from the evaluation was a vital component of the new design iteration. As already highlighted in [6], designers had inappropriate expectations of the players’ ability, who manifested their limitations during gameplay. A strong simplification of the tutorial and the controls were given priority. The instruction messages were simplified, and graphical information was added to help the children to identify the keys in the keypad. Moreover, the developers fine-tuned the training difficulty balancing algorithms to achieve better personalization. In parallel, following the children’s suggestions, they built the upgrade system, which was only sketched in the design document, adding some general proposals elicited by the children. However, not all the suggestions coming from the children were implemented in the game. Some of them needed mitigation based on the design context and its surrounding values. A conflictual example was related to the request of stereotyping the character embodiment as the “muscular hero” and the “nice princess”. This suggestion was discarded, but the team designed a more extensive roster of embodiment for both male and female characters (Figure 6). The player could choose between a few variations based on the aesthetics of their hair and their clothes. Similarly, other requests concerning the use of violent content were moderated. Some aerial battles were introduced, yet all the enemies were non-organic entities, reducing the violent act of killing to dismantle lifeless objects.

6. Conclusions

This article suggested that the design of serious games is an art [15] that requires the harmonious development of five constituent dimensions, including the story, mechanics, aesthetics, technology and purpose. Appropriate balancing of these dimensions can be supported by children’s involvement in the early phases of the design process, with a special emphasis on their contribution to the story and the aesthetics. The ideation workshops informed the story of SoM, in terms of genre, characters and context. The children acted as muses [3,29], providing inspiration that enriched the design space. However, their ideas had to be mediated by the designers when the final decisions were made to satisfy the functional requirements of serious games and the designers’ values. Some of these decisions proved unpopular in the evaluation study as they contrasted with strong gender stereotypes and the requirement for increased violence. Participation as co-designer [5,6] was considered too demanding for the resources of the project. Still, the authors believe that—depending on the context, the resources and the commitment of both designers and players—the latter could become true design partners [26], ensuring a path of growth for both designers and gamers in line with the mutual learning principle of participatory design [85].