Scrum VR: Virtual Reality Serious Video Game to Learn Scrum

Abstract

Education is crucial for the growth of society, and the usage of effective learning methods is key to transmit knowledge to young students. Some initiatives present Virtual Reality technologies as a promising medium to provide active, effective, and innovative teaching. In turn, the use of this technology seems to be very attractive to students, making it possible to acquire knowledge through it. On the other hand, agile methodologies have taken an essential role within information technologies and they are key in Software Engineering education. This paper combines both areas and presents prior research about Virtual Reality experiences with educational purposes and introduces a serious VR video game that aims to promote the learning of agile methodologies in Software Engineering education, specifically the Scrum methodology. This application tries to bring students closer to their first days of work within a software development team that uses the Scrum methodology. Two evaluation processes performed with university teachers and students indicate that the developed video game meets the proposed objectives and looks promising.

1. Introduction

Agile methodologies are currently widely used in the software development industry, and they have become extremely important in Software Engineering [1] and, consequently, in Software Engineering education. At the same time, Virtual Reality (VR) has started to be used in many sectors to train their professionals, allowing them to become familiar with their work environment and to improve their performance. For example, we can find applications that simulate medical operations [2], handle control panels in power plants [3], or perform psychological exposure therapies [4]. On the other hand, as Daniel [5] point out, these COVID-19 times have forced education to largely adapting to an online format where digital material works best for students. At this point, it is possible to propose the use of VR as a key point for education.

The benefits of using educative VR applications include the promotion of skills development or knowledge acquisition as well as the enhancement of the learner motivation and the whole teaching-learning process. The first approaches of this technology applied to education have been studied for several years [6]. However, not all educative VR applications succeed in promoting the retention of the concepts presented to the user [7,8]. Consequently, it is important to focus the design of educational VR applications on fostering learning.

Keeping this in mind, this paper presents a serious VR game designed to teach trainees the key concepts of Agile software development methodologies, concretely Scrum. The game, named Scrum VR, aims to improve the teaching-learning process of the Scrum methodology by immersing the user into an experience addressing the key concepts of Scrum. Likewise, this application is proposed as an innovative way to motivate students to learn more about this type of methodology. Scrum VR is intended to be integrated into future classrooms or online lessons with university students, but its usefulness is not restricted to the university environment. The serious game could also be used by trainees who have just arrived at a company that adopts this methodology.

However, the development of an educational, serious VR game is challenging and a series of technical, artistic, and educational decisions need to be considered. In addition, it is necessary to perform an evaluation of the developed application to ensure that it is adequate and it would be useful in Software Engineering education. Therefore, Scrum VR was evaluated through two independent experiments. The first experiment was conducted by university teachers who are experts in agile methodologies. The second was carried out by students who had recently taken the course that contained the Scrum syllabus. Both experiments were carried out in order to guide and improve the development of the game. The results show that Scrum VR is an innovative and promising educational resource that may complement traditional masterclasses and educational resources.

This manuscript describes the technical and artistic decisions made to achieve the proposed objective through the following structure. Firstly, some examples of the use of VR technology in education are analyzed. Secondly, the details of the design and development of the tool are thoroughly described. Finally, the empirical evaluation of the serious game is presented and the obtained results are discussed.

2. Related Work

The narrative capacity of the VR technology allows displaying the didactic contents in a realistic way to the users, managing to develop valid lessons in this type of virtual environment. In this way, modern examples of studies achieving greater user retention using VR technology can be found. An example of this is the research conducted by Ebert and Tutschek [9], which used immersive technologies to improve the effectiveness and retention of the ability to perform a medical fetal ultrasound, achieving a significant improvement in user learning. Furthermore, some studies such as the one made by Smith et al. [10] indicate that the learning improvement achieved through educative VR applications seems to be especially significant in the initial phases of a learning process when it is crucial to engage and motivate students. Accordingly, from the motivational perspective, the use of educative VR applications seems to be highly recommendable at the beginning of a learning process since it favors motivation and creates interest in the learners.

Regarding this issue, prior research suggests an increase in motivation when teaching using this technology, obtaining a consequent improvement in learning [11,12]. This enhancement of motivation may be due, in part, to the fact that VR technologies are generally perceived as appealing, especially to young people [11,12]. Furthermore, some studies [7,8] found this type of virtual didactic experience beneficial for student motivation, although they did not always significantly affect the knowledge acquisition with respect to other traditional media. Consequently, it is important to focus the design of educational VR applications on fostering proper learning in order to get benefits beyond learner motivation.

There are far more initiatives bringing VR technology closer to the classroom and examining its academic impact. For example, Liou and Chang [12] conducted a study with 105 high-school students, achieving positive results in terms of knowledge acquisition and motivation in scientific subjects. As an example, in higher education, Tzong-Hann et al. [13] used VR technology to develop global collaborative project-based courses, providing knowledge commonly taught in a theoretical way applied to practice. Many more educative VR applications such as these can be found in the recent survey performed by Kamińska et al. [14].

Among the different educational virtual studies, it is worth highlighting those that compare the suitability of VR technology versus other media formats. An interesting example is a study by Raya et al. [15], who compared the motivation and learning of users who experienced three-dimensional representations of pictorial art. This comparison was made between different types of educational formats, including VR. As a result, an increase in motivation was highlighted, although this visual representation did not stand out in knowledge acquisition. In the comparison of different technological educational formats, Moro et al. [16] conducted a study comparing VR, Augmented Reality, and tablet-based. This study was conducted in relation to sciences and medical anatomy, and the results showed that extended realities have a higher level of immersion and engagement in students.

Furthermore, some learning scenarios that require many resources, thus implying high costs, can be simplified thanks to VR technology. Indeed, Román-Ibáñez et al. [17] suggested that this kind of learning approach often reduces the cost of expensive practical lessons commonly presented in engineering education. That is why we can find contributions [18,19] presenting complete teaching platforms using this technology for different types of scientific subjects and provide a series of VR design ideas.

A low-cost approach to VR teaching is the use of mobile platforms for development. Moro et al. [20] found no significant differences in learning in the use of mobile devices using the same medical and health science education experience. However, they found worse symptoms in relation to cybersickness with the use of mobile platforms. Nevertheless, the authors recommend the use of mobile platforms, although it is necessary to pay special attention to virtual locomotion to avoid these symptoms.

There are long-standing guidelines that address these and other considerations, thus facilitating the design of effective teachings through virtual environments [21]. In addition, Radianti et al. [22] recently conducted a systematic review to identify the design elements and lessons learned in the development of VR applications for higher education. Moreover, they proposed a research agenda recommending to increase research on this type of teaching for its future inclusion in the classroom. However, as technology evolves so quickly, new guidelines and recommendations are needed to properly develop educative VR applications.

On the other hand, focusing on Scrum learning, there are gamification initiatives through classical games to achieve similar objectives to the one in this article. The development of serious games to teach Scrum is of great didactic interest. For example, Souza et al. [23] presented a serious board game to teach Scrum and they validated achieving good results. A precedent to this project was the card game produced by Fernandes et al. [24], which also resulted suitable for teaching Scrum. The results of these studies show that Scrum seems a good topic to be taught through games, but will VR video games be a good medium to learn Scrum?

The closest example found in the literature is the virtual experience presented in 2020 by Caserman et al. [25]. This experience uses modern VR devices to teach some aspects of the Scrum methodology. However, the authors and their results pointed out that the developed virtual experience should be more engaging. Our application, Scrum VR, addresses this problem and differs from that application in several aspects:

• Scrum VR is oriented to be used in academic environments, and it can be applied in real classrooms, rather than in professional contexts only.
• Instead of using high-cost VR devices, Scrum VR uses mobile devices, providing greater access and availability.
• Scrum VR is fully focused on making it attractive to promote the motivation of students and engage them in the course. To achieve that, the development and validation processes were carried out in an intermingled and iterative manner.

3. VR Serious Game Development

Agile methodologies are currently widely used in the software development industry and they have become extremely important in Software Engineering [1] and, consequently, in Software Engineering education. Many learning methods can be used to teach this topic, including some involving gamification and serious games [26]. However, to the knowledge of the authors, there are no VR applications aimed at teaching agile methodologies and more specifically the Scrum methodology.

Scrum VR was carefully developed to impact both knowledge acquisition and learner motivation. This serious VR game aims to improve the teaching-learning process of the Scrum methodology by immersing the user into an experience that addresses the key concepts of this methodology. This serious game can be useful for university students who are learning Scrum as well as for apprentices who have just joined a company using Scrum.

3.1. Design

A trainee could learn through the designed serious game about the key aspects of Scrum: the roles that are usually involved in the development process (i.e., Scrum Master, Product Owner, and Developers), the meetings that are commonly conducted during a sprint (i.e., Daily Meeting, Sprint Planning, Sprint Review, and Sprint Retrospective) and some artifacts or techniques frequently used throughout a Scrum project, such as Kanban boards, Burndown charts, or Poker Planning [1].

The user, represented by the main character of the story, is a young worker who has just joined a software company. He/she will learn about the target topics through observation and interaction with the actors participating in a Scrum team, and especially through the Scrum Master (SM) character. The SM will tell the user about the most important aspects of Scrum and will guide him/her through the processes of the Scrum methodology. Furthermore, other characters such as the Product Owner and the Developers are also relevant during the experience because they will interact with the user along these processes.

The whole experience is narrated linearly, and it is divided into four scenes with an approximate duration of 8 min each. The following is a summary of the subjects covered in each chapter of the serious VR serious game developed.

• Chapter 1: Scrum introduction

  -

  Agile and Scrum foundation.

  -

  Presentation of the Scrum Master and Developers.

  -

  Operation of a Kanban board.

  -

  Software development life cycle.

• Chapter 2: Daily meeting

  -

  Conduction of a Daily Meeting.

  -

  Selection of User Stories based on their prioritization and estimation.

  -

  Usage of the Kanban board.

  -

  Usage of the Burndown Chart.

  -

  Usage of a dashboard involving quality metrics.

• Chapter 3: Sprint Review and Retrospective

  -

  Presentation of the Product Owner.

  -

  Conduction of the Sprint Review, including a Software product demo.

  -

  Conduction of the Sprint Retrospective, including an analysis of the Sprint performance supported on the Burndown Chart.

• Chapter 4: Sprint planning

  -

  Definitition of User Stories through dialogue between the Product Owner and the developers.

  -

  Estimation of User Stories using the Poker Planning technique.

  -

  Prioritization of User Stories using the MSCW technique.

As presented, each scene addresses several learning objectives covering the main Scrum concepts so that by the end of the story, the user will have learned about how Scrum works. The scenes are described in detail below in order to introduce the narrative and to explain some VR narrative resources that guide the user’s attention: audio explanations, visual support, character movements, object lighting, and interaction through diegetic elements.

The story begins with the newcomer being welcomed into the hall by the SM, who explains to him/her about the Agile principles and the Scrum framework. The audio explanations are accompanied by some posters hung on the wall that support them visually. The user’s view (that is a first-person perspective) is guided through the movements made by the SM and the lighting of the objects, as depicted in Figure 1A. After the explanations, the SM offers the possibility to repeat them to reinforce the concepts if necessary. When the foundation of Agile and Scrum is established, the SM asks the user to enter the office to meet his/her colleagues, a question that can be answered through an interactive diegetic button. Once inside the office, the developers are introduced to the user. The majority are programming on their workstations, but some of them are chatting and having coffee on the office couch (see Figure 1B). Following the presentation, the SM proceeds to explain the operation of the Kanban board and the software development life cycle, as shown in Figure 1C. The speech is dynamized through interactions and curious facts (e.g., the SM asks the user if he/she wants to know the origin of the word Kanban, which is Japanese and means something like “card you can see”). Once again, the audio explanations are supported by visuals and guided by the movements of the SM and object lighting (See Supplementary Materials for clarity).

In the second scene, the user is involved in a Daily Meeting, which is being facilitated by the SM. He/she discuss with his/her peers the next actions to be performed by considering the information provided by the Burndown chart and some quality metrics. The user must select which user story to perform from the existing ones in the Sprint Backlog based on their priority and estimation. This interaction is performed through diegetic elements, as depicted in Figure 1D.

The third scene is focused on Sprint Review and Retrospective. In this scene, the PO is introduced and she takes a leading role. This scene is very dialogue-based, and the user has a high level of interaction thanks to many diegetic menus, as shown in Figure 1E. Moreover, the user participates in the demonstration of the software product increment and the analysis of the Sprint supported on elements such as the Burndown chart (see Figure 1F).

Finally, during the fourth scene, the user attends a Sprint Planning, in which the PO defines and prioritizes the user stories to be covered in the next sprint. The PO explains to the user the MSCW estimation technique through audio explanations and visual support, as depicted in Figure 1G. Furthermore, the user participates in the estimation of these user stories through the Poker Planning technique. This involves high interaction with other participating actors and the cards to be picked in order to estimate the story points of a user story (see Figure 1H).

3.2. Technical Decision Making

Beyond the educational design of the experience, there are several technical decisions to be made in order for the serious game to be successful and meet the planned objectives. Next, the most critical aspects are presented.

3.2.1. Platform and Programming Language

Scrum VR was developed for mobile devices with an Android Operative System using the Unity 2019.4.13f1 game engine. Thus, almost everyone can easily install the application on their own mobile and enjoy it just on a Cardboard, which is a very inexpensive instrument. In this way, it is possible to bring the experience closer to a large number of users. The choice of this engine was due to the fact that it is not complicated to learn, using its C#-based scripting language to create a low quality graphic appearance, such as the one needed in mobile devices [27]. Its main competitor engine, Unreal, is better for the development of applications with professional graphics, but with a higher computational load and file weight, etc.

3.2.2. Cybersickness and Locomotion

In order for the virtual experience to be effective, it is critical to avoid or minimize factors, such as the feeling of cybersickness [28].

Firstly, as a high duration of an experience can lead to greater dizziness, it is important to shorten the times when possible and split the experience into chapters. For example, the experience presented above was divided into four chapters of approximately 8 min.

Moreover, special attention has to be paid to the method of locomotion selected for the entire virtual experience in order to avoid or minimize cybersickness. Specifically, methods such as the on-rails-guided locomotion [29] are highly recommended. This method is of special interest when the experience is developed for mobile devices since it does not require additional external hardware to interact with the virtual environment. In the on-rails-guided locomotion, the user does not control the movements, being able only to observe the environment that is shown to him/her during the experience.

According to the good practices in the methods of locomotion [30], the speed of movement of the user should be always lower than a human walking during the whole experience (less than 1.4 m/s). These displacements always follow a linear path, without applying any kind of rotation to the user. In this way, the user is in charge of observing where the action of the virtual experience occurs.

3.2.3. User Interaction

The user interaction is critical to creating a real application in which the user can interact, going beyond a video in which the user just observes.

Since mobile platforms do not usually carry additional devices, the interaction should be purely visual. The interactions should always be done with diegetic menus that are fully visual and contained within the scene. Several examples can be found in our serious game, as shown in Figure 1D,E,H. These interactions can be designed for different purposes, for example, to pick an element or to confirm the understanding of a concept. The visual method uses a reticle visible to the user with which he/she could select one of the options. To do so, the user should observe one of them for some time, for example, 3 s. The progress of that action is displayed on the reticle.

While the user’s on-rails-guided locomotion could be done with a preset animation, the interactions should be programmed to be reused during the whole experience. Since all interactions usually follow the same mechanics described above, a simple interaction was developed based on two main classes: Interactable classes, which can be activated and are responsible for launching certain actions, and Selector classes, responsible for activating the Interactable objects. In Unity, the scripts are associated with the elements of the scene and, in this case, the activation of the Interactables can be done through a RayCast generated from the point where the Selector script is associated. On the other hand, the Interactable is in charge of measuring the time left to be activated and stores what actions should be triggered when this happens. In this way, the effect of activating elements with the sight was achieved. This process is explained through the sequence diagram depicted in Figure 2.

3.3. Development Phases

As mentioned before, the development of a proper serious educational VR game is challenging. Therefore, the development of Scrum VR was divided into two phases, and it was closely linked with two evaluation processes.

During the first phase, the foundations of the project were laid. Educational decisions were made by elaborating a first complete version of the complete script of the experience. Technical decisions were also made regarding the development, technologies and devices to be used, as well as the locomotion and interaction design. In addition, artistic decisions were made by creating the working environment and acquiring the character assets. With all these decisions taken, Chapter 1 was produced. After that, the first evaluation with teachers was performed to identify problems and gather ideas potentially useful for the next chapters.

The second phase involves the development of the remaining three chapters. Some modifications were made with respect to the initial plan thanks to the teachers’ feedback. Once the script was refined and the technologies and artistic elements were tested, developing new chapters was easier. Indeed, it took approximately the same amount of time to develop the first chapter as it took to develop the next three chapters. Once the game was completed, a second evaluation with students was performed to check the correctness of the application in educational, technical, and artistic terms. Some minor modifications were made based on feedback from the students.

4. First Evaluation

This first phase focused on the first chapter to serve as a basis for developing the following chapters. The first chapter of the VR serious game was validated in technical and academic terms by Computer Science teachers. They tested the first chapter and evaluated it through an ad hoc questionnaire.

4.1. Evaluation Instrument

The questionnaire included some initial questions about personal information (e.g., age, gender, mobile device, VR experience, etc.) and a list of statements with which teachers needed to (dis)agree using a Likert scale from 1 (totally disagree) to 5 (totally agree). The items of the questionnaire are presented together with the results.

4.2. Sample Description

The demographic information about the 15 Computer Science teachers who tested and evaluated the serious game is as follows: there were 11 males and 4 females, and the mean age was 38.92 with a standard deviation of 9.27. Furthermore, 75% claimed to have a sound knowledge about agile methodologies and Scrum, but they had a disparate experience in the use of VR applications. Approximately 50% of the teachers had some or a lot of experience using VR applications, while the remaining 50% had never used this technology.

4.3. Results

In general terms, the users did not experience performance problems (the 100% declared that they did not suffer them), neither dizziness problems (the 25% got a little discomfort and nobody got dizzy). This allowed them to properly evaluate the serious game in technical and educational terms.

Table 1. Quantitative results: Teacher’s evaluation.

Item	M	SD
Evaluation of Scrum VR
My overall opinion is positive	4.17	0.94
The general narrative is adequate	4.67	0.89
The provided explanations are adequate	4.42	0.67
The artistic design is adequate	4.33	0.89
The objects shown (e.g., furniture, posters, blackboard, etc.) are adequate	4.25	0.97
The method of locomotion (i.e., how you move) is adequate	3.83	0.84
The accompanying sound is adequate	4.58	0.52
The stage lighting is adequate	4.75	0.45
The mechanisms of interaction are adequate	3.58	1.17
I have felt present in the experience (i.e., surrounded by the virtual world)	3.75	0.97
Potential Academic Impact of Scrum VR
It can promote learning about Scrum	4.27	0.94
It can help to understand a Scrum professional context	4.17	0.72
It can be engaging and motivating for learners	4.5	0.52
It can help to energize the teaching-learning process	4.33	0.65
Evaluation of This Type of VR Application
My general opinion about this type of application is positive	4.45	0.93
This type of application is innovative	4.42	1.24
This type of application is a good complement to traditional learning methodologies	4.5	1.0
I would like to use this type of application with my students	4.08	1.38
It would be useful to incorporate more applications like this in the University	4.42	1.24

presents the quantitative results obtained through the evaluation of teachers for each item, the mean (M) and the Standard Deviation (SD) are shown.

The overall opinion provided by the surveyed teachers about the educative VR serious game is very positive, and in general terms, they considered that the main elements of the application (e.g., explanations, artistic design, interaction, etc.) are adequate. Almost all questions related to the evaluation of the serious game are above 4 out of 5. The aspects valued slightly worse are those related to locomotion (3.83) and interaction mechanisms (3.58), which are the most critical elements from the technical perspective in this type of application. Anyhow, these results indicate that the experience is correctly designed from the narrative and technical point of view and it would allow the fulfillment of the educational objectives pursued.

Regarding the potential academic impact of the serious game, the teachers considered that it can promote learning about Scrum (4.27 out of 5) and help to understand a Scrum professional context (4.17). Moreover, they thought that the serious game would be engaging and motivating for learners (4.5) and it would help to energize the teaching-learning process (4.33). These aspects are the key objectives of the serious game, which aims to facilitate the knowledge acquisition of the learners, enhance learner motivation, and enrich the teaching-learning processes.

On the other hand, the teacher’s opinion about this type of educative VR experience was also very positive (4.45 out of 5). The teachers found this type of application innovative (4.42) and a good complement to traditional learning methods (4.5), so much so that they were eager to incorporate them into their teaching (4.08) and some of them requested the implementation of another educative VR serious game to support their courses. In short, it seems that the use of VR for educational purposes has potential.

Finally, it is worth mentioning that the gathered comments were fully aligned with the quantitative results presented above. Some of them are highlighted next: "The explanations are clear, the graphic recreation is pleasant and the narrative is sound", "I think it is a very motivating learning resource and it will be greatly appreciated by the students", "I loved it and I find it very useful, great work!". Moreover, some useful suggestions related to the locomotion and interaction mechanisms were gathered and they were considered in the development of the following chapters.

5. Second Evaluation

The complete VR serious game was tested by students from advanced courses who already knew about agile methodologies as they had recently taken the course that contained the Scrum syllabus. Once these students tested the game, they perform the evaluation method defined below.

5.1. Evaluation Instruments and Techniques

To continue evaluating the serious game, two different evaluation methods have been used. The first one was quantitative, while the second one was qualitative. Thus, a mixed approach [31] was used in this evaluation as the quantitative data gathered through a questionnaire was complemented by the opinions gathered through the focus group method.

Firstly, a questionnaire similar to the one used by teachers was completed by the students. However, as in this case, the students were testing the full game, new items were added to the questionnaire. This questionnaire asks about personal information and presents a list of statements with which students needed to (dis)agree using a Likert scale from 1 (totally disagree) to 5 (totally agree), questions to select an answer from several options and open questions to gather comments about the aspects of the serious game they liked the most and the least. The items of the questionnaire are presented together with the results.

Secondly, several focus groups were conducted to collect qualitative data from the students about the usage of the application in VR. The focus group method started to be used in social sciences and currently it is widely applied in many domains [32], including Software Engineering [33]. Focus groups are defined as a discussion involving 3 to 12 people and guided by a moderator who follows a predefined script to keep the discussion focused [33]. In our case, three focus groups were carried out, and each of them lasted approximately 25 min. The following topics were explored from a qualitative perspective:

• General appreciation. What do you think about the application?
• Knowledge. What concepts about Scrum do you think are the most and least clear?
• Usefulness. How useful do you think it will be for the students who use it?
• Improvement. What aspects of the application do you think could be improved?
• Striking features. What aspects of the application have caught your attention the most?

5.2. Sample Description

The demographic information about the 12 students who tested and evaluated the serious game is as follows: there were 10 males and 2 females, and the mean age was 21.00 with a standard deviation of 1.04. All of them were third-year students enrolled in Computer Science degrees offered by the Politechnical University of Madrid, Spain (ETSISI-UPM). In total, 83% were taking a Software Engineering degree and they passed a course that teaches Scrum, while the remaining 17% were taking a Computer Engineering degree and had no knowledge about Scrum. Regarding their background in the use of VR applications, they had disparate experiences. Approximately 60% of the students had some experience using VR applications, while the remaining 40% had never used this technology.

5.3. Questionnaire Results

In this evaluation, the users also experienced no performance problems or dizziness problems ( 50% experienced a little discomfort and nobody got dizzy). This allowed them to properly evaluate the game in technical and educational terms.

Table 2. Quantitative results: Student’s evaluation.

Item	M	SD
Evaluation of Scrum VR
My overall opinion is positive	4.92	0.28
The general narrative is adequate	4.92	0.28
The provided explanations are adequate	4.67	0.49
The artistic design is adequate	4.75	0.45
The objects shown (e.g., furniture, posters, blackboard, etc.) are adequate	4.83	0.38
The method of locomotion (i.e., how you move) is adequate	4.25	0.75
The accompanying sound is adequate	4.75	0.62
The stage lighting is adequate	4.92	0.28
The mechanisms of interaction are adequate	4.83	0.38
I have felt present in the experience (i.e., surrounded by the virtual world)	4.92	0.28
Evaluation of Key Aspects of Scrum VR
The explanations provided by the Scrum Master are adequate	4.92	0.28
The explanations provided by the Product Owner are adequate	4.83	0.38
The elements of the environment (posters, whiteboard, information monitors, pop-up explanatory cards) are adequate	5.00	0.00
The interactions or dialogues with other characters through menus are adequate	4.67	0.49
The choice of user stories on the Kanban board (Chapter 2) is adequate	4.33	0.98
The estimation of user stories through Poker Planning technique (Chapter 4) is adequate	4.83	0.38
Potential Academic Impact of Scrum VR
It can promote learning about Scrum	4.83	0.38
It can help to understand a Scrum professional context	4.92	0.28
It can be engaging and motivating for learners	4.92	0.28
It can help to energize the teaching-learning process	4.83	0.38
Evaluation of This Type of VR application
My general opinion about this type of application is positive	4.92	0.28
This type of application is innovative	5.00	0.00
This type of application is a good complement to traditional learning methodologies	5.00	0.00
I would like to use this type of application during my studies	4.92	0.28
It would be useful to incorporate more applications like this in the University	4.92	0.28

presents the quantitative results obtained through the evaluation of students for each item, the mean (M) and the Standard Deviation (SD) are shown.

The overall opinion provided by the surveyed students about the educative VR serious game is very positive. It can be seen that students reported values above 4.25 out of 5, and most of them are around 4.8 out of 5.

The students considered that the main elements of the application (e.g., explanations, artistic design, interaction, etc.) are adequate. Almost all questions related to the evaluation of the serious game were rated above 4.25 out of 5. The students also positively valued aspects related to locomotion (4.25) and, especially, the interaction mechanisms (4.83). Once again, the results reported in this evaluation indicated that the experience is correctly designed from the narrative and technical point of view, and it would allow the fulfillment of the educational objectives pursued.

Similar responses can be observed in the group of questions related to the key aspects covered by Scrum VR. The students rated all items above 4.33 out of 5. Furthermore, students were asked to select the part of the game they found most interesting. The six key aspects of the serious game presented in the second section of Table 2 were offered as options. Approximately 60% of the students selected the estimation of user stories through the PokerPlanning technique presented in Chapter 4 as the most interesting part of the serious game.

The highest scores were reported in the following groups of questions. Regarding the potential academic impact of the serious game, the students rated every item of this dimension above 4.80 out of 5. On the other hand, in the case of the evaluation of this type of VR application, they rated every item above 4.90 out of 5. Based on these promising results, the use of this application in a real environment is of interest, as it could be easily accepted by students.

Finally, several comments were gathered from the students through the open questions posed in the questionnaire. Some of these comments emphasized the immersion feeling, for example, "It was great to feel like you are in a real company", "The immersion helps a lot with the introduction and familiarization with how Scrum works" and "The story is very good, it’s great to feel part of a real development case and to be able to participate in it". Many comments remarked on the interaction possibilities of the serious game: "I loved being able to interact with the environment and that depending on the choice I made, the application would have an appropriate response" and "I found very useful the discussion with the poker game and how the others interacted with me depending on what I had done". Finally, other comments like the following highlighted the educational purpose of the game, "The experience is very complete in general because it covers all important aspects of Scrum, I found the explanations and interactions very accurate". Moreover, comments related to general aspects about accessibility or usability were gathered and they were considered to refine the serious game.

5.4. Focused Groups Results

The results of the focus group questions that were defined in Section 5.1 will be shown separately.

5.4.1. General Appreciation

In overall terms, Scrum VR was very well received. No students indicated that they disliked the game. Some general comments can be highlighted, such as “The application provides a vision of what a Scrum team will actually look like” or “It is super innovative and has a lot of potentials”. Regarding the presence produced by the VR experience, some students explained “I love the interaction with other characters and be able to make different decisions” and “The experience is very realistic as if I was there”. Many general opinions were aligned with the feedback provided by the students through the questionnarie.

5.4.2. Knowledge

In terms of knowledge acquisition, the serious game was also very well valued. Students who took the course expressed comments such as “I have understood the topic of estimation better through the application than during the course” and “The application is incredible, it has summarized half of the course”. Anyhow, some comments like this “There are other prioritization methods such as dot voting that are not explained” highlighted that the serious game did not cover all the knowledge acquired in the subject, which is understandable as the length of the virtual experience is much shorter than a complete course. On the other hand, other students said “Until now I had not understood what scrum was, but through the application, I think the main concepts are clear to me”.

5.4.3. Usefulness

This aspect was also very positively valued in several ways. The students stated that this type of educational material captures their attention and interest very well, which is expressed in comments like “It is very easy to maintain concentration with the application, more than in most classes” and “The application can make you learn a lot in a very short time and generate interest”. Furthermore, the students indicated that “Prior knowledge was not necessary to understand what was being explained”, indirectly expressing that the game could be used at different times during the course (e.g., to introduce a specific subject or to make a general review).

5.4.4. Improvement

There are some improvements pointed out by the students. Some of them were implemented, but others were not due to duration or technology constraints. For example, “There is some knowledge about the Burndown chart that is not deeply explained”, “In terms of accessibility, subtitles could be enabled” or “More visual metaphors could be added”. As can be seen, these opinions showed broad limitations due to technology.

5.4.5. Striking Features

In general, the game has managed to impress students, but some features stand out. All groups shared the same opinion and were expressed through comments such as “The selection of user stories and, especially, the poker planning were my favorites parts of the experience”. Another group stated “It’s interesting to see how you make a mistake and feel bad about it, as well as to appreciate how the team downplays it and you feel supported by them”. In a similar vein, another group indicated “It was interesting to see someone developing in a hurry and to see how it was detected that the code coverage was dropping”.

6. Discussion

The evaluations performed with teachers and students report positive results and suggest that the game will be well received by students and will be useful both in terms of motivation and knowledge acquisition. It can be observed that while the ratings obtained in the first evaluation are good, those in the second evaluation are excellent. In that case, the users find the application extremely attractive, motivating and useful. These differences are to be expected since the users of the second evaluation tested the entire application and those of the first evaluation tested only one of the chapters. In addition, it should be taken into account that the second evaluation was carried out by young students, who tend to see the VR technology as more appealing. This strengthens the idea that the experience is attractive enough to be applied in the classroom. In this way, the difference with respect to Caserman’s [25] work is fulfilled, finally achieving a Scrum application that is attractive enough to its users.

The evaluations of Scrum VR indicate that this serious game is useful to convey the general concepts about Scrum (i.e., roles, meetings, artifacts, techniques, etc.), as well as certain details that show the spirit and values of agile methodologies (e.g., team support, deadline and stress management, etc.). The interaction mechanisms and the highly interactive parts of the game (e.g., selection of user stories, poker planning, etc.) were very well valued by the users, who believe that certain concepts can be understood in this way better than through traditional methods such as a teacher’s explanation. This was the main objective of the application as it was intended to go beyond an explanatory video and create a really serious video game in which users have multiple possibilities of interaction and learn concepts in an engaging way.

From the technical point of view, the teachers’ evaluation was slightly lower in terms of locomotion mechanisms. However, this was not reflected in the evaluation made by the students. This may be due to the fact that the teachers tested only the first chapter of the experience, as explained above. This chapter is the only one with user locomotion, since it is necessary to take the user on a tour around the office. To avoid this possible dizziness, the rest of the chapters do not contain any locomotion. This is reflected in the students’ evaluations. After the analysis of the state-of-the-art and as mentioned above, the chosen locomotion method is the most appropriate according to the needs of the project. This on-rails-guided locomotion method may cause a slight sensation of motion sickness compared to other methods using more advanced VR technologies [34]. However, those locomotion methods are not applicable to mobile devices such as the one used for this serious game [29].

Scrum VR can be used at various times with different objectives during a course that addresses the traditional Scrum syllabus. Many usage strategies could be adopted: the application can be used at the beginning of the course to introduce Scrum and awaken the attention of the students, at the final part of the course to review the concepts studied, and chapter by chapter in different sessions to complement or reinforce the teacher’s explanations, etc.

7. Conclusions

The usage of VR in educational contexts is promising for many reasons. Prior research shows a fair number of VR applications used at different educational levels (i.e., secondary education, higher education, and professional training) with very positive results in terms of knowledge acquisition and/or learner motivation. The areas of application are also varied, and VR applications can be designed to teach concepts from several knowledge fields, such as basic science, health, engineering, or arts.

However, to the authors’ knowledge, none of the applications developed so far address the learning of agile methodologies, and this is one of the novelties of this contribution. Moreover, another distinctive feature of Scrum VR is that it was developed for smartphones with the Android Operative System, thus allowing the application to be used by a large number of users. Moreover, technical recommendations regarding the most critical aspects of VR video games such as cybersickness, presence, locomotion, and interaction were considered to develop this application. On the other hand, the programming architecture applied has proven to be useful for the generation of new chapters. The architecture has proved to be versatile, able to be developed with relatively low effort of new interaction mechanics with the view, such as Poker planning or user story selection.

The performed evaluations report very positive results and indicate that Scrum VR would be very appreciated by students and would be useful both in terms of motivation and knowledge acquisition. The video game seems to be highly suitable for the teaching-learning process of Scrum because it facilitates the acquisition of general concepts such as roles, meetings, artifacts, or techniques typically used in Scrum, as well as the underlying values of agile methodologies. The evaluations presented here suggest that the highly interactive parts of the game allow students to learn key techniques such as the user to select user stories or to estimate them in a more effective way than by using traditional learning methods. Thus, Scrum VR goes beyond an explanatory video to transmit theoretical knowledge, and it is a real serious video game to develop skills in an immersive and interactive way.

On the other hand, although the results are mostly favorable, this application has some limitations inherent to VR technologies. The most notable one is referring to the cybersickness problem. Although locomotion techniques are used to minimize this problem as presented above, it is very difficult to eliminate it completely. In the case of Scrum VR, none of the users reported getting dizzy, but some did experience slight discomfort.

Lastly, future work is presented. On the one hand, it involves conducting randomized controlled trial studies to test the instructional effectiveness and motivational impact of Scrum VR, as well as to compare different usage strategies of the application. On the other hand, it involves developing new educative VR video games based on the guidelines and the experience accumulated making Scrum VR. This kind of application could be used in face-to-face and online modalities, and they would be potentially useful for high school, university, or working students from many knowledge fields. As technology evolves, new modalities of human–computer interaction are included. In the future, it would be possible to carry out studies that assess newer technologies such as facial interaction or gesture recognition [35], evaluating their usefulness for education.