The Potential Educational Value of Mobile Augmented Reality Games: The Case of EduPARK App

Abstract

New teaching methodologies are nowadays integrating mobile devices, augmented reality (AR), and game-based learning in educational contexts. The combination of these three elements is considered highly innovative, and it allows learning to move beyond traditional classroom environments to nature spaces that students can physically explore. The literature does not present many studies of this approach’s educational value. The purpose of the study is to present an illustrative case of a mobile AR game in order to analyse its educational value based on the users’ opinion, both teachers and students, and on logs of game results. Through a mixed method approach, the educational value scale was applied to 924 users after playing the EduPARK app in a Green City Park. Results revealed high educational value scores, especially among teachers and students of 2nd and 3rd Cycles of Basic Education (83.0 for both). Hence, this particular software seems to be more suitable for 10–15 years-old students who highlighted motivational features, such as treasure hunting, points gathering, the use of mobile devices in nature settings, and AR features to learn. This study empirically revealed that mobile AR games have educational value, so these specific game features might be useful for those who are interested in creating or using games supported by apps for educational purposes.

1. Introduction

Game strategies are used in Education with the aim of improving the learning process by making use of the motivating effects of game elements and techniques. Both strategies of game-based learning and gamification intend to motivate students for learning. Engagement and motivation are key factors that influence the students’ performance during a learning process [1]. However, game-based learning is not the same as gamification, as the first makes use of a game for learning, and the latter integrates a set of technical concepts (such as points, badges, and leaderboards) in the learning process. Games have been used for a long time in teaching and learning; however, they have not always been properly investigated.

The increasing use of mobile phones makes it possible to explore digital educational games in outside environments, and when combined with augmented reality (AR) content, the educative effect may be exponential and result in best practices [2]. With the use of AR, the real-world environment can be augmented by providing users with accurate digital overlays. AR is a promising technology that has the potential to encourage learners to explore learning materials from a totally new perspective. Additionally, technological advancements along with the proliferation of wireless mobile devices, such as smartphones and tablets, allow for widening the scope of educational AR applications [3].

The integration of each of the above-mentioned elements (games, mobile devices, and AR) in educational contexts is considered innovative [3,4,5]. Moreover, their combination has revealed diverse benefits for learning, both at the cognitive and emotional levels [6,7,8], and supports learning to move beyond traditional classroom environments to nature spaces that students can physically explore [9].

Regarding off the shelf games, Lin, Huang, and Lin [10] analyse the game design of “Pokémon GO” in contexts where parents play together with their children, and found that some of the most valued features are its user-friendly interface, going outdoors, visiting places of interest, and combining games with exercise.

In what concerns games developed for educational purposes, Preka and Rangoussi [4] combine quick response (QR) codes with AR to create a collaborative game for music learning in Early Childhood Education in indoors and outdoors activities. Evaluation results indicate that the AR–QR technology is a powerful tool that triggers and sustains children’s interest during the learning process and can enhance their cognitive and collaborative skills, as well as their social interaction.

Another study [11] presents a technical framework, Mobile Augmented-Reality Games for Instructional Support (MAGIS), for the development of this type of game. The authors illustrated the usefulness of the framework for implementing outdoor location-based educational games through the analysis of the game “Igpaw: Intramuros” for History learning. The authors highlighted that players’ enjoyment of the game tended to be adversely affected by weather conditions, long walking distances between markers, and devices’ battery life.

The EduPARK app is an example of a successful mobile AR game for learning [12]. It was developed to foster collaboration [13], and authentic [14] and situated learning [15] outside the classroom, under the umbrella of constructivist learning [16], as the learner or app user assumes an active role by constructing new knowledge within the articulation between the learning experience and previous knowledge. The EduPARK app aims to be explored in situ, an Urban Green Park, which is a context that can be used to promote new modes of learning in science education, since the ability to understand ecosystems is enhanced by experiences in real environments [17]. The app gives access to excellent cross subjects’ educational materials, both in Portuguese and in English. It comprises a very useful tool for Portuguese teachers and students to explore scientific knowledge by accessing contextualised and appealing information on biological and historical references that augment the experience of exploring a local Urban Green Park. This type of innovative educational resource is not common in Portuguese speaking countries, hence, adding to the relevance of the EduPARK app.

Users’ subjective perceptions about educational software can be evaluated with different tools, for example, System Usability Scale (SUS) and Educational Value Scale (EVS). The SUS is a robust, effective, and inexpensive tool, developed by Brooke in 1986, to quickly measure users’ subjective perceptions of computer systems usability [18]. With only 10 questions on a 5-point scale (ranging from “strongly agree” to “strongly disagree”), it is one of the most used tools for measuring perceptions of usability, with a scale varying from 0 to 100 [19]. Sauro [20] reviewed 500 studies and found out that a SUS score of 68 could be considered average.

The tool used in this study, the EVS, was developed based on the SUS to quickly and easily collect users’ subjective rating of the educational value of an app for outdoor green settings, considering the following dimensions: (a) Learning value; (b) intrinsic motivation; (c) engagement; (d) authentic learning; (e) lifelong learning; and (f) conservation and sustainability habits. Taking this into account, two items from each dimension were included in the scale, as described in [9,21]. As this is a new tool, there is a need of further studies to establish standards to support values interpretation.

Previous studies ⁹ showed that the EduPARK game achieved an average Educational Value Scale (EVS) of 83.8 and an average System Usability Scale (SUS) of 80.2, according to 244 students attending the 2nd or 3rd Cycles of Basic Education. This demonstrates its high educational value and usability for students of these school levels, indicating that the app can be used, as reference, by the international community, namely those who are interested in designing educational apps based on previous good experiences. As the EduPARK app is intended to be used by several school levels (as explained in the next section), there is a need to aggregate data in order to yield a wider picture of its educational value and its adequacy for different target publics. Hence, the purpose of this study is to present the EduPARK app, an illustrative case of a mobile AR game, in order to analyse its educational value based on the users’ opinion and on logs of game results. Aggregated data includes 924 questionnaires, filled in by teachers and students of non-higher education contexts.

The analysis sustains a reflection on the enhancement of the educational value of mobile AR games by presenting the specificities of the EduPARK game, including its educational resources, such as 2D and 3D models that mix real and virtual worlds, combining familiar technology with outdoor learning strategies. Finally, some guidelines arise that might be useful to inspire other educational game producers by providing theoretical and practical frameworks that can be useful in other natural environments to open horizons and opportunities for Education.

2. The EduPARK Game

As the main purpose of this paper is to analyse the educational value of a mobile AR game accessed through the EduPARK app, there is a need to present the educational principles that guided its development and the app’s features that may enhance learning.

The EduPARK app is the main product of the project with the same name. It was developed with the aim of supporting social constructivism approaches to teaching [16] in a game-based approach for student engagement and motivation. These are well known factors influencing learning [22]. Hence, the app was created to support the users’ construction of meaning through experiences in an Urban Green Park, being meaning influenced by the interaction of the learners’ prior knowledge with the new experiences, as well as by their interactions with others. When designing the users’ interaction with the app, the followed principles were: (i) To stimulate the app users to become active participants in their own learning, in a student-centred learning process with hands-on activities; (ii) to foster collaboration among app users from the same work group, through debate of ideas before answering questions, instead of making use of only competitive approaches; (iii) to embed learning in a real context, a park, which is rich in biology, history, and mathematics learning opportunities, thus providing authentic and situated learning experiences; (iv) to offer multiple modes of representation, namely through AR contents with video, audio, text, and 3D models; (v) to allow users to progress at their own pace, not establishing limited time to answering the challenges posed through the app; and (vi) to provide feedback with a scientific explanation after answering the proposed challenges.

Besides the social constructivism approach, game features, such as the treasure hunt format with a friendly mascot giving hints and feedback, accumulating points by correctly answering the challenges, and the leaderboard to show the best performance, can promote motivation by making boring content more enjoyable [22,23]. All these features were integrated in the EduPARK teaching approach. Moreover, allying these game features with the use of mobile devices (still widely forbidden in formal education contexts) and AR technologies is another factor increasing motivation and engagement with learning [3].

As mentioned before, the real outdoor context where learning is promoted by the app is a park. These areas have high ecological and environmental value, so they should be preserved. The use of this educational mobile AR game in the selected park promotes positive attitudes towards nature conservation and sustainability in the community [22], thus adding to the app’s educational value.

For educational relevance of the app and game approach, it was important to carefully analyse the National Curriculum to identify multidisciplinary issues (e.g., integrating Biology and History) to integrate in the educational guides (games), so that students may correlate the experiences promoted by the app with the aimed curriculum learning. The fact that the educational guides were designed to be explored in the park provides an example of a truly authentic context for situated learning, where the location is essential for learning [23].

The EduPARK app can be installed from the project site (edupark.web.ua.pt/app). It requires the update of quizzes after downloading; dismissing the use of mobile web in the park. The app was developed for Android using Unity 5 and Vuforia SDK, integrating AR and quiz games, in the logic of a treasure hunt [24]. Therefore, it was designed to provide a learning experience that requires the exploration of a Green Park, the Infante D. Pedro Park (Aveiro, Portugal).

The app is intuitive and can be used autonomously, either individually or in a group, at any time using the game mode or explore freely mode. The last one allows accessing AR content, without the requirement of following a predetermined path. In the game mode, users are welcomed by the EduPARK mascot, a female monkey who explains the rules to the players. The game objective is to gather points by correctly answering quiz questions. The user, or group of users, who gets the higher score is the winner. So, the game is more interesting when several groups play at the same time in a friendly competition climate [12,25].

For motivational and engagement purposes, the app allows the creation of several profiles that record the progress of the explored games. For each profile, it is possible to know all completed games, percentage of correct answers, number of visited markers, and number of found treasures [12].

For the first-time experience of playing, the user is guided by a tutorial that introduces the app’s features (Figure 1). The game includes: (i) Instructions for users to find locations in the park, in order to follow a predetermined path; (ii) questions whose answer requires observing the surroundings or analysing multimedia resources, sometimes in AR format; (iii) feedback providing an explanation about the correct answer; (iv) the number of accumulated points; (v) challenges to find virtual treasures (caches), enclosing extra credits—bananas—the number of assigned bananas decreases with the time needed to find the treasure; (vi) the accumulated bananas can be exchanged for clues to help players answer later questions or are converted into points at the end of the game [12].

The logic of searching for virtual treasures/caches is based on Geocaching principles, integrated into the game to increase user involvement and motivation. An AR icon is always available so that, at any time, users can point the camera on their mobile device to specific images, called AR markers, to access the AR content. The compass can be used at any time during the game, to support users’ orientation through the park. It is also possible to access an interactive map to view the four game areas and various locations to visit during the game. These locations are usually associated with botanical species of the park, marked with physical plaques with AR markers giving access to AR content about the species. Other locations for players to visit are historical interest points with “natural” AR markers, for example pre-existing tiles or information boards to access additional AR information [12,24].

For different target publics, there are different educational guides that encourage users to follow a path to promote learning relevant for the curriculum of Sciences, Mathematics, and History, among other subjects. The project created three guides aimed at students and teachers of different school levels: (i) 1st Cycle of Basic Education (CBE) comprising school years from 1 to 4 (attended by children with 6 to 9 years-old), (ii) 2nd and 3rd CBE, comprising school years from 5 to 9 (attended by children with 10 to 14 years-old), and (iii) Secondary, from 10 to 12 school years (attended by students 15 to 17 years old), and higher education. A fourth guide was also created for the public who visits the park, including questions of general culture, for example, about the typical aesthetic style of Aveiro [25].

Each guide consists of four stages that correspond to a specific area of the park, in which the user is challenged to search for points of interest, collect information, answer multiple choice questions, receiving immediate feedback whether or not they answer the question correctly. The questions may have associated multimedia resources, such as audio, photography, illustration, video, or 3D objects in AR [12].

At the end of each stage, the user has five minutes to find virtual treasures in a “treasure hunt” inspired by Geocaching principles. After this period, if the treasure is not found, the game proceeds normally initiating a new stage. Points are accumulated throughout the game, whenever the users answer the questions correctly and whenever they find an AR marker. At the end of the game, the users have access to the number of correct answers and the total number of accumulated points [24].

AR combines the real world with a virtual world, which can be three-dimensional and interactive in real time. Usually, the camera of a mobile device is used to detect a previously defined marker (image) that activates associated virtual content. The physical AR plaques, which support the AR markers, have a digital laser engraving on laminated vinyl with ultraviolet protection on galvanized plaques. These are fixed to the ground by means of external piles, in order to constitute permanent elements in the park.

The markers about the park’s plants present a menu (Figure 2) that includes information about the plant, the leaf, the flower, the fruit, its origin, ecology, and curiosities. The users can choose what they want to have access to systematized information with a photo or image. The words specific to Botany, which may not be known to ordinary people, are underlined in blue, and their meaning can be known by clicking on them, like a glossary. The user can digitally interact with the 3D models of the plant leaves, being able to rotate them and observe the top and bottom page of the leaf, which is often useful to identify species. This feature is particularly advantageous when the specimen is leafless, in the case of deciduous species. All this information, associated with AR markers, is available in Portuguese and in English, so that foreign tourists can also use the EduPARK app to know more about the Park’s species [12,24]. For greater user convenience, it is possible to freeze the AR information that appears on the mobile device’s screen. This way, the user can turn away from the marker without losing the associated information.

Additionally, the app recognizes “natural” markers, pre-existing in the Park, such as tiles and information signs for monuments. These were frequently used to provide 3D contents produced by the EduPARK project, which aim to provide information that complements the reality observable in those locations. Next, a brief description of the produced contents is presented.

• Moliceiro (typical boat of the city) (Figure 3): On the ancient moliceiro’s tile, the AR functionality overlays a real photograph of a current moliceiro to highlight the change of use from transportation of aquatic flora to fertilize farmlands to tourist transportation.
• Santo António (saint adored by the Portuguese population) (Figure 4): On this tile, the AR functionality overlays several buttons on parts with religious significance, which become interactive and give access to a brief description.
• Symmetric tile (Figure 5): On this tile, the AR functionality overlays an animated three-dimensional tile that demonstrates its axes of symmetry, through folding.
• Torreão (ancient water deposit) (Figure 6): On the building identification plaque, the AR functionality overlays a three-dimensional reconstruction of the building, animated by its decomposition into the three main geometric solids that compose it.
• Ducks’ House (wooden construction for the ducks in the parks lake) (Figure 7): On an identification plaque, the AR functionality overlays a three-dimensional reconstruction of the ducks’ house. The object allows its rotation on all axes.
• Monument to Dr Jaime de Magalhães Lima (local personality) (Figure 8): On the monument, the AR functionality overlays the three-dimensional reconstruction, which has interactivity to allow the exploration of the geometric solids that compose it. This model is triggered by pointing the mobile device camera directly at the monument, not requiring a physical AR plaque. This was the marker that constituted the greatest challenge from the technological point of view, since the overlap of the 3D model on the physical monument requires a high precision in terms of dimension and positioning.
• EduPARK Mascot (the inspiration for this mascot was the fact that a female monkey lived in the park for several years, so it is commonly known as the Monkey Park) (Figure 9): On an identification plaque, the AR functionality overlays an animated three-dimensional model of the mascot and her living cage, allowing the visualization of the monkey that remains an iconic symbol of the park.

In summary, the EduPARK app innovation relies on the articulation of the following components: (i) The use of new and easy to explore technologies, mobile supported AR; (ii) Geocaching-based learning in outdoor environments; and (iii) cross subjects educational materials (guides specific for different educational levels) [25].

3. Materials and Methods

Mixed method research approaches are known for combining qualitative and quantitative elements to achieve a level of understanding and corroboration in breadth and depth, not possible through either approach on its own [26]. So, a mixed method approach is used to analyse the effect of using the EduPARK app into the following dimensions of educational value: (a) Learning value; (b) intrinsic motivation; (c) engagement; (d) authentic learning; (e) lifelong learning; and (f) conservation and sustainability habits by triangulation of teachers and students’ opinions, after playing the EduPARK game, with game results.

This section comprises four sections: (i) A context introduction regarding the EduPARK activity and study participants; (ii) data collection methods; (iii) data analysis strategies; and finally, (iv) sample description.

3.1. The EduPARK Activity and Study Participants

To collect data, the research team organizes activities involving schools and other educational entities (such as after school study centres) for a period of about one year, from March 2018 to April 2019, in Infante D. Pedro Park (Aveiro, Portugal). The EduPARK activity, organised under the project, comprises:

(a)

A small introduction about the activity program and some instructions on how to use the EduPARK app to play;

(b)

the EduPARK app does not require internet connection, and the game is usually played by teams of three or four students accompanied by an adult (mostly teachers, but also other school staff or EduPARK team members);

(c)

the leaderboard construction and announcement to participants, with small prizes distribution to the three groups with higher scores, such as medals of 1st, 2nd, and 3rd places or project’s merchandising items (for example, pen, bracelet, mobile phone holder, yo-yo, etc., depending on the age of the players).

For the game playing, the EduPARK project provided mobile devices (smartphones or tablets) for participants to use during the activity, whenever needed.

A total of 44 activities for 1007 students, from the 1st Cycle of Basic Education (CBE) to Secondary Teaching, and 122 accompanying adults, usually teachers, were organised both in formal and non-formal educational contexts. After student distribution by groups and resources checking, the groups started playing the educational guide for their academic level, in a lagged departure organization. Five activities (out of 44) were mixed in what concerns students’ school level, and the games played in these sessions were either the one for the 1st CBE or the one for the 2nd/3rd CBE.

Table 1. The relation of number of activities, participant students, and accompanying teachers in the EduPARK activities, per school level. CBE: Cycle of Basic Education.

	1st CBE	2nd/3rd CBE	Secondary Teaching	Total
Number of activities	23 (5 mixed)	21 (5 mixed)	3	42
Number of students	476	396	74	1007
Number of teachers	91	24	7	122

shows, per school level, the number of activities (total of 42), students (total of 1007), and teachers and other adults (total of 122) who accompanied the students in the activities between March 2018 and April 2019. Considering both students and teachers, the total of participants was 1129.

3.2. Data Collection

Data collection included a paper questionnaire and automatic app logging mechanisms, supporting triangulation of users’ subjective points of view with their performance during the game (for example, number of right answers), which are objective results. At the end of the game playing activity, each participant (both students and teachers or other accompanying adults) was invited to complete an evaluation questionnaire. The questionnaire was similar for all the school levels and types of users (students and teachers); however, it included adaptations to the age and educational context of the respondents. Teachers who accompanied classes in the activity more than once filled in only one questionnaire, in the first activity, so the number of teacher questionnaires is lower than the number of effective participant teachers. This option supports the respondents in avoiding biases caused by increasing familiarity, as, according to Sauro [27], having prior experience with the system increases scores. For example, when analysing websites usability with Brooke’s System Usability Scale [28], Sauro found that users who had previously experienced the website, tended to generate higher SUS scores (11% higher) than first-time users.

The set of questionnaires analysed in this study were used in previous studies of the EduPARK project [9,29]. The tool comprises four sections, with mostly closed-ended questions in a Likert scale, where 1 corresponds to ‘strongly disagree’ and 5 to ‘strongly agree’, and a minority of open-ended questions that complement the quantitative data and provide a level of insight not captured by the closed questions. One section collected basic demographic data, such as age and gender, their familiarity with mobile devices, and their opinion on mobile learning advantages and disadvantages. Another section is about the interest regarding the activity of playing the EduPARK game in the park; although this is not the focus of this work. Another section refers to the Educational Value Scale (EVS) (presented in [9] and reliability demonstrated in [21]), and the last one consists of the System Usability Scale (SUS) [18,28,30] with the minor adjustment of replacing the word “system” with “app”, which does not appear to have an effect on the resulting scores [19]. The same authors highlighted SUS high reliability and pointed it as the most sensitive poststudy questionnaire, designed to assess perceptions of usability.

The questionnaire for students of the 1st CBE did not have questions on mobile learning advantages and disadvantages nor an open answer question to justify their opinion regarding the interest of the activity. Teachers’ questionnaires had additional questions regarding the interest of the activity for their own practice and for their colleagues.

Only fully answered questionnaires were included in the study. All teachers’ questionnaires were considered, however, some students’ questionnaires were discarded, as some questions were not answered.

The app includes automatic mechanisms of game log generation. These mechanisms allowed the collection of anonymous information from finished games during the observation period (between March 2018 and April 2019). The information includes: (a) Final score (points gathered through correct answers and points gained through collecting bananas); (b) game time; (c) number of questions answered correctly and incorrectly; and (d) number of hunted Geocaching treasures.

All data collection, processing, and storage procedures respected research ethics principles. Data were collected anonymously and did not include any personal information or set of information allowing the identification of specific participants.

3.3. Data Analysis

The focus of this study is the educational value of mobile AR games, analysed through an illustrative example, the EduPARK app. So, the questionnaire data analysis focuses the section comprised by the EVS, EduPARK activity interest, including qualitative data. Game logs were also analysed to triangulate the users’ opinions.

Data were analysed through scores computing, descriptive statistics, and content analysis of open response questions.

The computing of EVS scores is similar to the SUS computing process described by Brooke [18,28]. However, as described by Sauro and Lewis [19], to get a 12 items scale to range from 0 to 100, the sum of the 12 EVS items contribution is multiplied by 2.0834 (the result of the division of 100 by 48).

As questionnaires’ open questions aimed solely to collect the users’ reactions to the EduPARK activity, content analysis was performed to present illustrative citations of participants’ answers.

Document analysis of the app game logs was performed. A table with the overall results, regarding game scores attained, number of correct and incorrect answers, game time, etc., is presented in the next section.

Finally, the questionnaire and app log data were triangulated to provide a more comprehensive knowledge of users’ opinion regarding the EduPARK app educational value. This analysis is presented in the next section.

3.4. Sample Description

The survey allowed gathering information about the users’ profile.

Table 2. A characterization of each type of participant on the EduPARK activities, from March 2018 to April 2019.

	Type of User	1st CBE	2nd/3rd CBE	Secondary Teaching	Accompanying Teachers
General Information
Valid questionnaires		405	384	65	70
Age	Range	6–11	10–15	15–20	23–64
	Mean	8.1	11.0	16.6	42.1
Gender (female)		192 (47.4%)	204 (53.1%)	36 (55.4%)	54 (77.1%)
School year/Academic degree		year 1: 28 (6.9%) year 2: 177 (43.7%) year 3: 127 (18.0%) year 4: 73 (18.0%)	year 5: 294 (76.6%) year 8: 61 (15.9%) others: 29 (7.6%)	year 10: 22 (33.8%) year 11: 28 (43.1%) year 12: 15 (23.1%)	Graduation: 47 (67.1%) Masters: 15 (21.4%) others: 8 (11.4%)
Owns Android		287 (70.9%)	331 (86.2%)	55 (84.6%)	65 (92.9%)
Mobile learning	Often	108 (26.7%)	77 (20.1%)	29 (44.6%)	28 (40.0%)
	Sometimes	194 (47.9%)	261 (68.0%)	33 (50.8%)	32 (45.7%)
	Never	108 (25.4%)	46 (12.0%)	3 (4.6%)	10 (14.3%)

presents the characterization of each type of user in terms of number of returned questionnaires (valid and fully filled in), age, gender, school year, Android mobile device ownership, and its use to learn or to promote learning. The response rate was high in all types of users, more specifically, it was 85.1% for 1st CBE students, 97.0% for 2nd and 3rd CBE students, 87.8% for Secondary Teaching students, and 57.4% for accompanying teachers. It is worthwhile to note that several teachers accompanied groups of students in more than one activity; hence, they were counted more than once in the participants section. However, these teachers were asked to answer the questionnaire only in their first participation, capturing their immediate perceptions. As highlighted in the Materials and Methods section, the literature reports an increase of SUS scores with the increasing familiarity with the system under evaluation [27].

Students’ ages vary within the expected values for their respective schooling levels in Portugal: 6–11 and mean value of 8.1 for 1st CBE, 10–15 and mean of 11.0 for 2nd and 3rd CBE, and 15–20 and mean of 16.6 for secondary teaching. Teachers’ ages vary between those that are expected for graduated workforces, between 23 and 64 years-old, with a mean of 42.1.

The proportion of female and male students is balanced (47.4%, 53.1%, and 55.4% of females); however, the teaching class is composed mainly by female teachers (77.1%) according to the general scenario in Portuguese schools, where the female teachers percentage was 77.9 in 2019 [31].

In the 1st CBE, most participating students frequented year 2 (43.7%), followed by year 3 and year 4 students (both 18.0%), and fewer students in year 1 (6.9%), which are expectable results, as the EduPARK app requires reading skills. In the 2nd and 3rd CBE, most students frequented year 5 (76.6%), followed by year 8 students (15.9%), with a small participation of students from other school-years (7.6% for years 6, 7, and 9). This participation disparity can be associated with the National Directives for Natural Sciences Curriculum, as environment-related learning content is the focus of school-years 5 and 8. In the secondary teaching context, the participation in the EduPARK activity involves one class of each school-year. Comparing with the other school levels, this smaller level of participation may indicate a lower educative value of the app for secondary teaching. An alternative explicative hypothesis can be teachers’ fears of adopting new teaching methodologies with classes of students that will be submitted to mandatory national exams. Regarding participating teachers, all had higher education qualifications, which is a prerequisite in the Portuguese Education System. A considerable portion (32.8% = 21.4% + 11.4%) continued studies further (Post-graduation, Master’s degree, and Doctorate).

The majority of students and teachers owned Android mobile devices, such as smartphones or tablets. The lower Android device penetration rate is in the 1st CBE group (70.9%), and the higher rate is in the teachers’ group (92.9%).

Finally, most participants mentioned using mobile devices to learn or to promote learning. Most students and teachers claimed they used mobile devices for learning purposes either sometimes (47.9%, 68.0%, 50.8%, and 45.7%) or frequently (26.7%, 20.1%, 44.6%, and 40.0%). The smallest answer proportion was registered in the “Never uses” option for all types of participants (25.4%, 12.0%, 4.6%, and 14.3%). According to the results, most of these students and their teachers are already quite familiar with mobile technologies and usually employ them for learning. The results seem to support the literature, regarding the proliferation of mobile devices [29], especially in what concerns the young population.

When questioned about advantages of the use of mobile devices to learn, students and accompanying teachers mostly indicated a positive perspective regarding mobile learning, and Figure 10 shows the percentage of agreement with each advantage sentence.

All sentences related with positive aspects of using mobile devices to learn achieved a frequency of at least 244 students of basic education from a total of 384 (63.5%), 36 Secondary students from a total of 65 (55.4%), and 46 accompanying teachers from a total of 70 (65.7%). Only 3.6% of basic education students did not recognize any advantage in mobile learning. Among the most acknowledged advantages are ‘you can learn in a fun way’ (81.8% for Basic Education students and 88.6% for accompanying teachers), and ‘it is easy to find the information I want’ (84.6%), for Secondary students. Moreover, 9.9% Basic Education students added new advantages, such as: ‘Learn quickly’, ‘It’s much more interesting’, ‘It’s not boring’, ‘We can work as a team’, and ‘We can learn on other locations besides school’. On the other hand, accompanying teachers (5.7%) reported ‘The use of images’, ‘It is fast’, and ‘It promotes discovery learning’.

Regarding the difficulties of mobile learning, Figure 11 shows that 141 Basic Education students (36.7%), 29 Secondary students (44.6%), and 16 accompanying teachers (22.9%), recognized not having any difficulties in the use of mobile devices to learn. The most stated difficulties are the need for an internet connection (mentioned by 62.9% of accompanying teachers and 59.9% of basic education students), and increased battery consumption (mentioned 55.4% of secondary students). The EduPARK project approach contributes to reduce these constrains, as: i) The game supporting app was conceived for offline use, not requiring internet connection, so this is not an issue; ii) in activities promoted by the EduPARK team, the project provides full charged mobile devices, thus, not interfering with the mobile devices’ battery of participants.

Finally, 5.5% Basic Education students added new difficulties, such as ‘This is only for androids’, ‘It is not possible to explore the game in the classroom’, ‘running out of battery’, and ‘preference to study in books’. The first mentioned difficulty will be considered in further similar apps, intending to produce games for both Android and iOS operating systems.

4. Results and Discussion

This section reports this study’s main results and their discussion in light of the consulted literature. The data were collected in the EduPARK activities conducted between March 2018 and April 2019. First, the authors analysed the EduPARK app EVS for the entire aggregated dataset and for each of the considered target public: Students of 1st Cycle of Basic Education (CBE), 2nd and 3rd CBE, and secondary teaching, and their accompanying teachers and other adults (other school staff, parents, etc.). Data from open questions of the questionnaire about the interest of the EduPARK activity are also analysed through content analysis. Finally, app game logs of finished games uploaded to the EduPARK web platform were submitted to document analysis.

4.1. Users’ Perceptions: Educational Value and Interest

Figure 12 presents the participants’ general perceptions on the app educational value. The EVS items are worded in positive and negative sentences alternatively. Overall, participants’ perceptions are positive, as most respondents (strongly) agree with the scale positive formulated items and (strongly) disagree with the negative formulated items. For example, 629 (68.1%) participants strongly agree with the sentence “This app helps you/students learn more about topics we study/I teach at school”. A similar amount, 639 (69.2%) participants, strongly disagree with the sentence “This app does not help to realize that it is important to protect nature.” These results indicate that participants considered that the EduPARK app comprises all the dimensions of the educational value analysed in this study: Learning value, intrinsic motivation, engagement, authentic learning, lifelong learning, and conservation and sustainability habits. Subsets of students and teachers from this dataset achieved similar results in previous studies [9,29,32].

EVS data are submitted to exploratory data analysis.

Table 3. EVS scores descriptive statistics for each type of user of the EduPARK app, in the activities organized by the project, from March 2018 to April 2019.

	Type of User	1st CBE	2nd/3rd CBE	Secondary Teaching	Accompanying Teachers	All Users
Descriptive Statistics
EVS	Mean	83.0	83.0	69.1	83.0	82.0
	Median	85.4	85.4	70.8	86.5	85.4
	Mode	91.7	91.7	72.9	89.6	91.7
Standard deviation		13.9	13.6	13.4	11.9	14.0
Minimum-Maximum		16.7–100.0	20.8–100.0	39.6–100.0	54.2–100.0	16.7–100.0

presents the computed EVS scores arithmetic means, medians, modes (measures of central tendency), standard deviation (measure of variability), as well as minimum and maximum values, for each type of user and for the entire dataset. This table reveals high arithmetic means and medians of EVS for all participants in this study: All above 80 for students in basic education and accompanying teachers, and around 70 for students in secondary teaching. Additionally, the median is 85.4 for two EduPARK’s target publics and for all users. It is worth to note that all groups of participants included several users that attributed the highest value possible for EVS (100). This preliminary analysis indicates that the EduPARK game educational value is considered high by the questionnaire respondents; however, it seems to be considered less relevant for secondary teaching students.

Further analysis included the multiple boxplot display shown in Figure 13. The boxplots in this figure present both summary statistics (minimum value, lower hinge, median, arithmetic mean, upper hinge, and maximum value) and raw data. According to Theus and Urbanek [33], boxplots show robust measures of location and spread of datasets, and they are known for providing visual aids to compare two or more datasets. In each boxplot, the vertical lines represent ordered data (from the minimum to the maximum values); the boxes contain the middle range data, from the 1st quartile (25th percentile) to the 3rd quartile (75th percentile); the middle lines in the boxes indicate the median value for each dataset; and the cross below the median indicates the arithmetic mean. Basic education datasets present potential outliers, which are the dots located 1.5 times below the size of the box. Therefore, the arithmetic mean of these two datasets must be treated with scepticism, as this central tendency measure is affected by outliers. In this case, the median is a suitable central tendency measure [33].

It is possible that the presence of potential outliers in the younger students’ datasets is due to the high level of excitement related to the timing and location where the questionnaires were applied (just after playing the game, in the park). This was reported before ⁹, and may have hindered students’ concentration during the questionnaire filling. Moreover, possible reading difficulties of young students may also have biased the results. These hypotheses are supported by a previous study [21], where a relatively low Cronbach’s coefficient α (0.653) was found for this dataset. According to Hair et al. [34], this value indicates a not yet acceptable reliability (0.7), however, other authors consider the value 0.6 as the lower bound of reliability acceptance, particularly in the early stage of research [35,36], which is the case of the EVS. To reduce the impact of reading difficulties, children were supported by the adults who accompanied each group, to assure they understood the questions and answer options, whenever needed.

As mentioned above, the medians of the two datasets of basic education students are 85.4, the one of the accompanying teachers is very similar, 86.5; and the median of secondary teaching students is 70.8, which is the lowest. Moreover, as secondary teaching students’ median lies entirely outside the interquartile ranges of the remaining target groups, this dataset is likely to be different from the other three.

The interquartile ranges (box lengths) are small and similar for all the datasets: 16.7 for 1st CBE, 14.6 for 2nd/3rd CBE, 19.8 for secondary teaching, and 16.7 for accompanying teachers. Hence, all the datasets seem to concentrate near their median values.

Regarding distribution of the datasets, the box and whiskers considered together reveal the range of each dataset. Secondary Teaching students are the participants with the most scattered data, as its range is 60.4. Therefore, the consensus regarding the Educational Value of the EduPARK app among Secondary Teaching students was lower than among the other types of participants.

As the interquartile ranges of basic education students and accompanying teachers’ datasets roughly overlap each other, these datasets seem similar. On the other hand, the interquartile range for Secondary Teaching students’ dataset is located below all the other interquartile ranges, also indicating there is a difference between them. These results support the initial analysis (presented in Table 3), as the EVS median values for basic education students and accompanying teachers are higher than the EVS median value for secondary teaching students. These results are supported by previous studies, where the mean EVS value was 83.8 for 244 students in 2nd and 3rd CBE [29] and 88.2 for a cohort of 45 teachers in a workshop [32]. This indicates that the EduPARK app has a high educational value. Moreover, in this study, teachers also assigned high EVS scores, revealing the EduPARK app is a mobile learning resource with a high educational value for practitioners.

To understand further the case of the educational value of the EduPARK app for Secondary Teaching, the cohort of teachers who accompanied Secondary Teaching students was analysed separately to collect indicators of possible differences in teachers’ perspectives of the educational value of the app, according to the school level (basic education vs secondary teaching) they accompanied. These teachers constitute a small portion of this type of participant (7 in a total of 70), and assigned the following EVS scores: 81.3; 81.3; 89.6; 89.6; 93.8; 97.9; and 97.9. This yields a mean of 83.1 and median of 87.5, which is in line with the overall teacher dataset. Hence, despite this small number of teachers, this result supports the idea that, although the EduPARK app educational value for secondary teaching may not be immediately identified by teachers, the practitioners who participated with secondary teaching students revealed a very positive opinion regarding the app’s educational value.

4.2. Game Logs: Educational Value

App game logs of finished games uploaded to the EduPARK web platform are presented in

Table 4. The app game logs of finished games uploaded to the EduPARK web platform from March 2018 to April 2019.

	Type of User	1st CBE	2nd/3rd CBE	Secondary Teaching
General Information
Number of returned logs		125	100	28
Final score	Mean	234.4	271.1	213.0
	Standard deviation	42.2	44.3	39.3
	Minimum-maximum	134–320	175–361	134–260
Game time	Mean	01:28	01:19	01:15
	Standard deviation	00:17	00:13	00:24
	Minimum-maximum	00:52–02:12	00:43–01:55	00:34–01:54
Right answers	Mean	21.6	24.4	18.5
	Standard deviation	3.4	3.6	4.0
	Minimum-maximum	11–29	17–31	11–25
Wrong answers	Mean	5.3	6.6	11.5
	Standard deviation	3.2	3.6	5.6
	Minimum-maximum	0–16	0–14	4–23
Found treasures	Mean	3.6	3.8	3.8

to triangulate with data gathered from the questionnaire for more comprehensive knowledge regarding the EduPARK app educational value. Each log corresponds to the performance of a group of students (usually 3 or 4) who played the game collaboratively, in teams. These values (final score, game time, right and wrong answers, and found treasures) are generated automatically by the mobile devices and uploaded to the project web platform, after game over. The data are anonymous, and accessed only by the project team.

Table 4 presents the app game logs organized per target group. The final score is calculated by the sum of points gathered through correct answers and points gained through collecting bananas in the hunted treasures. The achievement of higher scores indicates higher educational value, as teams need to observe their environment, to select information provided by the AR contents accessed through the app, to analyse alternative solutions for the challenges, to negotiate meaning among group members, and to provide the answer considered correct by the collective.

From the table, the final score average is higher in the 2nd/3rd CBE group of students, as well as the minimum and maximum final score, compared to the other type of students. This is in accordance to the EVS scores for this group, reinforcing the high educational value of the app for this school level.

Game time, presented in the format hour:minute, corresponds to the time consumed from starting the game to its end (when the final scores are displayed in the last screen). The average game time is decreasing with the age of students, as older students are quicker to read and to answer questions (reaching less minimum and maximum time to finish the game), when compared to the younger ones. The game time does not indicate better performance, because students could finish the game very quickly, for example, in just over half an hour, without answering correctly the questions and not properly exploring all the educational resources available in the game, for example, AR contents, 3D models, images, or additional information. Moreover, the variance in game time can also indicate that students progress in the activity at their own pace, providing enough time for students to interact with each other and their prior knowledge with the new experiences, and thus, supporting learning. The fact that all groups were able to finish the game also indicates that the educational guides supported by the app are adequate to their respective school levels.

The average of questions answered correctly is higher in the 2nd/3rd CBE group of students (24.4) comparing to the 1st CBE (21.6) and secondary teaching group of students (18.5). Accordingly, the minimum and the maximum number of right answers is also higher for this group of students. The opposite situation occurs with the wrong answers, showing that some basic education groups answered correctly all the questions (zero wrong answers), contrasting to 4 minimum wrong answers for secondary teaching students. Accordingly, the older students reached higher values of maximum number of wrong answers (23), indicating that this type of student revealed more interest in finishing the game quickly than having a good learning performance. It is worthwhile to note that the AR contents, which are mostly provided before the proposed challenges, support the students in finding the solutions. It is a matter of students deciding to explore them properly, in order to select and analyse the contextualized information, to have a good performance in the game. The differences in the means of right and wrong answers, once more, indicates that the app is more suitable for the 2nd/3rd CBE group of students.

The averages of hunted Geocaching treasures are quite similar in the three types of students (varying from 3.6 to 3.8), which means that almost all group of students found the four virtual treasures available in the game. The Geocaching treasure hunt is a feature introduced in the game for motivational purposes, so the fact that the groups attempted to find the treasures is a strong indicator that they are engaged with the activity. It is worthwhile to note that finding the treasures is not essential to proceed with the game, as the groups could keep playing without finding any treasure.

Finally, the above presented game logs analysis triangulated with EVS scores (presented in the previous sub-section) reinforces the finding of the suitability of the EduPARK game as an educational resource, particularly for 2nd/3rd CBE students. Moreover, participants’ perceptions are in consonance with students’ performance in the game, as they achieved, overall, a good game performance.

5. Conclusions

This work addresses an identified need of research reported before [9], as the previous analysis of the Educational Value of the EduPARK app, with a smaller sample of 1st CBE students, is now expanded to include a higher number of respondents and other target publics.

The EduPARK game achieved an average EVS of 82.0, with higher values for the subsets of data referent to basic education students and for teachers, who experienced this mobile AR game in loco, that is, in a Green City Park. This applies to a sample of users who claimed: i) To have their own Android mobile devices (the lower penetration rate was registered in the younger students, with 70.9%); ii) to use them to learn or promote learning, at least sometimes (particularly among secondary teaching students, with 95.4%); and iii) to have a quite positive perspective regarding mobile learning, considering it is more advantageous than disadvantageous. The high educational value of this mobile AR game is supported by the data collected through the app logging mechanisms, as the groups of students achieved a good performance, overall. Educational resources that combine this set of innovative features, as being mobile, designed for outdoor use (namely in urban parks), with contextualized AR contents, and supporting game-based activities, may promote learning, both at a cognitive level and at an affective one, increasing motivation for learning.

The game revealed a particularly high educational value for basic education students, as secondary teaching students assigned it lower EVS scores. Moreover, the small number of activities with secondary teaching students (3 out of 42) seems to support the lower adequateness of this methodology for this school level. However, both basic education and secondary teaching teachers, who accompanied students in the activity, revealed a very positive opinion regarding the ability of the app to promote learning. This result may indicate that secondary teaching teachers are more reluctant in trying out new approaches with students of a school level that comprises national exams. This issue needs further study, as the number of teachers accompanying this school level was low (7 in a total of 70).

Secondary teachers’ reluctance in adopting this mobile AR game may be due to dominant mentalities associating mobile devices, games, and parks to distraction, play, and leisure [12]. Nevertheless, early adopters among teachers seem quite optimist towards this approach and may promote changes in their colleagues’ mentalities on how their students can learn. Still, more studies involving a higher number of students and teachers of this school level are needed.

Limitations of this study are related to the young age of most participants (405 students in 1st CBE and 384 students in 2nd and 3rd CBE). As discussed before, factors such as level of excitement and reading difficulties may have influenced the results, for this age group. Compensation methods may include adapted questionnaires in terms of vocabulary and supporting children in the interpretation of the items, which were the main strategies in the EduPARK activities. In future studies, the use of face emojis instead of the numbered Likert may be a powerful alternative for young students [19].

Another issue to consider is groups’ constitution, which may also have influenced the results, as each student’s participation in the game is lower in bigger groups. This factor may have an impact on how the activity is perceived by the app players. However, this variation in groups’ constitution could not be addressed, particularly with activities with a lot of participants, as it was related with the human resources available to accompany each group of children in each session.

Finally, as the EVS is a new data collection tool, more studies involving the rating of other educational mobile AR games are needed, to both improve and consolidate this data collection tool, and also to establish benchmarks and scale norms. Future research must address this issue, by implementing the EVS with more users and by analysing the educational value of other mobile AR games.

In terms of implications for research, this paper contributes to the mobile game-based AR learning literature, as it is an empirical study with evidence on the educational value of the integration of these elements in teaching practices. For practitioners, this work also bears the report of an example of excellent cross-subjects’ educational materials—the learning game—that comprises a very useful tool for teachers and students to explore scientific knowledge by accessing appealing information on cross subjects references (such as biological and historical) of a local Urban Green Park.