In this section, we give an account of the architecture of the PlanetarySystemGO platform and present the implementation tests of the game that were performed with the targeted public and their results.
5.2. Implementation Tests of the Game
As mentioned before, the MAR platform has been in development since 2016 by higher education students supervised by higher education teachers in the context of their final project before graduation. Throughout the platform development, several implementation tests were performed with the targeted public to evaluate its impact and identify any performance problems (Table 4
This research is inserted on a broader project of intervention targeted to promote STEAM hands-on experiments in primary schools. During students’ holidays in the polytechnic campus, children’s parents signed a document about agreement or not related to data collection. In primary school classes, the school leaders authorized the study and informed the parents about it. When we go to primary school classes, the teachers always have the documents signed by the children’s parents, and data is only collected from the children whose parents authorized.
In a first stage, the game’s target audience was primary school students and the first version aimed at exploring our Solar System. In the school year 2016/2017, two implementation tests took place on the campus of the Instituto Politécnico de Tomar during Christmas 2016 and Summer 2017 holidays. In this version, it was possible to visualize the planets on the smartphone but, because this was an incomplete version, QR code-based markers were used to give information about the planets of the Solar System [11
]. Also, we used different models of smartphones according to their availability, and concluded that low level models may seriously compromise the game performance. This preliminary study allowed the authors to conclude that this approach was efficient to catch children’s attention and promote learning about the planets of the Solar System. For this reason, it was decided to upgrade the game in the following school years.
In the school year 2017/2018, an information system that communicates with the app (Figure 3
) was developed and may provide the player with different experiences every time he plays the game. For example, it is possible to create pedagogical events with other planetary systems of the Universe besides our Solar System. In addition, the back-office allows teachers and instructors to introduce contents according to the grade level they teach and, for this reason, it is possible to extend the experience to any grade level [30
]. Furthermore, the mobile app informs the Platform server about the results of the pedagogical event, namely the scores of the players.
To implement the game on the field, we decided to buy 6 medium-level smartphones that guarantee a good experience with the app. After some tests, the system chosen was the smartphone SAMSUNG Galaxy J5 2017 16 GB model. This device made possible to begin deploying the game outside the polytechnic campus. On the International Children’s Day (2018 1st June), an experience took place at Mata dos Sete Montes with 76 children (Figure 4
and Figure 5
) with the support of Tomar City Council.
In this experience, we had limited space and many trees around it, which made it difficult to find the planets (Figure 6
). Also, every 30 min a new class of students arrived to join the activities, which gave limited time to play the game with each class: ~20 min. However, again, children were very excited when playing the game.
In the end, a questionnaire was applied to them and to the eleven teachers who accompanied the classes. First two questions applied to students were: “Did you have fun playing the game?” and “Would you like to play it again?”. As can be observed in Figure 7
, most students answered that they enjoyed playing the game and would like to play it again. The blue color (on the left) is for “Yes” and the orange color (on the right) for “No”.
Another question applied to students was: How much did you like the game? Following a Likert scale, the options to answer this question were: “I did not like it at all”, “I liked it a little bit”, “I somewhat liked it”, “I liked it”, and “I really liked it” (Figure 8
). From the 76 students, three did not answer the questionnaire because they wanted to keep looking for planets. According to Figure 8
, no one answered “I did not like it at all” and “I liked it a little bit”, six students answered “I somewhat liked it” (gray color), fourteen students “liked it” (yellow color), and fifty-three students “I really liked it” (light blue color). Explanations for not liking the game were related to difficulties to find some planets and also because in some groups some children did not have the opportunity to hold the smartphone.
The questionnaire applied to the teachers included the following questions. “Do you think your students had fun playing the game?” and “Do you find this game adequate to primary school curriculum?”. Results are in Figure 9
. All eleven teachers answered “yes” (blue color) to both questions.
After this experience the designers of the game improved some aspects that they found in the last test. For example, planets were above the orbits (Figure 5
). Also, there was a need to adequate the game to the available game arena. In this regard, four scale options, representing the distance of the last planet (Neptune) to the Sun, were introduced: 50 m, 100 m, 500 m, or 1000 m (Figure 10
). With this option, the player may choose the game arena according to the outdoor space available to play the game.
A few weeks later, another deployment test took place at the Polytechnic campus with a group of twenty students during their Summer holidays. The chosen scale was 100 meters which allowed to have a much wider game arena than in the city garden with no big trees around. However, there were some buildings and cars spread in the game arena. After the game that occurred for approximately 40 min, some interviews were applied to the participants. One question was: what did you like more in the game? Below are some answers.
In what concerned the students’ major dislikes, we highlight the following answers.
Not be able to find all the planets
Planets crossing walls and cars
Last planets were far away and it took too much time waiting for its journey around the Sun
Having to wait for the planets that were lost in the walls
After reflection and discussion, several adjustments were performed to improve some of the identified issues, namely, last planets velocities around the Sun. Also, to better identify the orbits and planets that were captured, several improvements were performed. For example, when the player gets closer to the planet its size is increased. In addition, when the orbit is found, it changes its color from yellow to green (Figure 11
), and when the planet is captured, it receives a flag with a logo from the broader project (http://www.academiacap.ipt.pt
Finally, based on the good results in informal learning environments we decided to implement the game in a formal learning environment at a primary school in a 4th grade class with 20 students (Pilot experience). In class, after a small presentation about the Solar System, the students were organized into five groups according to the number of available mobile phones with the application. Figure 11
shows some sequences of the game.
After the students played the game, they completed a questionnaire in order to assess the impact of the game on the class. First two questions were: “Did you have fun playing the game?” and “Would you like to play it again?” (Figure 12
). As can be observed in Figure 12
, most students answered that they enjoyed playing the game and would like to play it again.
Another question was: How much did you like the game? Following a Likert scale, the options to answer this question are: “I did not like it at all”, “I liked it a little bit”, “I somewhat liked it”, “I liked it”, and “I really liked it” (Figure 13
). According to Figure 13
, no one answered “I did not like it at all” and “I somewhat liked it”, only one student answered “little” (orange color), three students “liked it” (yellow color), and fifteen students “I really liked it” (light blue color). The student who answered “I liked it a little bit”, was an autistic child. He explained that he did not like to run, and his team was running all the time.
Also, the teacher answered a questionnaire, containing the following questions: “Do you think your students had fun playing the game?”, “Do you find this game adequate to primary school curriculum?”, “Does this approach favour learning comparing to traditional methods?”, and “What are the advantages of this approach to the Solar System?”. The answer to the first three questions was “Yes”. In the last question, she referred that the game provided the students with a real experience by visualizing the planets that improved their learning about the Solar System contents in a ludic approach.
Another feature that was tested in the last experience was the information transmitted by the mobile app to the Platform server. Besides providing the development of the application in the mobile phone, the back-office is in charge of processing and presenting data collected during the game. Therefore, it gives access to the scores of each student and to the answers that each student gave to each question. Figure 14
shows the results of the final score of each of the five teams who played the game with the information provided by the back-office.
The winner team, named “Mistério” (Mistery), scored 39 points and the worst score was 19 points from “Grupo Especial de Planetas” (Special Group of Planets). Unfortunately, one of the mobile phones had a lower performance than the other four and this was one the main reasons for this worst result. In fact, variables related to sensors such as gyroscope and accelerometer may compromise the efficacy of finding orbits and planets and the problem increases with lower performing mobile phones.
Because of the positive feedback of this pilot experience, since 2018 the game is being implemented with success with students aged 8 to 10 years old, in the context of a broader intervention project targeted to implement STEAM hands-on experiments in primary schools. Until June 2019, the game was implemented in several 3rd and 4th grade classes. Questionnaires were applied to students and to teachers in charge of classes after previous participants consent. A total of 102 students and nine teachers agreed to answer the questions and an exploratory data analysis was performed. Concerning the question “Would you like to play it again?”, 97 students answered “yes” and the other 5 answered “no”. Results of the answers to the question “How much did you like the game?” are presented in Table 5
As can be observed in Table 5
, 86 students answered “I really liked it”, which means that most of the students really enjoyed playing the game. Again, some students complained they did not had the opportunity to use the smartphone and for this reason it was not a great experience. To improve the results we intend to buy more smartphones to provide a better experience in the future.
The teachers’ questionnaires contained the following questions. “Do you find this game adequate to primary school curriculum?”, “Try to quantify on a scale of 1 (very little) to 5 (very high) the importance that this approach may have in favoring learning comparing to traditional methods of teaching?”
The responses were organized on a Likert-type scale with five categories (organized from one to five). For the first question “1” means “Not adequate” and “5” means “Very adequate”. For the second question “1” means “Not important” and “5” means “Very important”. The ten teachers only chose the “4” and “5” options for both questions as can be observed in Table 6
and Table 7
, which confirms the opinion of the teacher from the pilot intervention in a primary school.
The sample size available is small, but the teachers involved recognized the game is adequate to primary school curriculum and it favors students learning comparing to traditional methods of teaching. Currently, more sound instruments of measuring are being considered in order to provide a more complete statistical analysis.
During the implementation of the game, it was observed the great enthusiasm of the children. From the beginning, they were very interested in the topic and showed great commitment to play the game. In addition, the spirit of mutual help and communication between peers was observed, for example to choose the correct answer to the question displayed in the smartphone screen. After playing the game in primary schools, focus group was performed between the students, the teachers and two of the researchers (first two authors of this paper). As well as various manifestations of enthusiasm about the game, suggestions for improvement are also made, such as the introduction of asteroids, among other celestial bodies. Almost all students end up expressing interest in continuing to play by asking how they can install the game on their smartphone or tablet. Teachers also participate by reinforcing the importance of introducing these approaches to motivate children to learn.
After the good experience with students and teachers, the priority was to provide a back-office that would be user-friendly for primary school teachers, because our final goal is that they may implement the game with their students adapting the information about the planets and questions to the grade level they teach.
The celestial objects and the planetary systems that group them together are stored in repositories that may allow the sharing of pedagogical contents among teachers. Figure 15
is an example of our Solar System that is included in the public repository.
gives an example of a webpage where questions about the planets may be introduced. In this particular case, the multiple-choice question is related to Mars. Other webpages with questions about other planets have a similar image.
Menus are still in Portuguese but our next priority is to introduce multi-language support in the next version of PlanetarySystemGO, allowing instructors to set information and questionnaires in their own language, thus reaching audiences at a more global level (http://PlanetarySystemGO.ipt.pt
In summary, the game consists of discovering a planetary system that includes a set of celestial objects and is parametrized to be played in a real environment in the context of augmented reality provided by the application in a smartphone. The back-office is still under development but it already allows teachers to introduce information about the planets and questions to assess students’ knowledge about the contents they introduce.