Educational Uses of Augmented Reality (AR): Experiences in Educational Science

: Augmented Reality (AR) is an emerging technology that is gaining greater inﬂuence on teaching every day. AR, together with mobile technology, is deﬁned as one of the most e ﬃ cient pairs for supporting signiﬁcant and ubiquitous learning. Purpose of the study: the Instructional Material Motivational Survey (IMMS), by Keller, was used to determine the degree of motivation possessed by the Pedagogy students on the utilization of the notes enriched with AR in the classroom, available for their didactic use through mobile devices. Methods: through an app designed for the courses Education Technology (ET) and Information and Communication Technologies (ICT) Applied to Education, the motivation gained when participating in this experience, and how it inﬂuences the improvement of academic performance, was evaluated. Results and conclusions: the most notable main result was ﬁnding a strong relationship between the motivation of the students when using the enriched notes and the increase of performance in the academic subject where it was used. Likewise, it was proved that the use of Augmented Reality beneﬁted the learning process itself.


Introduction
The document presented investigates both an original and novel field that contributes, exponentially, to the scope of the sustainability of this magazine, especially considering that Augmented Reality in the educational field has a perfect place in the present monograph "Online and Ubiquitous Training, Mobile Technology in Education and Sustainability".
Augmented Reality (AR) is one of the emerging technologies that have rapidly been incorporated into the education sphere due to the diverse possibilities it offers [1][2][3][4][5][6][7][8]. Among other reasons, this is due to the ease of accessing information offered by this tool, as it is normally accessed thanks to mobile devices, with these technological resources being highly present with students in the Ibero-American context [9,10].
AR offers the possibility of mixing and combining two environments: the physical and the digital, and all this in real time, through the use of emerging and easily accessible technologies, such as smartphones or tablets. All this makes possible the generation of another reality, a new reality [11] "Augmented reality introduces the knowledge that the student must learn into his or her real-world environment, in a visible way" [12].
How the combination is established and the integration of realities is due to several factors. Among them, physical and digital realities are mixed through different resources, such as images, different three-dimensional objects, the incorporation of QR codes, so-called thermal signatures, or GPS coordinates [5]. On the other hand, the AR systems can be classified according to its location or experiences in the development of AR learning objects, the size of the viewing screen, the lack of conceptual frameworks to rely on for searching for innovative educational practices for the application of AR, the need to create support centers for teachers to facilitate the production of AR learning objects and the maintenance of the servers, the need for the students to have basic technology for its viewing, and the overload of information [4,22,24,31].

Research Objectives
The objectives proposed in the study are: a) To understand the degree of acceptance of the AR technology measured through the Technology Acceptance Model (TAM) by Davis, and the dimensions that comprise it (perceived usefulness, perceived ease-of-use, attitude towards use, and intention of use) that the use of AR objects fostered in the students enrolled in Education courses.(See Table A1) b) To understand the motivation it created in students through the use of the Instructional Materials Motivation Survey (IMMS) by Keller, and the dimensions it comprises (attention, confidence, relevance, and satisfaction), that the use of AR objects promoted in the students who were enrolled in Education courses. (See Table A2) c) Analyze if the degree of acceptance and the dimensions that comprise it, through the TAM model by Davis, and the motivation and dimensions in agreement with the instrument by Keller, had repercussions in the acquisition of knowledge by the students. d) To analyze if there were significant differences in the performance reached by the students through the use of AR objects. e) To know if the participation in the experience was similar, in the scores from the degree of acceptance of the technology, the motivation, and the performance, with the different objects produced.

Research Design
Two pre-experimental designs were used in the investigation, the two single-case designs, which analyzed the degree of acceptance of AR technology students had with the learning objects produced, and point out that the experience was developed through different studies. This analysis was conducted with a post-test measurement; and a second one utilized to understand if the degree of acceptance of the AR technology had repercussions on the academic performance of the students. This was conducted for the analysis of the academic performance, with pre-test and post-test measurements [50]. In both designs, a treatment was administered (AR object produced) and a measurement was performed (degree of acceptance of the technology by the students after participating in the experiment, and the academic performance).
In the study carried out, different AR objects were produced, which were used in the research, and which responded to three different contents from the courses "Education Technology" and "Information and Communication Technologies (ICT) applied to Education": manners of using video in teaching, design, and production of ICT for teaching, and educational uses of the Web 2.0 tools. The courses were part of the Pedagogy, Education of Children, and Primary School Education Degrees taught in the Faculty of Education at the University of Seville.
The experience was developed following the same sequence every time: a) Explanation for the students about what AR is. b) Application of the knowledge pre-test. c) Demonstration of the different objects produced and their possibilities. d) Presentation of the place where the different guides and "app" of the objects produced could be downloaded. e) Individual work by the students with their mobile devices with the different objects. f) Completion of the TAM instrument by Davis, the IMMS instrument by Keller and the knowledge post-test.

Research Hypothesis
The hypotheses of the research formulated with respect to the TAM were the following: Hypothesis 1: The way in which students verify the perceived ease of use in AR is positively and significantly related to the perception of enjoyment through the use of learning objects in AR.

Hypothesis 2:
The way in which students verify the perceived ease of use in AR is positively and significantly related to the perceived utility of the use of learning objects in AR.

Hypothesis 3:
There is a positive correlation between perceived ease of use and attitudes of use of learning objects in AR.

Hypothesis 4:
There is a positive correlation between the perceived utility of the use of AR learning objects and the perception of enjoyment.

Hypothesis 5:
There is a positive correlation between the perceived utility of the use of AR learning objects and their intention to use them.

Hypothesis 6:
There is a positive correlation between the ease of use of learning objects in AR and the attitudes of use by students.

Hypothesis 7:
Students' attitudes towards the use of learning objects in AR can positively and significantly increase their perception of enjoyment.

Hypothesis 8:
The perception of enjoyment can positively and significantly affect the intention to use AR learning objects.

Hypothesis 9:
The intention of using the learning objects in AR by students can be modified in a positive and significant way by the attitude towards their use.
The hypotheses that were established for performance were the following: Hypothesis 1 (null hypothesis): the scores obtained in the pre-test and post-test do not present significant differences-with an alpha risk of rejecting the null hypothesis of 0.05-after the participation of the students in the experiment with AR objects.
Hypothesis 2 (alternative hypothesis): the scores obtained in the pre-test and post-test show significant differences-with an alpha risk of rejecting the null hypothesis of 0.05-after the participation of the students in the experiment with AR objects.

The Sample
As mentioned, the sample was composed by students from the Pedagogy, Education of Children, and Primary Education Degrees from the University of Seville. The samples were non-probabilistic, convenient, or causal [51,52], and the criteria for choosing it were the ease of access that the students who were part of the study carried out by the researchers presented. Table 1 shows the number of participants in each of the experiments.

The Objects Produced
The experiment was conducted with three AR objects that were especially created for the study, and they referred to the content from the Education technology discipline: "ways of using video in the teaching-learning processes", from Anatomy (Medicine) and the field of Art. For each of these objects, a guide of the program was created, which offered information to the students about how they could access the downloads of the apps to be able to use the learning objects, what possibilities they offered them and where were the bibliographic references that allowed them to expand the subject studied.
The three objects were of different types; the ones from "Education Technology" content (Roles of the educational use of the video) and "Anatomy" could be classified as type II [5], in an "AR-enriched notes" format, with the first one linking to a video resource, and the second to a 3D object that could be manipulated by the student. Figures 1 and 2 show images of the object produced for the "Education Technology" discipline.  For the production of the different objects, different programs were utilized, and Table 2 shows the types of uses they were created for. The programs worked in Android as well as iOS devices. Table 3 shows the URLs where videos showing their functioning can be viewed, and where the corresponding apps can be downloaded. Table 3. URL for downloading marker and app.

Program URL
Ways of using video in teaching http://ra.sav.us.es/index.php/realidad-aumentada/49-videointegrado-en-apuntes-formas-de-utilizar-el-video ICT design and production http://ra.sav.us.es/index.php/realidad-aumentada/36-rafodiun Educational use of the Web 2.0. http://ra.sav.us.es/index.php/realidad-aumentada/50-web-2-0 It should be mentioned that for each of the AR objects, a guide was created, where the different objects were included within the notes, the students were shown where to download the app, and the complementary bibliography for delving into the subject was included. Figure 3 shows an example of the guide for the subject "Design, production, and evaluation of the ICT used for education".

Information Collection Instruments
Three instruments were used for the research study: the "Technology Acceptance Model" (TAM), formulated by Davis [53], and utilized for analyzing the degree of acceptance of the AR technology by the students; the "Instructional Material Motivational Survey" (IMMS) created by Keller [54] for the analysis of the motivation created in the students by their participation in the experiment; as well as the realization during the experiment of a multiple choice test, whose purpose was to measure and analyze the student's academic performance.
The TAM instrument seeks to obtain information from four dimensions: perceived usefulness (PU), perceived ease-of-use (PEU), perceived enjoyment (PE), attitude towards use (AU), and intent to use (IU). The proposal by Davis suggests that the pre-disposition an individual has towards the use of any technology is determined by the attitude towards it, which is conditioned by the "perceived ease-of-use", the "perceived usefulness" and "perceived enjoyment". Its starting point is that the beliefs and standards of each person are responsible for a specific behavior. Figure 4 shows the graphical version of the author's proposal. There is a large number of research studies carried out on this subject in the educational field related to RA, as well as a high number of meta-analyzes of the different educational models which allow to verify that they offer validity and solidity to how the intention of developing use of different technological environments in subjects. Recently, this has also been confirmed when analyzed with the structural equation models [55][56][57][58][59][60][61][62][63][64][65].
In any case, it should not be ignored that like other educational models, it also has certain restrictions [35,66] that should be assumed: the context in which the technology is incorporated and used will determine, to a large extent, the results obtained in the study, which is evident taking into account the distinctive characteristics of the subjects participating in it For example, there are different contexts of use: mandatory and voluntary. There is an extensive review of whether or not the concept of attitude itself should be incorporated into the model as it creates confusion with the theory of planned behavior. In educational research, the data obtained through tests such as self-reports, generate in the researcher some limitations, especially in the relationship established between the intention to use and its actual use, the latter being able to be conditioned by variables of another type. Without forgetting that the limitations also include the difficulty that exists to obtain measures that are objective to establish the degree of acceptance of a technology and its conceptual simplicity with the model.
The application of the TAM model to AR will essentially indicate if the student, when utilizing it in teaching-learning situations, will consider its use to be relatively easy and if it will contribute benefits, and if an increase in the intent to use will be produced, which will signify that the student will utilize it in the future.
The instrument that was used was configured by 15 questions (four for the perceived utility category, three for perceived ease of use, three for perceived enjoyment, three for attitude towards use, and two for intention to use), Likert-type where there were seven valid response options ranging from "Extremely Unlikely/Disagree" (1) to "Extremely Probable/Agree" (7).
As for the IMMS by Keller, it should be underlined that it is composed by 35 items which collect the information from four dimensions: attention (12 items), confidence (9 items), relevance (9 items), and satisfaction (5 items). The instrument has a Likert-type construction with seven degrees of response (1 = Extremely unlikely/in disagreement to 7 = Extremely likely/in agreement).
As shown by Loorbach, Peters, Karreman, and Steehouder [67], the model formulated by Keller: "has been used countless of times to apply motivation strategies for teaching materials and to test their effects. Although the model was originally designed to influence the student's motivation in a classic learning environment, with the personal interaction between the professor and the students, it has also been applied and thoroughly tested, such as for example in computer-aided teaching and virtual and distance education". More specifically, the model has been used to understand the degree of motivation that different technology formats created in the students: videogames [68], the addition of video to e-learning training sessions [69], audio podcast [70], the MOOC [71], or AR [72][73][74].
Two multiple choice tests were created to measure students' academic performance through the use of different learning objects with AR, a pre-test and a post-test, consisting of 15 questions that were configured around the three Bloom's first categories of taxonomy, which are: apply, remember, and understand, and where the items of the two tests were the same, simply modifying their order of presentation.
For the TAM instrument, as well as the IMMS, an index of reliability was obtained with Cronbach's Alpha, as this test is the most appropriate for Likert-type instruments [75]. Table 4 shows the values for each instrument as well as the different dimensions that comprised them. The resulting values allow us to indicate that these instruments possess a high level of reliability, in their entirety as well as in their different dimensions [75,76].
For the analysis of academic performance, a multiple choice test was created, comprised of 15 items, which agglomerated information from the categories application (4 items), remember (6 items), and understand (5 items) from Bloom's taxonomy. The items of the pre-test and the post-test were identical, although the order of presentation was changed.

Results
The results will be presented first by showing the means and standard deviations obtained in the dimensions of the TAM and IMMS instruments (Table 5), and will finalize with the results on academic performance.
As observed for the overall results of both instruments, and in the different dimensions that comprise them, the scores obtained exceed the value of 4.5. At the same time, it should be pointed out that the general evaluation of both instruments has been very broad, with a score of 5.77 for TAM and 5.49 for IMMS, which out of 7 possible points suggests a high degree of acceptance and motivation of the experiments conducted with the AR objects that were produced for the research study.
As for the academic performance achieved, Table 6 shows the mean scores and the standard deviations of the pre-test and post-test for the three objects, as well as for the overall experiment.
In order to contrast the hypotheses that were previously formulated with respect to the TAM, Pearson's correlation coefficient was applied, with the resulting values for the different AR utilized shown on Table 7. Table 5. Mean and standard deviation of the TAM and IMMS instruments for their entirety as well as the dimensions that comprise them (note: 1 = Ways of utilizing video, 2 = Information and Communication Technology (ICT) design and production, and 3 = Educational use of the Web 2.0).  Table 7. Pearson's correlations between the different TAM dimensions ("Technical quality", "Perceived enjoyment", "Perceived ease of use", "Perceived usefulness", "Attitude towards use", "Intent to use"), for the different objects utilized (note: * = significant at 0.05; ** = significant at 0.01). The results allow us to reach different conclusions, but before their presentation, three aspects will be mentioned: a) That all the correlations were positive, which means that when one of the dimensions increases, the other does as well. b) That the correlations were all significant at a significance level of 0.01.

c)
And that the correlations, except for a few cases, can be considered "high" [77,78].
As for the hypotheses formulated, the following conclusions can be made: 1) The perceived ease of use can positively and significantly affect the perceived enjoyment of the AR learning objects produced in the different courses. Lastly, it should be pointed out that the results were significant for the three objects produced in AR.
Next, the hypotheses that referred to the question of: if the motivation and the degree of acceptance, and the dimensions that comprised the instrument, had a relationship with the performance achieved. Pearson's correlation was used again, with the results obtained for the IMMS shown on Table 8. Table 8. Correlations between performance and motivation created by participating in the experience of the students (IMMS) and the dimension that comprise the instrument. As for the degree of acceptance of the technologies, and their relationship with the performance, the results obtained with Pearson's correlation are shown below (Table 9). Table 9. Correlations between performance and the degree of acceptance of the technology by the students (TAM) and the dimensions that comprise it. The values obtained show for both types of instruments, according to the studies carried out by Etxeberria & Tejedor [78], different elements to be taken into account, such as the direct relationship between the performance and motivation variables and the degree of acceptance of the technology, and that this relationship moves between "moderate" and "high" values, which is significant for our study, especially considering that these relationships are significant at p ≤ 0.01.
In other words, it was found that through experience with AR objects, students obtained a high degree of motivation and acceptance of this technology, both increased by their incorporation and use by students in the designed training action. This is indicated by the very high mean values in the total for the instruments, as well as their different dimensions, and this has resulted in a positive relationship with the performance acquired with the three objects produced in AR.
For the contrast of these hypotheses, Student's t-test was applied, achieving values that are shown on Table 10, for each of the experiences conducted. Table 10. Student's t-test for the relationship between performance and IMMS.

Object Student's t-Test
Ways of using video 16.001 (**) Design and production 15.531 (**) Educational use of the Web 2.0 tools 17.072 (**) The results reached allow us to reject H0 with a level of significance of p ≤ 0.01; and as a result, it can be concluded that the student's participation in the experience allowed them to achieve the goals of the previously-mentioned courses. This was found for all three content blocks.
As the null hypothesis (H0) was rejected, the size of the effect was calculated in order to understand the strength of the differences found between the pre-test and the post-test, and this was obtained with Hedges' g [79,80], with the following results obtained: • Use of the video: 3.008951 • ICT design and production for teaching: 2.101745 • Educational use of the Web 2.0 tools: 3.757971 These values, in agreement with the statistical proposal made by Cohen [81], are very high.

Discussion and Conclusions
The experiment was developed with three different types of AR objects as for their design and with the incorporation of different resources, which were especially created for the study. These objects referred to content from the disciplines "Education Technology", in the Pedagogy Degree and the "ICT applied to education" from the Education of Children and Primary School Education Degrees from the University of Seville, allowing us to obtain conclusions about different aspects.
Firstly, it should be noted that the reliability results of the diagnostic instruments used showed that both Keller's IMMS and Davis TAM were good predictors for diagnosing variables relevant to our study, such as motivation, confidence, attention, relevance, and satisfaction. All this linked to the interaction of students and the object of AR in the specific case of IMMS. Note that the findings of the present research coincided with the works conducted by different authors [82][83][84][85][86].
Regarding the use of the TAM instrument, this research has shown that many of the values obtained coincide with the results of other authors who have researched the same subject and at the same educational levels [40,64,87,88]. Along this line, it should be mentioned that these results were independent of the AR object with which the students who participated in the study interacted with, as well as the course they were enrolled in; this supports the robustness of the results.
Ultimately, it should be pointed out that these were two instruments, and two models of diagnosis of the degree of acceptance and motivation that a technology created. This can also be observed in its recent use in other research studies that analyzed other technologies, i.e., in the case of the TAM, through the different works conducted for videogames [89], for Moodle [90], or for virtual training [91]; and for the IMMS, the study with videogames [89,92]. Therefore, our work points to the usefulness of both models.
Another of our conclusions indicates that the use of AR objects has resulted in a high degree of acceptance and an acceptable degree of motivation, as indicated by the high mean values reached, as well as the high scores shown in "the intention of use it in the future" and their "satisfaction". Therefore, it should be concluded that negative attitudes towards its use were not found.
On the other hand, the study does not indicate that these objects are easy to use by the students, as the mean values found in the dimension "perceived ease-of-use" of the TAM clearly exceeded the middle value of 3.5 (4.77, 6.23, and 6.46). These scores increased as the experience progressed, even with the objects increasing in difficulty due to the number and types of objects incorporated.
As for the degree of acceptance of the technology, as previously mentioned, starting with the TAM model formulated by Davis, the results obtained allowed us to conclude that the students, independently of the courses they were enrolled in, had a high degree of acceptance of AR. This indicates a strong attitude towards its use in training, pointing to a strong intention of use in the future. These results were confirmed with the analysis conducted to contrast the hypothesis with the scores obtained jointly with the two objects produced, just as when it was done for each of them, thus re-enforcing the findings even more. To a certain degree, it should be pointed out that the study confirmed the results found by other authors [18,19,42,72], which suggested that the students have high degrees of satisfaction when they interact with objects produced in AR, also mentioning that they would like to learn with AR in the future. This is reinforced by the perceived ease of use, which means that the participating students enjoy the use of this resource and the usefulness of this tool increases in the same way.
The study conducted shows that the participation in AR experiences increases the motivation of the students, measured through the IMMS and the different dimensions that comprise it: confidence, attention, satisfaction, and relevance. A significant relationship was found between the degree of motivation and the increase in performance. The greater the motivation (confidence, attention, satisfaction, and relevance), the greater the performance or the retention of the information learned. These results have also been achieved in other research works that utilized AR for the training of university students conducted in our context [47].
As for performance, it should be indicated that the students significantly improved their scores in the knowledge tests that were administered at the start of the experience. The results reached indicate that the use of notes enriched with AR objects can facilitate the student's learning and the acquisition of knowledge. This result is concordant different authors, in that the construction of books and notes with AR can be greatly useful for training [85,93]. This opens strong possibilities for observation in the new scenarios of learning, when the students receive complementary information through this technology, which will facilitate their understanding of the content, and will make easier the increase in performance. Thus, this work contributes with references for justifying the incorporation of notes enriched with AR in university training, where, aside from increasing or awakening the motivation and satisfaction of students, it can facilitate the acquisition of content presented with it, with the students not showing difficulties in their management.
These possibilities suggest that they can be useful objects that can be incorporated into emerging technologies such as the "flipped classroom", where the students work at home with these learning objects, and the in-person sessions are dedicated to resolving doubts and problems and conducting hands-on tasks or for the implementation of "collaborative learning" activities [94,95].
Therefore, it is necessary to consider that we must forget the novelty effect of technology and think more about its implementation and proper use. It is interesting that its integration in the classroom is within an educational project that always puts the pedagogical before the technological. It is advisable to include with the material some indications to take into account for its use, as well as a brief explanation of the characteristics of the object, without forgetting the importance of training the teacher in both a technical-instrumental and methodological and pedagogical dimension.
Nevertheless, it should be underlined that its use is dependent, on the one hand, on the existence of good internet connections in the education centers, and on the other, that in its design the following be included: principles of accessibility, ease of use, and flexibility for its adaptation to different contexts. Funding: This research was funded by the Spanish Ministry of Economy and Competitiveness by the R&D+i Research project from the entitled: "Augmented reality for increasing training. Design, production and evaluation of augmented reality programs for university teaching" (EDU-5746-P -Rafodiun Project) Acknowledgments: The authors are grateful to the three anonymous Sustainability reviewers, who have provided constructive ideas and insightful comments for this article.

Conflicts of Interest:
The authors declare no conflict of interest.
Appendix A Table A1. TAM diagnostic instrument. Perceived usefulness (PU) The use of the AR system will improve my learning and performance in this course (PU1) The use of the AR system during the classes will facilitate my understanding of certain concepts (PU2) I believe that the AR system is useful when one is learning (PU3) My performance would increase with the use of AR (PU4)

Perceived ease of use (PEU)
I think the AR system is easy to use (PEU1) Learning how to use the AR system is now a problem for me (PEU2) Learning how to use the AR system is clear and understandable (PEU3)

Perceived enjoyment (PE)
Using the AR system is fun (PE1) I had fun using the AR system (PE2) I think the AR system allows us to learn by playing (PE3)

Attitude towards use (AU)
The use of an AR system makes learning more interesting (AU1) I have been bored using the AR system (AU2) I think the use of an AR system in the classroom is a good idea (AU3)

Intention to use (IU)
I would like to use the AR system in the future if I have the opportunity (IU1) I would like to use the AR systems to learn anatomy, as well as other subjects (IU2)