The Past, the Present, and the Future of the Evolution of Mixed Reality in Teacher Education

Abstract

The authors in this article provide a historical view (past) on the development of mixed reality (MR) simulation in teacher education as well as a brief history of simulation from other fields along with foundational knowledge on the evolution of simulation. The authors provide a systematic review of the current state (present) of the research in MR for teacher education within the past 5 years aligned with the research question “What are the uses, practices, and outcomes of MR simulation in teacher preparation?”. Three themes were identified, i.e., simulation to this point is designed by teacher educators, feedback matters in impacting outcomes, and practice is safe and reflective for those who prepare teachers in these environments. A summary is provided of these key articles and the findings. The authors conclude the article by sharing the potential evolution (future) of aspects of the model of MR, focusing on the use of AI agents and multi-modal data collection, including biometric signals, providing insights into simulation in teacher education.

1. Introduction

The use of artificial intelligence (AI) is making significant inroads into all disciplines; however, before the emergence of AI, numerous intersections of humans and technology already existed. For example, in earlier times, people went to a telegraph operator to send a message to another person, and human telephone operators connected humans in conversation [1]. More recently, ONSTAR agents assisted automobile drivers in the navigation of directions. Currently, AI can provide these interactive services without the need for humans. However, the authors of this article do not believe AI can replace humans, but instead argue for how the power of the human brain serves as the driving force for stronger connections with technology and are forever relevant for the intersection of human emotions, interactions, and learning. Despite AI’s rapid evolution, this concept is not yet ready to prepare teachers or to remove teachers from the classroom. The evolution of technology-driven mixed reality (MR) for teacher preparation has reached a point of stability but is still not fully realized. Therefore, we provide the evolution of work in MR in teacher education, a summary of the current research on this topic, and a discussion of how future work in simulation, inclusive of all forms of extended reality (XR; virtual reality (VR), MR, and augmented reality (AR)) and AI could inform, shape, and potentially evolve into a deeper understanding of teacher behavior and student learning.

This team was asked to share, as part of a Special Issue, a historical lens of the evolution of over 15 years of interdisciplinary collaboration across computer science, mathematics education, and special education to create the first MR simulator in teacher education. However, the authors of this article want to be clear on how their existing research and investment in the evolution of MR in teacher education may reflect a potential bias in this publication. This statement does not mean simulation did not already exist before this team’s approach emerged, with prior and current work including case studies, role plays, and microteaching [2], but this article focuses on the narrow definition of simulation, like how the use of virtual environments involving a human-in-the-loop model allows nimble prototyping, while providing consistent experiences in teacher education to allow for standardized reflection, experimentation, practice, and research.

The work in teacher education aligns with all aspects of work in simulation (medical, flight, military, and education) which were never created to replace “real” practice, but to provide opportunities to rehearse specific skills before putting “real” humans at risk whether being used as a learning curve for a novice or the retooling for a practicing professional [3]. We provide, in this article the context for this wok in teacher education based upon a historical perspective of how the field of simulation began to how this emerged in teacher preparation.

Mixed Reality (MR), as we use it in this paper, covers multiple ways to experience virtual environments. The choice of how a scene is displayed depends on the scenario and the goals of each user within the context of the study they are conducting. In our research, we started with the use of a video see-through head-mounted display (HMD) that allowed us to blend the real and the virtual, delivering the merged imagery directly to the user’s visual field. However, we quickly found that many of our teachers were more comfortable with a virtual scene that existed in a real setting in which they had a whiteboard and manipulatives, and in which their movement in the real space could optionally affect their point of view in the virtual space. Others preferred a surround experience provided by a CAVE (Cave Automatic Virtual Environment) [4]; some wanted a blended real/virtual experience using either a video or optical see-through display, and yet others desired an experience that used an HMD to immerse their entire visual field in virtual content. As such, TeachLivE the first MR environment used in teacher education, was developed to support a wide range of virtual/real experiences, with most teachers preferring the screen-based approach that avoids their feeling encumbered by an HMD or in need of the specialized physical setup of a CAVE. The teachers instead noted the value of an environment free of physical encumbrances, yet creating the suspended reality between having real children versus an avatar as a tool for practicing the art of teaching.

Within this work in MR, this team, from the start, embraced the potential bias produced due to human involvement in the creation and delivery of the simulated activities. All aspects of this discussion and research, when developed by humans, is embodied in bias [5], i.e., the bias of the programmer, the bias of the interactor or puppeteer, the bias of creators of the tools, and the bias of scenario designers and what the scenario represents. At the core of this development of MR in teacher education has been through a social justice lens [6] for preparing more effective teachers that can impact the learning outcomes of students.

2. Evolution of Simulation

The world of interactive virtual simulation for training existed before it was applied to teacher education. The earliest use was by Edwin Link who developed the “Link Trainer”, also known as the “Blue Box”, in the 1920s [7]. This flight simulator played a significant role in pilot training and aviation history. The Link Trainer aimed to provide a safe and controlled environment for pilots to learn and practice flying by instrument skills without the risks associated with actual flight. To accomplish this goal, the Link Trainer design mimicked the movements and controls of an actual aircraft including a cockpit with instruments and controls that responded realistically to the pilot’s inputs. While the simulation experience did not perfectly replicate all aspects of flying, pilots did have a chance to practice basic maneuvers, instrument readings, and navigation techniques. The device was initially used for training for flying during adverse weather conditions or at times when it was impractical to conduct real flight training.

The success of the in-flight simulation work was a catalyst for the emergence of use in other fields expanding upon the lessons learned from aviation. In the medical field, “dummy patients” first appeared followed by robotic patients [8]. These MR simulations allowed novice doctors to practice diagnoses and treatments with little risk to real patients. The overall theme initially was the use of simulation training to practice a structured and linear activity (surgery, landing a plane, or driving) but most often in areas that rely upon life and death decision-making (crash landings, entering a hostile environment, first time performing surgery, or driving in adverse conditions). Despite successful use in other fields, the ability to apply simulation training in teacher education provided unique challenges as many activities in teaching are not linear but evolve through complex interactions between humans (the teacher and multiple learners in the classroom setting). Hence, this team over 15 years ago set forth to explore options to apply the power of “virtual rehearsal” [9] into use in teacher education [10]. Training airplane pilots and automobile drivers are uses of simulation that provide a chance to rehearse various scenarios without causing harm to self or others. Similarly, the novice teacher or real students are unharmed when practicing teaching skills in a MR simulator much algined with use in many other professions.

3. Moving toward Mixed Reality in Teacher Education

Many seemingly sudden appearances of a phenomenon occurring in a field of disruptive change often is traced to a “tipping point” [11]. In retrospect, the development of a MR simulator for use in teacher education is one of those cases. Pathways for the use of simulation existed that increasingly made the use of MR simulation feasible, and perhaps, a necessary tool for teacher education, due to the rapid development and access to lower cost technology. In 2005, faculty members from various departments met weekly to explore how the development of a MR simulator might make sense and be employed in teacher education not to replace the presence of teaching but to practice discrete skills (much like a pilot) in a safe and reflective environment.

There was and still is a need for a better way to prepare teachers using MR aligned with increased accountability in public education. In the past two decades, extensive initiatives converged to improve student outcomes in American public education, such as the way student success is measured and reported has undergone a significant transformation over time [12]. The relatively simple process of collecting student data on learning has evolved from a classroom activity overseen by a teacher into a system of high stakes testing that carries substantial implications for all involved. In the early days of American public education, student assessment was primarily focused on the mastery of subject matter within the confines of the classroom. Teachers assessed students’ learning of content previously covered in class through assignments, observing students, classroom discussions, and periodic classroom testing. Teachers used these insights to guide instruction and provide meaningful feedback to parents and guardians. The evaluation of teachers was based solely on a limited number of classroom observations.

Globalization, technological advancements, and increasing competitiveness led states to provide a heightened emphasis to academic standards and accountability for student learning and teacher performance, alike [12]. Policymakers and education reformers sought ways to measure educational outcomes on a larger scale, looking for tools that could not only evaluate individual student performance but also gauge the effectiveness of teachers, schools, and districts [13]. The intention was to improve school performance measured by student state test performance, and, thus, “grades” for schools were and are reported to the public based on the student performance in statewide testing [14].

Since the beginning of American teacher education programs usually found in “Normal Schools”, all teacher candidates were required to successfully complete “practice teaching” [15]. Practice teaching is usually the culminating experience in a teacher education program now called student teaching or teacher internship. This experience occurs in public schools and is typically the final assessment for teacher candidates [16]. Successful completion of this final assessment indicates readiness to become a state-certified teacher. Political pressures caused states to extend education accountability and required testing of teacher education graduates with bachelor’s degrees to further qualify them for state teacher certification. Data from these tests for college graduates became the tool to rate state-approved teacher education programs.

If students’ test scores were used as the sole success indicators to measure the teacher, then any low student performance could affect their job security and/or pay increases. Consequently, many teachers were reluctant to host student teachers. Also, some school administrators were reluctant to accept interns because their performance might affect the school grade adversely. Teacher education programs are, therefore, challenged to assure teachers and school administrators that student performance will not be diminished by allowing interns to teach under the direction of a certified classroom teacher. Teacher education candidates need a way to improve and ensure their effectiveness in key teaching skills before their student internship. The need for a new way that teacher education instructors could coach teacher candidates to become proficient on key instructional skills became a challenge to develop a MR simulation in teacher education.

The team discussed many current phenomena and how they impacted the nature of any future simulator that might be built for teacher education. How could technology help teacher educators observe and assess a teacher candidate’s readiness to teach children in a public school classroom? Many variations of using technology were proposed, but a plan only began to form while keeping in mind traditions and trends in education, society, and technology that might impact the design of an effective simulator for teacher education, and having a human in the loop was at the core of this development. The team also grounded all the work in the theoretical framework presented by Kolb’s experiential learning cycle (2015), which requires four stages of interaction for learning to occur. This process requires the participant to learn from (a) concrete experiences; (b) reflective observation; (c) abstract conceptualization; and (d) active experimentation. Many online environments or classroom activities start and stop at the first stage of the learning cycle with concrete experiences but incorporating Kolb’s stages of reflective observation, abstract thinking, and active experimentation required the involvement of a deeper level of human interaction and reflection by the participant for learning to occur in the MR environment.

4. The Emergence of TeachLivE^TM

Introducing the bridge of humans and technology were key in creating a MR simulation in teacher education that was nimble, reliable, and impactful. At the time, the team added a human puppeteer, most simulators were static and linear in nature, using an approach of asking a human to follow a script or a specific teaching lesson. That strategy did not reflect the complex and pedagogical nature of teaching [17]. The team immediately started to look for new and non-traditional approaches by employing highly educated humans with acting and improvisational experience, called interactors, to puppeteer the avatars in the MR-simulated experience. These interactors were challenged to operate the simulator while playing the virtual students. After several efforts, the team created a model where one human could inhabit and control up to six avatars (students) simultaneously. Of course, this human inhabitant of the avatars introduced a potential bias, but much like in a Broadway production, the team created a way to standardize the avatars and the backgrounds with the interactors embracing the rules of improvisation to ensure no two experiences were exactly the same while never straying from the purpose of the lesson (e.g., managing behavior or teaching content) or the backstories of the virtual characters. The goal was to avoid the linear map of technological practices occurring in other fields. Instead, the team focused on creating a nimble, unpredictable, yet standardized scope of scenarios that could not be learned, like those in a gaming situation, but instead, the human had to respond to the traits, behaviors, attributes, and nature of each of the avatar students in the classroom and the human teacher who interacted with the students. Unlike most online games or even well-developed simulators, the scenarios were not static but instead were dynamic in nature allowing the user (teacher candidate) to reflect, change, react, and even try again in a safe environment. Much like those pilots learning in the Link Trainer [7], teacher candidates tried as many times as necessary to meet the academic and behavioral needs of students in the simulated classroom no differently than the use of simulation in flying or driving.

The MR simulator that emerged, called TeachLivE^TM (Teaching and Learning in a Virtual Environment), is designed to provide educators with a safe and controlled virtual environment to practice and improve their teaching skills. The MR platform created a virtual classroom populated with simulated students, each exhibiting a range of behaviors, learning styles, and personality attributes controlled through both technological components and the human-in-the-loop. This MR environment allowed teachers, both novice and experienced, to practice various discrete teaching strategies, classroom management techniques, pedagogical skills, and content instruction in a risk-free environment. The goal of the team was to enhance teachers’ pedagogical skills, content skills, and classroom interactions through repeated practice and constructive feedback without “harming real children” or the reputations of the teachers.

The virtual students in TeachLivE^TM are controlled by the interactor, allowing a variety of scenarios to be played out, ranging from typical classroom situations to more challenging scenarios involving student behavior, emotional reactions, and diverse learning needs. Teacher educators can experiment with different populations, lesson approaches, and teaching techniques, allowing teacher candidates to receive feedback on their performance. If the same lesson needs to be repeated, the avatars can begin again as if it is a new lesson.

The TeachLivE^TM environment is an MR classroom simulation composed of up to six 3D virtual students, who respond in real time and are known as avatars. The avatars are cognitively and behaviorally modeled based on an adolescent psychologist’s work, i.e., of William A. Long, M.D. Using Long’s work [18], the characters are based on his categorization of adolescent personalities that have a combination of passive or aggressive and independent or dependent traits. These same avatars have been developed at elementary, middle, and high school levels with a plethora of other avatars (parents, college students, English learners) in addition to two avatars, each with a disability: one male avatar, Martin, with ASD, and one female, Bailey, with intellectual disabilities. Both avatars with identified disabilities were developed with input from actual individuals with disabilities and their families along with focus groups of experts in the field [19]. The array of avatars seen in Figure 1 provides the field with multiple ways to create and design scenarios with the TeachLivE system.

Research on TeachLivE^TM focused on various aspects, including its effectiveness in improving teacher preparation, the impact of practice in virtual environments on real classroom performance, and the development of teaching skills and strategies. Studies also explored the potential of TeachLivE^TM for professional development, especially in areas such as special education, classroom management, and cultural competence [19,20,21]. This work, after being fully developed and funded by the Bill & Melinda Gates Foundation, continues to allow the UCF team to conduct research. The scaling up to commercialization levels occurred through a UCF licensing agreement with Mursion, Inc based in San Franciso, CA, USA. Mursion offers simulation training and professional development in teaching as well in many other industries (e.g., hospitality and corporations). Specifically, Mursion specializes in creating realistic and immersive scenarios where individuals can practice various skills, such as leadership, communication, problem-solving, and interpersonal interactions, in a safe and controlled environment.

Mursion’s platform includes new avatars used in commercialization of the tool, which also involves real-time interactions with virtual avatars who respond dynamically to the user’s actions and choices. This technology is particularly valuable for preparing individuals in fields where interpersonal skills and human interaction play a significant role, such as education, healthcare, customer service, and management. This work, in both the TeachLivE^TM and Mursion versions, has led to numerous research studies being conducted in teacher education with most building upon the earlier work in the field of case studies, role play, and microteaching.

To reflect the current status of this work, the authors provide a summary of the last 5 years of work on the topic of MR simulation and its use in teacher education. This review is provided to highlight the current status of the research and to build upon these findings for further discussion of the future of work evolving into AI-enhanced simulation in teacher education.

5. Method: Systematic Literature Review on the Use of Mixed Reality in Teacher Education

The criteria used for the systematic analysis of the literature were articles published as empirical studies in peer-reviewed journals, containing the search terms “simulation” and “teacher education” and “mixed-reality”. Both qualitative and quantitative studies were included. The time frame that the search covered included articles from 2018 to 2023 due to the evolution of these tools with research partners using both the TeachLivE and Mursion environments, and to show the most current research in the use of MR simulation. Prior use of simulation reflected activities such as videos, role plays, and case studies that we acknowledge were and continue to be used in teacher education, but for this review, these terms were excluded to reflect the specific research questions driving this review aligned with this special issue, “What are the uses, practices, and outcomes of MR simulation in teacher preparation?” Studies including student use or teacher use for student learning were excluded. Studies focused on teacher preparation at the in-service or pre-service level in peer-reviewed journals were included in the final analysis. After the initial search, 28 articles emerged and the researchers then excluded articles based on the following criteria: (a) not relevant or empirical, (b) duplicates, which appeared in multiple databases, and (c) not including teacher preparation. These exclusion criteria were chosen to identify the most salient research that aligned with the research question.

The researchers used a university library system to navigate relevant databases in teacher education including ERIC (EBSCOhost), ERIC (ProQuest), APA PsycINFO, Social Science Database (ProQuest), Education Source (EBSCOhost), and ScienceDirect. After compiling the initial list for these educational focused databases, we assessed if each paper aligned with rules for relevancy that included if the study met the following criteria:

• Grounded in preservice or in-service teacher preparation.
• Used simulation that aligned with the definition of MR.
• Examined teacher practice in the MR environment.
• Focused on a research study; the team excluded reviews, technical papers, non-peer reviewed papers, book chapters, or research reports.
• Published in English.

Table 1 consists of a summary of the 12 research articles that were deemed to be included and were both quantitative and qualitative in nature and published in peer-reviewed journals, and all aligned with the research question proposed in this systematic review. The population, methodology, measures, and outcome are provided for each study in Table 1.

5.1. Results: Teacher Preparation in Simulation

Simulations can come in simple formats, such as role play and case studies, or more complex formats, such as MR and even fully immersive virtual reality settings. The most common use in teacher preparation is through either online simulations (gaming) or through MR. Live simulations typically occur in natural settings with humans and/or the equipment appropriate for the environment (i.e., role play). Virtual simulations with humans and/or equipment in a computer-controlled environment might involve a human-in-the-loop (i.e., flight simulator).

Teacher preparation programs have used simulation for decades with Twelker in 1967 [22] describing the use of simulation activities to support teachers in basic skills in behavior management and decision-making. The published work in the last 5 years on the use of simulation specifically in MR is provided to discuss themes in the current work to build a platform for future research and exploration. As noted in Table 1, the overall trends in the use of MR simulation appear to be in a positive direction for use and are evaluated through qualitative, quantitative, and mixed-methods studies. The majority of the research in the past 5 years has originated from the United States (U.S.) and Australia. With large scale use in Australia examining self-efficacy and beliefs as well as use with parents and, in the U.S., studies focused on various aspects of co-teaching, practicums students’ reflections using simulation, and looking at time in the simulator as a variable for change. Reflecting upon what exists in Table 1, the authors share three themes emerging from the research in the past 5 years.

5.2. Design Led by Teacher Educators

The research studies, as reviewed, clearly reflected the design of the study, and the simulation activity was driven by the teacher educators. Throughout the studies, authors noted potential limitations (the lack of movement throughout the room of the avatars or the inability of the teacher to share manipulatives with the avatars), but the interesting theme noted was the way the researchers addressed the limitations of simulation to provide the most authentic practice possible within the confines of mixed reality. In the studies provided, all researchers established clear procedures for use by the teachers and for the interactor to allow for slight human variations while maintaining reliability as well as the validity of the research.

Feedback matters as seen in [23]. Several of the researchers, noted in Table 1, studied teacher reflection through various means such as focus groups, video reflections, and written summaries. Other researchers focused on the importance of feedback with one looking at the time in the simulator as a variable and another looking at practicum students’ perceptions. Like the initial work established in the field and emphasized in the research on simulation is the use of a cycle, often referred to as the ARC cycle [24], continues to emerge as important in impacting change in teacher education.

Practice is safe and reflective. Much like feedback is important, so is a safe and reflective space for use by novices or even practicing teachers in any professional development. The studies presented in Table 1 had a cross-cutting theme of reassuring participants of their safety or even studied how the reflective nature of the simulators impacted practice.

Table 1. Summary of research publications on mixed reality in the past 5 years.

Article	Population	Methodology	Measures	Outcome
Barmaki and Hughes, 2019 [25]	30 college of education students	Quantitative counterbalance	MR with and without feedback	A significant change in the education students’ body gestures occurred when provided feedback in the MR environment.
Driver et al., 2018 [26]	7 students	Quantitative	Embedded four simulation experiences into a special education collaboration course	Significant shifts in readiness for collaborative settings and improved communication skills.
Spencer et al., 2019 [27]	90 preservice teachers in elementary, secondary, or special education	Compared simulation to role play in a co-teaching scenario	Practice in co-teaching	Students in the simulator through regression analysis had higher outcomes in average gain in skills, realism, and usefulness than those in the role play group.
Gundel et al., 2019 [28]	53 preservice teachers	Repeated measure	Compared self-efficacy scores of 30, 60, and 90 min of simulation exposure of three groups	The greater the exposure, the higher the self-efficacy scores, with even just 30 min of exposure significantly impacting self-efficacy scores.
Dieker et al., 2019 [9]	102 science teachers	Quantitative	Four 10 min sessions in simulator and transfer to “real” classroom	Behavior changes observed in the simulator transferred back to the real classroom for those in the simulation group.
Rosati-Peterson et al., 2021 [29]	15 preservice teachers	Mixed methods	Immediacy of feedback over three lessons and case study	Significance changes in immediacy between lesson 2 and 3 and qualitative found themes of confidence, reflection, and objectivity from the simulation.
Liaw and Wu, 2021 [30]		Mixed methods	Immediacy of feedback over three lessons and case study	Significance changes in immediacy between lesson 2 and 3 and qualitative themes of confidence, reflection, and objectivity.
Walters et al., 2021 [31]	30 undergraduate students in a course in ASD	Quantitative	Experimental and control groups in system of least prompts	On a specific activity with a scoring rubric, those in the mixed reality simulator significantly outscored those in the control group.
Gundel and Piro, 2021 [32]	49 students and 5 professors	Mixed methods	Perceptions of increase in self-efficacy due to use of simulation	Self-efficacy increased through four themes “being the teacher, peer observation, feedback, and managing emotions”. These themes represented thoughts of outcomes for students and observations by professors involved with the study.
Scarparolo and Mayne, 2022 [33]	28 preservice teachers	Focus group	Perceptions of students in working with parents after simulation	Pre-post focus groups showed that simulation was a way to expose teachers to parent conferences. Some PST in the study did not have prior experience working with parents and found this type of activity of value.
Fischetti et al., 2022 [34]	2000 new teachers across 3 university sites	Mixed methods pre-post surveys and Flipgrip interviews	Experience with simulation prior to practicum	Helped build candidates’ self-confidence and prepared novices for teaching practicum.
Rappa and Ledger, 2023 [35]	57 preservice teachers	Mixed methods	Working with parents in the simulator	Recommendations for future use of the simulator are provided (practice, performance, preparedness, and reflection on talking with parents) as well as on the outcome of the simulation, helping students with difficulty conversations with parents.

Table 1. Summary of research publications on mixed reality in the past 5 years.

Article	Population	Methodology	Measures	Outcome
Barmaki and Hughes, 2019 [25]	30 college of education students	Quantitative counterbalance	MR with and without feedback	A significant change in the education students’ body gestures occurred when provided feedback in the MR environment.
Driver et al., 2018 [26]	7 students	Quantitative	Embedded four simulation experiences into a special education collaboration course	Significant shifts in readiness for collaborative settings and improved communication skills.
Spencer et al., 2019 [27]	90 preservice teachers in elementary, secondary, or special education	Compared simulation to role play in a co-teaching scenario	Practice in co-teaching	Students in the simulator through regression analysis had higher outcomes in average gain in skills, realism, and usefulness than those in the role play group.
Gundel et al., 2019 [28]	53 preservice teachers	Repeated measure	Compared self-efficacy scores of 30, 60, and 90 min of simulation exposure of three groups	The greater the exposure, the higher the self-efficacy scores, with even just 30 min of exposure significantly impacting self-efficacy scores.
Dieker et al., 2019 [9]	102 science teachers	Quantitative	Four 10 min sessions in simulator and transfer to “real” classroom	Behavior changes observed in the simulator transferred back to the real classroom for those in the simulation group.
Rosati-Peterson et al., 2021 [29]	15 preservice teachers	Mixed methods	Immediacy of feedback over three lessons and case study	Significance changes in immediacy between lesson 2 and 3 and qualitative found themes of confidence, reflection, and objectivity from the simulation.
Liaw and Wu, 2021 [30]		Mixed methods	Immediacy of feedback over three lessons and case study	Significance changes in immediacy between lesson 2 and 3 and qualitative themes of confidence, reflection, and objectivity.
Walters et al., 2021 [31]	30 undergraduate students in a course in ASD	Quantitative	Experimental and control groups in system of least prompts	On a specific activity with a scoring rubric, those in the mixed reality simulator significantly outscored those in the control group.
Gundel and Piro, 2021 [32]	49 students and 5 professors	Mixed methods	Perceptions of increase in self-efficacy due to use of simulation	Self-efficacy increased through four themes “being the teacher, peer observation, feedback, and managing emotions”. These themes represented thoughts of outcomes for students and observations by professors involved with the study.
Scarparolo and Mayne, 2022 [33]	28 preservice teachers	Focus group	Perceptions of students in working with parents after simulation	Pre-post focus groups showed that simulation was a way to expose teachers to parent conferences. Some PST in the study did not have prior experience working with parents and found this type of activity of value.
Fischetti et al., 2022 [34]	2000 new teachers across 3 university sites	Mixed methods pre-post surveys and Flipgrip interviews	Experience with simulation prior to practicum	Helped build candidates’ self-confidence and prepared novices for teaching practicum.
Rappa and Ledger, 2023 [35]	57 preservice teachers	Mixed methods	Working with parents in the simulator	Recommendations for future use of the simulator are provided (practice, performance, preparedness, and reflection on talking with parents) as well as on the outcome of the simulation, helping students with difficulty conversations with parents.

Overall, the research studies identified in Table 1 show statistical or qualitative impact in the use of the MR environment in teacher education, yet these are only the published studies from the past 5 years. The authors note the potential that exists in work that is not published and has demonstrated negative outcomes for the use of MR in teacher education and emphasized the potential bias created by all human-created scenarios. Yet, the team argues that using simulation to talk about bias is a safe and reflective place to provide feedback if led by experts in teacher education who understand, acknowledge, address, minimize, and discuss openly with teachers about their thoughts, reflections, and understanding of the bias in simulation and in their own practice.

5.3. Discussion: The Future

How does the current status of MR reflect on the potential future use of technology in teacher education? What are the lessons learned from the current literature base and how can that work be used to build upon technology use in the future? Of course, there is not a crystal ball into what is the future of technology and teacher education, but the lessons learned from the literature to this point reflect the need for teacher educators to work alongside technology developers in the design of scenarios inclusive of experiential components aligned with feedback. The purpose of the MR environment should be to provide tools and processes that are safe in understanding practice and yet providing feedback into teacher development.

These authors, who try to stay at the cutting edge of technology use, observe forward pathways for reflective practice, already established in the literature on MR, being blended with and further enhanced through the rapid development of AI tools. The team notes AI is not without use in most existing MR simulations as Artificial Narrow Intelligence (ANI) already exists in automated facial gestures and interactive statements (canned laugh and talking during a think-pair-share activity). These behaviors are programmed into the system and can be triggered by an interactor or even simple scene processing. Moving from this simple narrow AI (narrow range of abilities) to more Artificial General Intelligence (AGI), which reflects human behavior, is the next step to consider as the future of mixed reality in simulation and teacher education begins to integrate AI into preparation and practice [36]. So, what is the future of AI and other emerging technologies in teacher preparation and how do they align with existing research and outcomes in preparing teachers in using technology?

6. Discussion: The Future of Technology and Mixed Reality Environment in Teacher Preparation

6.1. Contexts for Automation

When a teacher interacted with virtual students, changes were noted in all 12 studies in Table 1. Those changes varied from body gestures to self-efficacy [25,28,32,34], to performance in tasks [9,27,29,30,31], to communication skills [26], and to working with parents [32,35]. The reactions of the virtual characters in either evolving MR or emerging AI environments need to be realistic, context dependent, appropriate to the character’s backstory, within the scope of the existing learning scenario, and designed by teacher educators to meet the mission of what they are trying to improve in teacher practice as seen by all of the research provided in Table 1.

A theme that emerged from the research studies reviewed is the importance of the ARC cycle [24] to support learning with feedback during and after the experience. Feedback during an experience is usually provided by a master teacher, an experienced coach, or an education faculty member. Feedback after an experience often occurs by viewing a video of the encounter so a subject-matter expert can provide annotations (often called tags) that are temporally aligned with clips in the video. The resulting tagged video can then be used, with or without a coach, to help the teacher reflect on the performance, a component of both the ARC cycle [24] and Kolb’s experiential learning theory. The assumption, supported by the studies noted in Table 1, is that such reflection leads to the improvement in performance both in the simulated environment and as noted by one study ensuring transfer back into the real classroom [37].

Despite the importance of the feedback process [24], the challenges in providing the experience and the feedback at scale (available to everyone when and where they want it) is one of resources. Evolving technologies can annotate or enhance MR experiences, creating ways to provide more objective versus potentially subjective feedback. A typical observation model in the current MR environment would require two to three people, an interactor, a Coach, and an Annotator (who could be the Coach). The Annotator would tag specific skills or take observation data and notes to provide feedback in studies on teacher performance. Yet, using humans in a technology-driven system was the solution these authors noted to create simulations in teacher education, but the evolution to overcome the resource constraints of the cost of human tagging, is to tap into the virtual characters behaviors, which are driven automatically. With the evolution of machine learning and simple tools such as video tagging and language analysis, automation of feedback in MR environments in future studies needs to consider using software-automated annotations to provide constructive feedback, rather than having to employ a subject-matter expert or adding a human in the loop. In the study by [9], they found that providing teachers with automated tagging provided less reaction to the data than in their earlier work where teachers were provided human-coded data. The ability to provide non-judgmental feedback from automation is a strong consideration for future research, but with the understanding of potential bias or even miscommunication of the feedback when humans are removed from the equation.

The way to reduce the cognitive load of the humans needed in the process for use of simulation in teacher education and to reduce the need for multiple simultaneous meetings of humans is to consider the integration of both AI and sensor data. Taking into account the studies by [32,34], the authors conducted extensive pre-post surveys that were perception based upon changes in self-efficacy. Could the field of teacher preparation begin to look at how data on stress, anxiety, and even brain waves (EEG’s or MRI’s) provide further understanding of teacher behavior, but also be more robust feedback and reflection for the teacher educator in simulation environments? How could AI and the integration of a variety of sensors (visual, auditory, physiological) provide a potential pathway for scaling studies beyond a hundred into thousands to better understand both the use of simulation in preparation and also in effective teacher behavior(s)? The goal of future integration of technology is, much like what occurred initially when trying to apply MR to teacher education, to provide creative solutions that are needed to prepare and address extreme shortages in the field for those who aspire to be teachers or who wish to improve their practice or evaluate their skills anytime, anywhere, any place, and at any pace. This type of robust system that is less human dependent and more robust in the data acquired could further elevate the impacts of simulation in teacher education.

6.2. AI and Interactive Performances

Will humans ever be removed from the delivery of teacher preparation using simulation? Online games and AI agent-driven scenarios are emerging, but the robust nature of a human brain reacting in real-time will be difficult if not impossible to replace. The current research shows collective effectiveness in this approach of having a human-in-the-loop, so what are the risks of removing humans or what are the risks of overlaying more technology for feedback? This question is one that has no satisfactory answer, but is the next natural expansion to the foundation research already established on teacher preparation in simulation as well as in other professions built upon simulated practice (flight, medicine, driving). As noted in these studies in this review, trained interactors directed by teacher educators use well-honed observation skills and improvisation techniques to pick up emotional cues, both verbal and nonverbal from participants, and use these to direct the experience as appropriate to some pre-determined objectives. However, as each interactive session requires its own interactor, scalability and cost of this model have been and continue to be an issue. AI has the potential to provide an effective alternative to having human-controlled MR environments in teacher education, but research in comparing the robust nature of learning and outcomes will have to be re-evaluated and re-established to ensure equal outcomes and impact on performance. The basic idea is that, in the future, virtual characters, in a scenario, are driven by a software agent. To accomplish this removal of humans, as seen in the current research, the software controlling the virtual setting needs the AI to be able to read both verbal and nonverbal interactions to drive character behaviors in ways that are contextually correct, stay within the backstories of each character, and maintain the scenario’s goals established by the expert, the teacher educator. This type of multi-modal data system is a heavy load for today’s narrower AI but reflects some of the steps needed to make virtual learning economically available to all and potentially more robust than one human can handle.

These types of robust technology are emerging, but no such tools are clearly present in the teacher preparation arena. While technology such as Wolfram Alpha [38] has been successful in helping in mathematics teaching and learning, and Grammarly [39] in spelling and grammar, being a useful coaching tool to shape a teacher’s behavior involves much more than direct content and structural assistance. An effective coaching tool, which typically involves human coding data, is needed to understand each teacher’s personal goals, current strengths, and existing weaknesses, and to provide support appropriate for that individual teacher’s objectives and needs. For example, ref. [31] found they could work on a discrete skill of increasing the teacher’s use of fewer prompts in working with students with ASD in the simulator. Yet this study required the presence of an interactor in the simulator, the expertise of the teacher educator to create the scenario, and the use of a graduate assistant to tag data and provide feedback. These steps in the future could be automated in many cases, but the level of simulation that is effective without the presence of humans is still at a frontier unknown in relation to both how it can occur and the impact of these tools as they emerge void of the human interaction.

The emerging evidence of such tools to influence a more automated ARC cycle [24] could emerge from the power of conversational agents. Today, most humans have experiences with these agents through the use of Alexa, Google Assistant, Siri, or other Chatbots. While these tools provide interfaces to knowledge sources, their power is limited to verbal interactions, is rarely creative, and fails to account for the nonverbal intent of the message. A human can yell at Alexa, but the reaction by this conversational agent is the same no matter the tone, voice, or body language of the user. Yet in the current research on MR, the human in the loop can respond differently to someone with their arms crossed and yelling “What is the temperature today?” versus someone smiling and in a quiet relaxed voice asking the same question. Although, ChatGPT [40], Bard [41], and other generative AI systems seem to exhibit human or, in some case, even super-human skills and creativity, they fail to understand the nuances of the power of combined verbal and nonverbal expressions and gestures provided in the complex world of being a teacher. Clearly these type of AI systems have the potential to create contextually meaningful dialogue between virtual students and teachers, but at the risk of out of context or even highly inappropriate responses potentially reinforcing negative behaviors that could have been reduced with a human involved at several points in the process. Efforts to build fences around AI tools that could be integrated into MR systems include assessing and, where needed, censoring their responses, with such actions being used to further train the underlying models, but this training specific to teacher education is not at the forefront of any tool emerging at this time. A new tool aligned with more robust discussion is being developed by Kahn Academy (Kahnmigo) and other places related to student learning, but the realization of this work in teacher education is one that is yet to be fully realized in the field.

Currently, in MR environments in teacher education, the interactor is an extremely useful alternative model that can deal with unexpected events during a session or rapid development of new scenarios as needed. Every study provided in Table 1 was built around the use of an interactor in the simulated environment. The alternative use of AI could potentially provide suggestions to the interactor, with the interactor’s actions helping to train the AI, so it eventually can take over without human assistance. Variants of this approach have been used to develop agents that control virtual companions in several research projects involving children and adolescents who are on the spectrum [42], yet how this will evolve between the interactor and the role of shaping what occurs in the simulator by the teacher educator is yet to be realized.

Building on this paradigm, an approach that has succeeded in early testing is to develop AI characters using human control to identify the most effective verbal and nonverbal behaviors. The interactor’s actions are not scripted but emerge in the context of interactions with many users. Using observations of verbal reciprocity, word frequency, and nonverbal actions, master teachers and coaches could determine the most effective expressions of the virtual characters and the triggers from the participants that led to these productive interactions. These interactions could then drive the development of the AI that controls virtual characters, including vocalization, verbalization, facial poses, and gestures. How this shift impacts learning is not clear but is a potential pathway to reduce both the financial and human capital required in the current research and use in teacher education.

Despite the potential of such a model, research challenges faced during the evolution of such AI-enabled characters would include the following: (1) developing novel machine learning (ML) algorithms using labeled datasets populated with the verbal and nonverbal behaviors of members of various student communities; (2) creating an autonomous agent that effectively controls automated behaviors of the virtual characters; (3) testing the effectiveness of these virtual characters in supporting teacher development; and (4) integrating these improvements into a web-based automated virtual learning environment in ways that are informed by Universal Design for Learning (UDL) principles [43].

6.3. AI and Coaching

All the current research studies identified in this review used some type of coaching or coding of observational data with all requiring a human to perform this work. Existing coaching software can use automation blended with human input and is another logical next step for work in MR in teacher education. For instance, if the simulation captures the time intervals in which each virtual student talks and in which the teacher talks, these data could be used to automate annotations in a corresponding video. They also could be used to provide an after-experience pie chart demonstrating how much the teacher allowed for student input and which students may have dominated or been left out of the conversation. With the summary in hand, a teacher or teacher with a coach might go back to instances of interactions in the annotated video to see where they effectively triggered good student involvement, where they may have let one student dominate, or where they may have left a student out of the discussions. This type of automation in teacher education is already commercialized by some companies and a current U.S. Department of Education grant will further develop a tool such as an Open Education Resource for the field to use and further develop automation of the feedback process in MR http://ucf.deviws.edu (accessed 22 October 2023).

This type of automation is something that can easily be performed with today’s technology and, with the integration of additional tools and machine learning algorithms, could assess a teacher’s facial expressions or the verbal intent of a message (e.g., friendly or unfriendly). Of course, these examples require the AI system to determine a person’s emotional cues correctly, which is a challenge filled with bias and misinformation to a level which the field must proceed with caution. A more complex problem to solve is the analysis of a teacher’s utterances to determine if they are asking students high-order or low-order questions or their communication with parents [33,35] or with students [31,35] is on target and relevant. At present, this categorization of utterances seems to require human input.

6.4. Emotional States of Participants: Challenges and Opportunities with Sensors and AI

Beyond utterances, the ultimate learning about teachers in MR environments that could help inform the practices of experts versus novice teachers or help the field consider behaviors related to stress, an area that is currently not present in the literature, is the use to understand teachers’ emotional states. Studies on behavior [26,32,34] or changes in perceptions by teachers exist, but the direction of research on sensors and AI that have displayed the most promising results on emotion recognition are multimodal ones that fuse verbal signals (vocalization and verbalization often using sentiment analysis) [44,45], facial expressions [46], eye gaze [47], body poses and gestures [48], and physiological signals [49] such as heart rate [50] and respiration [51]. The use of these emotion data could provide cues to further enhance either an automated system or a human-in-the-loop or a combination of the two. These cues could also affect the actions of virtual characters and provide automated annotations to further inform the ARC cycle [24] or to create a new model for simulation in teacher education.

An area of AI research that has shown great promise for future integration into simulation and teacher education is natural language understanding and generation, as well as computer vision in the development of transformer-based models [52]. Transformers also can be employed to classify emotions in an arousal (high, neutral, or low)/valence (positive, neutral, or negative) two-dimensional space that categorizes people’s emotional states. For instance, if someone is high on the arousal scale and negative on the valence scale, that person is normally assumed to be tense or even angry. On the other hand, if an individual is low on the arousal scale and positive on the valence scale, that person is normally assumed to be in a calm state. One of the advantages of the transformer approach is that it can learn temporal dependencies over long intervals. As with other ML-based approaches, it can also be used to fuse multiple modalities, combining signals from each modality to achieve a more confident assessment of a person’s state of mind [53]. The fusion of complex data on teacher behavior from AI and ML tools has the potential to move from a subjective state of teacher observation to an objective data-driven state, but only if guardrails against bias or misuse of such tools are put in place, and this is plausible.

An example of work that is already occurring with this type of data collection [42] is an approach to capture biosensor data using a Polar Verity Sense [54] to acquire a PPG (Photoplethysmography) signal. The Polar is worn on one’s arm, in current work by students in the classroom. The Polar then uses an optical sensor to detect volumetric changes in blood in peripheral circulation. The resulting PPG signal contains information about various aspects of a person’s current physical state, including heart rate and respiration. In recent studies, PPG signals have been fused with facial expression information, which, when integrated with personal baseline data, achieves a high level of accuracy about a subject’s emotional responses. Details, including performance comparisons of this with other multimodal approaches, are presented in [55], and the potential for use in teacher education is a new frontier for exploration.

Despite the promise of tools for understanding emotional states of teachers, the current machine learning-based emotion recognition and natural language processing tools are lacking in three major areas (1) ethical considerations, (2) trustworthiness, and (3) transparency. Large Language Models (LLM) like ChatGPT, Bard, and LLaMa [56] discover, condense, and stitch together diverse sources of information found in their dynamic databases. However, few filters exist on whether the result is ethically supported. For instance, in early releases of some of these engines, they reported that the primary attribute correlated to business success is gender. However, these models now nuance this with factors such as stereotypes, work–life balances, and pay inequities. A challenge is providing such nuanced and ethical responses in arbitrary interactions. Trustworthiness is whether the information provided is properly vetted. Initially, the work in simulation in teacher education took over a decade of vetting before widespread use, adoption, and research began to emerge. This is not to say that LLMs cannot evolve to a level of transparency or as it often called, explainable AI [57], in that the model can explain why it came to the conclusions it expressed. Yet, without transparency, the use of AI to make critical decisions like, should we launch a missile, or should we provide a real estate loan to this family, is potentially based on statistical correlation and faulty causation, but, as there is no way to see how the decision-making took place, society could be left with conclusions for which there are unknow underlying parameters. This need for transparency and trustworthiness is currently accepted for the use of simulation through the creation of the scenarios and studies by content experts in teacher education working with a human interactor to deliver safe and effective practice. How this will evolve and change is important to consider to save time and resources, but not at the expense of faulty causation in outcomes for those trusted with one of the most valuable resources in our society, the children they are educating.

7. Conclusions

The evolution of simulation is not new, but the use of MR environments occurred only in the last decade and a half in teacher education. The next evolution of technological advances in teacher preparation were halted to some extent due to the pandemic. For example, the TeachLivE environment was set to move into more immersive virtual reality but, with the pandemic, the ability to share virtual goggles, and the ability to use the work in a room where VR and MR headsets were installed was not possible. Instead, the team pivoted to offer a close to similar experience in a non-immersive virtual platform allowing for more flexible MR experiences. However, research comparing an experience through a video-based platform (e.g., Zoom, Microsoft Teams) has not been compared to settings that are fully immersive in movement. The team, despite interest in immersive virtual environments, believes, however, that the greater leap is to combine this work using the current MR platforms with various AI and biosensors to further impact teacher education.

For example, an area the team is already exploring with students that could be applied to teachers is to use automated character behaviors and coaching data sets to create personalized virtual companions that are always at a teachers’ side, providing coaching, where helpful, and personal assistance, e.g., recommendations to do breathing exercises, as needed. This evolution of this line of research involves both opportunities and threats. The opportunities are clear but so are the threats. If the AI agent makes wrong assumptions or gives bad advice, the impact is not just on the teacher but on their primary role of impacting student learning. For these reasons, we encourage researchers to be conservative, for instance, never using AI or biosensor data to tell a teacher that they are stressed but rather suggesting options such as a virtual agent asking if the teacher wants to join in performing a breathing exercise. This type of collaborative, friendly, and helpful AI agent puts the teacher in control, providing agency rather than criticism.

As explained, the promise of AI and sensors (or more generally the Internet of Things) is great in simulated learning environments but is also fraught with danger if technologist fail to understand the potential harmful effects on human participants. A single bad interaction can negate a series of positive ones. The key to success is effective and respectful partnerships that include, at a minimum, technologists, subject-matter experts, psychologists, ethicists, and teachers, both novices and experts, ensuring the intent of the evolution of this work meets their needs and not that of the research team.

The use of simulation in teacher education is well documented ranging from case studies to role play, to micro-teaching, to MR, to future AI experiences. The evolution of the integration of technology, in not just its use by teachers, but in preparing and retooling practice that exists and continues to evolve needs to occur through an established and emerging research base. The bias, ethics, understanding of how teachers best learn, and more importantly, how their behavior impacts learning outcomes is and will continue to be a question that the field must answer as this research is conducted. However, until teachers can assure, and researchers can attest that technology is a better tool to prepare teachers, then everyone involved in preparing teachers needs to listen to the voice of the teacher in what helps versus harms the most important job they have, changing the learning and social emotion outcomes for all students who enter their classrooms. If MR, AI, and biosensors, or any other tool can improve a novice teacher, decrease burnout, or change the practice of veteran teachers at a faster and more efficient rate than past PD tools, then any or all should be considered but with caution of always ensuring the changes in teacher performance positively impacts student learning outcomes.