Evaluating Virtual Reality in Education: An Analysis of VR through the Instructors’ Lens

Abstract

The rapid development of virtual reality (VR) technology has triggered a significant expansion of VR applications in educational settings. This study seeks to understand the extent to which these applications meet the expectations and pedagogical needs of university instructors. We conducted semi-structured interviews and observations with 16 university-level instructors from Oregon State University to gather insights into their experiences and perspectives regarding the use of VR in educational contexts. Our qualitative analysis reveals detailed trends in instructors’ requirements, their satisfaction and dissatisfaction with current VR tools, and the perceived barriers to broader adoption. The study also explores instructors’ expectations and preferences for designing and implementing VR-driven courses, alongside an evaluation of the usability of selected VR applications. By elucidating the challenges and opportunities associated with VR in education, this study aims to guide the development of more effective VR educational tools and inform future curriculum design, contributing to the enhancement of digital learning environments.

1. Introduction

Over the past decade, the rapid advancement of digital technologies has transformed the educational landscape, introducing novel electronic methods of delivering and experiencing learning. The expansion of online university education has significantly contributed to the body of knowledge on electronic learning (e-learning). However, the preponderance of this research has highlighted more limitations than strengths in the remote classroom setting. While e-learning surpasses traditional in-person instruction in terms of cost and accessibility, it often lacks in providing practical, hands-on experience [1,2]. Additionally, the physical separation between students and instructors can hinder communication, collaboration, and engagement—elements crucial to the learning process [3,4,5].

In this context, immersive and interactive technology has emerged to enhance these digital learning environments by simulating real-world experiences, thereby addressing some inherent limitations of e-learning [6,7]. Virtual reality (VR) seems to emerge as a particularly promising technology, offering immersive environments that potentially enhance student engagement and facilitate a deeper understanding of complex subjects [8]. However, despite VR’s growing accessibility, its integration into educational settings remains in the preliminary stages, particularly in higher education, where traditional methods still dominate.

In fact, although considerable enthusiasm surrounds VR’s capabilities, a significant gap exists in the empirical understanding of its effectiveness in real-world educational contexts. This augmentation has been explored from various perspectives, including the impacts of synchronicity, return on investment, and the modeling of barriers and facilitators within e-learning environments [9,10,11]. However, most existing research focuses on theoretical potential rather than practical application, leaving educators uncertain about the true benefits and limitations of VR [12]. Moreover, there is a lack of comprehensive analysis on how well current VR applications meet the specific pedagogical needs of instructors, which is critical for fostering meaningful educational experiences [13,14]. Given this robust research foundation, it is crucial to assess how well existing virtual reality (VR) products meet instructors’ expectations.

To address these gaps, this study evaluates the effectiveness of current commercial collaborative VR applications from the perspective of university-level instructors. By employing a mixed-methods approach, including semi-structured interviews and observational studies, this research aims to capture a detailed picture of educators’ needs, their satisfaction with existing VR solutions, and the pedagogical outcomes of using such technology. This approach not only highlights the practical utilities and drawbacks of VR in education but also maps out the expectations educators hold toward this emerging tool.

The primary aim of this study is to identify educators’ needs when using immersive and interactive technology and to verify the efficacy of VR solutions in meeting these needs. To achieve this goal, we utilized current commercial collaborative VR applications in the educational sector to assess their potential effectiveness from the instructors’ perspectives. Our analysis is intended to inform the development of future educational software systems. The remainder of the paper is organized as follows: Section 2 reviews related work in educational VR and identifies knowledge gaps. Section 3 describes the study design and methodology. The results are discussed in Section 4, followed by the practical and theoretical implications of our findings in Section 5. Finally, Section 6 presents our conclusions, along with the study’s limitations and directions for future research.

2. Related Work

During the COVID-19 pandemic, most educational institutions in the United States adopted emergency remote teaching, a shift whose long-term impact remains uncertain [15,16]. Despite the transient nature of this urgent need for effective remote education, the potential of e-learning to transform STEM education endures beyond the immediate crises. The constraints placed on synchronous classes during COVID-19 caused educators and learners to experiment with digital tools for delivering engaging learning experiences and nurturing an interest in new technology [17]. In this context, simulation-based learning has proven to be highly beneficial [18,19].

Simulation tools offer immediate visual feedback, help students understand abstract concepts that are otherwise difficult to grasp [20], and provide a safe and cost-effective way for students to experiment and learn. However, the information conveyed through these 2D simulations is limited, as they fail to represent the complexity of 3D concepts [18]. VR technology can help students effectively visualize abstract complex concepts and create experiential learning opportunities [21,22,23,24].

Furthermore, VR also supports the implementation of use cases that can potentially extend the conventional mode of instruction and satisfy pedagogical needs that are not possible to meet with traditional in-person instruction [13,14]. For instance, constructivism states that the learner is an active participant in the knowledge acquisition process. Learners construct their knowledge through experiences rather than just acquiring and processing concepts [25]. The use of simulations, virtual tours, and hands-on experiences in the virtual environment, extending geographical, financial, and safety barriers, provides contextual experience to learners and involves them in the learning process rather than having them act as mere observers [26,27,28,29,30]. An additional element to consider is the decline in students’ attitudinal factors, such as motivation, confidence, and interest, which may act as a catalyst for attrition in STEM courses [31]. Previous research works have demonstrated how VR can complement traditional classroom teaching methodologies by facilitating engagement, motivation, and inclusiveness [32,33,34]. Virtual learning environments support a range of capabilities for students to explore their creativity and imagination, including immersion [35], storytelling, and avatar representation [36,37]. VR in learning may change the way a student approaches the subject. This attitude shift can create a positive environment for students to learn a complex topic. This can open up a plethora of options for student engagement, resulting in career opportunities, an increase in the number of students in STEM, and a boost in the creative economy. This opportunity to engage in collaborative learning derives from social-constructivist theory, which describes learning as a social, participatory process [25]. Learners are diverse, and through collaboration, they can co-construct knowledge from different perspectives to expand their eventual understanding. Collaborative learning has benefits across social, psychological, and academic facets [38]. There are five fundamental elements involved in collaborative learning: positive interdependence, individual and group accountability, interpersonal and small-group skills, face-to-face promotive interaction, and group processing [39]. These factors are positively related to academic achievements [40], social competence, and a sense of community, all of which are crucial for the holistic development of a student. Collaborative instruction supports a shift from thinking of instructors as merely “expert transmitters of knowledge” to viewing them as “expert designers of intellectual experiences” [41]. Therefore, they can improve the learning outcome compared to traditional collaborative learning methods [42] and have the potential to promote non-traditional learning scenarios [43,44].

Within the rich field of 3D immersive learning environments, two distinct application domains can be observed: the training sector, such as employee training in manufacturing, retail, and public service, and the educational sector, including K-12 and higher educational programs. Virtual reality HMDs, as a key enabling technology, have already found a home amidst commercial training applications. In the industrial sector, it is more cost-efficient to deploy tailored training software from scratch to attain critical advantages in the conservation of resources, the safety of training environments, and learner engagement, all while maintaining the efficacy of traditional (either in-person or electronic) training methods [45]. This is likely possible due to a combination of interest from VR researchers in specific use cases coupled with the minimal flexibility required by specific industries, which results in software being developed in a rigorous, costly, and non-retrofittable manner.

The same cannot be said for the educational sector. Even within STEM, the diversity of disciplines, the breadth of required knowledge, and the scope of activities quickly make VR costly as a teaching tool. For example, it is often infeasible for instructors to develop educational materials from scratch on a granular use-case basis [46,47]. Although it has been studied immensely in controlled laboratories, VR has yet to be adopted as an educational tool in operational contexts. This is despite the fact that laboratory experiments show the same tremendous theoretical maximum yield for education as for training: in other words, the potential for saving resources, ensuring safety, and promoting engagement while maintaining the strength of in-person learning [48,49,50].

This lack of adoption is particularly relevant considering two main factors: a sudden drop in the cost of VR HMDs and the aforementioned pressing need for more effective and efficient e-learning solutions. Until very recently, VR HMDs have been prohibitively expensive and not capable of running intensive simulations at a high level of fidelity. These accessibility limitations produced a limited pool of users, in turn eliminating the incentive for the production of highly scalable, customizable VR educational solutions. The acceleration in the VR market has fueled the development and release of several educational VR products, but the adoption of VR in learning environments is still impeded by other factors, such as technological and organizational constraints, operational costs, and content availability [51]. Teachers’ attitudes and beliefs toward technology have also been considered a deciding factor in the successful adoption of VR in learning environments [52,53,54]. Ashrafzadeh and Sayadian [55] listed several factors that may influence the instructor’s perception of technology usage, including the usefulness of the technology for classroom instruction, the institutional support provided, and individual characteristics. In their study of 91 Iranian educators, the authors mention concerns with technology integration and the role of culture in instructors’ perception of technology. Specifically, the authors emphasize the need for awareness of the technology, its benefits, and implementation scenarios to ease these concerns.

This is perhaps exacerbated by the fact that it is hard to employ suitably qualified instructors who have the technical knowledge to develop such applications (in addition to the domain content they teach), and it is likewise unrealistic to train existing faculty to develop these skills [56,57,58]. Many educators, perhaps due to their lack of familiarity or otherwise, push back against the inexorable technological creep of modern e-learning, which naturally further impairs their ability to participate effectively within it [59,60,61]. As such, there is a strong demand for technologists who understand learning theories and for instructional designers and educators who understand technologies and how to effectively integrate technology into learning and instruction.

Theoretically, there are also issues that might inform some of these struggles related to the adoption of VR technologies in the classroom. For example, only a small number of studies test applications and prototypes developed for HMDs in real-class settings. Many papers instead consider only virtual environments made for web and desktop platforms but still call them immersive virtual environments. Radianti et al. [30] noted this ambiguity in the use of the term “immersive” in the literature. There also appears to be a lack of a comprehensive overview of existing VR applications for educational purposes, which might better inform development and tool selection. Additionally, the number of studies evaluating commercial applications is also limited. Konstantinidis et al. [62] evaluated collaborative virtual learning environments by reviewing current systems based on popularity and reviews; however, the paper evaluates web-based virtual environments and does not account for applications that can support VR capabilities. Further, the paper was published a decade ago, with a few applications discontinued, indicating the need for a current review.

Jensen and Konradsen [63] described two barriers impacting the use of HMD experiences in education: lack of content and hardware. The authors further underscored that most educational applications are not classroom-ready, are directed at self-learners, and demand a certain degree of technical expertise for effective classroom usage. Furthermore, with the ever-growing development of VR applications in the educational space, it is pertinent to understand their effectiveness as instructional tools rather than mere engagement devices.

Educators who are interested in introducing immersive technologies to their curriculums need to have sufficient technological knowledge and be abreast of the market of suitable VR applications. The widening stream of devices and applications with uneven quality and uncertain utility reaching the market these days calls for a periodic assessment of the current state of the art. Hence, it seems necessary to review current off-the-shelf applications to assess their quality, effectiveness, affordability, and utility in the educational environment with the goal of identifying viable candidates and discovering challenges and opportunities for future products.

3. The Rationale and Justification for the Research Approach

The current study seeks to move closer to the integration of VR with the curriculum by understanding the needs and challenges perceived by instructors. For this purpose, first, we identified existing VR applications that support collaborative virtual learning. We chose hardware (Oculus Quest) for the study based on constraints such as the standalone nature of the hardware and costs (less than USD 400). The rationale behind these constraints is based on the assumptions surrounding the university budget that is usually provided for technological devices commonly shared with students. Subsequently, we recruited university instructors from various fields to gather software requirements for such apps. These requirements were collected using questionnaires filled out before and after exposure to VR apps for teaching activities. Lastly, we assessed the effectiveness of the applications by reviewing instructors’ reflections and recommendations. Furthermore, the derived user requirements inform the design and development of future collaborative applications for learning activities.

Specifically, we focused on the following two main research questions (RQs), which are broken down into sub-RQs as follows:

• RQ1: What is the current state of the art in commercial collaborative VR applications for educational activities?

  –

  What are the features provided by commercial applications to support instructional activities?

• RQ2: How do current commercial collaborative VR applications meet the expectations of educators to conduct teaching activities?

  –

  What are the software requirements from the point of view of instructors?

  –

  How do instructors perceive the usability of these selected applications?

  –

  Barriers to adoption: What are the challenges perceived by instructors for deploying VR in classrooms?

The contributions of this paper are timely for two reasons. First, the global pandemic has highlighted the necessity of a digital transformation in education that improves the learning experience in remote collaborative settings. Second, VR technology has become noticeably more affordable in recent years, which makes the use of VR headsets as educational tools much more feasible. Furthermore, this research is relevant since it can assist instructors in finding existing educational VR applications that meet their needs and expectations, which will ultimately stimulate a broader deployment of immersive technologies as educational tools. Moreover, designers of collaborative VR learning applications can benefit from the gathered user requirements to better address instructors’ needs.

4. Methodology

This section outlines the comprehensive methodology, see Figure 1, used to investigate the impact of virtual reality on educational practices, detailing the systematic approach adopted to gather and analyze data from instructors.

4.1. Participants

The participants in the study were 16 Oregon State University (OSU) educators (9 women, 7 men) recruited from various departments across OSU: Electrical Engineering and Computer Science (5), Mechanical, Industrial, and Manufacturing Engineering (3), Civil and Construction Engineering (3), Animal and Rangeland Sciences (2), Nuclear Science and Engineering (1), Wood Science and Engineering (1), and Nutrition and Public Health (1). Participants were recruited via a multi-pronged (email invitations, website announcement, university newsletter) recruitment strategy in an attempt to recruit a diverse and multidisciplinary population of instructors. The study required participants to be of legal age to consent, fluent in English, and an instructor at a university. Additionally, the recruitment survey included questions to verify any susceptibility to motion-induced sickness and digital-screen-related illness.

We recorded reasons for those instructors who declined to participate, if possible, to potentially understand non-response bias. The time commitment was the top reason. However, a minority of participants rejected the invitation due to other concerns related to age or sensitivity to motion-induced sickness/nausea. One of the rejection responses we received indicated a disinterest in VR technologies for education. The individual stated that they have never played video games and the technology does not appeal to them. Such responses highlight how educators’ perspectives on the usefulness of VR as an instructional tool can impact even considering the use of such technologies in the classroom. Such comparisons were also observed during the study and are consistent with the challenges found in the literature [64]. They may highlight educators’ varied teaching styles, individual differences, and concerns with technology. Future research is warranted to study these perceptions in detail.

4.2. Phases

4.2.1. Application Search and Selection

We performed a search for commercial collaborative VR applications for teaching activities. The applications were identified by conducting both a web search and/or searching through various app marketplaces. The search was conducted using keywords such as collaborative, learning, education, or edutainment. We followed the below criteria during the search and selection process:

• Inclusion criteria

• Shared collaborative space available for learning activities.

• Exclusion criteria

• Does not support the HMDs selected for the study;
• Free license not available;
• Limited to a certain subject (e.g., chemistry or anatomy).

We conducted the search process in an iterative way. Based on the features supported, we compiled a number of user scenarios that participants could experience in the virtual environment to collect further requirements. Constraints to the search and selection process were applied, bearing in mind the practicality of classroom adoption along the lines of budget and portability with the untethered nature of HMD. The reasoning behind this decision is that the current cost of Oculus Quest (starting at USD 299) is considerably lower than other headsets, such as HTC Vive, comparable to the cost of technological devices such as tablets/smartphones. Furthermore, Oculus Quest is a standalone device, unlike headsets such as Rift or Vive, providing the added advantage of not requiring a powerful VR-ready PC for its utilization. Utilizing a mobile phone combined with a passive headset, such as Samsung Gear VR or Google Cardboard, may also seem like a viable, affordable option. However, the diversity of available phones and the corresponding software and hardware compatibility make it virtually impossible to find applications that support all platforms while ensuring a comparable learning experience. Furthermore, the freedom of interactions within the virtual environment and graphics differentiates the Oculus Quest from the other aforementioned headsets. As the hardware market evolves, future studies will include other HMDs and hardware to evaluate the effectiveness of educational VR applications.

After the search process, we explored all identified applications individually to evaluate their quality and confirm the available features. In addition to the features, information such as cost, hardware support, and license information was also considered. Applications that were either early-release or unstable were excluded, as this might hinder the user’s experience in the subsequent hands-on experience session. A table of features that we observed in these collaborative applications was compiled. Through the table, we selected three applications, all of which included capabilities such as generic learning space, classroom collaboration, and content presentation tools.

4.2.2. Initial Needs Assessment

In this phase, we identified the initial user requirements and expectations for collaborative virtual environments through the lens of instructors. We collected these requirements utilizing the pre-exposure questionnaire, a measure that consists of a mix of close-ended and open-ended questions that was administered to participants. These questions covered participants’ current methods of teaching, their experience using VR applications, their motivations and expectations from a VR classroom, and potential usage scenarios of integrating VR into their teaching.

4.2.3. Application Walkthrough

During the application walkthrough session, participants saw the applications in action remotely through Zoom and observed different usage scenarios demonstrated by the research team. To achieve the above objective, the research team created a video consisting of several VR use cases across the 3 selected applications. A video presentation was used to maintain consistency across all participants. After watching the video, the participants were further interviewed by the researcher. This semi-structured discussion was designed to clarify any questions about the technology and VR suitability for their courses. Subsequently, participants filled out the post-exposure questionnaire. The responses were compared to the pre-exposure questionnaire to determine whether this initial exposure to the potential of VR resulted in any change in participants’ expectations or perspectives. The session concluded with a brief device tutorial introducing the participants to the technology and HMD hardware (Oculus Quest) and demonstrating how to use it.

4.2.4. Hands-On Immersive Experience

To conduct the immersive experience, we observed participants remotely through Zoom as they performed predefined tasks in a virtual mock classroom within the identified applications on the VR equipment. Participants were asked to “think aloud”. Initially, participants explored a “warm-up” application to familiarize themselves with VR interactions and the environment, such as teleporting, avatars, and loading content. After completing the warm-up exercise, participants then began to interact with the target applications. Participants experienced the two selected VR applications for the study, and the order was randomized among participants to avoid sequence effects.

Each application had three tasks; one researcher read out the instructions over Zoom, and the other researcher joined the virtual environment along with the participants to stream the participants’ behavior over Zoom for synchronous observation and ad hoc troubleshooting. The tasks included using 3D drawing, 3D models, PDF slides, quizzes, videos, and a whiteboard (2D drawing). For each task, subjective feedback was collected on perceived ease of use, task completion time, and the relevance of the feature for their teaching activities. Additionally, suggestions for improving any features were collected through an open-ended discussion. Once all the tasks for an application were completed, participants disconnected from the VR environment and were debriefed, where they provided additional feedback and filled out the System Usability Scale (SUS) survey. This process was repeated for both applications. At the very end of the session, a final semi-structured interview was conducted to collect any further comments or reflections.

4.3. Data Analysis

This study included components of quantitative and qualitative data collection. The quantitative data were analyzed using R, and the qualitative data were analyzed using Excel spreadsheets and R scripts for frequency analysis. Non-parametric tests were used to analyze the feature preference ratings across stages, and other descriptive measures for quantitative data gathered are provided in the following sections.

Qualitative data were collected at four points during the study: the application walkthrough discussion and open-ended responses collected about use cases, the think-aloud protocol used during the hands-on experience session, verbal debriefing questions, and semi-structured interviews. The research team transcribed the video recordings of all sessions and segmented the verbal protocol data based on stanzas. A stanza refers to a group of lines dedicated to a particular event, topic, or perspective [65]. The researchers created rules and templates to maintain consistency during the segmentation procedure. During transcribing, the researchers added observer comments and maintained analytical memos. Subsequently, a priori codes, derived from the research objectives, were applied to code the data into three overarching categories: Needs, Satisfaction, and Challenges. Two researchers coded the data from three participants, which roughly covered 20% of the data, and calculated inter-rater reliability (IRR). IRR was calculated using Jaccard’s index [66]. During the coding, the researchers discussed their decisions and calculated the partial IRR for roughly every 5% of the data. The average IRR for these 20% of data was 83%.

Next, affinity diagramming was used to generate the codebook on another set of roughly 20% of data for the next phase [67]. Affinity diagramming is a functional tool to identify relationships and patterns in data. The researchers used three participants’ transcripts (18.75%) to generate the codebook. First, the coders individually annotated the initial pool of participants’ transcripts. Subsequently, the coders read the annotations and had discussions to create the codebook. The description and label were revised throughout the discussion. This process was repeated for each of the a priori categories of stanzas coded in the previous phase. Subsequently, the researchers tested the codeset on another 2 participants’ transcripts to calculate the inter-rater reliability. The IRR achieved in the second coding was 90.8% using the Jaccard index. Finally, the researchers coded the rest of the data using the codebook. Researchers used affinity diagramming to organize the codes into themes and labeled them after discussion; then, they created a structure of categories, themes, and sub-themes for organizational purposes. Subsequently, these sub-themes were iteratively analyzed to understand the nature of the responses, and verbatim responses were provided.

5. Results

5.1. Participants’ Technology Usage and Current Teaching Methods

Participants’ experiences with technology for classroom usage and their current instructional styles were collected in the pre-exposure questionnaire. Most participants expressed using cameras (11), followed by tablets (e.g., iPad, Surface Go) (8) and graphics drawing tablets (5). The least common were VR headsets (1), Smartboards (1), and 3D printers (1). When asked about their current classroom instructional delivery, 43.75% of participants mentioned that they use prepared instructional materials for more than 90% of lectures, and 37.5% of participants mentioned the use of real-time demonstrations for more than 90% of lectures.

Table 1. Summary of current instructional methods for delivering in-person classes.

Delivery	Frequency
Method	More than 90% of Lectures	More than 60% of Lectures	More than 30% of Lectures	30% of Lectures or Less	Never
Real-time demonstration	6	1	4	3	2
Prepared instruction	7	3	1	5	0

summarizes the typical class instruction for the participants in the study. Some participants also mentioned additional methods, such as live working on problems (1), the usage of whiteboards/docCAM/fill-in-the-blank notes (1), and in-farm practices (1). Prepared instructional materials require that educational applications afford content integration (e.g., Drive), presentation (e.g., document viewer), reusability, and support for authoring. Real-time demonstrations highlight the importance of classroom collaboration, communication, and content presentation tools for in-class activities.

5.2. Participants’ Prior VR Experiences

Among the participants, seven instructors had used VR previously for personal use (43.75%), and of those, one had also used it for classroom instruction. We inquired about possible opportunities and challenges observed during their usage to understand participants’ prior experiences, including the headsets and applications used. The headsets used for personal use were Google Cardboard or GearVR (2) and HTC Vive (2). Two participants reported that they did not know the headset they had used previously. The headsets used by the participant for classroom usage were Oculus Quest 1/2 and Google Cardboard/GearVR. In terms of the domain, educational applications were used by most of the participants (4), followed by applications in the areas of games (3), arts and creativity (2), travel and discovery (1), and productivity and collaboration (1). The opportunities noted by the participants for VR during their personal use were a feeling of immersion, 3D models, visualization, a sense of community, and cultural appreciation. In terms of challenges observed, participants mentioned the following: non-intuitiveness, navigation-related concerns, language support, and concerns about motion sickness/eye strain/disorientation/physical space constraints. Such opportunities and challenges were also present in participants’ responses during the study, indicating their wide prevalence.

5.3. Expectations for Teaching a VR Course

We asked participants their expectations for delivering a VR class for a selected course and asked them to rate the importance of VR features for that class. We collected these metrics across two stages through the pre- and post-exposure questionnaires (

Table 2. Expectations of optimal duration for class taught in VR.

Descriptive Statistic	Pre-Exposure (Mins)	Post-Exposure (Mins)
Mean	40	48.75
Median	35	50
STDEV	24.77	40.64
MIN	0 (asynchronous)	10
MAX	90	180

). We noticed that few participants modified their preferences after a short exposure to the VR applications. Though we have a higher range for the optimal duration for a class taught in VR, participants during the hands-on experience phase indicated concerns about discomfort and fatigue while using the headsets for prolonged periods.

We calculated the results of the preliminary exposure based on the ratings of importance given to the individual features across pre- and post-exposure responses. The Wilcoxon signed-rank test indicated a significant difference (p < 0.05) in eight ratings, with higher ratings given during the post-exposure questionnaire. The features were 3D models, web browser, video player/YouTube, 360 videos, cloud integration, webcam sharing, creation/personalization of avatars, and customization of the environment. Figure 2 illustrates boxplots of the features that were significantly different between pre- and post-exposure questionnaire responses.

After the walkthrough discussion session, we noticed that an overwhelming majority of participants voted the virtual field tours and simulations to be the most promising use case (14), followed by immersive environments (3), 2D tools (2), and shared experiences (1). Use cases that did not meet their expectations: typing in VR, the inability to see students’ faces, skepticism toward the usefulness of features such as chat, in-app quizzes, and slides, and lack of integration with university-supported tools (e.g., Canvas) were mentioned. These findings are consistent with participants’ responses during the hands-on experience.

We asked participants what kind of support they would expect for teaching their VR class. The Wilcoxon signed-rank test indicated no significant difference between the pre- and post-exposure questionnaires. Figure 3 illustrates the distribution of preferences for four different types of support methods. Finally, we asked our participants about their preferred methods for taking notes inside the VR environment, as traditional methods (paper, laptop, etc.) may not be possible. Eight out of sixteen (50%) participants preferred to provide students with lecture materials (texts, slides, videos, etc.). Five out of sixteen (31%) participants preferred that students take notes using virtual notebooks within the environment. Finally, 3 out of 16 (19%) participants preferred to provide students with video recordings. No participants favored students taking screenshots within the environment. No other alternative methods were proposed either.

5.4. Usability Assessment

A general measure of usability was assessed for the applications through a SUS questionnaire. The average SUS score of application-1 was 72.60 (sd = 13.77) 95% CI [65.27, 79.94]. The average SUS of application-2 was 71.98 (sd = 13.45) 95% CI [64.81, 79.14]. Both of these applications reported a SUS above 70, marking them acceptable under the Good category in terms of usage [68]. Additionally, we collected participants’ satisfaction with the amount of task time, ease of completion, and relevance of the feature for their teaching activities.

Figure 4 shows the distribution of the ratings provided for satisfaction with the ease of completion, the amount of task time, and feature relevance, categorized by tasks. It can be noted that the ease of completion has the largest variation, with most users providing lower scores compared to the relevance ratings. A pattern consistent with the qualitative responses is that, despite finding features relevant, participants rated the ease of completion low. For instance, participants found the whiteboard relevant to their teaching. However, they had trouble interacting with it due to a lack of precision/control and availability of features. Further, the ratings for relevance depended on participants’ course, teaching style, and, at times, the effectiveness of the tool and its advanced features.

5.5. Themes of Requirements

The highest proportions of themes under Needs were 2D media (26.57%), 3D media (22.54%), and immersive and interactive experiences (21.83%). Figure 5 illustrates the association between the themes and sub-themes. The themes and sub-themes are provided along with participants’ quotes in the Appendix A.

5.5.1. Need for 2D Media

A popular sub-theme was the need for 2D media delivery tools. These include a wide range of traditional media tools, such as images, videos, slides, screen sharing, web browsers, etc. Participants expect the VR environment to support their current instructional delivery methods but raised questions about the advantage of using VR for delivering 2D content over existing methods. A frequent requirement was the support for multiple file formats and advanced features for presentation and control. Further, participants expected fluidity and seamless interaction at the same comfort level as traditional technologies for 2D content delivery. Other sub-themes mentioned by the participants were 2D drawing and 2D drawing support, with the majority of responses toward requiring assistance while drawing. As mentioned before, participants found the whiteboard extremely relevant but were not satisfied with the current implementation (drawing with controllers). Participants emphasized the need for advanced features, including color palettes, predefined shapes, auto-complete, smart drawing/erase, text, and better precision.

5.5.2. Need for 3D Media

Participants highlighted their need to visualize semantically enriched 3D models, noting that the generic repository available in current applications was insufficient. They stressed that the utility of this functionality hinges on access to an extensive library of models or the ability to use custom models.

For instance, a participant from the Animal and Rangeland Sciences department imported a 3D model of a cow into the virtual environment during the task. While impressed with the model’s quality, she noted that the detail and availability of only the animal’s exterior were inadequate for her teaching needs. Further, participants emphasized the importance of interactive 3D model features, particularly the ability to view and manipulate these models from a student’s perspective. They also expressed a strong interest in the creation of 3D models, a topic that emerged frequently in their feedback.

The 3D drawing capabilities were found to be engaging, with uses ranging from annotations on virtual objects to teaching complex mathematical or abstract concepts. Although participants were relatively inexperienced with 3D compared to 2D sketching, they envisioned potential educational applications. Some expressed a willingness to use these features even without specific use cases in mind. Similar to 2D drawing, they requested enhancements, including a color palette, grids for adding 2D shapes and text, and improved comfort and control during drawing. Concerns were also raised about the preciseness of tools and the ability to seamlessly switch between 2D and 3D drawing modes, echoing common challenges in 3D visualization [69].

5.5.3. Need for Immersive and Interactive Experience

Participants expressed strong interest in virtual field tours and simulations, which allow students to explore environments that are otherwise risky or inaccessible. This interest aligns with motivations for using VR technologies, as identified in the literature [26,28]. Examples of virtual field trip ideas proposed by participants include visits to manufacturing factories, mills, dairy farms, incubators, construction sites, data centers, nuclear power plants, steel fabrication shops, and even outer space. Such virtual field trips and simulations are central to authentic learning approaches, facilitating the transformation of abstract concepts into concrete, tangible understanding through multi-sensory experiences, which VR technology enhances [27,29].

Immersive learning environments, both formal and informal, also generated significant interest among participants. However, opinions varied regarding the setup of these environments, especially concerning the presence of furniture, indicating a need for customizable settings based on specific educational needs. While some participants appreciated minor details like the layout and familiarity of the virtual environment, others raised concerns about elements such as closely seated chairs, oversized presentation boards, and lack of personal space, which could affect student comfort.

Participants additionally highlighted the need for enhanced visualization and spatial understanding. They expect VR to aid students in visualizing abstract concepts, performing compare-and-contrast analyses, and interacting with 3D artifacts for better spatial comprehension [26]. This need extends to the integration of 3D drawing, virtual simulations, design walkthroughs, and interaction with 3D models into the learning process. The feedback underscores the demand for domain-specific learning activities that go beyond the generic features of current VR applications.

5.5.4. Social Collaboration

Shared learning spaces and collaborative tools were frequently recommended by participants in the study. Moore [70] identified three key types of interactions in distance education programs: learner–learner, learner–content, and learner–instructor. Our findings particularly highlight the need for tools that facilitate both learner–learner and learner–instructor interactions, such as verbal and non-verbal feedback mechanisms (voice, chat, emojis), interactions with virtual objects, team break-out rooms, collaborative presentations, and brainstorming tools.

The principles of social constructivism and experiential learning are evident in participants’ preferences for learning environments where students actively engage and communicate with peers [29,30]. Most participants expressed flexible preferences regarding avatars, prioritizing students’ choices to enhance personalization and inclusivity within the virtual space. However, concerns were raised about the potential for avatars’ movements or appearances to be disruptive. For instance, unintended hand or leg movements when the Head-Mounted Display (HMD) is removed or disconnects could distract users. Participants emphasized the critical role of avatar representation and customization in creating an inclusive learning environment. One participant noted the importance of allowing students to personalize their avatars to reflect their identities, which supports a more engaging and inclusive educational experience.

5.5.5. User-Centric Design

Participants highlighted the importance of being able to customize and organize the virtual environment using tools that mimic real-life counterparts, such as tablets and smart displays. They emphasized the need for the ability to easily switch between different content presentation modes to effectively manage the classroom. Additionally, concerns about privacy, safety, and personal space within the virtual environment were prevalent, underscoring the role of environmental design in creating a safe and inclusive space for students. Participants also expressed a desire for personalization options that respect individual differences. From the early stages of the study, there was a noticeable interest in incorporating external tools into the virtual environment. Participants were particularly curious about features like keyboards for text entry and note-taking capabilities, although these had not been explored during the study. Furthermore, there was curiosity about non-HMD-related experiences, such as using Google Cardboard VR, desktop or browser companion applications, and the integration of hardware like voice input and web cameras. While these features were not part of the study, participants expressed a keen interest in understanding how these technologies could enhance the user experience in virtual environments.

5.5.6. Need for Instructional Support

Participants recognized the potential value of reusing content within the virtual environment, such as recordings and session saves, as a means for instructors to share and reuse course materials effectively. This capability is especially valuable given the significant time and effort required to create virtual spaces, particularly for those who teach in asynchronous or hybrid formats, where content reusability can greatly enhance course delivery. Additionally, participants discussed the need for the VR application to support large class sizes, maintain fluid interactions with existing content delivery methods, and accommodate varying durations for VR experiences to integrate successfully with the current curriculum. They emphasized the importance of compatibility with university tools like Google Drive for content creation and delivery. Alongside these integration capabilities, there was notable curiosity about authoring solutions outside the VR environment. Participants were interested in how these tools could connect with existing university systems for managing and creating content, suggesting a need for seamless integration between VR technologies and traditional educational resources.

5.6. Themes of Barriers to Adoption

The most popular themes identified as challenges in Figure 6 were “attitude toward technology”, accounting for 27.4% of responses, and “expertise and effort concerns”, making up 25.9% of responses across participants.

5.6.1. Attitude Toward Technology

Among the sub-themes, a significant majority of participants were skeptical about the appropriateness of VR technology for specific activities, especially when compared to existing methods. In line with Rogers [71], if educators do not perceive clear advantages over existing methods, this could become a significant barrier to its adoption. Many described the use of 2D features in VR as either “overkill” or “not worth the investment”. They expected VR to enhance, rather than merely replicate, the capabilities of traditional educational technologies. This sentiment was exemplified by participants’ criticisms of simply using videos and slides in VR. The adoption of VR technology requires a substantial investment of time and effort.

5.6.2. Learner’s Experience

A trend in participants’ responses was a concern about students’ experiences with the learning environment and their potential frustrations with the technology. Participants expressed that their top priority was to ensure equal learning opportunities for all students in the classroom. Responses indicated the need for accommodations for individual differences for an inclusive and safe classroom experience. Participants also expressed concerns about certain interaction mechanisms and the effort required to use them for students. Further, women’s safety in the virtual space was stated by one participant.

5.6.3. Technology Deficiency

Participants were observed adjusting their headsets multiple times during the sessions, along with showing redness on their faces after headset removal due to the firm fit that is required to prevent possible slip-offs. Headset fit and comfort, along with blurry vision and fatigue observed while wearing the headset, were discussed by participants. The most prevalent reason for discomfort was the headset slipping off.

Concerns related to fatigue induced by the headset over prolonged wearing surfaced during conversations with the participants. Additionally, equipment discomfort was often linked with screen quality limitations due to the blurriness caused by the headset slipping off. Most concerns about motion sickness were with respect to students or other instructors who may face motion sickness. Three participants reported dizziness associated with the experiences. Internet disconnections and bandwidth and latency concerns were expressed by participants and also observed during the study.

5.6.4. Expertise and Effort Concerns

When participants interacted with 3D models, they frequently expressed concerns about the time, effort, and expertise required to create VR content, identifying these factors as significant challenges. Responses also revealed a lack of exposure to authoring solutions or resources for VR content creation, underscoring the need for external support during the development process. Instances were noted where participants felt that their inexperience with VR contributed directly to their interaction challenges. Any interface inconsistencies or unintuitive interaction mechanisms led some to question their own ability to navigate the VR environment effectively.

Specifically, participants raised concerns about the impact of low technology awareness and demographic factors, such as age, on the implementation of VR in classrooms. They also worried about students’ technological competencies and the time needed to familiarize them with VR technology to ensure its effective use in classes. This highlights the limited usage of VR technology among students and its dependence on variables such as students’ domains of study and instructors’ teaching methods [72]. The assumption that young adults naturally adopt new technologies quickly may be misleading, considering the complex factors influencing adoption. Therefore, training and support programs could be crucial for successfully onboarding students and instructors. Another notable observation was participants’ comparisons of VR learning environments to video games, which suggests that VR is often viewed more as an engagement tool than an educational one.

Despite these challenges, there was a strong belief among participants that continuous use of the technology would enhance their comfort and proficiency with its interactions, aligning with findings from the literature [73].

5.6.5. Logistics

Logistical concerns were a primary focus of participants’ responses, centering on the challenges of providing VR headsets to all students and setting up a VR-enabled classroom. Participants debated the feasibility of expecting students to own VR headsets, given socio-economic barriers and the limited return on investment for students required to purchase a headset for just a few minutes of VR experience per lecture. They also raised concerns about potential inclusion barriers if the university does not provide the headsets, which could create disparities based on ownership. Conversely, one participant noted that the decreasing costs of VR hardware over the years have made these headsets comparable to the cost of textbooks, suggesting that the expense should not be a significant limitation. However, the logistics of organizing headsets, collaborating with startups, and other elements necessary for establishing a VR classroom remain unclear. Additionally, participants questioned the practicality of integrating VR technology in large class settings.

5.7. Themes of Satisfaction and Dissatisfaction

5.7.1. Dissatisfaction with Unfulfilling Experience

A majority of participants indicated that the existing features were limited and lacked advanced options available in traditional drawing and media presentation tools. For instance, the absence of an “undo” option while drawing in 3D space was mentioned. The responses highly suggested that the effectiveness of a feature depends on the availability of advanced options for customization and how this affects the outcome generated. Participants expressed dissatisfaction with how they interacted with the whiteboard in one of the applications. Participants had to use their controllers to draw on the board, and the precision and control were monumentally hard. We observed participants expressing frustration while attempting to redraw their diagrams. Due to the hurdles associated with unfamiliar or unoptimized interactions, bugs/technical glitches, and lack of advanced features, participants felt disappointed with the outcome of using some of the features. Participants expressed disappointment through emotions, criticizing their drawing skills or giving up on refining their drawing.

5.7.2. Dissatisfaction with Non-Intuitiveness

Misconceptions with respect to the usage of icons and labels present in the interface options misled the participants into choosing the wrong options while performing the tasks, emphasizing the importance of using consistent language and avoiding jargon. Responses under this sub-theme indicated that participants were unsure how to activate certain features in the application. We notice that these responses usually follow the pattern of “How can I do <feature>?” or “Can I do <feature>?” Similarly, we observed several “Woah” and “Not sure” moments when participants performed incorrect interactions because of a lack of consistency or mismatched mental models. For example, accidental teleportations while interacting with virtual objects were common during the experience. A trend we noticed across participants was the mention of the lack of clearly visible indicators of the availability of features or information about the status of the system [74]. For instance, many participants could not see the text they typed in the search bar due to its placement.

5.7.3. Satisfaction with Immersive Experience

Participants compared the virtual environments to real-life formal learning environments present in the university based on classroom design, leading to a sense of immersion.

5.7.4. Satisfaction with Fulfilling Experiences

Participants identified various features in the selected applications relevant to their classroom teaching. However, simply having these features available is not enough to guarantee their use; their perceived relevance often hinges on the effectiveness and completeness of each feature. Participants strongly preferred the system’s 3D capabilities (such as 3D drawing, 3D models, and avatars) over 2D features. From the interactions with the three different applications, we noted preferences for specific types of interactions: they favored manipulating models by grabbing instead of using button selections, typing on a virtual keyboard rather than tapping, and drawing freehand rather than using controllers at a distance. Despite some challenges in their user experience, many participants reported a positive overall experience with the applications. For most participants (56.25%), this study marked their first encounter with HMD-based VR. They entered the study without strong expectations, but during the hands-on experience, they recognized potential opportunities for integrating VR into their teaching practices.

6. Discussion

In this work, we have identified themes of requirements and perceived barriers to adoption, which we hope will guide future design and development efforts for educational VR systems.

For most of our participants, the current study was their first exposure to HMD-related VR experiences, placing them in the phase of familiarization according to the adoption of the technology model [75]. However, as participants watched and experienced these features, they proposed new ideas for the usage of VR in their current course settings, consistent with the belief that exposure helps potential adopters become aware of the technological capabilities [71]. The literature suggests that teachers’ attitudes and beliefs are a strong indicator of the use of technology in the classroom [76,77]. A majority of participants expressed some form of skepticism toward the use of VR within the classroom. Participants were unsure of the applicability and suitability of certain VR features for their current instruction style. Educators questioned the relative advantage of using VR for 2D features compared to existing tools that support advanced features. For P17 Hands-On, one participant stated, “..I think [VR] has potential... I will use it if I could bring in a more immersive experience instead of just replicating what’s in real life. Because the magic of the virtual world is beyond what we see in real life. So that’s the component that I really hope to see come alive. And that freedom to create that. Yeah. So if [VR application] can do that, then I will... most likely bring people in there. Otherwise, I will... have to answer the question, ‘Why are we doing this in VR, but not in person or 2D?’, right?”.

Further, we examined participants’ pedagogical beliefs and their applicability to VR to understand VR’s compatibility with their existing beliefs. Several promising use cases along the lines of virtual field tours, walkthroughs, and shared experiences were proposed by the participants, hinting at ideas of constructivism. Our findings regarding the learning curve, concerns with the input devices, and intuitiveness are consistent with the usability problems identified in the literature for collaborative VR environments [73]. Further, the barriers to adoption identified in the current study, such as logistics, expertise, and effort concerns, are consistent with the challenges influencing technology integration in classrooms [77,78,79].

Our research trajectories and timelines have been affected, as the initial study design had to be restructured multiple times with steps taken to ensure a smooth experience for the participants. Such measures include creating device guides, familiarization tutorials, warm-up applications, regular checks for headsets and applications, and additional time reservations during the sessions. As the study involved participants interacting with VR hardware, we delivered the device to the participants’ locations and followed protocols for sanitization and storage. The remote modality enabled us to conduct the study at the participants’ preferred locations, which were locations that the participants typically use to deliver their classes or attend online meetings, thus adding a certain degree of external validity to our results. As expected in remote studies with the additional involvement of an HMD, the research team and participants were at risk of facing Zoom fatigue and were susceptible to difficulties resolving technical issues. The think-aloud protocol was primarily used as the mode of communication and observation. We tested alternative methods for observing participants’ screens (screencasting). However, our pilot studies demonstrated unpredictable results (bugs) with these methods. Such challenges raise the need for remote research tools for HMD-related VR experiences to support research endeavors.

During the hands-on session, one member of the research team joined the participant in the virtual classroom for two reasons: (1) to simulate the feeling of belonging in an actual classroom setting and (2) to observe the participant’s actions during the study for contextual and troubleshooting purposes. However, participants expressed that it is pertinent to experience the applications with students to understand the feasibility of incorporating VR into their courses, highlighting the necessity for future VR interventions in the current educational setting. A majority of our participants were new to the technology. However, we have identified a wide range of specific use cases and challenges, hinting at the fading novelty effect of VR from instructors’ perspectives upon initial exposure.

To ensure validity in our process, we followed audit trails by maintaining tracking sheets and study scripts. Further, analytical memos were used to keep track of the research process and were helpful during the coding process. However, the analytical memos were used only for the walkthrough videos due to the time constraints and volume of data in the hands-on experience phase. We also transcribed participants’ verbatim responses to reduce the potential bias of researchers while interpreting the data and strived to produce multiple perspectives to ensure authenticity. During familiarization, we added observer comments while transcribing to ensure the presence of context while coding. We followed a template to stay consistent between the coders. We conducted two pilot studies with representative participants, aiding us in the preparation of logistics and study scripts. Further, the researchers ran multiple dry runs adopting the persona of a target participant with minimal VR familiarity to record any potential problems with the applications or the script. Finally, we conducted two rounds of inter-rater reliability assessment for initial coding and codebook generation using the Jaccard index, achieving an IRR of 83% and 90.8%, which is greater than the standard 80%. We encountered technical difficulties with a few of the participants. These include internet disconnection, technical glitches, and problems that could not be reproduced. We could not conduct the session for one participant due to numerous technical issues.

Our findings reveal three primary trends: attitude toward technology, expertise and effort concerns, and technological integration and support. Each of these trends directly addresses our research questions and sub-questions, providing critical insights into the state of VR in educational settings.

• Attitude toward technology directly correlates with RQ2: “How do current commercial collaborative VR applications meet the expectations of educators to conduct teaching activities?” This trend substantially addresses the sub-question regarding instructors’ perceptions of the usability of these applications. Many participants expressed skepticism about the added value of VR over traditional methods, which often reflects a cautious attitude toward adopting new technologies without clear pedagogical benefits.
• Expertise and effort concerns reflect the practical challenges instructors face, which ties back into RQ2’s sub-question about the barriers to VR adoption in classrooms. The significant effort required to develop and implement VR content, coupled with a lack of requisite skills among instructors, highlights critical obstacles that need addressing for VR to become a viable teaching tool.
• Technological integration and support speak to RQ1: “What is the current state of the art in commercial collaborative VR applications for educational activities?” This trend delves into the existing features of VR applications that support instructional activities, and it also aligns with the sub-question concerning software requirements from instructors’ viewpoints. The discussion emphasized the need for VR applications to seamlessly integrate with current pedagogical practices and university systems, such as Google Drive, to enhance their utility and adoption.

The findings have to be seen in light of a few limitations. Firstly, our study included only OSU instructors and did not include participants outside Corvallis (Oregon) due to the need to physically deliver the headsets to participants. We believe such HMD limitations may be alleviated in the future with the affordability and popularity of the headsets. Though the remote nature of the study enabled us to conduct the study in the participants’ natural settings, we believe that participants’ experiences and responses may be influenced by Zoom and VR fatigue. The study included testing a subset of relevant apps and a subset of available features, and we could not test all features that participants indicated an interest in in the pre-exposure questionnaire due to time constraints. Finally, we had participants across various departments, with a majority from the College of Engineering. However, the evaluation of usability and the elicitation of expectations for VR classes may benefit from a larger sample size. These results underline the importance of addressing both the technical capabilities and the user-centric aspects of VR applications to better align with educators’ needs and expectations. By focusing on these areas, VR developers can enhance the relevance and effectiveness of their educational tools, fostering more robust adoption and utilization in academic settings.

7. Conclusions

This study has provided critical insights into the utilization and perception of commercial collaborative VR applications in higher education, highlighting three main areas: attitude toward technology, expertise and effort concerns, and technological integration and support. Each of these areas reflects significant implications for the adoption and effectiveness of VR in educational settings. Firstly, the cautious attitude toward technology among instructors suggests a need for VR developers and educational institutions to actively demonstrate the pedagogical benefits of VR technologies. Recommendation: Educational leaders and technology providers should collaborate to conduct pilot projects that showcase the educational gains from VR applications and provide empirical evidence of their benefits. Workshops and seminars can also be employed to educate instructors about the potential and practical usage of VR in teaching. Secondly, the expertise and effort concerns highlighted by participants underscore the barriers related to the technical challenges of creating and implementing VR content. Recommendation: Institutions should consider providing technical support and training for instructors to lower these barriers. Additionally, VR platforms should be developed with user-friendly interfaces that require minimal technical knowledge to encourage wider use by educators who may not have advanced computing skills. Thirdly, the need for technological integration and support indicates that VR applications must be compatible and integrative with existing digital tools used in education. Recommendation: VR developers should focus on enhancing the interoperability of VR systems with widely used educational platforms and tools, such as learning management systems and content creation software. This integration will facilitate a smoother transition for educators moving into VR-based teaching. Lastly, the feedback from some potential participants in the study underscores the necessity for VR developers and educators to consider physiological impacts when designing and implementing VR in curriculums. Recommendation: Future research should focus on developing strategies to minimize such adverse effects and to make VR experiences more inclusive. Addressing these concerns not only broadens the usability of VR technologies but also ensures more equitable access to innovative educational tools.

In conclusion, while VR holds considerable promise for enhancing educational experiences, its successful implementation hinges on addressing these identified challenges. By focusing on clear, actionable strategies to improve acceptance and utility, the path to integrating VR into mainstream education can be significantly smoothed. As this technology continues to evolve, ongoing research and adaptation to educators’ feedback will be crucial in realizing the full potential of VR in education.