Teaching Methodology for Understanding Virtual Reality and Application Development in Engineering Major

Abstract

This study proposed a virtual reality (VR) course that addresses the overall understanding and application of VR technology. After investigating previous studies, we found that two technologies must be applied to design a VR course that fits the latest trends. One is hardware technology dealing with the technical background, while the other is software utilization and development using the merits of VR technology. To accommodate these needs, we designed a VR course consisting of three steps: VR-related theory, TA-led content creation training, and team projects. Through this course, students will improve their ability to develop applications that apply to their research fields after studying the technical background and courses of VR. We conducted a semester-long study with nine students to verify the proposed method and then evaluated them through an in-depth interview and a questionnaire with a five-point Likert scale consisting of nine items. Considering this feedback, we have added several steps to improve the educational effect among students.

1. Introduction

With the rapid development of emerging technologies such as augmented reality (AR) and virtual reality (VR), technology-enhanced learning (TEL) has been attracting increasing attention from the education community in an attempt to enrich the traditional learning experience with an interactive and multimodal component [1]. Recent studies indicate that the educational environment is changing to respond to social, technological, and environmental changes, and it can enhance students’ spatial and cognitive learning effects [2,3]. In the field of TEL in science, technology, engineering, and mathematics (STEM) education, studies are underway to design applications, appropriate technical devices [4], and appropriate learning designs considering student characteristics [5], methodological characteristics [6], and pedagogical uses across various age groups [7,8]. Among them, engineering education is undergoing the most dynamic changes due to the rapid introduction of new technologies and the adoption of modern education methods. [9]. For this reason, there is a demand for course redesign to enhance students’ potential to succeed in engineering education [10]. Course design in fast-growing fields, such as Industry 4.0 technology, should be focused on mastering the major’s overall technical framework and specific technical details [11]. Various advanced learning techniques are emerging to enhance learning, enhance the hands-on experience, and increase interest in engineering in science and engineering technology.

In some research, project-based learning (PBL) is mainly used as an educational method to promote student participation [12,13,14,15]. PBL is a student-centered method, a widely used educational strategy in engineering education that fosters students’ practical problem-solving, design, and teamwork skills as they take on the role of active explorers [16,17]. In addition, PBL is the preferred educational strategy in engineering education to encourage the development of various skills and competencies essential to the profession, integrating both theoretical and practical content and applying them to practical problems to achieve high-level learning goals and solve real-world problems [18,19,20,21].

Unlike problem-based learning, which focuses on the application of knowledge, PBL has a multidisciplinary orientation centered on self-initiative and collaboration [22,23]. PBL successfully approaches engineering education courses by helping students achieve advanced learning goals and handle real-world troubleshooting activities. In addition, PBL is the preferred educational strategy to encourage students to practice self-determination learning while simultaneously encouraging them to develop various skills and abilities essential to their profession in engineering education [23,24,25].

However, while PBL has clear advantages by giving students autonomy in determining the nature and scope of the projects they wish to undertake, the lack of expert guidance and clear direction can lead to confusion, frustration, and unachieved goals [26]. For this reason, the instructor is required to play the facilitator rather than a teacher who delivers the content in a PBL-based class, and through this, students cultivate higher-order thinking skills [27,28].

In this paper, we present a course about VR technology, which is mainly used as an integrated technology in STEM education. With the development of computing systems (e.g., VR hardware), projections and interactions of visual elements, and various operating environments, it is rapidly emerging as the core technology of Industry 4.0. Utilizing such VR-related technical fields requires developing practical applications according to the type of hardware and platform used, which leads to the technical background and process of VR. Afterward, we collected improvement points for the designed class through in-depth interviews.

In the next section, we review the articles on course design with VR technology. Section 3 describes the course design in this paper to create pedagogically meaningful activities. In Section 4, the outcomes and feedback of the course using the proposed method are discussed. Finally, in Section 5, we outline limitations and directions for future research and conclude our paper in Section 6.

2. Literature

Engineering-related technologies are evolving rapidly. This situation has brought about significant changes in STEM education, which is actively embracing new technologies. VR is a typical technology that has led to such changes [29]. Even before VR technology became widespread, there was a basis for using this technology in engineering education [30]. VR technology can motivate students to experience a virtual world with a high degree of freedom without space–time constraints [31]. In addition, classes that combine this technology will enable students to experience new technology naturally, improve their familiarity with the technologies, and increase their interest in engineering [32].

Recent studies in the STEM field consider applications, technical devices, and suitable content and learning through the VE, considering student characteristics [33], methodological characteristics [34], and educational uses [35] at various ages from K–12 to higher education [36,37]. Research to advance the course design is conducted in the context of various majors, such as computer science [38], engineering [39,40], and medicine [41]. Moreover, we found a report about the positive effects of education through VR regardless of the age group. VR hardware technology has evolved with a cave automatic virtual environment (CAVE), which projects 3D images onto an indoor display, and a head-mounted display (HMD)-based computing system, which utilizes an optical system and display to be worn on the user’s head. VR software technology has evolved into computer-graphics-based 3D image projection technology and virtual object modeling technology.

Alhalabi investigated the appropriate environment for improving student achievement when using VR in engineering education [42]. This study compared CAVE and VR HMD with tracking systems that provide three degrees of freedom (DoF), VR HMD without tracking systems, and traditional teaching methods. As a result, all courses that utilize VR received higher scores on the exam compared to the existing course method, of which the VR HMD environment including the tracking system received the highest score.

Jensen and Konradsen reviewed studies in which courses with VR HMD influenced learner attitudes [43]. This study examined learner attitudes toward HMDs, which were, in most cases, based on the self-reported opinion of the study participants. This study examined whether the experience helps to learn and is interesting. Across all studies, the study participants were very positive toward both of these aspects. The purpose of most VR-related courses is to develop content that suits the students’ majors.

Wang designed the process of applying virtual reality to modular education in the context of civil engineering education [44]. This study used 3ds Max as a 3D modeling tool and practiced software composed of Quest3D for applying and coding physics engines. This paper presented modeling for building information modeling (BIM) education and logic and code to utilize it but did not verify it by testing it with students.

Häfner et al. designed a hands-on course focused on virtual reality learning to develop interdisciplinary industrial projects to solve industry-centric challenges in engineering education [45]. This study constructed a high-end VE that enables distributed stereoscopic visualization through CAVE consisting of three sides. This course adopted the definition of technology consisting of four weeks, CAD software such as modeling tools Blender and CATIA, and software using VR technologies such as 3DVIA Virtools and PolyVR. The next phase was to proceed with a team project involving small groups for a given task for nine weeks. This study confirmed its course goals by quoting student opinions and lecture evaluations written in the project documents. This paper emphasizes the importance of updating lectures and practices every semester for the rapid development of new hardware devices and software and application for VR education.

Vergara et al. have spent many years researching the software and development sequence for building virtual reality learning environments (VRLEs) [28,32,46]. This study defined a general-purpose content design method that constitutes VRLE modified to modernize Pantelidis’s method [47]. This study confirmed that it has reached its goal and made the necessary corrections. This process provides guidelines in many VR education systems, and this method is modified and used in this paper.

Most of the previous research focused on learning methods to understand 3D representations based on computer graphics theory or to implement 3D modeling based on architectural theory. However, these studies exclude theories about hardware and do not present it as an issue. For students to fully understand and apply VR technology, it is necessary to include a course that understands the principles and courses of an HMD creating an IVE. For this reason, VR courses should be designed to understand the technical background and process. However, it is common for most classes to use VR as a single application or environment to harness the training potential of immersive technology [48,49]. Some classes also teach VR-related hardware or software perspectives. However, most courses are biased toward either hardware or software [50]. For this reason, existing VR-related course methods do not meet current educational and industrial requirements [51]. Therefore, we proceed with a course design that can develop a holistic understanding and application ability to face VR technology in this paper.

3. Methodology

We proposed the course design, including VR-related theory, teaching assistant (TA)-led content creation training acting as a facilitator, and projects and virtual exhibition. First, we presented an introduction to VR technology to help students understand VR-related theories and a visual recognition process that optically interprets the process of transferring images from hardware to the user’s eyes. Second, we presented software-based technologies required for content creation to operate within a virtual environment (VE). Finally, the students formed a team to conduct a team project based on the theory and practice they had previously learned and provided an environment where class members could freely share and provide feedback on each other’s content. After the class was over, to verify the validity of this class design, evaluation was conducted through a questionnaire composed of a five-point Likert scale.

The total amount of time is 60 h during the semester (15 weeks). The prerequisites for this course include display engineering to technically understand VR hardware and OOP to understand software for developing programs. However, since there are no restrictions on attendance depending on the elected subjects, essential content for students to proceed with the team project was provided in the theoretical and practical training process to proceed with the course. All students who attend the course are graduate students and meet the minimum prerequisites for advancing a project on a particular subject through the majors they take in their undergraduate studies and the comprehensive design tasks over two semesters for graduation. The VR course proposed was designed in three steps according to Bloom’s taxonomy, which has six levels. Details about each step are provided in the next section.

3.1. Step 1: Introductory Theory and Method

Step 1 introduces the theories and methods for developing an understanding of VR from a hardware perspective in the course, which progresses for 3 weeks (20%) of the total 15-week process. Figure 1 shows the sequence of step 1 objectives in the VR course.

In Knowledge 1, the instructor explains the hardware-related theory of VR that composes this course. In addition, the instructor explains step-by-step the process of progressing the team project through software learning through the practical course conducted by the TA. Knowledge 2 introduces the basic technology of VR. In Knowledge 2, the instructor introduces the definition of VR technology and various actual cases using it and helps students understand the advantages and disadvantages of VR technology. It also helps students understand the ripple effects of these technologies and guides them to exercise their imagination freely. In the Understand process, the instructor teaches theories that are related to hardware. The goal of this course is to create immersive content that considers the characteristics of the VE. It considers that the content creation of virtual reality is an environment that covers the entire field of view of the user with a VE, unlike an environment that uses a digital device such as a computer or mobile environment. The image that the user sees in the virtual reality environment is determined by the characteristics of the VR device’s performance and the person’s visual effect. Therefore, in the process of Understand 1, the instructor teaches visual recognition according to the performance of the VR device, and in Understand 2, the instructor teaches the process of the image of the VR device reaching the user’s eyes. Through this process, students understand the VR content delivered from the VR device through the 3D image distribution process. Additionally, it guides students to propose ideas based on their understanding of the characteristics that occur according to the performance of VR devices in the project design process.

3.2. Step 2: Content Creation Training by Teaching Assistant

In step 2, the TA leads a practical course on the primary technologies required for VR content creation while the team project is in progress from a software point of view. Before the course started, the instructor provided students with a personalized VR device (Oculus Quest 2) for practice and project progress. The students were guided to fully understand the VR environment through device experience and to take the initiative based on their experience in advancing practical training and projects in the future. In this paper, we configured the components needed to create VR content, as shown in Figure 2.

The elements that develop the content are divided into external elements that can intervene in the application, various data that can be used within the application, and software that turns it into the content. The external elements consist of an infrared (IR) tracker to distinguish between position and behavior, a user’s hand and controller to implement the actions they want, and a 3D object that can be used in addition to the application. The data available in the application are the movement and position data that can record the user’s movement so that the position of the virtual object existing in the virtual reality environment can be manipulated and the controller so that the user’s movement can be transmitted to the application. It consists of generated input data and gestures. In this paper, we used Unity3D software to create the VR application. This software can implement logic in an application through C # script-based coding. It also has a built-in rendering engine that can give a sense of reality to other modeling software or 3D object models created with Unity3D. It also has the advantage of taking advantage of various virtual objects and features through the 3D object asset store. In this course, we defined two main technologies for content creation. One is a technology that implements a VE by utilizing virtual objects and the surrounding background; the other is to implement dialogue with virtual objects via gestures between the controller and the user to perform various actions in the content.

Step 2 consists of three detailed stages, which are equivalent to five weeks (33%) of the total 15-week process. Figure 3 shows the sequence of step 2 objectives in the VR course.

Apply 1 introduces Unity 3D-based application development tools for creating content. In the process, students learn basic C # scripting examples to give them the background knowledge needed to implement the skills needed to create content and to give them what they lack. In Apply 2, students learn to implement a VE by leveraging virtual objects and surrounding backgrounds. During this process, students are guided to create a 3D model and VE that incorporate their ideas and to bring external resources into their VE. Students learn to build their target VE freely. In addition, students apply the visual perception of people in the VR environment learned in the process of Understand, and they learn through their own experience that the visual changes of virtual objects and environments depending on the usage environment are different. In Apply 3, students learn techniques for implementing various actions in content by implementing interactions with virtual objects. In this course, the design difficulty is relatively low, and the emphasis is on operation with a controller that does not require additional technology. In the case of gesture recognition technologies, we provided students with an example to give an explanatory course. If students wanted to take advantage of this, we provided additional examples to study individually. After the training, students learn to acquire the know-how to create straightforward content together through the training of the TA and apply it to content that creates interactions suitable for the idea proposed by the student in the team project. After completing steps 1 and 2, we evaluated whether the students understood the theory and practice in Analyze 1. Based on this evaluation, the highest-ranking students with a high degree of understanding of the previous learning content were selected. We chose them as leaders who can lead the team project and guide them to have equal abilities for each team.

3.3. Step 3: Team Project-Based Learning

In academia, PBL is mainly composed of team-based learning (TBL) because PBL includes learning content for information transmission and skills acquisition and acquiring soft skills such as teamwork, leadership, and troubleshooting [52]. TBL is known to be a successful educational design pattern and was initially proposed to encourage participation in large-scale collaborative face-to-face activities with a higher student participation rate than existing lectures [53].

TBL has several key characteristics [54]. First, TBL requires a strategically organized team [55]. Second, it is an activity that promotes as much critical thinking as collaborative work does. Finally, it involves a complementary process through project evaluation and colleague feedback. This method improves student awareness [56], enhances students’ understanding of the subject they are studying, and helps improve their grades [57,58].

To improve the educational effect using TBL, four matters must be considered. First, assignments should always be designed around a significant problem to students. Second, all the students in the class should be working on the same problem. Third, students should be required to make a specific choice. Finally, groups should simultaneously report their choices [59]. In this paper, PBL verified through many cases of engineering education is included in the course design to construct a student-centered course. In addition, we included a questionnaire addressing these four points to evaluate the team project performance during the proposed VR course.

Step 3 involves proceeding with a team project based on what the student has learned through theory and practice, which is equivalent to seven weeks (47%) of the 15-week process. Figure 4 shows the sequence of step 3 objectives in the VR course.

Figure 5 illustrates a flowchart that proceeds from the derivation of ideas to the creation and verification of content. Students confirm the detailed process for content creation through the flowchart and construct a team project based on the theory learned and the practical content. In Evaluate 1, students suggest ideas for team projects and the benefits of implementing them in a VR environment. In proposing ideas, students can deal with the theory directly, practical content and hardware learned earlier, and compose free ideas based on what they have experienced. In particular, they can define the goals of providing users with the characteristics of the VR environment through immersive content, explain the benefits that the content has in the VR environment, and determine whether they will proceed with an accurate understanding. In Evaluate 2, students design the technical foundation to implement their ideas. In this course, students define the key technologies required for content design and then determine the software and surrounding environment required to implement them and additional development materials. In addition, they ensure that all the environments needed to take advantage of the goals and environments that were initially built are configured.

Finally, students develop and internally validate the project from Create 1 to 3. During this process, students make a three-dimensional model of a VE or object to interact with various virtual objects in the VE by using motion data such as controllers, gestures, and position data. Students import and experience the content they create within their devices. Moreover, they add and modify the features needed to reach their initially set goals based on feedback. Students can take the initiative in learning through team projects and develop their problem-solving abilities. In addition, the instructor guides students to present the possibility of fusion with other fields based on the course’s content and the content learned through the project.

4. Results

To verify the course design proposed in this paper, we gave a lecture in the spring semester of 2021 to graduate students majoring in the engineering series of Pohang University of Science and Technology (POSTECH).

Table 1. Student list and identification information.

Student	A	B	C	D	E	F	G	H	I
Prerequisite	O	O	O	O	O	X	X	X	X
Experience	O	O	O	O	O	X	O	O	X

summarizes the identification information of students before attending courses regarding whether they meet the prerequisites and their VR experience.

This course was conducted via distance learning utilizing an online platform (Webex) during the COVID-19 pandemic. In the case of team projects, students proceeded with the project with constant coordination without environmental restrictions. The team project involved nine students forming three teams, including students who did not meet one or more prerequisites.

After the team project was completed, each team submitted their results, as shown in Figure 6. After all the courses were over, we conducted a questionnaire through Google Forms, and the in-depth interview was a one-on-one meeting.

4.1. Questionnaires

Two evaluation questionnaires were constructed to verify the validity of the proposed course design. All items are evaluated on a five-point Likert scale. For the first evaluation questionnaire, each objective of the three stages of the course structure is produced as a questionnaire, and the course structure and contents are evaluated.

In Theory and Method (TM), based on the theoretical content learned in step 1, we evaluate whether students can understand the overall understanding and principle of VR technology and explain various fields of application utilizing related technologies. This evaluation determines whether the theoretical part is explained to the student. Then, we verify that students will secure the ability to plan and organize team projects.

In Content Creation Training (CCT), based on the practical content learned in step 2, we evaluate whether students can create a VE and whether interactions can be realized. This evaluation verifies that the course content and duration were sufficient for students to acquire the skills necessary to proceed with the project through practical training.

In TBL, evaluation is divided into three detailed classifications based on the contents of the step 3 course.

In Idea and Plan (IP), we evaluate students’ ability to set ideas and goals for team projects and design the structure and key technical skills needed to implement the content. This evaluation verifies whether the course content and duration were sufficiently provided so that students could take the lead in designing content that leverages the VR environment by deriving team members’ opinions.

In Development (DEV), students implement interactions with virtual objects and environment implementations that they have learned through the training content, and we evaluate whether they have been applied to the project. We then evaluate whether the student can explain the script structure designed to compose the content. Finally, we verify through the project’s completeness that students were provided a sufficient environment to reach the goals set at the planning time.

In Team-based Learning (TBL), students coordinate their opinions through sufficient communication and evaluate whether they have designed a project by sharing individual tasks to achieve their goals. Then, from the beginning to the completion of the team project, we check whether all the members have participated. The design of this course provided students with motivation and a sufficient educational environment to proceed with the team project.

The second evaluation questionnaire investigates the emotional effects of the course on students, separated into Engagement and Motivation, and the educational effects, separated into Cognitive Benefits (COG) and Perceived Learning Effectiveness [60].

In Engagement (ENG), we evaluate students’ entry barriers and preferences when experiencing new environments and methods [61]. The detailed items consist of Attraction to overcome the barriers of preference, Time Investment with positive recognition, and Usability for ease of use [62,63].

In Motivation (MOT), we evaluate key psychological factors that are actively associated with learning outcomes and experiences and show the impact of student synchronization on learning effectiveness in theory and practice and in a PBL environment [35,64,65].

Research manuscripts reporting large datasets that are deposited in a publicly available database should specify where the data have been deposited and provide the relevant accession number [66,67]. If the accession numbers have not yet been obtained at the time of submission, please state that they will be provided during review. They must be provided prior to publication.

In Perceived Learning Effectiveness (PLE), we evaluate whether students can actively accept new information as a positive entity in the course and actively compose new knowledge in conjunction with their knowledge [68,69].

Table 2. Questionnaire items for evaluating proposed VR learning module.

Category	Questionnaire
Theory and Method (TM)	(1) I can explain the technology and principles of VR (2) I can explain the trend in the field to which VR technology is applied (3) I can explain VR devices and applied technologies
Content Creation Training (CCT)	(1) I can build an environment (PC, VR HMD) for developing VR content (2) I can implement various contents such as virtual object generation in a VR environment (3) I can implement various actions by animating virtual objects in a VR environment (4) I can program to interact with content via operating devices such as controllers in a VR environment
Idea and Plan (IP)	(1) I clearly set the ideas and goals of projects (2) I structured the devices and technologies used in project design well
Development (DEV)	(1) I implemented actions and animations to improve the sense of immersion of content in a VR environment (2) I implemented it to interact with virtual objects through an operating device such as a controller within a VR environment (3) I can explain the content configuration and process and the coding for implementing it (4) The completeness of my project is consistent with the level at the time of project planning
Team-based Learning (TBL)	(1) I established a project design plan through sufficient communication with the members during the team project (2) I shared and solved individual tasks to solve the same problem during the team project (3) I selected the necessary elemental skills during the project after sufficient communication with the members (4) All members participated in the process of reporting the project results
Engagement (ENG)	(1) This class captured my attention (2) I liked this class because it was novel (3) The topic of this class made me want to find out more about it (4) The space in which this class took place was interesting (5) I was able to follow the flow of this class easily (6) I felt confident since I learnt how to use VR technology through this
Motivation (MOT)	(1) I enjoyed this class very much (2) Learning through this class was fun (3) I would describe this class as very interesting (4) I was satisfied with my performance in this class (5) I did try very hard while learning with this class (6) I could learn much through this class
Cognitive Benefits (COG)	(1) This class makes comprehension easier (2) This class helps me to better apply what was learned (3) This class helps me to better analyze problems (4) This class helps me to have a better overview of the content learned
Perceived Learning Effectiveness (PLE)	(1) Through this class, I became more interested to learn about VR topics (2) Through this class, I learned a lot of factual information in the topics (3) Through this class, I gained a good understanding of the basic concepts (4) Through this class, I learned to identify the main and important VR issues of the topics (5) Through this class, I became interested and stimulated to learn more (6) I was able to summarize and conclude what I learned (7) This class was meaningful (8) I can apply what I learned in real context

shows 41 survey questions divided into nine categories.

4.2. Discussion

Figure 7 and

Table 3. Mean difference of the questionnaires between P group and NP group.

Category	TM	CCT	IP	DEV	TBL	ENG	MOT	COG	PLE
Prerequisites	4.40	4.60	4.40	4.25	4.40	4.40	4.37	4.20	4.40
Non-Prerequisites	4.25	3.33	4.38	3.31	3.88	3.70	3.50	3.88	3.78
Mean	4.33	4.06	4.39	3.83	4.17	4.11	3.98	4.06	4.13
Standard Deviation	0.38	0.80	0.59	0.87	0.69	0.65	0.71	0.51	0.57
Mean Difference	0.15	1.27	0.03	0.94	0.53	0.70	0.87	0.33	0.62

show the difference between the survey results for all students and the standard deviations and the average values of the students who met the prerequisites (P group) and those who did not meet the prerequisites (NP group). Students who showed a significant difference from the average value for each item or gave a rating of 3 points or less were selected as a sample to analyze why the difference was shown in the in-depth interview.

From the results of Theory and Method (M = 4.33, SD = 0.38), it seems that this course provided students with a complete understanding of VR technology through hardware-based theory and case studies, which are the goals of the course.

The results of Content Creation Training (M = 4.06, SD = 0.80) show the largest mean difference of the comparison group among all the questionnaire categories (MD = 1.27). Students learned a key technology through examples to ensure practice and project progress. In the case of the P group, it can be confirmed that there were no other problems in advancing the training. However, the survey results showed insufficient training time for the NP group or a lack of course content spanning from Apply 1 to 3. From this result, we confirmed that the students of the NP group also need to be supplemented by redesigning the courses to facilitate the practical training.

In the case of Idea and Plan (M = 4.39, SD = 0.59), the mean difference of the comparison group is the smallest (MD =0.03). Through this result, we confirmed that all students could actively participate in the project planning stage of our method. In addition, after gaining a thorough understanding of the VR environment in the process of theory, practice, and experience, we evaluated the project and confirmed that students could design the project by classifying the features, strengths, and weaknesses of VR.

In the case of Development (M = 3.83, SD = 0.8), it showed the lowest mean, and the NP group also had the lowest mean. In developing the project, students need to design a broader structure that includes techniques that are not covered during the practice. In particular, we found that students had difficulty designing the project, as the average of the questions assessing the completeness of the project was found to be the lowest. To solve this problem, we confirmed that TA intervention is needed to improve the completeness of the project.

In the case of Engagement (M = 4.11, SD = 0.65), we confirmed that this course is a novel and interesting method for learning the new VR environment and related technologies, and the course design is attractive. However, although the item showed a high score in the P group, it gave a relatively low score in the NP group.

Moreover, in the case of the NP group, there was an evaluation that they could not easily follow the flow of the course, and in particular, they gave a low score on the item that evaluates their self-confidence that they can use VR technology well.

Motivation (M = 3.98, SD = 0.71) has the largest mean difference (MD = 0.87) in the comparison group among the four items for verifying the educational effect. In the case of the P group, they answered that they were interested in proceeding with the team project by combining previous theories, practical training content, and background knowledge from the prerequisites. As a result, they said they were more motivated to work harder, learned a lot, and were happy with their results. However, the NP group replied that it could not provide sufficient motivation because of the problem of not easily following the course flow mentioned in Engagement. This result showed that their satisfaction with their learning outcomes and experiences was slightly lower than that of the P Group.

In the case of Cognitive Benefits (M = 4.06, SD = 0.51), both groups met the expectations set by the course for a comprehensive understanding and analysis of VR technology. However, in the case of the NP group, the result showed that they experienced difficulties until applying what they learned. From this result, we confirmed that the course redesign should include a learning process that can combine the background knowledge that the student has with the knowledge learned while taking this course.

In the case of Perceived Learning Effectiveness (M = 4.13, SD = 0.57), the P group showed positive educational effects for students overall. In particular, they replied that this course was meaningful, expressed confidence in identifying and summarizing the main content of what they had learned, and were interested and inspired to learn more. This result shows that students can actively accept new information and organize their knowledge. In the case of the NP group, lower scores and even distribution results were shown for the overall courses compared to the comparative group. In particular, they showed confidence that they could put together the course content. This result confirmed that they had difficulties in the application process but they could actively construct the learned knowledge. In short, the process of introducing the theory and overall skills was well communicated to all students, but the practical and team-project courses seemed to be a bit lacking in practice time and content for the NP group.

4.3. In-Depth Interview

To receive student feedback on the course designed in this paper, we conducted an in-depth interview with the students selected.

Table 4. Excerpts from in-depth interviews for students (TM = step 1: introductory theory and method; CCT = step 2: content creation training by teaching assistant; and IP, DEV, and TBL = step 3: team project-based learning).

Category	Questionnaire
Theory and Method (TM)	“The trends of VR applications were somewhat lacking in explanation in class, which made it difficult to expand the scope of ideas in establishing project plans.” (Student E)
Content Creation Training (CCT)	“The VR environment should be expressed in three dimensions in perspective space. In order to express the image of the plane in three dimensions, design through graphical matrix calculation was required, which took considerable time to understand. It seems that there was a lack of understanding of this because he did not take the prerequisite course. Furthermore, due to the short class time, there is a limited amount of content that can be learned. In addition to class hours, an additional study was required, and it seems that more diverse practice is needed than the next time this class is conducted.” (Student G, F, I)
Idea and Plan (IP)	“In developing the project, various technologies were to be implemented in planning. However, it was difficult to clarify the technologies and codes required in the pipeline design process. Because they were inexperienced in project-based classes and lacked prior knowledge of content design.” (Student B)
Development (DEV)	“In order to improve the sense of immersion in content, we tried to implement various actions and interactions. However, there was much additional knowledge to learn in the process of designing content. In addition, it was difficult not only to learn this content but also to solve the problems that arise in the process of applying it to the projects we develop with the efforts of our team members. Although efforts were made to improve the completeness of the project through communication with team members, the goal of improving the sense of immersion of content through various interactions was not fully achieved.” (Student A, B, F) “In the project planning stage, designing the overall structure of the project based on the contents of VR-related technologies and the contents learned in the actual practice stage was not difficult. However, in implementing this as an actual program, prior knowledge required to implement the content was quite extensive. Not all of these contents were learned in the class, and this part was quite burdensome for students working on the project. In order to increase the sense of immersion in content, better quality 3D modeling was needed. However, these tasks required advanced techniques accompanied by a lot of time and expertise. Since this was not covered within the class, learning was needed through the searching internet and related books.” (Student G, H, I)
Team-based Learning (TBL)	“In order to develop content during the project, we tried to communicate frequently with our team members and share roles. However, not all team members had the same content design ability in this process because they did not take courses in the same environment, such as professional courses or related majors. In order to complete the project safely, much weight was reflected in some students with slightly superior development skills, which was a burden to some extent.” (Student C, H) “Programming-based projects seemed to have the advantage of proceeding from a distance project from each other because they were conducted through PCs, but when it comes to project design through smooth communication, it felt easier to proceed with the project face-to-face.” (Student G) “Each team member had different abilities. It was a project that required much prior knowledge based on advanced subjects and related majors, not just through what was learned in class. This process does not seem to have helped the team members much due to their relatively insufficient prior knowledge compared to other team members.” (Student G, I)

is an excerpt of the in-depth interview for the evaluation by the third stage of the course.

In the case of Idea and Plan, one student said that there were many technologies that the student wanted to implement when planning the project, but it was difficult to clearly explain the additional technologies and code required in the pipeline design process for content creation. To improve this, the student replied that the learning of various cases in the process of explaining the theory had to be longer.

In the case of Development, many students answered because of the low mean. Some students tried to implement various actions and interactions while the project was in progress to improve the immersion of the content. However, it was necessary to utilize the knowledge that was not covered in the training content to implement these functions. For this reason, it was difficult for students to solve problems that occurred during content development with the efforts of team members alone, and they had to solve the problems themselves by utilizing external learning materials. This problem occurred not only in programming but also in virtual object modeling to improve visual immersion. As a result, the students answered that they could not completely achieve the goals set in project planning. To improve this, we need to add a process to teach students useful skills while the project is in progress. We also need to add a process to identify the ideas that students propose during project planning and teach them the skills they need to develop.

In the case of TBL, many students answered about the division of roles. Some students answered that various kinds of background knowledge, such as prerequisites and related majors, caused various problems. As a result, some students took on too many roles in the project and felt burdened, while others took on relatively limited roles and were reluctant to participate. In addition, some students said that proceeding with the project in a non-face-to-face manner sometimes hindered smooth communication, and there were difficulties in adjusting the work contents. To improve this, the practical course should be structured step-by-step during the course redesign to reduce the gap in the students’ learning content. Another option is to clarify each other’s strengths at the project planning stage and set achievable goals and concentrate on them. Overall, through in-depth interviews, we confirmed the need for course redesign to balance students’ experience in practical and project courses. In particular, this problem is further exacerbated by students’ background knowledge, such as prerequisites and related majors. It is a factor that can affect student learning satisfaction and outcomes. To improve this, it is necessary to redesign the course by using a step-by-step structure in the practical training process to help the students.

Table 5. Excerpts from in-depth interviews of students (ENG and MOT = Emotional effects; COG and PLE = Educational effects).

Category	Questionnaire
Engagement (ENG)	“There were many difficulties in following the practice. As a result, my contribution to the project was relatively small compared to the same team members. For students with relatively little prior knowledge, such as prerequisite and related majors, this class was relatively burdensome in conducting classes based on practice and actual design.” (Student F) “Since it was conducted as a project-based class, a long time seems to have been allocated to the project design process. However, the project design process took more time to learn personally through searching internet and related books.” (Student B, H) “Since I took prerequisite and the research topics are similar, motivation and novelty for learning did not come much.” (Student C) “Due to the lack of prior knowledge, classes on VR-related and other technologies and content development were challenging to follow.” (Student G)
Motivation (MOT)	“Initially, satisfaction with the class was high until progressing the overall understanding of VR technology and conducting practiced-based learning. However, in carrying out the project, too much burden was placed on students. Designing a project took a considerable amount of time to search for and learn related content rather than what was learned in class. It was also difficult to apply new things directly to the project. As this period took about two months, fewer and fewer parts motivated students, and I felt that students were exhausted and less motivated.” (Student B, C, F, G, H) “I felt relatively tired and antipathetic to the class in this process.” (Student C) I gradually suffered from following the practice stage, satisfaction with the class was partially decreasing, and fatigue was further accumulated due to the too much project period. (Student F) “Confidence began to decline from the process of conducting practice. In taking this class, there were many parts that I had to try on my own, and it was difficult to keep up with the overall flow of the class, and I think this affected motivation. As a result of relatively less motivation than other students before the start of the project, I think the long project period felt rather dull.” (Student G)
Cognitive Benefits (COG)	“In developing the project, various technologies were to be implemented in planning. However, it was difficult to clarify the technologies and codes required in the pipeline design process. Because they were inexperienced in project-based classes and lacked prior knowledge of content design.” (Student B)
Perceived Learning Effectiveness (PLE)	“In order to improve the sense of immersion in content, we tried to implement various actions and interactions. However, there was much additional knowledge to learn in the process of designing content. In addition, it was difficult not only to learn this content but also to solve the problems that arise in the process of applying it to the projects we develop with the efforts of our team members. Although efforts were made to improve the completeness of the project through communication with team members, the goal of improving the sense of immersion of content through various interactions was not fully achieved.” (Student A, B, F) “In the project planning stage, designing the overall structure of the project based on the contents of VR-related technologies and the contents learned in the actual practice stage was not difficult. However, in implementing this as an actual program, prior knowledge required to implement the content was quite extensive. Not all of these contents were learned in the class, and this part was quite burdensome for students working on the project. In order to increase the sense of immersion in content, better quality 3D modeling was needed. However, these tasks required advanced techniques accompanied by a lot of time and expertise. Since this was not covered within the class, learning was needed through the searching internet and related books.” (Student G, H, I)

shows excerpts from an interview about some of the effects of the course design and methods on students.

In the case of Engagement, there were differences in some students’ opinions. In particular, students who did not meet the prerequisites answered that they lacked confidence in the content of the courses due to the difficulty caused by the disparity. Students who experienced such problems experienced a phenomenon in which their interest in courses fell. In other opinions, many of the contents were learned by themselves using external learning materials during the long project period, and the contents of the lectures were not very important.

In the case of Motivation, there was a common response. They were interested in and enjoyed learning new skills through experience in the early courses, but the long project period put a heavy burden on them. As a result, the students felt tired and replied that their motivation was relatively low compared to the early stage. In particular, students answered that they experienced many difficulties due to the trial and error that occurred by solving the problems themselves through external learning materials rather than interacting with them through the course. As a result, the students replied that being overburdened could cause fatigue and antipathy toward the course. Emotional effects are essential in evoking positive feelings for and interest in new learning methods and content and providing motivation. The answers from the students mentioned above confirmed that the period of practical training and projects needs to be partially adjusted during the course redesign process.

In the case of Cognitive benefits, they responded that there were difficulties in applying and analyzing learned content. Some students must have a well-founded theoretical amount of learning to apply VR-related technologies and must further address diverse and detailed application cases. They also answered that self-directed learning by external learning materials was mainly conducted, not the knowledge transfer that was advanced during the course.

In the case of Perceived Learning Effectiveness, some students answered that they had difficulty understanding the content of the course, identifying the problem, and applying it. They replied that the conceptual content they were provided in the course was too limited. They replied that the course redesigned to solve this had to deal with more profound and diverse theories and information. Moreover, some other students answered that their interest in the course decreased because they did not easily follow the flow of the course or the dependence on external learning materials increased. For educational effects, we confirmed changes in skills and knowledge levels before and after learning and evaluated whether a better understanding and application of what was learned was possible. We confirm that course redesign is needed to facilitate the overall understanding and analysis of skills, including the process of introducing more diverse and detailed skills through student evaluations.

4.4. Course Redesign

We checked some compliments to the existing course design through questionnaire evaluations and in-depth interviews. As shown in Figure 8, we have redesigned the course to reflect the students’ feedback.

The modified step 1 lasts three weeks as before. In step 1, in parallel with theory courses and student-led actual case analysis, we guide students to improve their understanding of the prior knowledge required before practical training and design. With the added Knowledge 3, after an existing theory step, students select the desired VR content and take the lead in investigating the components needed to build that content. Students conduct a preliminary survey to advance their understanding and analysis of existing VR content through this course.

This process allows them to learn content components based on their area of interest. In addition, the process of comparing with the overall concept map for content creation presented in Apply 1 allows students to supplement the undiscovered parts of what they need in the actual content creation process. Understand 3 is added for the same reason. After the hardware-related course, the VR hardware to be investigated for each student is selected, and based on the content of the previous course, the projection process and components of the hardware selected by them are investigated. To express a VE and an object three-dimensionally and factually in virtual reality, it is necessary to understand the form and process of the image that the user can see. In this course, students learn to understand the 3D image projection process based on understanding the hardware and identifying the factors needed to improve the results. Finally, based on what students learned earlier in Analyze 1 and 2, they evaluate their understanding of the course through a simple quiz. These steps evaluate precourse content to advance future training and team project-based design courses, understand students’ understanding of the course, and identify in advance students who experience difficulties. For students who lack understanding of the course, additional learning materials can be used to provide steady motivation to keep up with the course flow without losing interest.

Overall, step 1 focused on explaining the overall concept and theory to students through more diverse and detailed examples. Then, students go through directly researching and presenting content and appropriate hardware in their area of interest. Through this process, we solve the students’ difficulties with the theory and prior knowledge and guide them to maintain their interest and motivation for learning through their areas of interest.

The modified step 2 lasts for eight weeks. The modified step 2 is structured in stages to advance a sufficient project based on the actual course content through the comprehensive practical course and handle diverse and deep content. Additionally, we added time to understand the skills that students will need while the project is in progress and set aside another course time to study.

In Apply 1, we provide an overall concept map of the core elements required for content production before proceeding with the VR content production training course, which corresponds to the concept map and training process of the development process of the produced project. Through this course, we provide time for the Apply 6 course to anticipate the needs of our students. This guides students to judge and suggest the essential skills they need while the project is in progress and help design it.

Apply 4 makes the UI and the part that creates virtual objects in the VE more concrete. Through the evaluation of Development and Team-based Learning, creating and defining UI and virtual objects in the VE, which was briefly covered in the existing course, is carried out by one of the team members in the process of the actual team project. Since it was confirmed that consideration was given to the weight of burden through the division of roles, we guide this content so that the team project can proceed smoothly by adding the course process.

In Apply 6, we further practice the core skills students require for project design, including what they have suggested in the concept map. Most students replied that the course process alone was somewhat lacking in advancing the project, requiring additional learning from external learning materials such as internet searches and YouTube. We guide students through long hands-on courses to team projects to envision the skills they need to construct the project. Then, we encourage them to think about and propose the technologies they think are necessary for the content design process. We enable students to receive help through reinforced courses and reduce their reliance on external learning materials during the project design process.

In Analyze 3, we evaluate students’ practical knowledge based on the overall concept map. This course allows students to review their practice and follow the team project process flow before designing a team project.

Overall, step 2 provides step-by-step learning of the key skills needed while the project progresses. The knowledge provided through the additional training provides students with the ability to solve the project.

The modified step 3 lasts for four weeks. We confirmed that the existing long-running project elicited negative feedback on the emotional effects of the students. To solve this problem, we modified the project process to provide students with steady motivation. Furthermore, we provide support during the project construction stage when students experience difficulties and guide them to complete the project.

In Apply 7, students explore the key technologies needed to design a project after envisioning an idea and then learn these technologies through TA-led courses. This course was designed to encourage students to advance the project and explore and select core skills. For this reason, this process must proceed after the overall concept map previously provided by Apply 1 and the entire course of step 2 learned through it have been prelearned. We have reduced the design time of the project and have reorganized it so that students can quickly learn and apply the key skills needed for the project.

Finally, through the Create 4 course, we review the project deliverables before evaluating the student’s project. Through this course, we understand the progress of the project designed by the student and the problems and corrections that occurred during the design. After that, we technically assist students in achieving the goals set at the time of project planning. This course can increase students’ academic achievement and satisfaction.

5. Limitations and Future Work

This study focuses on designing and evaluating a course about VR technology, which is mainly used as an integrated technology in STEM education. Before concluding the experiment, we would like to acknowledge the limitations of some of the experiments.

First, the small sample comprising nine students to verify the course design is insufficient to verify the educational effect. Hundreds of students are needed to verify the effectiveness of course design statistically [70]. These issues imply that our research methods can be quite limited in assessing educational value.

To solve this problem, we received more detailed feedback from students through in-depth interviews to ensure the reliability of statistics throughout the survey. As a result, we confirmed that the findings and the in-depth interview results had a strong relationship with each other and supplemented the reliability of this study with a small sample with qualitative research.

Second, we did not evaluate knowledge in the same way as an exam to verify the learning effectiveness of the students who attended this course. To evaluate the knowledge, it was necessary to proceed with a detailed design of the controlled variables, such as the students’ departments, majors, and prerequisites. To evaluate the effect of the proposed method on the students, we focused on the objective and learning effect of the course, not the course results. Future research will provide more students with redesigned courses and investigate the educational benefits of this method on a large scale.

6. Conclusions

In this study, we proposed a redesigned course that deals with the overall understanding and application of VR technology, used as the recent core technology of STEM education. Through previous research, we confirmed that a VR course must deal with both the hardware-related technical background and the software technology, which can be applied to utilize VR-related technologies. We designed a VR course consisting of three steps: VR-related theory, TA-led content creation training, and team projects and exhibitions. The proposed method is structured so that they could implement their desired virtual environment in software based on what they understood about the hardware-based IVE generation process. To evaluate this course, we conducted a semester-long study with nine students to verify the proposed method and then evaluated them through an in-depth interview and questionnaire comprising nine items rated on a five-point Likert scale. The questionnaire results were categorized based on the presence or absence of prerequisites, and the average score for each item was used to evaluate whether all students were able to achieve their course objectives successfully. As a result, we confirmed a difference in the questionnaire results of students who took prerequisite courses and those who did not. In addition, the students with the lowest scores for each item were selected as in-depth interview subjects, and feedback was collected for course redesign. Based on the survey results, we proceeded to redesign the courses, which were improved through the results of the questionnaire and feedback. Future research can focus on a redesigned course that matches the latest trends in VR-related technology fields and plan further to verify this course with more students, comparing the course with various state-of-the-art teaching methods.