Evaluating the Use of 360-Degree Video in Education

Abstract

Virtual reality (VR) has existed in the realm of education for over half a century; however, it has never achieved widespread adoption. This was traditionally attributed to costs and usability problems associated with these technologies, but a new generation of consumer VR headsets has helped mitigate these issues to a large degree. Arguably, the greater barrier is now the overhead involved in creating educational VR content, the process of which has remained largely unchanged. In this paper, we investigate the use of ${360}^{\circ }$ video as an alternative way of producing educational VR content with a much lower barrier to entry. We report on the differences in user experience between ${360}^{\circ }$ and standard desktop video. We also compare the short- and long-term learning retention of tertiary students who viewed the same video recordings but watched them in either ${360}^{\circ }$ or standard video formats. Our results indicate that students retain an equal amount of information from either video format but perceive ${360}^{\circ }$ video to be more enjoyable and engaging, and would prefer to use it as additional learning resources in their coursework.

1. Introduction

In the past two decades, numerous studies have demonstrated the benefits of virtual reality (VR) over multiple levels of education [1,2] and in many application domains [3,4]. The term VR encompasses the array of software and hardware that creates a digital representation of a 3D object or environment, with which a user can interact and feel a sense of immersion.

Immersion is one of the primary motivations for educators employing VR, because a sense of immersion in students can foster engagement with the learning content [5,6,7].

The definition of immersion is widely debated. In this paper, we use an integrative definition from Mütterlein, describing immersion as “the subjective experience of feeling totally involved in and absorbed by the activities conducted in a place or environment, even when one is physically situated in another” [8]. This activity-centric definition allows immersion to be differentiated from the related concept of presence, which describes “how realistically participants respond to the environment, as well as their subjective sense of being in the place depicted by the Virtual Environment” [9]. A heightened sense of presence in a virtual environment is associated with increased immersion in the learning activities performed there [4].

VR has encompassed experiences on many different devices with different levels of immersion and interactivity, from low-immersion, 2D computer screens, to fully immersive, VR head-mounted displays (HMDs). However, there has been a recent surge in consumer interest in VR and the concurrent release of a new generation of affordable HMDs and peripherals, greatly reducing the barrier of entry to immersive VR [10,11,12,13]. This has transformed the landscape of research into VR and has led the educational literature to focus on immersive VR as a newly accessible option.

Immersive VR provides a complete simulation of a new reality by tracking the user’s position and creating multiple sensory outputs through a VR HMD [14]. It has been successfully applied to education in many fields, including engineering [15], architecture [16], medicine [17], history [18], and music [19]. Educators using immersive VR often rely on constructivist pedagogy, because the medium facilitates the experiential and environment-driven learning at the centre of this methodology [20,21]. Studies have consistently found positive learner attitudes towards immersive VR [22], and it has been shown to improve time on task [23], motivation [6], and knowledge acquisition [24].

Despite these findings, immersive VR has failed to achieve widespread adoption in education. Traditionally, this failure was attributed to the high cost and questionable usability of the technology [2,3,20,25]. With modern VR systems available, this hesitance now arguably comes from educators weighing two other factors: whether there is evidence that VR has enough of an impact on students’ learning to justify costs; and whether they can guarantee the availability of VR content in the long term [22].

Currently, if educators wish to create their own VR content, they either have to learn how to program 3D graphical applications themselves or hire content developers to create it for them [26,27]. Novel technologies such as neural radiance field [28] and photogrammetry [29] have made it easier to create customised 3D content, but content creation is still a major barrier towards a wider use of VR in education.

To address these barriers to VR adoption, we investigated ${360}^{\circ }$ video as an alternative for educational VR content. ${360}^{\circ }$ video, when viewed inside an HMD, has been considered by some authors as a subset of immersive VR [30,31], but it is more commonly defined as different from VR [32]. We consider ${360}^{\circ }$ video as a simplified form of VR, since it shares some properties of VR but lacks essential functionalities. ${360}^{\circ }$ video provides a movable, user-centric viewpoint inside an immersive, digitally projected environment. However, standard ${360}^{\circ }$ video only allows passive viewing, users can not interact with the 3D environment, and the view position is fixed and movements restricted to the prerecorded camera movements. This differentiates it from VR using 3D environments, where users can interact with the environment and actively change it. More complex interactions are possible, e.g., gaze-based point-and-click mechanics or possibilities to change the storyline [33], but this usually requires post-processing of recorded videos and specialized applications.

${360}^{\circ }$ video is much easier to create than interactive VR. Spherical cameras that capture ${360}^{\circ }$ video have followed a similar trajectory to VR HMDs, with a recent generation of consumer-focused models greatly reducing the cost barrier [34,35]. In response to this, popular video sharing platforms YouTube and Facebook added support for ${360}^{\circ }$ video in 2015, and later Vimeo in 2017 [36]. Once captured, ${360}^{\circ }$ video can be edited using traditional video editing software in the same file formats as standard videos [37,38]. This means that ${360}^{\circ }$ video is cheaper to make than immersive VR, more easily sharable, and requires less upskilling of current content creators. Recent research has looked into making ${360}^{\circ }$ video more useful by proposing more powerful editing algorithms [39] and mixing it with other media [40,41].

${360}^{\circ }$ videos can also be combined with VR environments, e.g., improving collaborations by sharing remote environments via ${360}^{\circ }$ videos [42,43,44], improving the realism of 3D environments by fusing them with (potentially real-time) video data [45], creating immersive virtual field trips [40], or creating gamified learning experiences by combining ${360}^{\circ }$ technologies with VR [46].

Finally, ${360}^{\circ }$ videos can be used to create immersive VR environment by reconstructing 3D scenes using photogrammetry techniques [47,48,49] or by training Neural Radiance Field (NeRF) models [50,51]. The advantages of using ${360}^{\circ }$ video for 3D reconstruction are a comprehensive (${360}^{\circ }$) coverage of the environment, efficiency, smooth data acquisition, and that the ${360}^{\circ }$ video itself can be immersively viewed (e.g., as fast preview of the reconstructed scene). Disadvantages of using ${360}^{\circ }$ video for 3D reconstruction are the data size requiring high processing power, limited resolution, and potentially calibration and stitching issues.

Research into educational ${360}^{\circ }$ video is relatively new, but it has already been applied to many fields, including medicine [52,53], language [30], sports [54], business [55], collaborative design [56], education and training [46,57,58,59], sustainability [60], and marine biology [61]. The literature suggests many of the same benefits as immersive VR, such as enjoyment, motivation, and improved learning outcomes [33], though more research is needed in this area [36,62,63].

Many of these studies were applied to domains that the researchers believed would particularly benefit from ${360}^{\circ }$ video [30,54]. The evaluated videos were often hard to compare to standard videos because they were embellished with features that did not translate between the two formats [52,61]. Additionally, the studies directly comparing ${360}^{\circ }$ video to standard video only evaluated learning through immediate knowledge tests [52,54,55].

In this paper, we describe a randomised, crossover study in which we sought to compare the effects of ${360}^{\circ }$ video and standard video. Twelve videos were recorded and then produced in both ${360}^{\circ }$ and standard video formats.

To minimise the effect of domain and prior knowledge, the content presented was believable but fictitious. To ensure a comparable learning experience from the same video presented in either video format, we placed certain restrictions on the production of the ${360}^{\circ }$ videos, which incidentally lead to parallel production processes between the two video types. And to measure learning outcomes that had not yet been investigated, we included long-term retention and special cases for active and passive visual recollection in our study design.

This paper reports on both the results of a User evaluation surveying participants’ experience and the comparative short- and long-term learning outcomes of participants who viewed the same video in ${360}^{\circ }$ and standard formats. This is, to the best of our knowledge, the first time long-term learning retention has been investigated for ${360}^{\circ }$ video.

Research Questions

RQ1:

What differences in user experience exist when presenting educational content in ${360}^{\circ }$ video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

RQ2:

What differences in short- and long-term learning retention exist when presenting educational content in ${360}^{\circ }$ video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

2. Materials and Methods

This is a randomised, crossover study intended to compare the learning experience of ${360}^{\circ }$ video, viewed on a VR HMD, to standard video, viewed on a desktop PC. We chose an HMD since we wanted to test whether the additional immersion provided by it affected learning outcomes, e.g., no distraction from the real environment and more intuitive navigation (move head instead of a mouse). The study was spread over a six-week period and collected results about the user experience and short- and long-term learning retention of participants.

2.1. Participants

In total, 20 tertiary students participated in this study, 16 of which were undergraduate and 4 postgraduate. Of these students, 18 were male and 2 were female. The mean age of participants was 25; however, the ages of the participants varied greatly (standard deviation = 9.02, minimum age = 18, maximum = 55). No participants dropped out of this study, and all successfully completed all activities.

2.2. Educational Content

There were two primary considerations when designing the educational content used during this study. Firstly, we wanted to limit the effect that the application domain may have on the learning outcomes of students. Participants may have prior knowledge in different domain areas, affecting the study, and we wanted to confirm that the benefits of ${360}^{\circ }$ video were not constrained to especially well-suited subjects. Secondly, we needed to create videos covering the exact same learning content in ${360}^{\circ }$ and standard formats.

As a result of these considerations, we created a series of 12 short lectures in both video formats, filmed at 12 visually distinctive locations (e.g., a forest, a lecture theatre, a playground, inside a car). In 10 of these locations, a teacher would present a series of facts about that particular location, sometimes referring to visible features around them. The other two locations will be discussed in Section 2.3.

To address the effect of domain and prior knowledge, the location names and information presented in the videos were designed to be believable but were actually fictitious. For example, in the “Whiterock Bush” video recording, participants were informed that the area had been the site of “the frequent illegal dumping of rubbish”. “Whiterock Bush” is not a real location, nor is the presented fact true. Participants were not informed that the information was fictitious until after the study was complete, as that knowledge could have an effect on learning outcomes.

Ensuring the two video formats provided the same information placed two limitations on video development. Firstly, as extra information is visible when viewing a ${360}^{\circ }$ video, we needed to ensure learning did not rely on content outside of the frame captured in the standard video. Secondly, the quality of accessible ${360}^{\circ }$ cameras varies considerably, and while the pixel resolution is high compared to standard video, it is stretched over the full sphere when viewed. This is compounded by re-projection through VR HMDs of varying resolutions, leading to ${360}^{\circ }$ video experiences often being of low quality with poor text legibility. It is for this reason that we chose to use verbally delivered lectures in visually distinct locations as the educational content.

We also decided to reduce the resolution of the standard videos to match that of the ${360}^{\circ }$ videos. Although the current technology results in a difference in resolution between typical ${360}^{\circ }$ and standard desktop videos, we anticipate that, in the future, the gap will close, and the experience of resolution in each technology will be more similar. We adjusted the resolution to ensure our results are more robust to changes in technology and not significantly influenced by the current technology limits.

Figure 1 and Figure 2 show screenshots of the ${360}^{\circ }$ videos participants watched and various questions about these videos. The questions covered specific locations (e.g., which location is shown in this image?), objects in these locations (e.g., which painting did you see in the building?), positions of objects (e.g., where was the swing located within the playground?), and information about these locations (e.g., which bus numbers stopped at this bus stop?).

2.3. Special Case Videos

There were 3 additional factors of interest we wanted to measure, for which we designed 2 special case videos and modified one of the location-based lectures.

Firstly, as arguably the most common setting for tertiary educational videos, we included a lecture theatre as one of the locations. Rather than presenting facts about the location in this recording, we instead performed a traditional lecture where information was presented both verbally and via an overhead projector. This was performed to evaluate whether increasing the feeling of presence in a lecture theatre would affect learning outcomes.

We were also interested in whether viewing a video in ${360}^{\circ }$ would affect the active and passive recollection of visual information. To investigate active visual recollection, we included a recording located inside of a car. Participants were not presented with any facts but were instead asked to memorise as many details as they could about the interior of the vehicle. They were later tested on this information.

To investigate passive visual recollection, a bright red ribbon was tied around a tree in one of the existing locations. The ribbon was clearly visible in the recordings; however, the ribbon was never explicitly mentioned by the speaker at this location. The participants were later asked to recall where they had seen this ribbon among 3 visually similar locations.

2.4. Technologies

The ${360}^{\circ }$ videos were captured using a Ricoh Theta S spherical camera. The camera features two fish-eye lenses capable of recording ${180}^{\circ }$ each. We imported the videos into the Ricoh Theta software, which stitches the video together to a ${360}^{\circ }$ video and exports it as a standard MP4 file, which we then edited with Adobe Premiere Pro.

We displayed ${360}^{\circ }$ videos on a Samsung Gear VR HMD (used in conjunction with a Samsung Galaxy Note 5). While more modern HMDs, such as the Meta Quest 2, offer higher comfort and display quality, very few students own such HMDs, and hence, we decided to use a mobile phone-based HMD, which we considered more accessible for education applications.

The standard videos were obtained using the free and open-source OBS Studio to take screen recordings of the ${360}^{\circ }$ videos, running on the free Ricoh Theta desktop video player software. They were recorded at 60 fps and adjusted to a resolution of 1920 × 1080, to match the ${360}^{\circ }$ video’s resolution. Standard videos were played on a 23″ PC monitor with a resolution of 1920 × 1080.

In most of the standard videos, all necessary information was contained within the intial field of view. However, in some locations, small rotations of the camera were needed to include other details. These rotations were integrated into the standard video recordings, not controlled by or performed in front of the participants.

In total, 24 video clips were created, including both the ${360}^{\circ }$ and standard versions of each location and lecture. For a detailed description of the content creation process see [64].

2.5. Challenges in Creating 360° Videos

We encountered several challenges when creating ${360}^{\circ }$ videos for practical applications. Finding a suitable position for the camera was non-trivial. Placing the camera on a table using a normal tripod resulted in an unnatural experience when viewing the video with an HMD. Ideally, the camera had to be located at the eye height of a participant. Holding the camera by hand or using a large tripod resulted in the hand or tripod being visible when looking down. In the end, we used a large monopod (a one-legged camera stand). This single-legged stand was almost invisible, hidden under the camera, and provided the most natural viewing experience.

Another problem we encountered was the low video quality caused by the use of a cheaper older camera model. While the device is capable of recording in 1920 × 1080 pixels, this resolution is spread across the entire ${360}^{\circ }$, resulting in a display resolution more similar to a 240p video than a 1080p video when viewed on a desktop screen. This limited the educational content we could display; e.g., we avoided situations requiring the recording of text.

2.6. Study Design

Participants were randomly divided into Groups 1 and 2. Both groups were shown all 12 recordings; however, Group 1 was shown locations 1–6 in ${360}^{\circ }$ video on a VR HMD and locations 7–12 in a standard video on a desktop PC. Group 2 was shown the opposite videos on the HMD and PC (see

Table 1. Distribution of the 24 video recordings among the 2 groups.

Group 1
Desktop Recordings	360∘ Recordings
(Shown First)
Cook Street Bus Stop	Rawene Nature Reserve
James Cook Domain	Inside a Car
Woodward Building	Ponga Falls
Swanson Park	Park View Road
Willis Lake	Playground
Lecture Theatre	Whiterock Bush
Group 2
360∘ Recordings	Desktop Recordings
(Shown First)
Cook Street Bus Stop	Rawene Nature Reserve
James Cook Domain	Inside a Car
Woodward Building	Ponga Falls
Swanson Park	Park View Road
Willis Lake	Playground
Lecture Theatre	Whiterock Bush

). This is a randomised, crossover design to evaluate user experience across video formats, with both groups experiencing both treatments and subsequently compared through a user evaluation.

This study is additionally a randomised design to evaluate the learning outcomes between the video formats, achieved by comparing the test scores of participants experiencing the same recordings in different formats. These scores were obtained from both a Short-Term Retention Test and Long-Term Retention Test.

The groups compared in this second design are not Groups 1 and 2, but rather the pseudo-groups HMD and PC. This is because the scores associated with experiencing ${360}^{\circ }$ video will include question scores from both Groups 1 and 2, depending on the recording being assessed by each individual question. In fact, all participants may appear in both pseudo-groups, depending on the specific condition, since all participants experience both ${360}^{\circ }$ and standard videos.

2.7. Assessment Instruments

Three assessment instruments were used in this study: a user evaluation, a Short-Term Retention Test, and a Long-Term Retention Test.

The user evaluation was 12 questions long, comprising 4 Likert-scale questions, 1 short-answer question, and 7 open-ended questions. In all questions, participants were asked to compare ${360}^{\circ }$ and standard video experiences, except the final question, which directly asked participants for their opinion on ${360}^{\circ }$ video as an educational tool.

The user evaluation included questions about participants’ senses of enjoyment, immersion, engagement, and their level of distraction while watching the different video formats. We also asked about their preference of format for lecture recordings and any issues that were caused by video and screen quality, motion sickness, or other factors. These are some of the demonstrated benefits of immersive VR, and we wanted to validate that they were evident in ${360}^{\circ }$ video as well.

The Short-Term Retention Test was 20 questions long, comprising 14 multiple-choice questions and 6 short-answer questions about the content of the video recordings. This test also included questions about the active and passive visual recollection from the special case videos.

The Long-Term Retention Test was 21 questions long, comprising 15 multiple-choice questions and 6 short-answer questions. It is identical to the Short-Term Retention Test except for 4 questions, which are slight simplifications of 3 of the original questions. Two of these asked the participants to recall information from an image instead of a location name, and one short-answer question asking for two pieces of information was split into both a short-answer and multiple-choice question.

The Short- and Long-Term Retention Tests were divided into 2 sections, “Content Retention” and “Location Recognition”. Questions about content retention focused on the recall of the information participants were taught during the lectures, while the location recognition questions targeted visual information about the recording locations. The questions were designed to be unambiguous and binary in nature, allowing for a rigid marking rubric.

2.8. Study Procedure

Participants were first asked to fill out a Demographic Questionnaire. This questionnaire collected basic information, including the participants’ age, area of study, experience with VR HMDs, and whether they had any issues affecting their vision (so the HMD could be adjusted accordingly).

Next, participants were asked to put on the HMD and instructed to complete both a pre-installed Oculus Tutorial as well as watch a 2-minute-long introductorylecture on the HMD. This was to familiarise the participants with ${360}^{\circ }$ video.

After the familiarisation protocols, participants were asked to watch the first set of videos on either the HMD or the PC, depending on the group they had been assigned to (see Table 1). These video sets each comprised 6 location recordings stitched together, forming one continuous recording approximately 5 min long.

Participants were then asked to perform an unrelated reading task for 3 min, designed to act as a distractor between the 2 video sessions. After the distractor task, they were shown the second set of videos in the other video format.

When they had finished viewing the video content, participants were asked to complete both the user evaluation and Short-Term Retention Test. Six weeks later, participants were asked to complete the online Long-Term Retention Test.

3. Results

This section is split into three primary subsections. Section 3.1 discusses the results of the score-based questions in the user evaluation, before Section 3.2 reports on our thematic analysis of the open-ended responses. Section 3.3 then outlines our statistical analysis of the results of the Short- and Long-Term Retention Tests.

3.1. Quantitative User Evaluation

The user evaluation was a combination of Likert-scale, short-answer, and open-ended questions. It was completed by all participants after they had finished viewing both video formats.

This section reports the quantitative results from the four Likert-scale and one short-answer question, discussed in two groups: Subjective Experiences and Study Habits.

3.1.1. Subjective Experiences

In the first three Likert-scale questions, participants were asked to rate the degree to which they felt enjoyment, immersion, and engagement while watching the two different video formats. These questions used a five-point Likert scale, from −2 (“Strongly Disagree”) to 2 (“Strongly Agree”), with 0 corresponding to “Neutral”.

Immersion was described to participants as the degree to which they forgot about their surroundings and “lost themselves” in the experience. Engagement was described as the degree to which a learning experience holds their attention and makes them actively focus on their learning.

Table 2. Comparative enjoyment, immersion, and engagement between ${360}^{\circ }$ and standard video formats. The mean and median are in the range of −2 to 2, and the mean is given with a 95% confidence interval.

Enjoyment
Format	Mean	Median	V-Stat	p-Value	Effect Size
Standard	0.25 ± 0.43	0.0	4.5	$5.3\times {10}^{-4}$	1.62
${360}^{\circ }$	1.50 ± 0.28	2.0
Immersion
Format	Mean	Median	V-Stat	p-Value	Effect Size
Standard	−0.65 ± 0.46	−1.0	0.0	$7.9\times {10}^{-5}$	2.56
${360}^{\circ }$	1.45 ± 0.28	1.5
Engagement
Format	Mean	Median	V-Stat	p-Value	Effect Size
Standard	0.00 ± 0.40	0.0	11.0	0.0026	0.99
${360}^{\circ }$	0.95 ± 0.49	1.0

presents the means and medians of the responses to these three questions, which range between −2 and 2. It also presents the results of Wilcoxon signed rank tests between the scores for ${360}^{\circ }$ video and standard video, as well as the Cohen’s d effect sizes where the tests were considered significant at the $\alpha =0.05$ threshold. Cohen’s d values of 0.20, 0.50, and 0.80 correspond to small, medium, and large effect sizes, respectively. Wilcoxon tests were used because Likert responses are ordinal, the sample size is small (n = 20), and each group of scores heavily defied a Shapiro–Wilk test for normality.

Based on these analyses, we found statistically significant results across all three questions, with participants expressing strong preference towards ${360}^{\circ }$ video over desktop video in all factors.

As Figure 3 shows, participants rated their sense of enjoyment much higher for ${360}^{\circ }$ video compared to standard video. The majority of respondents, 12 of 20, responded neutrally to standard video, with only 5 responding positively. Conversely, 18 participants responded positively to ${360}^{\circ }$ video, with 11 of them saying they “Strongly Agree” that they enjoyed the experience. This is strong evidence that the students enjoyed the ${360}^{\circ }$ versions of the videos more than the standard versions, with a large effect size in our sample.

Figure 4 shows the results of the participants rating their sense of immersion during the ${360}^{\circ }$ or standard videos.

Of these three subjective Likert questions, this one had the smallest p-value and largest effect size, suggesting that the participants felt much more strongly immersed in the ${360}^{\circ }$ video than the standard one. A total of 19 of 20 participants responded positively to this question regarding ${360}^{\circ }$ video, while 12 responded negatively regarding desktop video.

Regarding engagement, shown in Figure 5, the responses were again much more in favour of the ${360}^{\circ }$ video approach, with 16 of 20 participants rating their engagement level positively, compared to just 5 participants for the standard video.

Unlike the previous two questions, however, two participants responded negatively regarding their feeling of engagement with the ${360}^{\circ }$ content. While the p-value was significant and the effect size large, this question had the smallest difference between groups of these three subjective Likert questions. Potential reasons for this will be expanded on in the discussion, after evidence about attention and distraction is presented in Section 3.2.1 of the thematic analysis.

3.1.2. Study Habits

The next two questions concerned the potential study habits of participants using the two video formats. Participants were asked how long they believed they could comfortably study by watching ${360}^{\circ }$ video on a VR HMD or by watching standard video on a desktop PC. They were then asked how likely they would be to use educational materials of a similar nature to those they had been shown, as an additional (but optional) material for their own courses. These were a short-answer and a five-point Likert-scale question, respectively.

The differences between the responses to the two video formats were analysed using Wilcoxon signed rank tests, and the effect sizes were estimated by Cohen’s d when the result was significant. The results are given in

Table 3. Study habits of participants using standard and ${360}^{\circ }$ video, comparing the length of time they believed they could study and their likelihood to use either video type as additional learning material. The mean and median of supplementary use are in the range of −2 to 2, and the mean is given with a 95% confidence interval.

Study Times
Format	Mean (min)	Median	V-Stat	p-Value	Effect Size
Standard	87.1 ∓ 21.9	-	62.5	0.91	-
${360}^{\circ }$	79.5 ∓ 27.9	-
Supplementary Use
Format	Mean (min)	Median	V-Stat	p-Value	Effect Size
Standard	0.55 ∓ 0.42	1.0	14.0	0.003	0.99
${360}^{\circ }$	1.40 ∓ 0.38	2.0

and are illustrated in Figure 6.

These responses indicate that participants believed they could study using the different video formats for very similar amounts of time. The mean reported time was 87.1 min for the standard video and 79.4 min for the ${360}^{\circ }$ ($sd=57$) video. A Wilcoxon signed rank test on the reported times found $p=0.91$, showing no significant difference between these times.

Assuming that they had equal and easy access to a desktop PC and a VR HMD, participants showed a statistically significant preference for the idea of using ${360}^{\circ }$ video as supplementary content to their own courses. Most participants answered that they were “Likely” or “Very Likely” to use both ${360}^{\circ }$ video and standard formats, but this number was larger for ${360}^{\circ }$, with responses more heavily weighted towards “Very Likely”.

3.2. Thematic Analysis

This section outlines the results of our thematic analysis of the open-ended responses to the user evaluation. The evaluation included six questions in which participants were asked to comment on both the standard and ${360}^{\circ }$ video experiences, followed by a final question in which they were asked their opinion on ${360}^{\circ }$ video as an overall tool for education.

As explained in Section 2.7, participants were asked about their senses of immersion and engagement, their level of distraction, their preference of format for lecture recordings, and any issues that were caused by video and screen quality, motion sickness, or other factors.

Qualitative responses were examined for key themes, patterns, and outliers based on the “Coding in Detail” approach outlined by Bazeley and Jackson [65]. NVivo was used for the coding process, and as a result, we created 29 codes organised into four primary themes: “engagement”, “immersion”, “learning” and “technology limitations”. These codes were applied for a total of 268 times to the data.

To test the validity of this coding scheme, two reviewers independently coded a subset of the data. Cohen’s Kappa was then calculated, resulting in a coefficient of $k=0.8705$, suggesting a high degree of inter-rater agreement.

The four primary themes relate to the questions asked, but the codes for each theme were spread among responses for every question. The following sections will discuss the findings in each theme, with notable sub-themes presented in subsections where appropriate.

3.2.1. Engagement

Of the four primary themes we identified during our analysis, “engagement” was the most prevalent. Participant responses were coded 117 times as relating to engagement, accounting for 43% of the total coded data.

Responses in this category can be grouped into two major sub-themes: “enthusiasm and enjoyment”, which align with the expected trends observed in the quantitative results above; and “attention”, which revealed substantially more mixed opinions and interesting insights about the video formats. These sub-themes are discussed in further detail in the following two subsections.

Enthusiasm and Enjoyment

Mirroring the quantitative results from Section 3.1.1, participants expressed much more enthusiasm and enjoyment regarding learning in ${360}^{\circ }$ video. A total of 12 of 20 participants described it as “enjoyable” or “fun”, while 4 participants explicitly described the standard video as “boring”.

The major factor to which participants attributed their enjoyment in ${360}^{\circ }$ video was the interactive, immersive environment. One participant said that “the experience of being able to look around and follow visual cues was engaging”, while another described ${360}^{\circ }$ video as “dynamic” compared to the “stagnant” standard video. This idea of passivity in the standard video was mentioned by another participant, who stated that “it was quite boring to just stare dully at the screen, and try to just take in all the information”.

Another factor that three participants mentioned was the novelty of experiencing ${360}^{\circ }$ video. Only 7 of the 20 participants in the Demographic Questionnaire claimed to have used a VR HMD for more than an hour, and many others reported an increased sense of curiosity and interest in the immersive environment. Novelty will be discussed in Section 4 as it is likely to impact enjoyment, and also affected the leniency reported in the “Leniency” section below.

Overall, these responses clearly show that participants enjoyed the ${360}^{\circ }$ video format more, but whether this translated into engagement with the learning content itself will be discussed further after more evidence is provided.

Attention

While participants rated the overall engagement of ${360}^{\circ }$ video higher than standard video, by asking specifically about distractions, we were able to derive greater nuance from their responses.

Surprisingly, 11 of the 20 participants claimed to be more distracted in ${360}^{\circ }$ video. Only five felt more distracted watching standard video, while four claimed no difference or did not compare the formats directly. The interesting factors here, however, are the vastly different reasons students felt distracted.

Participants who felt more distracted in standard video often cited boredom, repetitive content, and a lack of focus or attention. They also mention distractions from the real world, and that ${360}^{\circ }$ video has an advantage because “it hides distractions in the surroundings”.

For the larger group who felt more distracted in ${360}^{\circ }$ video, the major reason was the extra information available in the surrounding environment. One participant reported they “can’t focus on the speaker because [they’re] curious with the surrounding things”, while another said that ${360}^{\circ }$ video “was very immersive...but I don’t know how much the distraction of looking around was combating the usefulness of it”. Another participant claimed they “enjoyed being able to look at random things while listening”, but conceded that they “got distracted more with this”.

Five participants additionally reported accidental encounters with the real environment as a distraction in ${360}^{\circ }$ video. For example, two of these participants specifically mentioned bumping their feet against the wall while wearing the HMD.

Interpreted together, the responses regarding distraction highlight the distinctions between the two video formats. Watching a standard video is a less engaging activity, but the limited information can help students identify what to focus on for their learning. ${360}^{\circ }$ video is much more fun and engaging, but the immersive environment can pull attention away from the learning content if it is not directly relevant. This is a key finding that will be expanded on further in the part “Content” of Section 3.2.3 and in Section 4.

3.2.2. Immersion

A sense of immersion was commonly mentioned by the participants, with responses coded to this theme 57 times (accounting for 21% of the coded data). A total of 16 of the 20 participants described experiencing immersion in the ${360}^{\circ }$ video, whereas only 1 of the participants described finding the desktop video “somewhat immersive”.

The responses regarding immersion were frequently strongly emotive in nature, with participants using superlatives including “surreal”, “incredible”, and “powerful” to describe their experiences. As noted in the part “Enthusiasm and Enjoyment” of Section 3.2.1, immersion in the environment was often given as the reason a participant enjoyed ${360}^{\circ }$ video more than standard video.

Immersion is a primary motivation for using ${360}^{\circ }$ video, so these results are encouraging but not surprising. However, the sources of immersion are useful to understand so that they can be taken advantage of.

Two distinct groupings emerged from this theme: immersion through a heightened sense of presence and realism in the environment; and immersion from a sense of intimacy and closeness to the presenter.

Presence

A feeling of presence in the ${360}^{\circ }$ video was explicitly mentioned by seven participants but did not apply to standard video at all.

One participant wrote “I can feel the surround[ings] with me, just like I am there”, while another said they “Forgot [they were] actually still in a room for a moment...”. Some participants noted very powerful feelings of presence, with one stating “I felt as if I was a new person and I’m in a different world”.

Two participants described the lack of presence in standard video, with one saying it “felt more like looking through someone’s perspective”. Another participant, when comparing the video formats, said of standard video “I imagine I was there, imagining what was there”, in contrast to feeling “like I am there” in ${360}^{\circ }$ video.

A sense of presence can be reinforced by a sense of realism in a virtual environment, and the realism of the ${360}^{\circ }$ video was mentioned by three participants. One said “it allows me to step into the content and explore...so it feels like the experience is more real”, and another described ${360}^{\circ }$ video as “a more ‘real’ experience” than standard video.

The sense of presence potentially afforded by ${360}^{\circ }$ video is a major motivation for its use over standard video, which is reinforced here by our study results.

Intimacy

Another way in which participants experienced immersion in ${360}^{\circ }$ video was through an increased sense of intimacy with the speaker. Three participants mentioned this directly.

One of the participants listed the following as a benefit of ${360}^{\circ }$ video: “I found the headset gave me a greater sense of interaction with the presenter, I could understand him a bit more”. Another wrote as a criticism of standard video, “The experience was almost less personal than in VR, so I felt like I could pay less attention to the facts”. This feeling of obligation to the presenter was echoed by a third participant, who wrote “I feel like I have to pay more attention because the person speaking feels more like a real person”.

3.2.3. Learning

This theme encapsulated any discussion about how the video formats may impact learning outcomes or the learning content of the videos themselves. This was the least common theme and was only coded 28 times during the analysis (approximately 10% of the coded data).

Three participants said they felt ${360}^{\circ }$ video helped with their learning outcomes, with one stating “it seemed like [my] brain learnt by itself as I do not need to concentrate too much”. The other two directly mentioned it helping with memory retention, with one postulating that “perhaps it also engages spatial processing in the brain more which creates more robust memories”. The other attributed the better retention to novelty and immersion, saying they “felt HMD is the more memorable experience hence [they] would be able to remember and recall more things”.

However, most responses under this theme were about which types of learning content are appropriate for standard and ${360}^{\circ }$ video.

Content

Generally, responses about content suggested that effectiveness of ${360}^{\circ }$ video would depend on the content being taught and how it was presented.

When asked about their preference of video format to watch lecture recordings, nine participants said they would prefer ${360}^{\circ }$ video, eight would prefer standard video, and two said it would depend on the content of the lecture.

When participants preferred ${360}^{\circ }$ video for lecture recordings, they cited increased immersion and novelty as the reason. One participant even claimed they would use ${360}^{\circ }$ video until their novelty wore off and likely switch to standard video after that.

The reasons participants would prefer using standard video for lecture recordings included that they can more easily take notes, they would be less likely to encounter distractions, and that a lecture recording is not appropriate content for ${360}^{\circ }$ video. Integrating these last two reasons, one participant wrote “If it’s a lecture recording; there’s no need to be aware of my surroundings—it’d only be an unnecessary distraction”. Another wrote quite directly “The content of the lectures doesn’t get improved by HMD. A lecture is a lecture!”

The topic of content appropriateness was again mentioned by several participants when giving their opinion of ${360}^{\circ }$ video as an educational tool. In one participant’s words: “For material which benefits from being in the environment, it’s a much more enjoyable way to learn it. For material which has no benefit from being in the environment (e.g., a traditional lecture), the desktop is more comfortable to use”.

A couple of participants linked this idea back to distractions, with one saying “the content should be related to the spherical video, otherwise it’s too easy to get distracted”. This was echoed almost verbatim by another participant, who responded “immersing students in a subject is how you best learn. If you could immerse a student and make sure the distractions are informational, you have a very powerful tool”.

The presentation of learning content is closely tied to the themes of attention and distraction reported above. The implications of these findings will be explored in the discussion.

3.2.4. Technological Limitations

The final theme in our thematic analysis encompasses any issues relating to the technologies themselves. Responses were coded as relating to this theme 85 times (approximately 31% of the coded data) and came largely from questions on the user evaluation about screen and video quality, motion sickness, and other issues encountered by participants.

The nature of the issues discussed varied considerably; however, they can be classified as relating to either the comfort of the participants or issues with the viewing experience and video quality. The other interesting sub-theme that emerged was leniency; multiple participants exhibited forgiveness towards technical issues with ${360}^{\circ }$ video. These three sub-themes are discussed below in the final sections of this thematic analysis.

Comfort

Problems with the physical comfort of the HMD were among the most common issues with ${360}^{\circ }$ video, being mentioned by eight of the participants. Of these, two described experiencing a sense of disorientation when removing the HMD, while another expressed difficulty swivelling the computer chair while watching ${360}^{\circ }$ video. The remaining five participants all mentioned that the HMD was heavy, with several stating that it caused discomfort in their neck.

Surprisingly, and despite being explicitly asked about it, only one participant reported any motion sickness while wearing the HMD.

The final issue of comfort while viewing ${360}^{\circ }$ video was not physical. Instead, it was the fact that students cannot take notes while wearing a VR HMD. This was explicitly mentioned by three of the participants, two of whom provided it as their justification for why they believed that ${360}^{\circ }$ video may not be an effective tool for education overall.

Viewing Experience

An equal number of participants reported issues with the quality of the ${360}^{\circ }$ video and the standard video. Six participants mentioned their learning was impacted by video or screen quality in each format.

This is surprising because the videos were of the same resolution, but the ${360}^{\circ }$ video is spread over the full visual range of the viewer, generally decreasing perceived quality. We anticipated more issues with ${360}^{\circ }$ video than standard video, but two participants even stated they believed the ${360}^{\circ }$ video had a higher resolution than the standard video. This shows a level of leniency that the participants had towards ${360}^{\circ }$ video, which will be noted in the next section.

Three participants encountered a specific issue with the viewing quality in ${360}^{\circ }$ video: the Screen Door Effect. These participants described being able to identify individual pixels, with one writing “Because the pixels were visible, my immersion and attention were both negatively affected”. This was the only viewing quality issue ascribed to the screen itself, instead of the resolution of the video.

Leniency

In reviewing responses about the limitations of ${360}^{\circ }$ video, an interesting trend emerged of participants expressing leniency towards these limitations. This forgiveness was expressed by six of the participants and usually came in the form of a criticism followed immediately by a justification as to why it was unimportant.

For example, when asked if video quality affected their ${360}^{\circ }$ experience, one participant wrote: “Yes! But overall the experience felt quite real so the low quality was a compromise”. Another similarly responded “Yes, resolution could have been better, but I think that being able to interact with the environment lessened the impact of this”.

This leniency could have a number of causes, including novelty and social desirability bias, which will be expanded on during the discussion in Section 4.

3.3. Learning Retention

The second major section of results involves learning retention. Learning retention was measured through the Short-Term Retention Test, taken by participants directly after watching the videos, and the Long-Term Retention Test, taken 6 weeks later. The tests comprised multiple-choice and short-answer questions separated into two sections: “Content Retention” and “Location Recognition”.

Test scores for short- and long-term retention were analysed by comparing the scores between participants that experienced the same recording in ${360}^{\circ }$ and standard video. This involved comparing the pseudo-groups HMD and PC, which conditionally contained all participants, but for questions about different locations. For example, for questions pertaining to the “James Cook Domain” recording, scores from Group 2 were considered HMD because they experienced it in ${360}^{\circ }$ video, while scores from Group 1 were considered PC. For questions about the “Rawene Nature Reserve” recording, scores from Group 1 were considered HMD, and scores from Group 2 were PC.

This setup effectively splits the sample into two groups, each with n = 10. The reduced sample size means that statistical comparisons are less powerful than would be desired, but they can still indicate if an effect may be present and could warrant further investigation. None of the score distributions for HMD and PC in either test was significantly different to normal, according to Shapiro–Wilk tests, so they were compared using two-tailed two-sample Student’s t-tests assuming unequal variances. These results are reported in Section 3.3.1 and Section 3.3.2.

The Short- and Long-Term Retention Test also included questions about the special case videos, which sought to test active and passive visual recollection and recollection from a lecture theatre environment. These questions were part of the “Content Retention” Section of the tests and are analysed individually in Section 3.3.4 below.

3.3.1. Short-Term Retention

The Short-Term Retention Test was scored out of 32 total marks. A total of 27 marks were available for “Content Retention”, and 5 for “Location Recognition”.

From

Table 4. Results of the Short-Term Retention Test, comparing the learning outcomes of the HMD and PC groups. Mean scores for each group and test section are given with a 95% confidence interval.

Group	Content Retention	Location Recognition
HMD	18.35 ∓ 1.57	3.3 ∓ 0.59
PC	18.75 ∓ 1.87	2.7 ∓ 0.56
Group	Total Score	t-Stat	df	p-Value
HMD	21.65 ∓ 2.00	0.504	19	0.6174
PC	21.45 ∓ 2.30

, we can see that the total mean marks of both the HMD and PC groups are very similar, at 21.65 and 21.45, respectively.

These means are 67.65% and 67.03% of the available marks.

The overall t-test found $p=0.6174$, suggesting that there is no statistically significant difference in the learning outcomes of the two groups.

While the HMD group did appear to perform better in the “Location Recognition” questions, a second t-test run on just these results found $p=0.2248$, indicating that the difference was not statistically significant.

3.3.2. Long-Term Retention

The Long-Term Retention Test was scored out of 33 marks, with 27 available for “Content Rentention” and 6 for “Location Recognition”.

From

Table 5. Results of the Long-Term Retention Test, comparing the learning outcomes of the HMD and PC groups. Mean scores for each group and test section are given with a 95% confidence interval.

Group	Content Retention	Location Recognition
HMD	12.15 ∓ 1.29	3.1 ∓ 0.49
PC	12.1 ∓ 1.11	2.6 ∓ 0.23
Group	Total Score	t-Stat	df	p-Value
HMD	15.25 ∓ 1.64	0.550	19	0.5854
PC	14.7 ∓ 1.16

, we can see that again, while the HMD group performed better on average than the PC group, the difference in learning outcomes was not statistically significant.

The HMD group achieved an average of 15.25 (46.21%), while the PC group averaged 14.7 (44.55%), and the t-test found $p=0.1615$.

As the HMD group appeared to perform better on the “Location Recognition” questions, a second t-test was conducted on these results for the two groups. However, the result was not significant at the $a=0.05$ threshold, with $p=0.3575$.

3.3.3. Question Performance

Interestingly, from

Table 6. Comparative performance on questions between the HMD and PC groups across the Short- and Long-Term Retention Tests.

Group Performance	Short Term	Long Term	Total
HMD performed better	12	12	24
PC performed better	5	6	11
Groups performed equally	3	3	6

, we can see that the HMD group performed better on more than twice as many questions as the PC group in both the Short- and Long-Term Retention Tests. We investigated the probability of this occurring by chance using a binomial test, assuming equal probability that either group would outperform the other on a question. The cumulative probability that the HMD group performed better on 24 questions across the two tests is $P(X\ge 24)=0.174$, under the null hypothesis that video format had no effect.

While this value is not significant at the $a=0.05$ threshold, this is an indication that a study with more power has the potential to detect an effect on learning, or that the impact of video format may vary considerably between types of question.

3.3.4. Special Case Questions

As outlined in Section 2.3, participants were tested for three special interest cases: recollection from a lecture theatre, and active and passive visual recollection. Content from the lecture theatre video accounted for four marks in both the Short- and Long-Term Retention Tests, and one mark was assigned to each of the visual recollection cases. The results of these questions are in

Table 7. Results of the “special case” questions, comparing between the HMD and PC groups across the Short- and Long-Term Retention Tests. Mean scores for each group and case are given with a 95% confidence interval.

Short-Term Retention Test
Case	HMD	PC	t-Statistic	p-Value
Lecture Theatre	4.0 ∓ 0.47	3.9 ∓ 0.45	0.156	0.8767
Active Recollection	0.8 ∓ 0.19	0.7 ∓ 0.22	0.493	0.6278
Passive Recollection	0.1 ∓ 0.14	0 ∓ 0	1	0.3434
Long-Term Retention Test
Case	HMD	PC	t-Statistic	p-Value
Lecture Theatre	2.5 ∓ 0.53	1.9 ∓ 0.43	0.895	0.3768
Active Recollection	0.6 ∓ 0.23	0.3 ∓ 0.22	1.342	0.1964
Passive Recollection	0.5 ∓ 0.23	0.5 ∓ 0.23	0	1

, along with students t-test results comparing the performance between the HMD and PC groups.

From Table 7, we can see that the HMD group performed better on all of the cases across both tests, with the exception of long-term retention of passive visual information, for which the two groups performed equally. Though this may suggest a trend, the t-tests again failed to identify any significant differences for the individual factors.

4. Discussion

• RQ1. What differences in the user experience exist when presenting educational content in ${360}^{\circ }$ video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

The results of our user evaluation analysis highlighted many of the key differences between using standard and ${360}^{\circ }$ video.

In terms of general preference, participants were found to enjoy ${360}^{\circ }$ video more, feel more engaged by it, and would prefer to use it as additional material for their learning, provided they could access it as easily as standard video. This was largely attributed to the increased sense of immersion and interaction in the environment in ${360}^{\circ }$ video, amplified by feelings of presence and realism. These results are consistent with existing research around immersive VR and validate our motivations for using ${360}^{\circ }$ video as a learning tool.

Looking further into the thematic analysis, we see a more complex relationship between immersion and engagement in the two video formats. While ${360}^{\circ }$ video was more engaging, participants commonly reported being more distracted while using it, primarily by the interesting, immersive environment itself. The extra visual information pulled attention away from the speaker in the video who was delivering learning content verbally.

This illuminated a distinction in attention and focus between the video formats. Participants watching standard video were more likely to get bored and be distracted by elements of the real world. However, they paid more attention to the core learning elements of the video. Viewers of ${360}^{\circ }$ video were more engaged, but that did not translate directly to engagement with the learning content itself.

While a couple of participants took this as evidence that standard video was the better teaching tool, more noted the potential strength of harnessing distractions in ${360}^{\circ }$ video. They noted that if the learning content is reinforced by the visual environment, instead of being distracted by it, ${360}^{\circ }$ video has the potential to create greater engagement with learning content than standard video.

This related directly to another major emergent theme: content appropriateness. Participants stated that ${360}^{\circ }$ video would be more suited to topics that relied on visual and environmental information, instead of those that are taught verbally or through text.

Our study design should be taken into account when discussing distractions and content appropriateness. We included videos that were heavy in visual information (e.g., memorising details inside a car), and heavy in verbal information (e.g. a traditional lecture recording), incidentally allowing participants to compare the two kinds of content. We also tried to create as comparable an experience as possible between the two video formats, by restricting the learning content in the ${360}^{\circ }$ videos to a frame that could also be captured as standard video. This meant most of the environmental information in the ${360}^{\circ }$ videos was not part of the learning by design, which would have increased the effect of distractions.

Interestingly, the sub-theme of intimacy suggested that ${360}^{\circ }$ video may have an advantage with some verbal content. Some participants felt an increased sense of connection to the speaker due to the presence and realism of the video format, which translated into a sense of obligation to listen to them. This kind of authentic connection can be particularly useful in situations like language learning [30].

Enhanced intimacy could even be harnessed to create better learning resources for students from cultural backgrounds that emphasise the relationship between student and teacher as a part of learning. For instance, Reynolds suggests that for students of Pacific Island background, “teu le va [the nurturing and valuing of a relationship] between a teacher and student is crucial because a student’s identification with a subject can come through a positive connection with a teacher” [66]. With the increasing prevalence of digital learning resources, consciously maintaining an element of human intimacy as part of their design will benefit some students.

The final major difference between the video formats to mention here is usability. Watching ${360}^{\circ }$ video was reportedly less comfortable, with a few participants noting the weight of the HMD or a sense of disorientation from using it. These drawbacks of watching ${360}^{\circ }$ videos with HMDs have also been reported in previous research using different types of HMDs [67]. Participants also noted that you cannot take written or typed notes while in immersive VR. Overall, however, these limitations did not appear to outweigh the benefits of ${360}^{\circ }$ video, as participants also claimed they could study from both formats for similar amounts of time, and would prefer to use ${360}^{\circ }$ video to do so.

• RQ2. What differences in short- and long-term learning retention exist when presenting educational content in ${360}^{\circ }$ video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

Our analysis of the results of the Short- and Long-Term Retention Tests found no statistically significant differences in learning retention between the HMD and PC pseudo-groups. This is true of the overall scores, the separate scores for the “content retention” and “location recognition” subsections of the tests, and the scores associated with the three “special case” videos.

This means that ${360}^{\circ }$ video was as effective as standard video in conveying learning content, even with the limitations we placed on production. There were no major differences in short- and long-term retention in the context of this study.

It should be noted that the small effective sample size (n = 10) limited the statistical power of this study. When looking at the results in aggregate, the HMD group outperformed the PC group in every measure except two (short-term content retention and long-term passive visual recollection) and scored better in over twice as many total questions. As mentioned, none of these results were statistically significant at the $a=0.05$ threshold, but there is an indication that a study with greater strength may be able to detect an effect.

Our results are interesting in the context of previous research, which has focused on evaluating the use of ${360}^{\circ }$ video for specific applications and/or different VR technologies in education.

Schroeder et al. [63] conducted a review investigating how ${360}^{\circ }$ video influences cognitive learning outcomes. The authors identified 26 studies and report that overall, there was no evidence of benefits or detriments on learning. The authors also looked at specific properties of ${360}^{\circ }$ video, such as interactivity and contextual information, but neither property had a significant effect on learning. These results are in line with our results for short-term and long-term learning retention.

Baysan et al. [67] reviewed the use of ${360}^{\circ }$ video for nursing education. The authors included 12 studies in their review and found that ${360}^{\circ }$ video can improve motivation, confidence, and task performance. A total of 4 of the 12 reviewed studies used a smartphone-based HMD for better accessibility as we did in our study. Only one study compared display options and found that viewing ${360}^{\circ }$ video with an HMD was preferred to using a touch screen, but the authors did not investigate the effect of these options on learning [68].

Rosendahl and Wagner [59] reviewed application areas of ${360}^{\circ }$ video in education. The authors found 44 papers and reported that ${360}^{\circ }$ videos are mainly for three teaching–learning purposes: presentation and observation of teaching–learning content, immersive and interactive theory–practice mediation, and external and self-reflection. Our research focused on learning retention, which is relevant for the first two of these purposes. The authors cite multiple studies that report that viewer engagement increases with the immersion level (e.g., [69,70]) but only Rupp et al. investigated the effect on knowledge retention and claimed that, ${360}^{\circ }$ videos, with the most immersive display, enabled users to remember more verbally presented information [69]. However, the authors did not investigate long-term retention.

Atal et al. reviewed the use of ${360}^{\circ }$ video in teacher education [71]. The authors analysed 17 papers and found that ${360}^{\circ }$ video is a preferred method for overcoming the limitations of standard video, as it offers viewers multiple perspectives and levels of decision-making and enables viewers to notice more details faster due to the ability to look around freely. Furthermore, the authors cite a few studies investigating different display technologies for ${360}^{\circ }$ video. Two studies suggested a potential usefulness of HMDs for perceptual capacity, reflection, and teacher noticing [72,73], and one study showed no added usefulness of HMDs [74].

Muzata et al. [75] reviewed 66 publications and reported that ${360}^{\circ }$ video constitutes one of the most beneficial learning environments due to (1) the low cost of the required equipment; (2) the ability of viewers to employ their expected sensory–motor contingencies, such as head movements; and (3) the encouragement for viewers to use more immersive technologies. VR is preferred when users need to interact with objects and explore them in detail, e.g., in engineering and medicine.

In summary, our study supports and builds upon previous research. Learners enjoy using ${360}^{\circ }$ video for education. There are few usability and accessibility issues. Given a choice, most students prefer using an HMD over a desktop display for viewing. In our study, the display technology had no significant effect on learning, although we did observe higher (non-significant) values when using an HMD. This supports results by Rupp et al, which, in a larger study, found that participants using an HMD remembered more verbally presented information [69]. In contrast to Rupp et al., we also investigated long-term retention, where we again found (non-significant) higher values. This indicates that, while no evidence was found to suggest that using an HMD is better, there is a suggestion that a larger study may reveal such an effect. Similar to previous research, we found that HMDs result in increased enjoyment and engagement, which might have a positive long-term effect on learning [76].

5. Limitations

Novelty is a commonly observed confounding effect in studies involving new learning technologies [77,78], including immersive VR [22]. It can lead to increased motivation or perceived usability of a technology, which may translate into increased attention and engagement in learning activities [7]. The responses to the user evaluation showed evidence of the novelty effect among participants, which should be taken into account with these results. The novelty effect also manifested itself as leniency towards issues with ${360}^{\circ }$ video.

Connected to this novelty effect is the potential for anchoring bias in our study results. Participants will have likely watched many standard videos in the past, giving them high-quality reference points or “anchors” against which to evaluate the study videos. However, they are much less likely to have encountered ${360}^{\circ }$ videos before and, therefore, may not have any expectations of quality in that format. The anchoring effect can lead to more severe criticisms of experiences that can be compared, as well as more lenient evaluations of novel experiences [79].

Another factor that could generate leniency is social desirability bias. This is where survey responses are informed by a participant’s desire to project a favourable image to others [80,81]. In this study, participants may have been less critical of ${360}^{\circ }$ video, aware that we (the researchers) were studying the format, and mistakenly believing we may perceive them undesirably for answering negatively.

As noted previously, the other major limitation of this study is the small sample size. This caused our statistical analysis to have limited power to detect differences in learning outcomes between the HMD and PC groups. Also, most of our user study participants were students or staff from Computer Science and all of them were at least 18 years old.

Future studies comparing ${360}^{\circ }$ and standard video will need larger sample sizes and more diverse user study populations, including children, to conclusively comment on any effects on learning retention in different learning applications. Furthermore, there is a need for more elaborate longitudinal studies assessing the long-term impact on motivation and academic performance.

6. Conclusions

In this paper we presented and analysed the results of a user study to evaluate differences in user experience and short- and long-term learning retention in tertiary students viewing educational videos in both ${360}^{\circ }$ and standard desktop formats. We found that participants retained the same amount of learning from both types of video, but engaged with and enjoyed ${360}^{\circ }$ video more. We also found that participants believed they could study using either format for a similar amount of time and would generally prefer to use ${360}^{\circ }$ video as additional learning materials for their coursework, provided it was accessible.

To create a more comparable experience between the ${360}^{\circ }$ and standard videos used in this study, we placed restrictions on the production of the ${360}^{\circ }$ videos. That means these results were obtained using ${360}^{\circ }$ videos produced in an accessible way, parallel to standard video production. This indicates that ${360}^{\circ }$ video is a viable way for institutions to continue generating value from an investment in immersive VR technology, without having to upskill their current learning content creators or pay for expensive bespoke VR development. We believe that ${360}^{\circ }$ video also offers an attractive alternative to other new learning technologies such as game-based learning, which, while highly engaging, suffers from high content creation costs, vendor lock-in, limited diversity in teaching, and often too much focus on entertainment [82].

User responses also suggested that ${360}^{\circ }$ video has much greater potential for learning engagement if additional effort is put into incorporating the learning content into the surrounding environment. Our results show that at its most accessible level, ${360}^{\circ }$ video is equally effective and more engaging than standard video. However, it is a different medium with a different user experience, creating engagement and enjoyment through a sense of immersion in the wider environment. To unlock the potential of ${360}^{\circ }$ video for education, this environment should be harnessed to reinforce learning objectives.

We see significant potential by combining ${360}^{\circ }$ videos with VR technologies, as demonstrated in previous research [40,42,43,44,45,46]. In an educational context, 360-degree videos can be used in VR to provide realistic backgrounds (skybox) and instructor information or display large-scale information which is difficult to manually model or reconstruct (e.g., archaeological sites), whereas VR content should be used for objects users need to interact with.

Furthermore, ${360}^{\circ }$ video could be useful in learning applications where cognitive load must be measured. Cognitive load measurements can be used for adjusting the difficulty of learning content [83]. However, measuring cognitive load in interactive VR applications is difficult since body movements can interfere with physiological measures of cognitive load [84,85,86]. ${360}^{\circ }$ video could be more useful in such applications since users usually don’t move a lot while watching. However, the fact that students have limited interactions with the video can also be a limitation, since it makes it difficult to measure users’ behavior, which is an important parameter in intelligent tutoring systems [87,88].