Investigating the Potential Use of EEG for the Objective Measurement of Auditory Presence

: Presence is the sense of being in a virtual environment when physically situated in another place. It is one of the key components of the overall virtual reality (VR) experience, as well as other immersive audio applications. However, there is no standardized method for measuring presence. In our previous study, we explored the possibility of using electroencephalography (EEG) to measure presence by using questionnaires as a reference. It was found that an increase in the subjective presence level was correlated with an increase in the theta/beta ratio (an index derived from EEG). In the present study, we re-analyzed the original data and found that the peak alpha frequency (PAF), another EEG index, may also have the potential to reﬂect the change in the subjective presence level. Speciﬁcally, an increase in the subjective presence level was found to be correlated with a decrease in PAF. Together with our previous study, these results indicate the potential use of EEG for the objective measurement of presence in the future. S.Z.; S.Z.; X.F. Y.S.; S.Z.; Y.S.;


Introduction
Presence is the participant's physiological sense of "being there" in the virtual environment while physically situated in another place [1,2]. As one of the factors that contribute most to the overall experience of virtual reality (VR), presence has been investigated in the context of VR in many studies [1,[3][4][5][6][7][8][9][10][11]. In addition to VR, research on presence could also benefit other fields such as video games, audio production, and immersive audio technologies.
Generally, presence is measured using questionnaires, of which the two most widely used are the presence questionnaire (PQ) by Witmer and Singer [2,6] and the SUS questionnaire by Slater, Usoh and Steed [1,3,7]. Subjective measurement based on questionnaires is both valid and reliable since it reflects the subjective evaluation of the participants. As an alternative, researchers have been exploring and improving objective methods of measuring presence. Objective measurement uses physiological signals from the body directly, and can be straightforward and in real time if conducted under strictly controlled conditions. Furthermore, it can prevent the break in presence (BIP) [3,12,13] caused by the act of answering the questionnaires, since participants are no longer required to leave the virtual environment.
Since presence is a sensation generated by the brain, objective measurement could possibly utilize electroencephalography (EEG) as a tool to reveal physiological changes. EEG records brain activities through multiple electrodes placed on the scalp. It has already been used in research on presence [8][9][10][11]14,15] and in other fields [16][17][18][19][20].
In our previous study [21], we explored the possibility of measuring presence objectively using EEG. Two experiments were carried out, in which a loudspeaker array of eight loudspeakers was used to reproduce urban soundscapes as the only stimuli for presence. Subjective measurements (by questionnaires) and objective measurements (by EEG) were both conducted during the process. It was found that the presence score (the result of the questionnaire) and the theta/beta ratio (an index derived from EEG) could both indicate a change in the perceived presence level, and that they were positively correlated with each other.
In addition to the theta/beta ratio, several other indices were also investigated in the previous study, and we observed a dramatic change in the power of the alpha band as the presence level varied. In this study, we therefore re-analyzed the original data, focusing on the alpha band.
Alpha is the dominant oscillation in the brain's EEG. Numerous studies have demonstrated that the alpha band plays an important role in cognitive processing [22][23][24][25]. Peak alpha frequency (PAF) is the predominant frequency of alpha-band oscillations (i.e., the maximal power value in the alpha band), and it can be viewed as a neurophysiological trait of cognitive functions [26][27][28]. It shows inter-individual differences [29][30][31][32][33], and also varies intra-individually with age [34][35][36], fatigue [37], and cognitive load or task demands [38,39]. For this reason, the present study used PAF specifically as an EEG index to investigate the correlation between EEG and subjective presence level.
In summary, the present study investigated the objective measurement of auditory presence using EEG on the basis of the previous study. Specifically, PAF was examined to determine whether it could reflect changes in the perceived presence level.
The rest of the paper is organized as follows. Section 2 explains the methodology of this study, and Section 3 presents the results of the experiments. Following that, Section 4 provides a discussion of the results. The conclusion and suggestions for future work are then presented in Section 5.

Materials and Methods
This study used the same experiment as that described in our previous paper [21]. The previous study focused on investigating the relationship between presence and the theta/beta ratio, whereas this study used the same raw data (i.e., the recorded raw EEG data and the results of questionnaires) to investigate the relationship between presence and another EEG index, the peak alpha frequency (PAF). Therefore, this section summarizes the experimental design while focusing on introducing the specific method for acquiring PAF. Details about the other aspects of the experimental design can be found in [21].
This study used ESMA-3D Immersive Soundscape Recordings by Dr. Hyunkook Lee's team [40] as programs to stimulate presence. Due to its nature, the 8-channel signal was reproduced by an 8-channel double-layer quad-speaker array (i.e., cube array) [41] in the room, as shown in Figure 1. No visual stimuli were used in this study. To shorten the duration of the experiment, 6 out of 13 scenes were selected from the original file. With each scene lasting 30 s, the overall duration of the program was three minutes.
As well as the original program described above (denoted Program A), two processed versions of it were also used to stimulate different levels of presence. Program B was created by combining signals from all eight channels of Program A and copying the sum to each channel. Since each channel had the same content, we should expect Program B to have a similar effect as a mono sound file. Program C was derived from Program A by muting the upper four height channels. Thus, the ESMA-3D degenerated into ESMA without height channels. We assumed that these three programs would stimulate presence levels in the following order: A > C > B. These three programs were then used in the subsequent experiments. The study was composed of two experiments. Experiment 1 used Programs A and B for comparison, involving a total of 18 experienced listeners from the lab (16 males and 2 females, ages ranging from 23 to 27 years old), and Experiment 2 used Programs A and C for comparison, involving a total of 25 naïve listeners (8 males and 17 females, ranging in age from 21 to 28 years old).
During each experiment, two programs were presented in succession. In the meantime, EEG was recorded using a portable EEG device (Emotiv EPOC X [42]). After that, participants were required to fill in the questionnaire at the end of the experiment. The questionnaire used in the study contained four questions as follows:

1.
Please rate your sense of being in the virtual environment on a scale of 0 to 10, where 10 represents your normal experience of being in a place; 2.
During your experience, did you often think to yourself that you were actually in the virtual environment? Please rate it on a scale of 0 to 10, where 10 represents you almost felt you were actually in the virtual environment; 3.
How well could you identify sounds? Please rate it on a scale of 0 to 10, where 10 represents you could clearly identify different kinds of sounds; 4.
How well could you localize sounds? Please rate it on a scale of 0 to 10, where 10 represents you could easily detect the location of each sound.
The first two questions were taken from the SUS questionnaire. The first question focuses on assessing how closely the virtual environment resembles the real world, and the second question aims to evaluate the participants' subjective impressions of their experience within the virtual environment. The last two questions were chosen from the presence questionnaire to evaluate the realism of the sound field of the virtual environment. The final presence score was the sum of the answers to these four questions. As the questions suggest, the higher the presence score, the greater the level of presence.
Thus, the subjective assessment of presence (represented by the presence score) and the objective measurement (represented by EEG) were both acquired. More detailed descriptions of the program selection, the experimental design, and the EEG apparatus can be found in [21].
The raw EEG data were preprocessed in MATLAB using a brain signal processing toolbox called EEGLAB [43] in the following steps. A bandpass filter of 1-50 Hz was first applied to the EEG data to remove the noise. The filtered data were further cleaned via a built-in plugin of EEGLAB called Clean Rawdata, to remove non-physiological artifacts (e.g., insufficient contact of electrodes with the head surface or sudden large movements). Then, physiological artifacts such as heart beat (ECG), eye movements (EOG), and muscle pulsations (EMG) were removed by running an independent component analysis (ICA) and deploying an automatic classification of artifactual ICA components plugin called MARA ("Multiple Artifact Rejection Algorithm") [44,45].
Finally, the peak alpha frequency (PAF) was calculated using the "restingIAF package" [46]. This is an open-source package developed by Corcoran et al. for the estimation of individual alpha frequencies. This algorithm first calculates the power spectral density (PSD) of each channel and then deploys the Savitzky-Golay filter to smooth out the signal noise while preserving the main alpha peaks. Then, peaks within each channel are extracted and averaged based on their peak quality to obtain the final estimate of the peak alpha frequency [47].

Experiment 1
First, data for 2 of the 18 participants were excluded since they reported being distracted during the experiment. Then, data for another 3 participants were excluded by the algorithm due to the strict criteria for estimating the PAF. The algorithm requires the following conditions to be met to produce a valid estimate of PAF: (1) the peak frequency must appear within the alpha band's frequency range; (2) the peak power must exceed the minimum amount of normalized power estimated by the algorithm to qualify as a potential PAF candidate; (3) the peak's height must exceed all other competitors by at least 20% ; (4) at least three channels should yield valid PAF estimates, in order to calculate cross-channel averages [47]. Thus, 13 sets of valid data were collected, as shown in Table 1. It can be observed that the presence scores for Program A were generally greater than those for Program B. A paired t-test was carried out, which showed that the difference in presence scores was significant at a significance level of 0.05 (p = 1.98 × 10 −4 ). In contrast, the PAF estimates for Program A were generally lower than those for Program B. The results of the paired t-test also demonstrated that the PAF estimates for Programs A and B were significantly different (p = 0.040). That is, when participants subjectively experienced higher levels of presence, they tended to show a lower PAF.
To further investigate the correlation between the change in presence score and PAF, a McNemar test was conducted. This is a non-parametric statistical test for paired data within the scope of chi-squared tests [48]. The data were first transformed into a dichotomous format, as shown in Table 2. The transformation rule was that if the presence score for Program A was greater than that for Program B, it would be marked as 1, and otherwise it would be marked as 0. Likewise, if the PAF for Program A was lower than that for Program B, it would be marked as 1. Please note that the transformation rules for presence score and PAF were different to account for their opposite change direction. In this way, the McNemar test was conducted and revealed no significant differences (p = 1.000). Furthermore, the Pearson correlation coefficient was 0.677 (p = 0.011). Both these results indicated that if the presence score for Program A was higher than that for Program B, it was highly likely that the PAF for Program A was lower than that for Program B. That is, using the presence score and the PAF to determine the level of presence yielded nearly the same results. Another statistical metric, Cohen's kappa, could also be used. In general, this is used to determine whether different observers would reach the same conclusion. Kappa is standardized to a scale of −1 to 1. Kappa values of 1 indicate perfect agreement, kappa values of 0 indicate agreement equivalent to chance, and negative values indicate agreement less than chance [49]. The results showed kappa = 0.629 (p = 0.015), which indicated that there was an agreement between these two criteria (presence score and PAF) for determining the level of presence.

Experiment 2
In Experiment 2, data for one participant were initially ruled out since she stated herself that she could not discriminate between the two programs. Then, data for eight participants were excluded during the estimation of the PAF for the reason outlined in Section 3.1. Thus, 16 sets of valid data were collected, as shown in Table 3 (the dichotomous  data are not shown in this table).
It can be observed from Table 3 that the trend that "presence score for Program A was higher than that for the other program, and the PAF for Program A was lower than that for the other program" from Experiment 1 also held true in Experiment 2, though it was not as evident as in Experiment 1. This accords with the fact that the difference between Programs A and C was smaller than that between Programs A and B; that is, Experiment 2 had a more stringent condition than Experiment 1. A paired t-test was carried out and showed a significant difference in the presence score between the two programs (p = 0.016). The difference in the PAF between the two programs was substantial and close to significant (p = 0.051). Similarly, the correlation between the change in presence score and the PAF was examined using the McNemar test, and no significant difference was found (p = 1.000). The Pearson correlation coefficient was 0.590 (p = 0.016) and the value of kappa was 0.590 (p = 0.018). These results indicated that the correlation observed in Experiment 1 also applied to Experiment 2. That is, using the presence score (subjective measurement) and the PAF (objective measurement) to determine the level of presence yielded significantly similar results.

Discussion
In Section 3, it was found that an increase in the presence score was correlated with a decrease in the PAF. The underlying neurological mechanism of this correlation remains unclear. However, we found some clues that might shed light on research into this mechanism.
It was found in the previous study that an increase in the subjective presence level was correlated with an increase in the theta/beta ratio [21]. Neurologically, both the theta/beta ratio and the alpha-band oscillations have been demonstrated to be associated with attentional processes [24,[50][51][52]. Wittmer and Singer [2] stated that a person's presence in a virtual environment depends on them shifting their attention from the physical environment to the virtual environment. Hence, it is likely that changes in the theta/beta ratio and the PAF are closely related to changes in attention.
In addition, the theta/beta ratio was found to be negatively correlated with the PAF [53,54]. Combined with the results of [21] showing that the presence level is positively correlated with the theta/beta ratio, it can be deduced that the presence level is negatively correlated with the PAF, which is consistent with the findings of the present study. Furthermore, an increase in theta/beta ratio and a decrease in PAF were observed simultaneously in patients with attention deficit hyperactivity disorder (ADHD) [32,33], which further supports the results of this study. Considering that there are relatively few studies that investigate auditory presence exclusively, particularly using EEG, it is useful to note that the findings of these two studies are in agreement with each other.
It is interesting to note that an increase in PAF has been found to be associated with good performance [35,55,56] or an increase in cognitive load (or task demand) [38,39]. We assume by intuition that when the program that stimulated greater presence was played, the participants' attention would rise and the cognitive load would also increase. However, the results would be just the opposite of those in this study if they were built on this assumption. We suppose this discrepancy arises from the difference between the brain's top-down processing system and bottom-up processing system, which is reflected in the experimental design. Top-down processing is related to directed attention, which focuses on task-relevant information and inhibits the processing of non-essential information [57]. In the above-mentioned studies, the participants were required to complete tasks with various difficulties, where a better performance (i.e., better memory or shorter reaction time) should result from an increase in intentional attention. In other words, the causal relationship should be "the more attention assigned to the tasks, the higher the cognitive load, and consequently the higher the PAF". Furthermore, alpha oscillations have been found to be associated with top-down processing [57,58], which may account for the relationship between PAF and task-relevant performance. On the other hand, bottom-up processing is related to involuntary attention (i.e., driven purely by the sensory input) [59]. As in the present study, there were no complicated tasks that required the participants to intentionally focus their attention in order to complete them. The change in presence should instead be the natural cause of the change in attention. This might explain the discrepancy between the results of this study and the previous studies. Nevertheless, the neurological mechanism of the change in EEG induced by presence should be investigated in future research.

Conclusions
The present study investigated the objective measurement of auditory presence using EEG, on the basis of the previous study. It was found in the previous study that an increase in the subjective presence level was correlated with an increase in the theta/beta ratio, while in this study it was found that an increase in the subjective presence level was correlated with a decrease in the peak alpha frequency (PAF). These results indicate that EEG is likely to reflect changes in the perceived presence level. Though limited with regard to the number of participants, these findings may shed light on research into the objective measurement of presence. Nevertheless, the neurological mechanism underlying the correlation between presence and EEG should be addressed in future work. In addition, future work should involve more participants and possibly develop a model of presence level estimation using EEG indices.

Acknowledgments:
The authors would like to thank all listeners who participated in the listening test.

Conflicts of Interest:
The authors declare no conflict of interest.

Abbreviations
The following abbreviations are used in this manuscript: EEG electroencephalography PAF peak alpha frequency VR virtual reality