Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience

Abstract

Virtual reality is increasingly used for tasks such as work and education. Thus, rendering scenarios that do not interfere with such goals and deplete user experience are becoming progressively more relevant. We present a physiologically adaptive system that optimizes the virtual environment based on physiological arousal, i.e., electrodermal activity. We investigated the usability of the adaptive system in a simulated social virtual reality scenario. Participants completed an n-back task (primary) and a visual detection (secondary) task. Here, we adapted the visual complexity of the secondary task in the form of the number of non-player characters of the secondary task to accomplish the primary task. We show that an adaptive virtual reality can improve users’ comfort by adapting to physiological arousal regarding the task complexity. Our findings suggest that physiologically adaptive virtual reality systems can improve users’ experience in a wide range of scenarios.

1. Introduction

Virtual Reality (VR) has, in the last decade, grown from being a niche technology to one that we see employed across diverse application domains, including work [1], training [2], and education [3]. VR can provide an immersive simulation of the physical environment enriched by digital artifacts, providing more control over environmental variables. On the one hand, developers can achieve higher realism by manipulating environmental parameters, such as increasing the number of polygons, texture details, lighting realism, and animation [4]. This even allows for creating realistic Non-player characters (NPCs), such as pedestrians or co-workers, with human-like agent behavior. Thus, adding a social component to the virtual environment [5]. On the other hand, increasing the immersive properties of VR simulations could introduce complexity that requires more user effort to process [5,6,7,8]. This could diminish users’ cognitive spare capacity to execute primary tasks. In other words, highly immersive VR scenarios could overload the user, potentially hindering their ability to achieve their objectives by distracting, or even overwhelming, the user.

Researchers have shown that experienced cognitive workload can influence human physiological activity [2,9,10,11]. For instance, when we perform cognitively demanding tasks, the physiological arousal is heightened, manifesting in increased physiological activity such as skin conductance level (SCL) [12], heart rate [13], or pupil dilation [14]. This has given rise to the plausible notion that VR complexity can be adapted in real-time [15] in response to the user’s physiological activity in order to maintain an optimal physiological arousal level and, hence, the optimal performance level [16] within the VR scenario.

This aligns with the Motivational Intensity Model (MIM), which describes a ’tipping point’ where allocated cognitive resources decline with overload or unachievable demands [17]. The MIM model theorizes that increased cognitive effort due to increased task demands results in increased engagement until the user reaches overload, and task disengagement occurs. Here, physiological measures of mental effort can detect this [18,19]. With this in mind, biocybernetic control loops could prevent users from reaching cognitive under- or overload by dynamically adjusting the source that induces arousal [20]. Such systems could accept real-time inputs from physiological sensors, such as electroencephalography (EEG) [21], electrodermal activity (EDA) [22], or electrocardiography (ECG) [21,23], and adapt task difficulty accordingly. EEG [24] and ECG [25] contributed to the physiological underpinnings of MIM within biocybernetics loops. Pope et al. [26] developed a pioneering version of such a system in which EEG signals were exploited to adapt the automation level in a simulated aviation task to support task engagement and avoid attentional drops. Muñoz et al. [27] implemented a training scenario to support the users’ readiness level by adapting the task difficulty based on the computation of R-to-R intervals and ECG frequency domain parameters. Although EDA is relatively unexplored, in our previous work [28] on which this work is based on, we presented an initial investigation into real-time adaptation exclusively using EDA in a Social VR scenario of which this paper presents a more in-depth analysis.

Social environments are an emerging trend in VR that is especially critical in supporting remote collaborations [5,29,30]. Social VR leverages the flexibility of VR to enable multiple players and NPCs to interact. The inclusion of multiple agents can increase environmental complexity. In addition, it can also lead to social crowdedness, whereby the violations of social norms (e.g., personal space) [31,32,33] could induce heightened physiological arousal [34,35]. In the following, we use social crowdedness in a Social VR environment to induce arousal [32,36]. Such environments typically have a primary task, e.g., learning, training, or group conversations, while other characters perform their task nearby. Here, we present an online physiologically-adaptive VR system, which adjusts the secondary task based on the users’ EDA, allowing for optimal performance of the primary task. In detail, the user is performing an n-back task as the primary task, simulating a high cognitive load such as training or work. The secondary task is to engage with a stream of non-player characters bypassing in close proximity. By adapting the stream (number of non-player characters per minute), we change the visual complexity, which is itself linked to task difficulty [6,7].

In this paper, we present an evaluation of our novel physiologically adaptive VR system that aims to increase user satisfaction by optimizing the secondary task’s complexity to maintain an optimal physiological arousal state. Our system recognized changes in EDA, which we treat as an index for arousal and cognitive workload. Based on this, the system then adapts the stream of non-player characters of a secondary task in real-time to minimize EDA changes. With this, we make the following contributions: (1) we investigated the feasibility and usability of VR adaptation based on EDA changes; (2) we show that online adaptation of the VR environment improves the user’s perceived workload compared to a non-adaptive system; and (3) we provide the recorded dataset of behavioral performance as well as the EDA, EEG, and ECG physiological signals. With this, our study offers evidence that physiologically adaptive VR environments can improve the experience by reducing the complexity of the surroundings, which allows the user to focus on the primary task.

2. Related Work

In the following, we review three areas essential for this work: (1) current social virtual reality experiences; (2) electrodermal activity and its link to cognitive workload; and (3) previous physiologically adaptive systems.

2.1. Social Virtual Reality

Social VR is an emerging research topic [29,37,38] with many facets and applications. Generally, all instances of Social VR support social interactions between multiple users [39] for an initial application taxonomy. Current investigations mainly focus on how the user characters should be displayed [5,40,41] and how to facilitate user interactions with each other [42], which are considered critical factors for a good user experience [43]. Established findings from real-world proxemics often generalize Social VR experiences [29,44]. Hecht et al. [44] could show that the personal space is shaped circularly in physical and virtual environments. McVeigh-Schultz et al. [45] could show that establishing boundaries in Social VR is important [31]. Thus, they need interactions that are genuine to Social VR, e.g., by restraining close-by VR teleportation. Therefore, characters respecting a personal space can be considered as important in VR as it is in physical spaces, and Social VR needs interaction concepts that can take into account human social capabilities to enable a satisfactory user experience.

Note, however, that not only proxemics is relevant, but also that the level of crowdedness affects how people feel with respect to their environment [32]. In VR, crowd simulation studies show that people restrain their space when walking in a crowd [46]. Dickinson et al. [47] show that crowdedness produces negative feelings and can, thus, disrupt the user experience in Social VR. Moreover, fine-grained experimental work from Llobera et al. [36] can put this into context by experimentally varying the number of NPCs approaching in VR. Here, four characters approaching produces more physiological arousal compared to one. In line with this, Latoschik et al. [5] describe that the number of users in the VR scene impacts subjective performance such as fluidity, synchrony, and, more importantly, annoyance in the user when displaying more than 25 social avatars.

While it is clear that crowded Social VR scenarios can increase arousal and affect the user experience, there are no interaction concepts that can mitigate this effect or define what the optimal number of characters is per unit of virtual space. Nevertheless, from a social-crowding perspective, we can infer that more users in a scene increase the user’s arousal, which, in turn, induces overload [32,36,47]. Finally, crowdedness inherently raises the visual complexity of the scene, defined as the amount of detail, clutter, and objects in a scene [48].

2.2. Electrodermal Activity and Cognitive Workload

When cognitive resources are allocated, they impact physiological arousal in the form of increased activation of the sympathetic branch of the autonomic nervous system [49,50]. This allows for the implicit evaluation of the demands posed by a VR environment to users [51]. Across sympathetic peripheral physiological measures, EDA represents a noninvasive, easy-to-use, and robust method for detecting the effect of cognitive workload on arousal [52]. It provides information related to availability or decrease in efficiency of cognitive resources such as workload detection [53], task engagement [54], and stress [51]. In particular, the relationship between workload, engagement, and arousal is consistent with the MIM [17], where optimal task engagement correlates with increased arousal state just before it drops, resulting in disengagement of the task at hand, cognitive overload, and hypoarousal [55,56,57]. As an index of physiological arousal, EDA is characterized by two components: phasic and tonic. Phasic activity is indexed by rapid skin conductance responses (SCRs) reflecting discrete and stimulus-specific responses for the measurement of novelty, significance, and intensity of the stimuli used [58]. Tonic activity has a slower and increased response; it is an index of the general state of parasympathetic activation.

Measures of tonic activity are particularly suitable for assessing the effect of continuous stimuli (such as long-duration stimulations or tasks) and have, therefore, been used to assess changes in arousal under conditions of high cognitive load or stress in HCI. Son and Park [59] estimated a driver’s cognitive workload with SCL that correlated with behavioral performance. Mehler et al. [60] showed how SCL correlated with changes in cognitive workload at different points in a demand curve, reflecting the spare capacity of participants’ cognitive resources. Fairclough and Venables [12] reported that high stress increased sympathetic activity, as indexed by tonic activity, in a high-demand multi-component task. Similarly, when increasing task load to an effective task in immersive virtual reality, tonic activity increased in conditions of high workload [61]. In conclusion, EDA can be widely applied to physiologically estimate and monitor a user’s workload [62].

2.3. Physiological Adaptation

A physiologically adaptive system can monitor its user’s physiological activity and iteratively adapt its interactions with the user to optimize some aspect of user performance or user experience. This approach is inspired by classic control theory [63], which encompasses physiological data acquisition and processing (i), transformation into a system response (ii), which then shapes the future or expected psychophysiological response from the user (iii). Specifically, biocybernetic control loops employ negative-control loops, where deviations from the optimal state are detected and exploited to cue changes in the system to promote a desirable user’s state [64]. Biocybernetic loops found applications in different domains, such as cockpit automation [65], computer-based learning [66], and robotics [67]. Nevertheless, VR applications are relatively overlooked, with a small number of exceptions, e.g., [68]. For instance, Stach et al. [69] inferred and supported the user’s motivation from Heart Rate Variability to provide an optimum exercise protocol in an immersive exergame. Alternately, adapting task difficulty based on peripheral and central measures of arousal can support task engagement in a Virtual Reality Stroop Task (VRST) [70] or in a virtual object selection task to increase training effects based on EEG alpha frequency [71].

In the context of the cognitive workload, increasing visual complexity can induce cognitive overload, which is reflected by increased physiological arousal [72,73,74]. Thus, adapting the environment to optimize the user’s arousal can support the user in processing information and executing actions. Visual complexity adaptation based on users’ sympathetic arousal can support the user experience in various VR applications, ranging from simulator sickness [75] to VR productivity and interruption management [76].

The goal of this paper is to evaluate if online dynamic adjustments of visual complexity based on a peripheral physiological measure of arousal can support user experience and performance as compared to stable levels of visual complexity in a Social VR scenario.

3. Online Physiologically-Adaptive System

Here, we propose an online adaptation of virtual environments based on EDA. More specifically, we developed a real-time physiologically adaptive system based on the user’s arousal as indexed by EDA. This system determines if its user can handle more visual complexity or reduce complexity to optimize cognitive load. Consistent with prior literature [77], the online signal processing pipeline derives a mean EDA value by applying a 20-s median moving-average filter that replaced each data point with the average of its 20 neighboring data points.

To enable the user-dependent adaptation of the system, the physiologically-adaptive system is initialized with a baseline recording $b_0\cdots b_i$. For later baseline comparison, we calculate $\mathsf{\Delta }b$ as follows (Note: we denoted the mean value as $\overline{x}$):

$$\mathsf{\Delta }b=\overline{b_{j-i}\cdots b_i}-\overline{b_0\cdots b_i},$$

(1)

where i is $\mathit{sampleRate}\ast \mathit{windowBaseline}$ and j is $\mathit{sampleRate}\ast \mathit{timeBaseline}$. The baseline period is used to compute the baseline slope. In our implementation, this is the slope between the averaged values of the first and last 20 s over the three minutes using the aforementioned signal processing pipeline.

$$\mathsf{\Delta }s=\overline{s_{j-i}\cdots s_i}-\overline{s_0\cdots s_i},$$

(2)

where i is $\mathit{sampleRate}\ast \mathit{windowOnline}$ and j is $-\mathit{sampleRate}\ast \mathit{window}$. Thus, this determines the current slope of physiological arousal. Due to the other external influences, such as baseline arousal of the virtual environment, this alone is not sufficient for adaptation.

In the final step to determine how to adapt the virtual environment, we compare the baseline measurement $\mathsf{\Delta }b$ to the online measurement $\mathsf{\Delta }s$. The difference then determines how to adapt the environment, which we define as follows:

$$adaptation\left(\mathsf{\Delta }s\right)=\left\{\begin{array}{cc}\mathrm{increase}\mathrm{of}\mathrm{Stream}\hfill & \mathrm{if}\mathsf{\Delta }s\le \mathsf{\Delta }b-t\hfill \\ \mathrm{decrease}\mathrm{of}\mathrm{Stream}\hfill & \mathrm{if}\mathsf{\Delta }s\ge \mathsf{\Delta }b+t\hfill \end{array}\right.$$

(3)

where t is the threshold parameter that enables the physiological-adaptation system to work with a delay, counteracting high-frequency changes in EDA and, thus, preventing rapid and unstable adaptations. Moreover, for a stable adaptation, Equation (3) is only used for adaptation every 20 s.

4. User Study

We conducted a study to evaluate whether our physiologically adaptive system can support users’ comfort and usability in the virtual environment. To do so, participants performed a visual working memory n-back task (primary task) [78] and a visual detection task (secondary task). Both tasks were simulated as real-world activities in a virtual environment. While the primary task simulates a high workload task, the secondary task simulates situations where the user is distracted by, for instance, other players [38], notifications [79], or graphical artifacts [80].

In the n-back (i.e., 1-back) task, participants determined if the color of a presented sphere matched the color of the last presented sphere, see Figure 1. Participants were presented with colored spheres that they had to place in either a left or right bucket depending on their color (mis)match with the previous sphere. Spheres were either green, red, blue, or black [81]. A sound cued the appearance of each sphere. Then the participant had to pick up the sphere within 4 s; otherwise, it would count as an error. New spheres would appear either after 4 s or when the current sphere was placed into one of the two buckets. We counted missing a sphere as an error.

In the secondary tasks, we asked participants to inspect the museum tickets of a Stream of virtual NPCs. In detail, participants were required to distinguish museum visitors who had a ticket either in their left or right hand from those without a ticket. Museum visitors were represented by virtual NPCs that were walking past the participant on either the left or right side. We set the percentage of virtual NPCs who have no ticket to be 15%, which we found to be a suitable value in the informal pilot tests for this task. These had to be identified by clicking on them with the right input controller. All NPCs had randomized visual characteristics (i.e., hair, shirt color), and their shirt color turned red when clicked on. The NPCs’ distance from the user was kept at a minimum of 2.5 m to avoid proxemics confounds with the level of arousal of participants [36].

The five non-adaptive conditions had fixed Streams of 7, 22, 37, 52, or 67 NPCs per minute entering. In the adaptive condition, the system adapted the number of NPCs and added more NPCs when the physiological activity of the user did not indicate physiological arousal or removed NPCs. We randomized the six test conditions. Participants were not aware whether they would experience the adaptive condition or one of the non-adaptive conditions.

4.1. Participants

Eighteen volunteers ($M_{range}=23-31$; $M_{age}=27.9$, $S{D}_{age}=2.9$; $\mathit{male}=9$, $\mathit{female}=9$) participated in our study, of which we had to exclude three from the analysis as the EDA electrodes lost contact. We recruited the participants using our institutional mailing lists and social networks and using convenient sampling. Exclusion criteria required participants to not experience intense physical activity or consume any caffeine or nicotine in the 3-h pre-study period [82]. None of the participants reported a history of neurological, psychological, or psychiatric symptoms. They were also required to submit a negative test for SARS-CoV-2 within 48 h prior to participation.

4.2. Apparatus

We designed the virtual environment for the study in Unity (Version 2019.4.24f1), see Figure 1, and presented it via an HTC VIVE VR headset with a display resolution of $2160\times 1200$ pixels, refresh rate of 90Hz, and an average field of view of 110°. The environment used for the experiment is a replica of the Neue Nationalgalerie in Berlin, Germany. The 1-back task takes place in the entrance of the building, see Figure 1 and, for the detection task, agents with and without tickets in their hands entered the entrance.

We performed the EDA data recording according to the guidelines by Babaei et al. [82]. We placed Ag/AgCl electrodes (7 mm surface diameter) on the participant’s non-dominant hand (inner distal phalanges of the index and middle fingers). An electrolyte solution (0.5% NaCl) was applied to the acquisition site to ensure proper hydration and to minimize the effect of individual differences. We attached the electrodes 10 min later, with double-sided adhesive collars. We exploited the exosomatic recording principle with a direct current (DC) of a constant 0.5 V voltage.

We used Equation (3) to adjust the number of NPCs in the adaptive condition. In detail, when the EDA slope computed in the 20-s window was greater than the baseline slope added to the threshold slope, two NPCs were removed from the scene. On the contrary, if the EDA slope was lower, four NPCs were additionally spawned. A fixed range of 7–67 NPCs was maintained regardless of the participants’ measured physiological activity. These settings were determined empirically during a number of test sessions.

4.3. Measurements

We recorded three physiological measurements: (1) EDA signal via the LiveAmp amplifier (BrainProducts GmbH, Germany), using a 250Hz sampling rate; (2) ECG (Polar H10 chest strap, Finland) at 130 Hz; and (3) EEG signal (DSI-VR 300, Wearable Sensing, San Diego, CA, USA) at 250 Hz. Physiological data were streamed and recorded within the Lab Streaming Layer framework (LSL) (https://github.com/labstreaminglayer/, accessed on 1 April 2022). We only used the EDA signal as an indicator for physiological arousal. In addition, performance accuracy metrics were computed for both tasks. For the 1-back task, errors were represented by the proportion of times the sphere was placed in the wrong bucket. For the visual detection task, errors were represented by missing an NPC or selecting an NPC with a ticket.

Responses on three standardized questionnaires and two custom Likert items were collected to evaluate user experience and workload. First, raw subjective workload measures (NASA-TLX [83,84]). Second, perceived gamefulness of system use (Game Experience Questionnaire (in-Game Core Module) [85]). Here, we only recorded the subscales on Competence, Immersion, and Positive Affection allowing for content validity [86]. Third, the Fast Motion Sickness scale (FMS) to control for motion sickness [87]. Finally, participants rated two general usability statements on a 5-point Likert scale (strongly disagree - strongly agree); questions: “I would like to use the system in the future,” and “The flow of the people was appropriate”.

4.4. Procedure

Upon arrival, we briefed the participants on the study procedure, and we answered any open questions, which were followed by signing the informed consent form. Next, they performed a Snellen visual acuity test. Finally, the EDA sensor, EEG-VR headset, and ECG chest strap were attached to the participants.

The study began with a trial phase to allow participants to familiarize themselves with the VR environment. The VR trial phase started with participants practicing the 1-back task until they reached an accuracy level of at least 95% within a sequence of 80 spheres. Next, three minutes of EDA recording was performed, during which the participant was asked to ignore the NPCs (Stream: 37 agents/min) and to perform the 1-back task. This provided the baseline measurement for the physiologically-adaptive system.

The testing phase consisted of six blocks for each test condition. Each block lasted for six minutes. Visual feedback for the 1-back task was provided, and participants were instructed to maintain 90% performance. In between blocks, participants filled in questionnaires (FMS, NASA-TLX, In-game GEQ subscales, ad-hoc questionnaires) on the previous block and rested for 2 min to stabilize their physiological state. The entire experiment lasted approximately one hour. We compensated participants with 10 Euros for their participation.

5. Results

We evaluated the usability of a physiologically adaptive system in a Social VR scenario. In the following section, we report the results of our study. As no participant aborted due to simulator sickness, however, electrodes were not attached correctly for three participants; the following analysis is based on the data of 15 participants. Results of the analyses can be found online on the DaRUS Open Data Platform, at [88].

5.1. Analysis

We analyzed indicators of physiological arousal, performance, and subjective experience across the adaptive and non-adaptive conditions in R [89,90,91]. First, we present the results for the five non-physiologically adaptive conditions and present a comparative analysis of the adaptive condition to determine if the physiologically adaptive condition produces: (1) superior performance; (2) a more enjoyable game experience; and (3) lower levels of perceived workload. We fit a linear mixed model (estimated using a restricted maximum likelihood approach and nloptwrap optimizer) [92,93] to predict our outcomes as a function of Stream. We computed the p-values using the Wald-Approximation for the calculation of degrees of freedom. To account for the repeated-measures structure in our data, we added a random intercept for every participant to our model. We use Welsh-corrected t-tests for comparing means or Wilcoxon-rank tests for ranks.

5.2. Non-Adaptive Conditions

For mean EDA, the model intercept is at −0.16 (95% CI [−0.67, 0.36], $t\left(71\right)=-0.60$, $p=0.551$). Within this model, the effect of Stream is statistically significant and positive (beta = 0.004, 95% CI [0.00, 0.007], $t\left(71\right)=2.54$, $p<0.01$. In other words, every additional NPC increases mean EDA by about 0.004 standardized µS units (see Figure 2). This is consistent with the raw NASA-TLX score, see

Table 1. Means across non-adaptive conditions with the slope of linear mixed models (LMM) and their t-values estimated by Wald approximation, as well as their respective p-value.

	Stream 7		Stream 22		Stream 37		Stream 52		Stream 67		LMM
	M	SD	M	SD	M	SD	M	SD	M	SD	b	t	p
n-back Acc. [%].	97.423	1.607	95.381	2.660	94.401	2.430	92.133	3.182	91.055	4.162	−0.107	−9.396	<0.001
Visual Det. Acc. [%]	97.208	3.073	95.579	4.647	94.834	2.670	93.067	3.116	90.949	2.613	−0.100	−7.083	<0.001
Raw NASA TLX	5.267	2.520	7.644	2.268	9.500	3.616	9.133	3.282	11.833	3.262	0.097	8.101	<0.001
EDA Mean (std)	−0.091	1.031	−0.083	1.001	−0.034	0.896	0.003	1.083	0.166	1.056	0.004	2.536	0.014
SCL	9.078	2.974	9.102	2.888	9.242	2.585	9.348	3.127	9.819	3.047	0.012	2.535	0.014
GEQ—Competence	2.667	0.939	2.600	1.039	2.200	0.862	2.600	0.687	1.967	0.834	−0.009	−2.149	0.036
GEQ—Pos. Affec.	2.400	0.806	2.500	0.655	2.467	0.550	2.433	0.729	2.100	0.761	−0.004	−1.318	0.193
GEQ—Immersion	0.833	0.724	1.200	0.841	1.133	0.876	1.067	0.821	1.167	0.939	0.004	0.906	0.369
Stream Appropriate	1.400	0.632	1.600	0.737	2.000	0.756	1.933	0.799	1.467	0.743	0.003	0.754	0.454
Desire To Use	1.467	0.640	1.800	0.775	1.800	0.941	2.267	0.799	1.333	0.617	0.001	0.309	0.759

. Increasing the number of NPCs in the simulation elevated the NASA-TLX score by about 0.1 per NPC. In contrast, Stream affected performance accuracy negatively in both tasks, see again Table 1. Increasing Stream reduced accuracy up to 10% in both the n-Back and the visual detection task. Together with this, we supplement our analysis with a SCL computation as a measure of tonic activity. To compute SCL, EDA data were filtered with a 3Hz, high-pass, fourth-order Butterworth filter to remove high-frequency noise and decomposed into tonic and phasic components by means of a non-negative deconvolution analysis [94]. For SCL, the model intercept is at 8.89 (95% CI [7.40, 10.39], $t\left(71\right)=11.86$, p < 0.001). The effect of Stream is statistically significant and positive (beta = 0.01, 95% CI [0.00, 0.02], $t\left(71\right)=2.54$, $p<0.013$. This result implies that the addition of every NPC increases SCL by about 0.01 standardized µS units. We found that the subjective experience scales (GEQ Immersion, GEQ Competence, GEQ Positive affect, desire to use, and Appropriateness of Stream) were not significantly by our manipulation for all $p>0.05$, see Table 1.

5.3. Adaptive Condition

On average, participants adapted to having a Stream of 32.89 ($\mathit{SD}=18.6$) NPCs in their environment. This represents a medium level of Stream, relative to the fixed range (i.e., 7-67) that was adopted for the non-adaptive conditions. Figure 3 illustrates how Stream adapted over time for one participant.

The recorded measures from the non-adaptive conditions cannot be directly compared to their counterpart in the physiologically adaptive condition. This is because Stream was a fixed value in the former and an adapted value in the latter. Thus, we computed individual expected values from the five non-adaptive conditions using linear mixed models with Stream as a predictor and calculated the expected value as predicted from our statistical model of the non-adaptive conditions, see Table 2 in the Prediction column and at the local maxima in Stream for the adaptive condition.

In other words, any aspects of the user experience that are different in the physiologically adaptive condition compared to the non-adaptive conditions would be revealed if the actual measurement in the condition is significantly higher or lower than the corresponding value that would be predicted from a regression of the measurements of the non-adaptive conditions.

5.3.1. Electrodermal Activity Measures

The physiologically adaptive condition did not significantly increase physiological arousal in either the EDA mean or SCL according to the linear mixed model. Individual regression predicted a mean EDA of -0.023 µS ($\mathit{SD}=0.957$), while in the adaptive condition, the actual mean EDA was 0.040 µS ($\mathit{SD}=1.074$) after a stable Stream was achieved, see Figure 4b. Similarly, the adaptive algorithm did not significantly increase tonic arousal as indexed by SCL. Actual SCL was 9.456 µS ($\mathit{SD}=3.100$), reflecting predicted values of 9.275 µS ($\mathit{SD}=2.761$), see Figure 4c. SCL and EDA results are summarized in Table 2.

5.3.2. Workload and Performance

The physiologically adaptive condition did not increase subjective workload. Individual regressions predicted a mean raw NASA-TLX score of 8.31 ($\mathit{SD}=2.25$), while workload was 8.57 ($\mathit{SD}=3.20$) in the adaptive condition after a stable Stream was achieved, see Figure 2 and Figure 4 and Table 2. This was mirrored for both the performances in the detection task and the n-back task. The adaptive algorithm regulated performance based on the individual linear fit to a local optimum and, thus, we found no significant differences between the expected and actual performance for the adaptive condition as seen in Figure 5 and Table 2.

Figure 6 illustrates the negative relationship between workload and mean EDA. The algorithm allowed participants to perform tasks at their individual optimal levels of physiological arousal. Participants with relatively higher mean EDAs experience lower subjective workload $r\left(13\right)=-0.62$, $p=0.013$. This shows the successful adaptation of the algorithm and that participants approached an individual local maxima in perceived workload.

Table 2. Mean predicted and actual means across measures with Welsh-corrected t-test or Wilcoxon-signed-rank test depending on the Shapiro test and Cohen’s d. We compare the predicted value from the LMM at an optimized Stream with the actual value at this rate of Stream.

	Actual		Prediction		Diff		t-Test/Wilcoxon			Shapiro
	M	SD	M	SD	M	SD	d	t/Z	p	W	p
n-back Acc. [%]	94.892	2.662	94.471	2.007	0.422	1.801	0.234	93.000	0.064	0.825	0.008
Visual Det. task Acc. [%]	95.478	3.837	94.696	2.766	0.781	1.706	0.458	1.774	0.098	0.944	0.440
Raw NASA	8.578	3.205	8.317	2.250	0.261	1.952	0.134	0.518	0.613	0.938	0.363
EDA Mean (std)	0.040	1.074	−0.023	0.957	0.063	0.364	0.172	0.668	0.515	0.976	0.937
SCL	9.456	3.100	9.275	2.761	0.181	1.051	−0.172	0.666	0.516	0.976	0.939
GEQ—Competence	2.767	0.623	2.441	0.374	0.326	0.436	0.746	2.890	0.012	0.933	0.298
GEQ—Positive Affection	2.300	0.978	2.396	0.262	−0.096	0.866	-0.111	52.000	0.679	0.864	0.027
GEQ—Immersion	1.500	0.707	1.067	0.334	0.433	0.574	0.755	2.924	0.011	0.903	0.105
Stream Appropriate	3.467	0.743	1.669	0.058	1.798	0.729	2.465	9.548	<0.001	0.902	0.104
Desire To Use	3.533	0.640	1.728	0.101	1.805	0.591	3.053	120.000	<0.001	0.827	0.008

Table 2. Mean predicted and actual means across measures with Welsh-corrected t-test or Wilcoxon-signed-rank test depending on the Shapiro test and Cohen’s d. We compare the predicted value from the LMM at an optimized Stream with the actual value at this rate of Stream.

	Actual		Prediction		Diff		t-Test/Wilcoxon			Shapiro
	M	SD	M	SD	M	SD	d	t/Z	p	W	p
n-back Acc. [%]	94.892	2.662	94.471	2.007	0.422	1.801	0.234	93.000	0.064	0.825	0.008
Visual Det. task Acc. [%]	95.478	3.837	94.696	2.766	0.781	1.706	0.458	1.774	0.098	0.944	0.440
Raw NASA	8.578	3.205	8.317	2.250	0.261	1.952	0.134	0.518	0.613	0.938	0.363
EDA Mean (std)	0.040	1.074	−0.023	0.957	0.063	0.364	0.172	0.668	0.515	0.976	0.937
SCL	9.456	3.100	9.275	2.761	0.181	1.051	−0.172	0.666	0.516	0.976	0.939
GEQ—Competence	2.767	0.623	2.441	0.374	0.326	0.436	0.746	2.890	0.012	0.933	0.298
GEQ—Positive Affection	2.300	0.978	2.396	0.262	−0.096	0.866	-0.111	52.000	0.679	0.864	0.027
GEQ—Immersion	1.500	0.707	1.067	0.334	0.433	0.574	0.755	2.924	0.011	0.903	0.105
Stream Appropriate	3.467	0.743	1.669	0.058	1.798	0.729	2.465	9.548	<0.001	0.902	0.104
Desire To Use	3.533	0.640	1.728	0.101	1.805	0.591	3.053	120.000	<0.001	0.827	0.008

5.3.3. Subjective Evaluation

The subjective feeling of being immersed, measured by the GEQ-immersion scale, was increased in the adaptive condition ($M=1.556$, $\mathit{SD}=0.705$) as compared to its expected value ($M=1.032$, $\mathit{SD}=0.323$), as predicted from the non-physiologically adaptive conditions. Stream was also considered to be more appropriate in the adaptive condition, compared to the predicted value derived from the actual measures of the non-adaptive conditions. This converges with a desire to use the system as it increased when compared to the actual and the expected desire to use the system, see Figure 7a. In summary, participants favored the physiologically adaptive condition in terms of Stream appropriateness, the desire to use, gaming immersion (GEQ), and their feelings of competence (GEQ) and positive affection (GEQ). Those results are graphically summarized in Figure 7b.

6. Discussion

We presented a design of an online physiologically adaptive VR system. In detail, our system can adapt to the complexity of a VR environment based on the user’s arousal as measured using EDA. We tested this system in a VR environment where the adaptive system optimizes the stream of NPCs. In our evaluation, participants completed a primary task (n-back) and a secondary task (visual detection) while either being presented with five static levels of Streams of NPCs or an adaptive Stream of NPCs.

First, we investigated the effectiveness of the Stream modulation. Here, we lend credibility to earlier results on crowdedness [32,36] and visual complexity [6,7]. In detail, a higher Stream of NPCs impacts EDA and the task performance, albeit in both the primary and the secondary task. Thus, the desire to use the system is negatively affected by increasing Stream. Thus, we found that an online adaptation of the Stream is feasible for modulating workload. Therefore, we investigated the effectiveness of such a physiologically adaptive VR system. Here, we compared the baseline from the non-adaptive conditions to the adaptive condition using a prediction model. We found significant differences between adaptive and non-adaptive conditions for the desire to use the system, GEQ competence, and GEQ immersion. Interestingly, participants also rated the flow significantly more appropriate than the baseline, while we kept the range of possible flows within the same bounds. Thus, we conclude that our physiologically adaptive VR system can improve VR experiences, especially in complex environments such as this simulated Social VR. This is a promising step toward online adaptive VR environments based on physiological sensor data. Our study adds to other findings in physiological computing, namely that sympathetic arousal, here indexed by EDA, can depict task engagement [22,95,96]. Additionally, our subjective results are in line with studies that have adapted VR environments based on physiological arousal. Thus, our results are in line with prior work in terms of perceived workload [24] and task engagement [27] when interacting in a physiologically adaptive VR system. These results are promising for the implementation of EDA measures in a biocybernetic loop for adaptation, confirming subjective results obtained with other measures i.e., EEG and ECG.

At first, it is counterintuitive that EDA was not lower when using the adaptive system, as the system optimizes in order to not overload the user. However, we argue that lowering the EDA was not expected nor intended. In our implementation, we used a −2/+4 adaptation i.e., if NPCs need to be removed, we removed two, while we added four in the more adaption. Thus, this pushed the user to always take a bit more at once while reducing it only gradually. This prevents the user from just tuning out but also keeps up the pressure while not overloading. Moreover, in a real-world scenario, keeping as many characters in the environment as possible is important to not lose an understanding of the Social VR scenario one engages in, but also to prevent lowering the level of immersion and presence. We further see this manifested in the significantly higher rating of immersion. Here, we argue participants were less likely to be bored or stressed, as we optimized for optimal arousal according to the MIM [17] where flow-experience and task engagement correlate with rising demands [97]. As a result, they focused on the environment and, thus, achieved a higher level of immersion.

Behavioral results indicated that Stream adaptation did not significantly impact n-Back and detection accuracies. This might point toward an insufficient increase in task complexity. However, it is not corroborated by either the EDA processed by our algorithm or from the non-negative deconvolution analysis. Second, as distractor processing is influenced by increased executive load [98], future studies should investigate task complexity adaptation under increased working memory load conditions. This conclusion is shared with the work of Dey et al. [71] and with Ewing et al. [24] when considering increased task demands.

Our findings have a broader impact on collaborative environments; virtual agents can assist or perform tasks within a collaboration VR environment in future VR applications. Such systems could exploit physiologically adaptive systems to modulate the VR complexity (e.g., amount of virtual coworkers or concurrent visually displayed information). This allows supporting concurrent goals without the need to prioritize performance over comfort while preventing cognitive overload. We argue that our results are not limited to Social VR per se. For instance, physiological adaptation can be used to transition on the reality-virtuality continuum [99]. Here, the VR complexity can overload the user with the virtual component, and, thus, it could gradually fade out the digital overlay. Thus, varying the blend of realities according to users’ physiological arousal can be the next step toward a physiologically adaptive mixed reality system.

EDA data acquisition is low-cost and unobtrusive; skin-interfaced wearable systems are easy to implement in real or virtual scenarios, i.e., biofuel cell-based self-powered wearable sensors [100] or conductive fabric gloves [101]. Consequently, we argue that the next step is to embed physiological sensing into VR controllers, allowing practitioners to use them without a tedious setup process. This is consistent with recent developments in entertainment computing, where a player’s emotional state as inferred from physiological correlates was used to predict actions in a game [102] or dynamically adapt the game narrative to induce states of arousal [103].

Limitations

EDA activity has a one-to-one relationship with the state of physiological arousal [95], which can serve as a workload indicator. However, the adaptation pipeline used in this study, although with a low computational cost, could benefit from more advanced and standardized methods of EDA component deconvolution [94,104]. So far, this has not yet been used for online adaptation.

Tonic changes occur with a slow frequency and, therefore, may not be adequate for adaptive interactions that occur at a faster pace. Here, other physiological measures that are experimentally more demanding, but have better time-resolution, such as EEG, could be more suitable for adaptive systems [24]. Nevertheless, for adapting visual complexity in our Social VR scenario, adaptation in a third of a minute was suitable with regard to fostering immersion; here, faster adaptation could potentially break immersion as layer characters would appear and disappear seemingly randomly.

We targeted the complexity exclusively in the visual domain. Still, evaluating multimodal physiologically adaptive systems could better support users’ comfort as multimodal integration facilitates task-relevant information selection [105,106]. For example, in a social VR scenario, the visual representation could disappear, and voices from afar could be toned down to support the user in focusing on the primary task.

According to Braithwaite et al. [107], about 10% of the general population do not exert a strong electrodermal response attuned to sympathetic arousal. This is especially relevant when considering commercial applications of EDA sensing, e.g., for adaptive VR games where potentially 10% would not benefit from a solely EDA-based system. Thus, supplementing EDA with other arousal measurements such as the heart rate or eye-gaze could make measurements more robust and make the technology more accessible to the general population. Together with this, we had to exclude three participants via medical-grade instrumentation for high data quality due to a lack of data quality. This is consistent with EDA guidelines [107] and could be attributed to either technical failures and to individual variations in EDA. While we controlled for other factors such as physical stress [108] or caffeine consumption [109], using EDA in applied scenarios can not be shielded from such influences.

Finally, we acknowledge that only the data of 15 participants were used in the analysis. While on the one hand, this is a low number, we could still show large effect sizes for the appropriateness of the stream, and most importantly for the desire to use the adaptation. As such, larger sample sizes will be needed only for the measures with small and medium effect sizes.

7. Conclusions

Our study reports on the utility and viability of a VR system that adapts the environment to the physiological input, namely the user’s electrodermal activity. By adjusting the visual complexity of the environment in response to the changing arousal levels of the user, we were able to maintain an ideal working environment in a complex scenario (i.e., Social VR). This is the first step toward a physiologically adaptive immersive system that supports users’ satisfaction and cognitive capacity without the need for explicit inputs (i.e., verbal, button-presses). Future studies could use physiologically adaptive systems to optimally blend mixed reality, enabling more use cases by ensuring signal processing robustness and by integrating complementary physiological measures (i.e., EEG) or embedding physiological sensors into the hardware of VR systems (e.g., VR-controllers).