Recommendations for Integrating a P300-Based Brain Computer Interface in Virtual Reality Environments for Gaming

Abstract

The integration of a P300-based brain–computer interface (BCI) into virtual reality (VR) environments is promising for the video games industry. However, it faces several limitations, mainly due to hardware constraints and constraints engendered by the stimulation needed by the BCI. The main limitation is still the low transfer rate that can be achieved by current BCI technology. The goal of this paper is to review current limitations and to provide application creators with design recommendations in order to overcome them. We also overview current VR and BCI commercial products in relation to the design of video games. An essential recommendation is to use the BCI only for non-complex and non-critical tasks in the game. Also, the BCI should be used to control actions that are naturally integrated into the virtual world. Finally, adventure and simulation games, especially if cooperative (multi-user) appear the best candidates for designing an effective VR game enriched by BCI technology.

1. Introduction

A video game can be defined as “a mental contest, played with a computer according to certain rules, for amusement, recreation, or winning a stake”. It has also been defined briefly as “story, art, and software” [1]. In some cases, for example in serious games, amusement is not the main goal, however, to date amusement still plays a major role in video game industry. Although by completely different means, virtual reality (VR) and the brain–computer interface (BCI) are both excellent candidates for enhancing the possibilities of entertainment and satisfaction in video games. Indeed, both enhance immersion and it is a common belief that this encourages the feeling of amusement. The concept of immersion was defined in [2], observing that everybody may enjoy a game with immersion, even if the gaming control seems to play the main role in the user’s enjoyment. According to [1] this immersion feeling is created by computer graphics, sound, haptics, affective computing and advanced user interfaces that increase the sense of presence. Virtual reality is a collection of devices and technologies enabling the end user to interact in three dimensions (3D) [3], e.g., spatialized sounds and haptic gloves (for example Dexmo, Dexta Robotics, Shenzhen, China). The particular type of experience that is created by VR is emphasized by [4]. Such experience is named telepresence, defined as the experience of presence in an environment by means of a communication medium [4], joining the concept of presence of [1]. A BCI can also enhance the feeling of presence in the virtual world since it can replace or enhance mechanical inputs. According to [2], immersive games are played using three different kinds of inputs: visual, auditory and mental; since a BCI may transform ‘mental’ signals into input commands, such an interface may play an unique role in the mentalization process involved in the feeling of immersion. However, considering the limitations of a BCI system (to be analysed later), it is still not clear to what extent current BCI technology may improve immersion. As pointed out in [2], “engagement, and therefore enjoyment through immersion, is not possible if there are usability and control problems”.

An element of amusement derives from the originality and futuristic aspect of BCI technology as compared to other traditional inputs, like a mouse, a joystick or a keyboard. Nonetheless, as often happens in the technological industry, BCI technology risks being dropped by the general public if the improvement it brings is not worth as compared to the effort needed for its use. Virtual reality has already enjoyed the “wow-factor” and VR systems tend to be employed nowadays in commercial events especially for raising this effect (Feel Wimbledon by Jaguar, Coca Cola’s Santa’s Virtual Reality Sleigh Ride, McDonald’s Happy Meal VR Headset and Ski App, Michelle Obama’s VR Video, XC90 Test Drive by Volvo, etc.) (http://mbryonic.com/best-vr/). The “wow-factor” is defined in the Cambridge dictionary as “a quality or feature of something that makes people feel great excitement or admiration”, and was previously studied in the domain of marketing and education (e.g., [5,6]). The recent development of dedicated VR headsets, that is, head-mounted devices (HMDs, e.g., the Oculus, Facebook, Menlo Park, CA, USA; HTC Vive, HTC, Taoyuan, Taiwan; Google Cardboard, Google, Mountain View, CA, USA) has paved the way to the commercialization of combined BCI+VR technology. Indeed HMDs provide an already built-in structure that can support the embedding of EEG (electroencephalography) electrodes, which are needed for the BCI. The Neurable Company (Cambridge, MA, USA) has recently announced a product combining an HTC Vive (Taoyuan, Taiwan) with an EEG cap. The HTC Vive (Taoyuan, Taiwan), as well as other HMDs such as the SamsungGear (Samsung, Seoul, Korea) use inboard electronics, thus herein we refer to them as active devices. On the contrary, passive HMDs consist of a simple mask with lenses in which a smartphone is inserted (Figure 1). Passive HMDs are particularly promising for the BCI+VR field since they are very affordable, and smartphones are nowadays ubiquitous.

Prototypes of BCI-based video games already exist [7,8,9,10,11,12,13,14,15,16,17,18]. They are mainly based on three different BCI paradigms: the steady-state evoked potential (SSVEP), P300 event-related potential (ERP) and mental imagery (MI). The first two necessitate sensorial stimulation of the user, usually visual, and are defined as synchronous because the application decides when to activate the stimulation so that the user can give a command [19]. In this article we focus on P300-based BCIs. As compared to MI-based BCIs, P300-based BCIs require shorter training, achieve a higher information transfer rate (amount of information sent per unit of time) and allow a higher number of possible commands [20,21]. As compared to SSVEP-based BCIs, they feature a lower information transfer rate, however, the flickering used for eliciting SSVEPs is annoying and tiring, besides presenting an increased risk of eliciting epileptic seizure [22]. P300-based BCIs are based on the so-called oddball paradigm. The oddball paradigm is an experimental design consisting in the presentation of discrete stimuli successively; most are neutral (non-TARGET) and a few (rare) are TARGET stimuli. In the case of P300-based BCI, items are flashed on the screen, typically in groups. A sequence of flashes covering all available items is named a repetition. The goal of the BCI is to analyze the ERPs in one or more repetitions to individuate which item has produced a P300, a positive ERP that appears around 300–600 ms after the item the user wants to select (TARGET) has flashed. The typical accuracy of P300-based BCIs has risen over the past years from about 75% after 15 repetitions of flashes [23] to about 90% after 3 repetitions using modern machine-learning algorithms based on Riemannian geometry [24,25,26]. In practice, this means that at least one second is necessary to such BCI to issue a command, but more may be needed to issue reliable commands.

We anticipate that the integration of BCI in VR games thanks to the development of integrated HMD-EEG devices will foster the acceptance of this technology by both the video game industry and gamers, thus pushing the technology into the real world. However, the development of a concrete application for the general public faces several limitations. In the domain of virtual reality, motion sickness appears to be one of the most severe limitation. However, this limitation seems relatively weak in comparison to those risen by the BCI system. Above all, BCI are often unsightly, and the electrodes are not easy to use. Also, users in virtual reality may move a lot and this jeopardizes the quality of the EEG signal. In this work we analyse these limitations and give recommendations in order to circumvent them. This work has been inspired by previous contributions along these lines concerning BCI technology [12,27,28] and its use in VR [29]. Here we will integrate these previous works with similar guidelines found in the literature on VR [30,31,32,33] and spatial mobility [14,34,35,36], focusing on P300 technology for BCI and gaming applications for VR. In the following section, we will present the limitations concerning general public use, divided according to the fact whether they are introduced by (1) the HMD, (2) the BCI system (in general or by the P300) or (3) both BCI and VR. For the purpose of clarity, we present the recommendations directly after each corresponding limitation. Numerous limitations and recommendations already discussed in the literature are considered and others are added here. Since there is not the same level of evidence for all of them, the limitations and recommendations we report will be labelled according to the following taxonomy:

-

Level of Evidence (LoE) A: The recommendation or limitation is a fact, or there is a strong evidence supporting it; for instance, it has been reported in a review paper or in several studies.

-

LoE B: The evidence supporting the limitation or recommendation is weak for one or more of the following reasons:

○

It appears relevant, but it still not exploited currently in BCI or VR.

○

The limitation or recommendation was stated in some papers but challenged in others.

○

The limitation or recommendation has been sparsely reported.

○

The limitation or recommendation has appeared in old publications and is now possibly outdated in light of technological improvements.

-

LoE C: The limitation or recommendation is introduced here by the authors, thus it requires independent support and validation.

The authors acknowledge that all listed limitations and recommendations are not equally relevant when designing a BCI+VR system. For example, the limitation of the field of view (discussed later) is very specific to the VR domain. It is not a major concern for a BCI+VR system in comparison to other limitations such as the need for an ergonomic EEG cap. In parallel to the LoE just defined, we then label the limitations and recommendations also according to their Level of Interest (LoI), that is, their pertinence in the process of designing a BCI+VR system:

-

LoI 1: The recommendation or limitation deeply impact the conception of a BCI coupled with VR.

-

LoI 2: The recommendation or limitation is relevant for the field, but might be ignored for a prototypical version of a BCI+VR.

-

LoI 3: The recommendation or limitation is secondary.

2. Limitations and Recommendation

2.1. Limitations of the Head-Mounted Device (HMD)

2.1.1. Inertial Measurement Unit (IMU) Accuracy

Limitation

The Inertial Measurement Unit (IMU) is not accurate enough (LoE A; LoI 3). The smartphone position and rotation in space is determined by the IMU, the accuracy of which widely varies in different models of smartphone (see [37] for a benchmark of different smartphones, and [38] for a deep review of this problem). As a consequence, a VR device may detect movement when the user is not moving. This creates the perception that the virtual world moves slightly around the user, forcing the user to rotate his/her head to compensate and follow the scene. In turn, these movements generate artifacts in the EEG signal.

Recommendations

This problem seems restricted to passive HMD (when the smartphone is the only device of virtualisation). It is solved in the active SamsungGear device (Samsung, Seoul, Korea), which incorporates an IMU of good quality. It may be solved also by tracking the user head by means of an external tracker (LoE C; LoI 3). A new generation of VR devices which are based on sensor fusion (sensor fusion refers to the combination of sensory data or data derived from disparate sources) may solve this problem as well as positional tracking by correcting the IMU bias according to a video camera input (LoE B: Sensor Fusion is not working yet; LoI 2). The main commercial products are:

-

The Daydream SDK (Google, Mountain View, US), which is associated with Lenovo (Lenovo, Hong Kong, China) and Vive (HTC, Taipei, Taiwan; Valve, Washington, US).

-

The Structure Sensor (Apple, Cupertino, US).

-

The ZED camera (Stereolabs, San Francisco, US).

-

The Windows Mixed Reality platform (Microsoft, Washington, US) (Among others, Lenovo (Hong Kong, China), HP (California, US), Acer (Taipei, Taiwan) and Sam sung (Seoul, South Korea) have already build headsets for the Windows Mixed Reality platform.).

2.1.2. Locomotion in Virtual Reality

Limitation

Tracking user position is problematic (LoE A; LoI 3). Tracking a user’s position in an indoor environment like a room is a global issue for VR devices. Low-cost or portable devices use a gyroscope to track the user’s head rotation, however, they cannot determine the user’s position. More expensive devices, like the Vive or the Oculus, can track user position in a specific area. This area is limited by the size of the room where the game takes place as well as by the position of the motion capture sensors (around 25 m² for the Vive and 5 m² for the Oculus). Often game designers and application creators want to allow movement in a virtual world that is substantially bigger than the size of the room where the user is. This limitation especially applies to VR systems only. Indeed, the use of locomotion with BCI is not recommended (see limitation of the BCI system in general).

Recommendations

We propose here five solutions for locomotion in VR. The reader is referred to [39] for a systematic review of locomotion in virtual reality.

(1)

Teleportation (LoI 1: very relevant for BCI since this technique does not require any movement), which for example is used in the following games: Portal Series in VR (Valve, Bellevue, WA, USA), Robo Recall (Epic games, Cary, CA, USA) or Raw Data (Survios, Los Angles, CA, USA); the user focuses on the area of interest and then clicks to teleport on the place s/he has selected. This solution is simple to implement, but the effect of motion sickness is not clear. In fact, the apparition of an unnatural cutting scene could induce motion sickness (LoE C). However, teleportation may also reduce motion sickness since it does not involve visible motion (hypothesis presented in [40] with mitigate results). (LoE B: According to the systematic review [39], teleportation is a mainstream technique, but there is a lack of empirical studies about it.)

(2)

Walk-in-place [30,41], e.g., VR-Step (VRMersive, Reno, NV, USA) and RIPMotion (RIPMotion, Raleigh, CA, USA). The user first focuses on the area where s/he wants to move and then performs a walk-in-place to execute the movement and arrive at the selected destination. This solution reduces motion sickness as the user receives the sensation of movement as the virtual world moves. However, it is more complicated to be implemented because accelerometer data are needed in order to detect vertical movements of the user (walking). The accelerometer ordinarily employed is not always sufficiently accurate to obtain information about the step length and the movement may appear unnatural, again possibly resulting in motion sickness. Moreover, the walk-in-place input is restricted to situations where the user has to walk, since the in-place gesture cannot be easily mimicked for applications where the player swims or flies, for example. (LoE A: Stated in many games and studies; LoE C: limited to a few situations; LoI 3: this technique is secondary since motion is not recommended with BCI.)

(3)

Gesture recognition, implemented by Raptor-lab (Lyon, France). For example, this reproduces the way skiing people push their poles to move forward. The company claims that their solution “offers human locomotion in VR with full freedom of movement, ‘as real life’ agility and liberty of action, which means, walking, running, climbing, jumping, crawling, and even swimming. And all that while avoiding motion sickness”. The gesture system was used in a game called the “Art of Fight” for the HTC Vive that has reached 10,000 players (http://steamspy.com/app/531270). (LoE B: Used in games, but not evaluated in scientific studies; LoI 3: it requires motion.)

(4)

Motion platforms (similar to a fitness treadmill) allow the use of movement in order to navigate in a restricted area. There are already commercial motion platforms such as Virtuix omni (Virtuix, Austin, TX, USA), WalkOVR (WalkOVR, Istanbul, Turkey) or VR Motion Simulators (Virtec Attractions, Balerna, Swiss)—Figure 2. So far, this technology has been restricted by the encumbrance as well as by the fact that different physical movements require different platforms and that movement are sometimes unnatural—in particular, Virtuix omni was criticized because it reproduces an unnatural walking. Thus, these VR motion platforms are likely restricted to arcade rooms. (LoE A; LoI 3.)

(5)

Sensor fusion (LoI 3: Sensor fusion is used when the player is expected to physically move). Currently employed accelerometers are not sufficiently accurate to determine the position of the user by integrating two times the sensor’s input. By adding information from a gyroscope, magnetometer and camera (image recognition), a next generation of devices may be able to determine accurately relative position without the use of any external sensors. This will allow the expansion of the game area and reduce the encumbrance of the system. Such use of elements from the real world into the virtual world is known as augmented virtuality (Figure 3) and is part of the mixed reality domain. The reader may refer to [31,42] for a classification of virtualisation technology and a description of mixed reality. (LoE B: sensor fusion is still in an early research phase.)

2.1.3. Motion Sickness

Limitation

Motion sickness (LoE A; LoI 1: the comfort of the user is a main concern). The HMD may provoke motion sickness [43,44]. In general, motion sickness arises when there is a mismatch between the visual and the vestibular systems, for example when travelling on a ship or a car without seeing the horizon and the road [45]. Sensory conflict is commonly used to explain such a sensation of discomfort in a VR context [33,46]; however, this is still unproven [33,47] (LoE B: There is no agreement concerning the cause of motion sickness.) Practical factors that have been found to induce motion sickness in virtual environments include:

-

The user observes a movement that does not happen in the real world [44]. (LoE B; LoI 1.)

-

The lag between the movement of the user and the movement of the avatar in the virtual world [43]. For instance, this lag may be due to the refresh rate of the screen and/or computation time due to high-quality graphics. (LoE B; LoI 1.)

-

Motion sickness increases with higher field of view, with an asymptote starting at 140° [48]. This figure is criticized in [49], but [49] did not use the same measure of motion sickness as [48]. (LoE B; LoI 2.)

-

Postural instability [50,51]. (LoE A; LoI 3: since movements are limited in BCI this factor does not really apply here.)

In evaluating the risk of motion sickness, the following factors should be considered:

-

Mental rotation ability and field in/dependence; better mental rotation and weak tendency to field dependence or independence may result in lesser motion sickness. (LoE A, LoI 3: this limitation applies for the end user, independently of the application design.)

-

People feeling motion sickness in real life are more likely to experience motion sickness in virtual reality environments [52]. (LoE B; LoI 3.)

-

Age and gender are correlated to virtual sickness; motion sickness is less common in the age range 21–50 [33,45,53] and women are more exposed than men [54,55,56]. (LoE B; LoI 3.)

The motion sickness limitation especially applies to VR. However, this limitation appears to be a major limitation for a BCI+VR system since people feeling sick will not be able to use the system.

Recommendations

The following recommendations are useful to reduce motion sickness in a VR setting:

(1)

Avoid motion parallax effect [57,58]. (LoE A; LoI 1.)

(2)

Avoid flickering, that is, visible fading between images of the display [33]. (LoE A; LoI 1.) Main constructors such as Oculus and HTC Vive use a refresh rate of about 90 Hz, suggesting that less than 90 fps may result in a flickering effect (https://www.quora.com/Why-does-Virtual-reality-need-90-fps-or-higher-rates-What-are-the-technical-problems-in-terms-of-computer-graphics-in-the-lower-rates). However, headsets such as Oculus (Facebook, Menlo Park, CA, USA) already provide a frame interpolation if the render-rate is not high enough (https://developer.oculus.com/blog/asynchronous-spacewarp/). (LoE B: The minimum requirement of 90Hz is going to be outdated.)

(3)

Avoid cutting scenes, since such transitions do not apply in the real world. (LoE C; LoI 3: see locomotion in virtual reality.)

(4)

Avoid extreme downward angle, such as looking downward at short distance in front of the virtual feet [59]. (LoE B; LoI 1.)

(5)

Try to use stroboscopic vision and overlaying glasses as suggested in [32]. (LoE C: Stated for motion sickness but not for virtual sickness; LoI 3.)

(6)

Take breaks out of the VR immersion as motion sickness increases with playing time [59]. (LoE B; LoI 1.)

(7)

Introduce a static frame of reference [60,61,62]. (LoE A; LoI 1.)

(8)

Dynamically reduce the field of view in response to the visually perceived motion [63]. (LoE B; LoI 2.)

(9)

Try movement in zero gravity (https://www.theverge.com/2016/10/13/13261342/virtual-reality-oculus-rift-touch-lone-echo-robo-recall). (LoE C, LoI 2: this technique may be relevant for application design but, has to be studied in depth.) The link we have given is a short description of the game “Lone Echo” (Ready at Dawn Studios, 2017) that was nominated at the Game Award for best VR game. This game takes place in space and reproduces the sensation of floating by disabling gravity and continuously moving the player with an endless drift. In the aforementioned link it is suggested that this might diminish motion sickness.

2.1.4. Unsolved Limitations: Asset Restriction and Field of View

The following limitation applies to VR in particular. The impact of these limitations on BCI+VR systems appears limited.

• Asset restriction (LoE A). The use of HMD does not fit large assets, with graphics restricted in term of polygon and texture size. This restriction is due to the graphic engine, hardware and stereoscopic vision. Stereoscopic vision appears as the main problem since a texture has to be drawn two times, one for each eye. Next, big assets require more computation or power, which is mainly limited by the hardware capability, in particular when using a mobile platform such as the SamsungGear (Samsung, Seoul, South Korea) (https://developer.oculus.com/blog/squeezing-performance-out-of-your-unity-gear-vr-game/). The graphic engine may also have a great impact on performance when either multithreading or batching (among others) are enabled. (LoI 3: The quality of the graphics may be considered secondary when designing an ergonomic interface.)
• Field of View (FOV) is limited (LoE A; LoI 3). The HTC Vive (HTC, Taoyuan, Taiwan) has the largest FOV among the currently available virtual reality headsets. Its FOV is about 100°, that is, around 80° less than the human FOV. Such restricted FOV limits the feeling of immersion, while a wider FOV causes optical distortion of the image and increases the sensation of motion sickness [48,49].

2.2. Limitations of the Brain–Computer Interface (BCI) Systems

2.2.1. BCI System in General

Limitations

• Comfort and Ergonomics of the electrodes (LoI 1: The use of electrodes introduces a major discomfort). Traditional EEG caps need the use of a gel or a paste to establish the contact between the electrodes and the scalp or the use of dry electrodes, which often are more uncomfortable (Figure 4). This limitation is a major concern for the end-user, since an unsightly, messy or painful product has little chance to be successful (LoE C). Research is ongoing to develop EEG caps that are easy to set up, easy to clean and comfortable, with the key requirement of allowing accurate EEG signal recording [14,35,36,64]. Among the commercial products, we can mention: the “Mark IV” (OpenBCI, New York, US) and the “Muse” headband (Muse, Toronto, Canada) are very easy to set up, but covering only a small portion of the head; the “Quick-20” (Cognionics, San Diego, US) a dry EEG headset that can be set up in a few minutes. Most of these systems diminish the setting time by providing an already built-in support structure and dry electrodes. Nevertheless, the affirmation that dry electrodes are easier to install was challenged in [65]. This study concluded that the set up time was equal or even longer using dry electrode since they do not easily adapt to the shape of the head. This observation does not take into account the fact that the user may have to clean their hair after using the gel-based electrode. A common concern for the aforementioned products is that the number of electrodes and/or the quality of the signal is not sufficiently high for P300-based applications ([36,66] have a mitigated point of view, whereas [67] is more optimistic). (LoE A: wet electrodes are more accurate, stable and comfortable than dry electrodes; LoE B: Dry electrodes are easier to install and remove if the user is experimented and the cap can easily adjust to the shape of the head.)
• Locomotion with the electrodes. The use of BCI when moving around is very disputable. It has been shown that the recognition of the P300 component is still possible when walking, but the performance of the system is reduced [36]. (LoE B; LoI 2: It is possible to design applications with limited movements, sacrificing the immersion feeling and the VR capabilities.)
• Tagging. An important technical aspect of P300-based BCIs is that the BCI engine needs to be informed about the exact moments when the stimulations (flashes) are delivered. Traditionally this is obtained by hardware tagging, that is, through a serial or parallel port by which the user interface (UI) sends a tag to the EEG acquisition unit at each flash, which in turn synchronizes the tags with the incoming EEG data. This is an accurate method, allowing, in general, a tagging error within ±2 milliseconds. The alternative is known as software tagging and can be achieved in several ways. Software tagging is better achieved synchronizing the clocks of the machine on which the UI software runs and of the EEG acquisition device, the lack of which may result in a rather high tagging error [6]. This kind of synchronization problem is well-known in the network domain where multiple servers must be synchronized or when dealing with an array of wireless sensors [68]. If hardware tagging is used, the wire connecting the UI to the EEG acquisition unit may limit the movement of the user. (LoE A, LoI 1: From the perspective of the user, it is important to guarantee the accuracy of the system while minimizing its encumbrance.)

Recommendations

• In order to diminish set up time, the EEG cap could be embedded in the VR device (LoE C; LoI 1). For this purpose, since pin-shaped dry electrodes result in being uncomfortable, a new generation of dry electrodes based on conductive polymers could be preferred (e.g., [69]) (LoE B; LoI 1). For gaming, this solution is preferable to wet electrodes, as flexible dry electrodes do not require a long set up and cleaning (LoE C; LoI 2: see previous discussion on dry and wet electrodes). The problem concerning the quality of signal might be addressed by a shield placed over the electrodes to prevent electromagnetic contaminations (prototype of A. Barachant: http://www.huffingtonpost.fr/2017/11/22/brain-invaders-le-jeu-video-qui-se-controle-par-la-pensee_a_23284488/) (LoE C; LoI 1). Another option is the use of miniaturized electrodes (LoE B; LoI 1). In [70], the authors conceived a system made of 13 miniaturized electrodes placed in and around the ear in addition to the traditional sites for P300 recognition (central, parietal and occipital locations). An offline analysis showed that the accuracy of this system was comparable to that obtained in a previous study with state-of-the art equipment [34]. However, the authors point out the lack of robustness of such a system if the user moves around.
• The camera and accelerators could be used to detect the user movement and remove the corresponding EEG signal from the analysis (LoE C; LoI 3). It has also been suggested to use an automatic online artifact rejection such as [71], or to use new features such as the weighted phase lag index when walking [72]. However, we note a lack of studies in real life such as [36] (LoE B: It works in laboratory conditions, but there is a need for out-of-the lab studies; LoI 1: The removal of movement artifacts, if effective, will allow the user to freely move in VR.).
• When the UI and the BCI engine run on separate platforms: if the EEG acquisition unit is not mounted on the head, like in [34], prefer software tagging over hardware tagging (LoE B; LoI 1)—Figure 5b.
• It would be even better to directly embed a wireless EEG acquisition unit on the VR device. With such a system, the tagging problem would be solved for good [34] (LoE B; LoI 1).
• Embed the EEG acquisition unit as well as the EEG analysis on the EEG cap, making the BCI completely independent of an external computer (Figure 5a). (LoE C; LoI 1). Such a system would also avoid dealing with problems related to wireless communication (e.g., data loss, signal perturbation, etc.) (LoE B; LoI 1). A simple application of this recommendation consists in placing the PC on which the BCI engine runs in a backpack [34]. For HMD linked to a PC, MSI (Zhonghe, Taiwan) has released VR-ready PCs placed into a backpack (http://vr.msi.com/Backpacks/vrone).

2.2.2. Limitations of P300–Based BCIs

Possibly the most severe limitations are engendered by the P300 paradigm itself. We list here four limitations, the first three being well-known [12] (LoE A; LoI 1). The last one is a suggestion from the authors (LoE C; LoI 1).

Synchronous BCIs

Limitation

As we have seen the P300 is a synchronous BCI, thus it is not possible to control a continuous process where constant error correction is required, for example when driving a car. Rather, it is possible to perform a goal selection task, like for example choosing the final destination of our vehicle.

Recommendations

(1)

Enable goal selection strategies and gradual control strategies [12]. Gradual control means controlling a continuous process in a discrete way by use of separated and limited goals. For example, the player may control speed by focusing on items such as SLOW, FAST and MODERATE. (LoE A; LoI 1.)

(2)

Use the concept of cone of guidance [27], inspired from a game described in [73], where the player has to guide a helicopter through floating rings. In the process of approaching a ring, the player is assisted by an invisible cone that improves the player performance, but this is not necessary to win the game. From a larger perspective, the cone of guidance may refer to any optional computer assistance that may help the user to perform a task, but it still requires enough input from the user to finish this task. (LoE B; LoI 1.)

(3)

Use high-level commands [29]; they drive faster toward the sought result, although they are less intuitive. An example of navigation in a museum using high-level commands is shown by [29] . In this example, the user has to select a point of interest in the museum using three commands: two commands are for selecting the point of interest through a succession of binary choices; the last one is for deleting the last binary choice. The authors compared this method of navigation to a navigation using low level commands (such as turn left/right and go forward). The results of a subjective questionnaire show that high-level navigation is faster and less tiring than low-level navigation, but that high-level navigation is less intuitive because of the succession of binary choices. (LoI 1; LoE B: the given reference is about MI-based BCI; it is not clear to what extent the result applies to P300-based BCI since the latter BCIs allows a higher number of choices.)

(4)

Do not separate stimuli and action and always incorporate them in the virtual-world [12]. A more radical solution is the use of the P300 BCI control for actions that are normally “synchronous”, such as stopping when a traffic light switches to red [12,74]. (LoE A; LoI 1.)

(5)

Limit the use of complex movements such as controlling speed and movement at the same time. (LoE C; LoI 1.)

(6)

Design cooperative BCI games (whenever possible) where each player controls one parameter of the game. For example one player could be responsible only for changing the direction and another could control the speed of a moving avatar or vehicle (LoE C; LoI 1). In [75] multiuser interaction using P300-based BCI has been studied using the game Brain Invaders [17], showing the feasibility of cooperative BCIs for gaming [75].

To show how these recommendations may be applied in practice,

Table 1. Examples of practical implementations of the design recommendations given in this section.

Recommendation	When It Applies	Example with a Car Race Game
Goal Control	Every time you use a synchronous BCI.	Do not control the movement but set objectives that the car must reach.
High-Level Commands	As much as possible, but trying to make them intuitive.	Control the speed of the car through a simple interface (SLOW, MODERATE and FAST). Avoid real-time commands such as “activate clutch, select driving gear from one to six”.
Incorporate stimuli in the game	As much as possible.	At the start of the race, incorporate the stimuli in the signal light. The car direction can be set up by looking at different billboards on the left or right side of the road.
Separate complex actions	When controlling an action that has more than two possibilities, or when each possibility can take too many values.	The user action is to control the trajectory of the car, which depends on speed and direction. Usually these are done simultaneously with keyboard or joystick, but have to be set one after the other when using a synchronous BCI.
Enumerate all possibilities for an action	When an action can only take a small set of discrete values.	The speed can be slow or fast and the direction can be right or left. These are two tasks to be accomplished by the user. They can be combined into one choice with four possibilities: right-slow, right-fast, left-slow and left-fast.
Multiplayer interaction	Whenever the game is multiplayer.	The first player can control the speed and the second the direction.

gives practical examples in relation to the design of a car race game. We have chosen this kind of game for the example because, due to its nature, it is not the best candidate game for introducing BCI control. This gives us an opportunity to show that an adequate design may make BCI control possible even in unfavourable situations.

Visual Fatigue

Limitation

The flashing in a P300-based BCI is more tiring as compared to a normal visual scene and risks of photosensitive irritation should be taken into account. Furthermore, since P300-based BCIs work thanks to brain responses to stimulation (e.g., flashing of items on the screen), it continuously elicits cognitive resources from the user.

Recommendations

Ways to reduce this fatigue include:

(1)

Incorporation of the flashing items in the game scene [12,74] (LoE A; LoI 1). In [12] it was concluded that stimuli should be natural discrete events that should occur at expected locations. Examples are: blinking lights in the sky, advertisements in a city during the night, attraction park and horror scenes (graphical reference from video games: Planet Coaster, Until Dawn) or a diving experience (graphical reference from video game: Sub Nautica). Moreover, in the process of a goal selection only the controls that are specific to the current context should appear. For example, only the navigation commands should be displayed when moving an avatar, whereas these commands should disappear when the user is not controlling the movement of the avatar anymore. A game could therefore automatically switch among different control panels depending on the context.

(2)

Adopting stimulation as less tiring as possible. The use of audio stimuli which, coupled with visual stimulation, may lower visual fatigue, was recommended in [76] (LoE B; LoI 2). This study shows that combining auditory and visual stimulation is a good choice for a BCI speller to lower the workload. This study also reports that the use of audio stimuli alone leads to worse performance and higher workload as compared to a unimodal visual stimulation, thus the use of audio stimuli alone is not recommended. We should mention a promising study [77], which as stimuli used spoken sounds representing concepts as close as possible to the action they represent. Again, audio stimuli should be natural for the gaming environment and should vary according to the type of game (horror game, game with enigma, infiltration game, etc.). Another solution could be to couple the visual P300 paradigm with other BCI paradigms such as motion onset visual evoked potential (mVEP) (LoE B; LoI 1). mVEP is a type of visual evoked potential (like SSVEP or P300) allowing more elegant stimuli [78]. In [78] moving targets had low contrast and luminance, still they could evoke prominent mVEP. The protocol was nearly the same as for P300, but using moving instead of flashing targets. The usability of this paradigm needs to be studied specifically in the VR context, where targets can move in 3D around the user, thus the user may have to turn the head to follow them (LoE C; LoI 3).

(3)

Lowering the stimulation time. BCI systems that do not need calibration are definitely preferable [75] (LoE A; LoI 1). Also, in designing a game, BCI control may be activated only in some situations, totaling a small amount of the gaming time (LoE C; LoI 1).

Low Transfer Rate

Limitation

The low transfer rate of a P300-based BCI refers to the fact that several repetitions of flashes are needed for achieving accurate item selection and that, unless a large number of repetitions are employed, selection errors are unavoidable [12]. This introduces the need for repetitive actions instead of single actions to achieve a goal and the frustration of not being able to issue a command immediately, which is important in critical gaming situations.

Recommendations

There are at least eight ways to circumvent this limitation:

(1)

Using a-priori, user and/or context information to improve item selection (LoE C; LoI 1). To this end we may employ a so-called passive BCI to monitor physiological information about the user and adapt the gameplay consequently [27]. A passive BCI is a cognitive monitoring technology that can provide valuable information about the users’ intention, situational interpretation and emotional state [79]. For example, in the game Alphawow [9], the avatar’s character changes its behaviour according to the player’s relaxation state. Statistics are also relevant to predict the user’s behaviour. The use of natural blinking objects, such as advertisements, may inform the system about the user’s preferences and help to determine his/her choices. Also, it can be useful to keep a database of statistics from other users. For example, if 80% of people answer “yes” to a form in the game, the “yes” button could be given a higher weight (visually, or by putting a weight in the BCI engine output) to facilitate this selection.

(2)

Using appropriate stimulation. Recommendations given in the section ‘Visual fatigue’ also apply here. In addition, the use of a spatial frequency in the visual stimuli is known to generate high-frequency oscillations in the EEG that can be used to help the detection in P300-based BCIs [80] (LoE C; LoI 3). The shape, colour and timing of the stimuli may also play a role: reference [81] showed that stimuli representing faces lead to better classification (LoE A; LoI 1) while [81,82] suggest that the use of contrasted colours and the modification of flash duration impact the accuracy (LoE B; LoI 1).

(3)

Reduce the time needed to trigger an action and make each action non-essential. Increasing the number of flash repetitions leads to higher classification accuracy, but this forces the user to stay focused for a longer time (LoE A; LoI 1). As a consequence, the fatigue of the user increases, the task is perceived more difficult and the application is less responsive. A compromise between accuracy and responsiveness is to keep the number of repetitions low while making the BCI commands non-critical. For example, in a car-driving application, at each repetition of flashes the trajectory may be slightly adjusted in the sought direction, thus, despite occasional errors, on the long run the player will succeed in giving the car the sought trajectory. Two other examples from previous studies are the Brain Invaders [17] and the Brain Painting [83]. In the Brain Invaders an alien is destroyed after each repetition, however such action is not critical since the player has eight chances to hit the target, hence to finish the level. Brain Painting is a game that is used by patient suffering from the locked-in-syndrome [28]. It consists in a P300 speller where the selection items are special tools for drawing. The concept itself retains our attention because the errors are not critical since the painting can be always retouched without the need of starting again.

(4)

Dynamic stopping (LoE B; LoI 2). Current P300-based BCIs usually make use of a fixed number of repetitions, forcing the user to keep focusing even if the BCI may have already successfully detected the target. Dynamic stopping consists in determining the optimal number of repetitions required to identify the target and thus it can decrease the time required for selection and provide higher robustness and performance [84,85].

(5)

Use feedback (LoE B; LoI 2). Study [27] recommends the use of positive feedback. Nevertheless, feedback is mainly used for motor imagery-based BCIs, while it finds little use in P300-based BCIs. In the Brain invaders [17] there is a binary feedback that indicates if the result is correct or wrong. However, the feedback does not indicate how close accurate selection is. Also, people playing video games are used to immediate feedback: when driving a virtual car, there is no appreciable delay between the command and its effect on the scene. That is to say: the feedback must be given as soon as possible. In the presentation of an EEG acquisition unit prototype, A. Barachant (http://www.huffingtonpost.fr/2017/11/22/brain-invaders-le-jeu-video-qui-se-controle-par-la-pensee_a_23284488/) used a probabilistic feedback that set the size of each item according to its probability of being the target chosen by the user. The feedback is updated after each item is flashed. This idea could be a starting point for designing an appropriate feedback for P300. Another established way to use a feedback is to analyse the error-related potentials, which are produced by the brain after an error feedback is delivered to the subject. This can be used to automatically correct erroneous BCI commands, effectively increasing the consistency and transfer rate of the BCI [86,87,88].

(6)

Control non-critical aspects of the game (LoE C; LoI 1). In a race game for example, the speed is a critical aspect of the game and should not be controlled by a BCI. However, a BCI may be used for triggering a “boost effect” that would help the player by temporarily increasing the speed of the vehicle. Such a triggered effect would have an incidence on the score, but would not be an obstacle to finish the game. Also, we suggest restricting the use of the BCI to a limited set of aspects (LoE C; LoI 1).

(7)

Use a cone of guidance, as already defined in section “Synchronous BCIs” (LoE B; LoI 1).

(8)

Define levels of difficulty (LoE C; LoI 1). The above parameters could be set as a level of difficulty in the game, with the following limitations. First, the expected behaviour must be known by the game. This is the case for Brain Invaders [17], were the player is expected to concentrate on a specific alien, but not the case for a P300 puzzle game for example, where the player can place the puzzle pieces in any desired order. Second, lowering the difficulty lowers the impression of control. In general, it is not recommended to use adaptive difficulty, as suggested in [89].

Intention to Select

Limitation

Looking at a stimulus does not mean we want to trigger an action. For example, one can look at a door without having the intention to open it.

Recommendation

Current designs of P300 applications suppose that the user is focusing on the target even if the user is not looking at the screen at all. It has been suggested to use motor imagery [74] or the analysis of alpha rhythms [14] as a supplementary input to enable the user to signal the intention to select (LoE B; LoI 2).

Another option is to define a threshold for the certitude of the P300 classifier, below which the application will not take any decisions. This is what dynamic stopping (see also the recommendation for diminishing the “Low Transfer Rate”) performs by dynamically changing the number of repetition according to the certitude of the P300 classifier. Again, study [90] presents a benchmark of the methods for dynamic stopping (LoE B; LoI 1).

2.3. Limitations That Are Common to VR and BCI Technology

A major limitation of both VR and BCI hardware is the price. This is currently steadily decreasing and we expect it to become very affordable in the next few years (LoE A; LoI 2). The tendency can be explained by the increasing interest for such technologies, which foster larger productions permitting the price for a single unit to be lowered. All studies are consistent in saying that the BCI and VR market are both increasing. To cite some examples, consulting company Business Insider (London, UK) and Grand View Research (San Francisco, US) predicts that the VR and BCI market will reach 60 million and 1.77 billion of dollars in 2022, respectively. This can be compared to the BCI market today, which is estimated to be about 807 million dollars and to the VR market, which has been negligible until 2015.

3. Type of Game Recommendations

Reference [27] studied the possible applications of BCI technology depending on the kind of game. In the following, we review the recommendations given by these authors.

-

Real-time strategy (RTS) games are too complex and need continuous control, thus P300 does not suit them. In RTS games, P300 can still be used, but restricting it to non-critical control aspects. In general, however, RTS is not particularly adapted to the VR context, for only 7% of existing VR games are of this kind (Figure 6a). For these games, the recommendation is to have a third-person point of view. For example, the player’s avatar controls a map representing the game field and this map is an object in a virtual room.

-

Role-play games (RPG) are also problematic for P300-based BCI because of their complexity (LoE A; LoI 1). The general recommendations are the same as for RTS. The RPG should be turn-based and the BCI should be restricted to minor aspects of the game. An existing example is “Alpha wow” [9] where the user’s mental state is used to change the avatar’s behaviour in the virtual world.

-

Action games are the most popular type of game employing BCI technology (LoI 1; LoE A: [27]). This is surprising since action games often include fast moving gameplay. For this reason, the use of BCI is not recommended with action games without specific adaptation.

-

Sport games meet the same requirements as action games [27]. They often require fast-moving gameplay and continuous control. As a consequence, we do not recommend the use of BCI for sport game without specific adaptation (LoE A; LoI 1), such as the one given as an example in Table 1. In VR, sport games represent a moderate percentage of the games (9%, Figure 6a).

-

Puzzle games are very well-suited for P300-based application (LoE A; LoI 1). They should be turn-based, allowing the users to make simple choices at their own pace. The use of popular existing puzzles helps players because they are already familiar with the game’s rules. However, the problem is the same as for strategy games and board games in general, i.e., it is not very useful to adopt a 3D perspective with a board game (puzzle games represents only 3% of VR games—Figure 6a). We suggest the use of the same workaround as for strategy games and to use a third-person point of view (LoE C; LoI 1). In such a scenario, puzzle games may be presented as a board game inside a virtual room. Another idea is to design a puzzle in 3D allowing the player to move the pieces in all directions (and to move inside the puzzle itself).

-

Adventure games are well-suited for P300-based BCI, if the player is given a set of limited options within a given time interval (LoE A; LoI 1).

-

Simulation games (for training or education purpose for example) are also well-suited for P300-based BCI, especially in the case of management simulation. Simulation games should feature a slow gameplay, allowing the player to adjust and learn how to control the BCI. In addition, simulation games are not based on “score”, and therefore the player can relax and obtain better performance using a BCI control (LoE A; LoI 1).

In conclusion, the P300 paradigm suits well turn-based strategy game (board games such as chess or some PC game as Civilization or Heroes of Myths’ and Magic). Adventure and simulation games appear to be the most adapted types of games for a BCI+VR game (LoE A; LoI 1). In Figure 6b, these two types of game are highly suitable for either VR or BCI technology.

4. Conclusions and Discussion

4.1. Summary

In this article we have exposed limitations of current BCI-enriched virtual environments and recommendations to work around these limitations. The recommendations address several software and hardware problems of currently available systems. We have proposed different ways to resolve or circumvent these limitations. Hopefully, this will help and encourage game creators to incorporate BCI in VR. We have focused on P300-based BCI since as per today this BCI paradigm features the best trade-off between usability and transfer rate. An essential recommendation is to use the BCI only for small and non-critical tasks in the game. Concerning software limitations, using actions naturally integrated into the virtual world is important. A cooperative game is also a good solution since it enables multiple actions and enhances social interaction and entertainment. In addition, the use of passive BCIs appears essential to bring a unique perspective into VR technology. In fact, only a BCI may provide information on the user’s mental state, whereas for giving commands, traditional input devices are largely superior to current BCIs. However, more promising results can be obtained combining different stimulations for BCI, such as coupling a visual P300 to an audio P300, or mVEP to a SSVEP. In general, BCI integrates easier with turn-based games that require high levels of concentration and logical thinking (for example: strategy, artificial life, simulation, puzzle and society games). Among these games, simulation and adventures games appear the best choice for VR. However, P300-based BCI technology may be used in other types of games to control a specific action in the game and for increasing the level of excitement (like for sport or an RPG game). Concerning hardware limitations, an ideal solution would be to use a VR device with an embedded EEG headset, together with sensor fusion capabilities to track precisely the position of the user and the rotation of the head. The main recommendation for avoiding user sickness is to avoid unnatural effects, like a lag in the animation, performing user motion when the user is not actually moving or modifying the natural parallax of the user. Furthermore, locomotion in a virtual world should imply a motion of the user itself, which is not always possible considering the gesture to be performed (such as swimming, climbing or flying) and the space where the game takes place (the virtual world can be bigger than the real space). In such case, designers can use teleportation or walk-in-place to help moving in the virtual world, keeping in mind that cutting the scene and translations should be avoided. Motion sickness also varies according to individual characteristics such age, genre or psychological abilities, but it can be diminished by experience and by taking regular breaks when using VR devices.

4.2. Consideration, Challenges and Perspectives

The recommendations we have listed are numerous. We believe that a framework is required in order to maximize their usefulness. As a matter of fact, the global picture is complicated by the heterogeneity of gameplay modes in different types of games. To build such framework, there is a need of a posteriori data and of a method that can evaluate the impact of recommendations. As stated in [7], the current methods and paradigms devoted to interaction with games and BCI based on visual stimulation remain in their infancy. Future investigations in the human computer interaction (HCI) domain are needed to overcome the limits of BCI and facilitate its use within virtual worlds. Along these lines, [27] suggests that Fitt’s law may be used to compare BCI application designs. Fitt’s law is based on the assumption that in any game the objective would be to minimize the time required to accomplish a mission as well as minimizing the concentration or effort required by the user. This law is a good way to evaluate designs and elaborate patterns for BCI games in conjunction with VR applications. For instance, in designing BCI technology in the healthcare domain, [28,65] describe several concerns about the daily usage of the BCI for patients suffering from disabilities, including ergonomics of the electrodes and functional requirement from the patient. Then, the authors designed and created a system meeting the requirement of the patient, before testing it through a standardized questionnaire of satisfaction [91]. The recommendations we have developed here appear to be a first and a necessary step in the creation of a BCI+VR game at an “out-of-the-lab” destination.

Finally, an extended discussion is needed concerning ethical concerns and the medical consequences of video game designs [92,93]. About the latter, reference [94] introduces the term of VRISE (virtual reality induced symptoms and effects) and reports than VRISE might be serious for a small percentage of people, even if the symptoms seems to be short-lived and minor for the majority of people. Virtual reality might modify heart rate, induce nausea and increase the level of aggressiveness [95,96]. It is not clear, however, if these effects persist (more than a few days) or if they are temporary. Concerning the positive effects of VR-based therapy, after analyzing 50 studies on the subject, [97] concluded that the effectiveness of such therapy remains still unproven. We are not aware of any study concerning the long-term side effects of control-oriented BCIs, however, it exists a long-lasting literature demonstrating the potential of BCI technology for neurotherapy (i.e., neurofeedback, see [98]). A natural question arises concerning the long-term possible side-effects and therapeutical effects of using BCI-technology in VR environments.