Virtual Reality-Assisted Awake Craniotomy: A Retrospective Study

Simple Summary

Awake craniotomy (AC) with brain mapping by direct electrical stimulation for tumors within or adjacent to eloquent brain regions is a surgical approach for minimizing the risk of postoperative neurologic deficits and preserving the patient’s health-related quality of life. Language and motor functions are frequently mapped, but mapping is less frequent for more complex functions, such as visuospatial and social cognition, despite the importance of these functions for daily life activities. This lack of mapping for these functions results at least in part from a lack of tasks fully compatible with the restrictive environment of an operating room and AC procedures. We show here that the use of a virtual reality headset with eye tracking opens up new possibilities for the mapping of these complex functions.

Abstract

Background: Awake craniotomy (AC) with brain mapping for language and motor functions is often performed for tumors within or adjacent to eloquent brain regions. However, other important functions, such as vision and visuospatial and social cognition, are less frequently mapped, at least partly due to the difficulty of defining tasks suitable for the constrained AC environment. Objective: The aim of this retrospective study was to demonstrate, through illustrative cases, how a virtual reality headset (VRH) equipped with eye tracking can open up new possibilities for the mapping of language, the visual field and complex cognitive functions in the operating room. Methods: Virtual reality (VR) tasks performed during 69 ACs were evaluated retrospectively. Three types of VR tasks were used: VR-DO80 for language evaluation, VR-Esterman for visual field assessment and VR-TANGO for the evaluation of visuospatial and social functions. Results: Surgery was performed on the right hemisphere for 29 of the 69 ACs performed (42.0%). One AC (1.5%) was performed with all three VR tasks, 14 ACs (20.3%) were performed with two VR tasks and 54 ACs (78.3%) were performed with one VR task. The median duration of VRH use per patient was 15.5 min. None of the patients had “VR sickness”. Only transitory focal seizures of no consequence and unrelated to VRH use were observed during AC. Patients were able to perform all VR tasks. Eye tracking was functional, enabling the medical team to analyze the patients’ attention and exploration of the visual field of the VRH directly. Conclusions: This preliminary experiment shows that VR approaches can provide neurosurgeons with a way of investigating various functions, including social cognition during AC. Given the rapid advances in VR technology and the unbelievable sense of immersion provided by the most recent devices, there is a need for ongoing reflection and discussions of the ethical and methodological considerations associated with the use of these advanced technologies in AC and brain mapping procedures.

1. Introduction

Brain mapping by direct electrical stimulation (DES) during awake craniotomy (AC) is generally used to evaluate language and motricity in order to preserve these functions [1,2,3,4,5]. For this reason, AC is mostly performed on the left hemisphere, in which most language processing occurs. However, regardless of the side of the brain on which tumors occur, they have a similar effect on the patient’s perceived quality of life [6,7,8,9]. This probably reflects the effects on other important cognitive functions, such as visuospatial and social cognition, which are mostly right-lateralized [10,11]. Visuospatial cognition supports spatial awareness, perception and the representation of space. Lesions of the network underlying visuospatial cognition are associated with various symptoms, the most common being unilateral spatial neglect (USN). USN is typically associated with right hemisphere damage. It is defined as a failure to report, respond to, or orient in response to stimuli delivered in the space contralateral to the lesion in patients with brain damage. Social cognition encompasses all the cognitive processes involved in social interaction through non-verbal communication, such as the recognition of facial emotion, emotional prosody, eye gaze, empathy and theory of mind (TOM). Empathy is the ability to understand and feel another’s emotions and TOM is the ability to understand and act according to the mental states (beliefs, intentions and desires) of other humans [12]. Visual field defects after tumor surgery can also affect the patient’s perceived quality of life. Permanent hemianopia, which has been largely underestimated by neurosurgeons, is a significant postoperative handicap that impairs daily life activities, including, in particular, the ability to drive a motor vehicle or to read [13,14,15].

Far fewer procedures have been published for the mapping of optic radiation and visuospatial and social cognition than for language and motor mapping [14,16,17,18,19,20,21,22,23,24,25,26,27,28]. This may be explained by the complexity of the corresponding functional anatomy. Unlike the motor areas of the brain, visuospatial and social cognition networks and verbal language networks cannot be localized on the basis of anatomical criteria alone. Indeed, there are structural and functional variations within and between subjects [10,29]. The lack of tasks fully compatible with the restrictive environment of an operating room and AC procedures provides another explanation for the lack of mapping for certain functions [30]. It may be difficult to find a space directly in front of the patient and the patient’s position may not be suitable for performing the tasks. The patient must also give an unambiguous answer within five sec, the maximum duration of DES.

Virtual reality (VR) has boomed in recent years, with increasing numbers of applications for this technology emerging in the field of cognitive and social neurosciences [31,32,33,34]. This computer technology generates realistic images simulating the physical presence of the user in a virtual environment. Access to this technology has increased recently, and virtual reality headsets (VRHs) in particular are now affordable and can be installed in front of the patient’s head [13,35,36,37]. In this retrospective study, we aimed to use illustrative cases to demonstrate the potential of VR to open up new possibilities for the mapping of language, the visual field and complex cognitive functions during AC.

2. Methods

2.1. Patients

We performed a single-center, retrospective study on 64 patients undergoing AC with brain mapping through VR tasks. All the patients came from two clinical trials: ClinicalTrials.gov NCT03010943 and NCT04288505. These two trials were approved by the ANSM (“Agence nationale de sécurité du médicament et des produits de santé”), the ethics committee and the CNIL (“Commission Nationale de l’Informatique et des Libertés”) (NCT03010943: ID-RCB: 2016-A01139-42, ethics committee: CPP OUEST II, date of approval: 18 October 2016; NCT04288505: ID-RCB: 2020-A00074-35, ethics committee: CPP Est II, date of approval: 17 April 2020). All patients signed a written informed consent form.

2.2. VRHs

One of the difficulties in the field of VR research is the rapid technological progress in this field, leading to the regular release of new VRHs. Since we first began using VR in 2014, we have successively used a number of different types of VRHs: the Oculus VRHs DK1 and DK2 (visual field: 100°, resolution: 1280 × 800, refresh rate: 60 Hz) (Oculus, Menlo Park, CA, USA); the Samsung Gear VR combined with a Samsung S7 smartphone (Android platform) (visual field: 96°, resolution: 1440 × 1280, refresh rate: 60 Hz); and the HTC Vive (visual field: 110°, resolution: 2160 × 1200, refresh rate: 90 Hz) combined with an eye-tracking device (Tobii Pro SDK, Danderyd, Sweden). The eye-tracking system collects data at a rate of 120 Hz for various aspects of eye movement, including gaze origin and direction, pupil position and absolute pupil size, with an accuracy of 0.5° in visual angle.

2.3. VR Tasks

VR applications have progressed at a similar rate to VRHs. We initially used non-dedicated VR applications developed as entertainment [37,38,39,40] or to simulate social interactions, with the social platform vTime^® [35,41]. We then developed three types of dedicated applications. The first VR application was developed from the DO80 picture-naming test widely used to assess language function during AC (VR-DO80) [36,37]. The second VR application was developed from the Esterman test for intraoperative visual field assessment during AC (VR-Esterman) [13,15]. The Esterman test is the current gold standard for testing binocular visual fields and is used officially for driving license authorization in accordance with European recommendations. The third VR application developed was an interactive application called VR-TANGO (task for awake neurosurgery exploring gaze and TOM), which was developed for the simultaneous assessment of visuospatial exploration and detection of social cues, such as low-level TOM, including gaze processing and emotion recognition. The scene displayed by the VRH shows five avatars in front of a variable background (Figure 1A,B). One of the avatars is in the center and the others are located in the four quadrants of the VRH field visualized. Each avatar has a different eye-gaze direction. The patients are asked to identify the avatar making eye contact with them. The avatar expresses a dynamic facial emotion 0.6 sec after the patient establishes visual contact with the avatar to be found or stares at any other avatar for longer than 0.6 sec. The patient must identify the emotion expressed: joy, surprise or anger. The patient can also describe their assessment of the avatar’s intention to communicate (mental state attribution). The four quadrants of the visual field and all the emotions (joy, surprise or anger) were presented at random to patients.

2.4. Operative Procedure

The procedure used has been described in detail elsewhere [36,37]. Before surgery, all patients underwent a neuropsychological evaluation and fMRI, and were trained in the use of VR applications. The operating room organization is depicted in Figure 1A; it provides the neurosurgeon with real-time feedback for the VR task. The entire AC procedure was performed in the presence of an engineer, a neuropsychologist and/or speech therapist. An additional technical resource was needed to place and maintain the VRH on the patient’s head during VR tasks. Under general anesthesia, the patient was positioned in a supine or lateral position, according to the location of the tumor, with a rigid pin fixation of the head. Once the craniotomy, guided by neuro-navigation, was complete and the dura had been opened, the patient was awakened. Cortical DES was performed with a bipolar electrode delivering a biphasic current in association with motor and/or language tasks, such as the DO80 picture-naming task, performed using a computer tablet. Movement and/or spontaneous language were continuously monitored during tumor resection and a second mapping with subcortical DES was performed if necessary. VR applications were proposed during cortical or subcortical DES, and sometimes during the closure time, to distract the patient. The use of a typical task on a computer tablet or through VRH during AC is highly “personalized medicine” and the tasks were selected before surgery (during presurgical planning) with the neuropsychologist and the speech therapist. Heart rate, blood pressure and electroencephalogram (EEG) signals were recorded continuously during the procedure.

3. Results

In total, we evaluated VR tasks performed during AC for 64 patients: 37 men and 27 women, with a median age of 51 years (range: 23 to 75 years) (

Table 1. Characteristics of the 64 patients and the 69 AC procedures (AC, awake craniotomy).

Patients (n = 64)		n	%
Age (years)	median (range): 51 years (23 years–75 years)
<70		61	95.3
≥70		3	4.7
Sex
Male		37	57.8
Female		27	42.2
Handedness
Right		58	90.6
Left		6	9.4
Tumor location
Hemisphere
Left		38	59.4
Right		26	40.6
Lobe
Frontal		31	48.4
Temporal		3	4.7
Parietal		14	21.9
Occipital		2	3.1
Insular		1	1.6
Temporo-parietal junction		8	12.5
Fronto-temporo-insular		5	7.8
Tumor histology
Glioblastoma		21	32.8
Anaplastic astrocytoma		16	25.0
Anaplastic oligodendroglioma		9	14.1
Oligodendroglioma grade 2		3	4.7
Metastasis		14	21.9
Benign cystic lesion		1	1.6
Awake surgery (n = 69)		n	%
Duration of surgery	median (range): 2 h 17 min (1 h–5 h 20 min)
Time awake	median (range): 1 h 39 min (38 min–4 h 30 min)
Intensity of stimulation (mA)	median (range): 2.0 mA (0.5 mA–8.0 mA)
Intraoperative seizures		14	20.3
Hemisphere location
Left		40	58.0
Right		29	42.0
Duration of VRH use per patient	median (range): 15.5 min (3.0 min–53.0 min)
VR tasks
Number per AC
One task		54	78.3
Two tasks		14	20.3
Three tasks		1	1.5
Type
VR-DO80		42	60.9
VR-TANGO		33	47.8
VR-Esterman		10	14.5

). Five of these patients underwent two operations: four for revision surgery and one for metastases in both cerebral hemispheres. Six patients were left-handed. The location of lesions was frontal (n = 31), parietal (n = 14), temporal (n = 3), occipital (n = 2), insular (n = 1), temporo-parietal (n = 8) and fronto-temporo-insular (n = 5). The lesions were of the following types: glioblastoma (n = 21), anaplastic astrocytoma (n = 16), anaplastic oligodendroglioma (n = 9), oligodendroglioma grade 2 (n = 3), metastasis (n = 14) and benign cystic (n = 1). AC was performed on the right hemisphere in 29 (42.0%) of the 69 operations performed. The median duration of surgery was 2 h 17 min (first incision to closing) and the median duration of the awake phase was 1 h 39 min. The median intensity of DES was 2 mA (range: 0.5 mA–8 mA). In total, one AC (1.5%) was performed with all three VR tasks (VR-DO80, VR-Esterman and VR-TANGO), 14 ACs (20.3%) were performed with two VR tasks and 54 ACs (78.3%) were performed with one VR task. The median duration of VRH use per patient was 15.5 min. The length of surgery with the use of VR tasks was similar to that previously reported with “traditional tests” (median (range): 2 h 43 min (1 h 28 min–5 h 25 min)) [42]. When we first began using this technique, we encountered difficulties positioning the VRH on the patient’s face due to the head holder. These difficulties were overcome by carefully positioning the VRH before the head holder, and before drawing the incision line. None of the patients experienced “VR sickness” during the 69 ACs. Intraoperative seizures of no consequence and unrelated to VRH were observed in 20.3% of cases. Some of the patients for whom VR was used as a relaxation tool during closure described it as pleasant and able to decrease pain and anxiety. Nevertheless, we have stopped using these applications during closure because the patients are generally exhausted at the end of the mapping phase and prefer to fall asleep.

VR-DO80 was applied in 42 ACs (60.9%) (Table 1), and two versions were used with the VRH (Figure 2A). One version was two-dimensional and included the same images as the DO80 presented with a computer tablet (an image accompanied by the sentence “this is…”). The second version included the same items, but in stereoscopy, rotating in an empty virtual space (Figure 2B). Eye tracking data showed that patients never looked at the sentence “this is…”. This observation suggests that the patient “said” the sentence to themselves automatically, focusing only on the naming task. We observed that some areas for which the result was unclear during DES with the computer tablet (hesitation or delay in denomination) were clearly not eloquent areas during mapping with the VRH [36,37].

Visual field assessment was performed using the VR-Esterman task in 10 ACs (14.5%). This procedure was reserved for rare situations, such as lesions close to optical radiations, patients with a pre-existing scotome or blindness of the contralateral eye, and patients for whom a normal visual field was essential for their professional activity. The first VR-Esterman task used was performed using an Oculus VRH [13]. We then adapted this software to the HTC Vive combined with an eye-tracking device, to explore a visual field of 80° with eight possible red dots on a black background (Figure 2C,D). The maximum visual field attainable in the HTC Vive is 110°. However, we chose to retain the foam and not to press the VRH as close to the face as possible, so as to maximize user comfort. This configuration gave a visual field of 80°. Eye tracking made it possible to track the patient’s gaze accurately and to ensure that the patient really did focus on the central visual axis during visual field testing, without eye saccades. The VR-Esterman task made it possible to identify optical radiations. DES generated “positive” and “negative” phenomena. The positive phenomena were phosphenes in the corresponding contralateral visual field, which were easy for the patient to detect on the black background in the VRH. The negative phenomena were scotoma, with non-visualization of the red dot in the visual field tested.

The VR-TANGO task was applied during 33 ACs (47.8%). By associating this task with the eye tracking data, we were able to detect difficulties exploring the space, difficulties locating the face of the avatar trying to make eye contact, and failures to recognize the facial emotion, resulting in a delay or a lack of response from the patient. During bedside evaluations, we found that the VR-TANGO task was not affected by homonymous hemianopia or other visual field defects, such as quadrantanopia or bitemporal hemianopia. Conversely, performance in this task was impaired in patients with USN and/or difficulties with emotion recognition or attention (Figure 3A,B).

During AC, several defects induced by subcortical DES with the VR-TANGO task were observed. For example, DES of the superior longitudinal fasciculus (SLF) induced mild USN, with the patient having difficulty exploring the left part of the VR screen (Figure 4B–D). Without DES, the patient was able to explore the entire VR screen in gaze exploration (Figure 4A). During DES of the inferior longitudinal fasciculus (ILF), one patient easily identified the correct avatar, but was unable to analyze or describe the facial emotion expressed (data not shown). This is not surprising, because the ILF and other ventral tracts have already been implicated in the recognition of facial emotions [43]. When the frontal aslant tract (FAT) was stimulated intraoperatively, one patient ignored the communicative cues. The gaze of this patient remained fixed on the first avatar encountered or went from face to face, with the patient unable to identify the avatar making eye contact or expressing a facial emotion (Figure 5A,B).

4. Discussion

This retrospective study shows that VR opens up new possibilities for the mapping of language, visual field and complex cognitive functions during AC. In previous studies, we demonstrated the feasibility and safety of the VR procedure [36,37]. The use of a VRH during AC does not specifically increase the rate of intraoperative seizures. Nevertheless, several precautions should be taken when using a VRH for brain mapping procedures, including the performance of these procedures by a well-trained team, with a well-trained patient and, despite the lack of consensus concerning its utility, intraoperative monitoring of brain electrical activity. The setup described here requires no additional hardware in the operating room other than the VRH, which is affordable for most neurosurgery centers. Its use adds to surgery time but, as for conventional awake testing (one set of the TANGO tasks takes five sec to perform), it does not significantly disrupt the surgical workflow.

We showed that the DO80 picture naming test and the Esterman test, the gold standard for language and visual field assessment, respectively, can be adapted to VRH combined with eye tracking. Patients were able to perform these VR tasks during surgery. Eye tracking was functional, making it possible to trace the gaze of the patient during the tasks. An exploration of 40° on either side of the central visual axis could be explored during the VR-Esterman task. This exploration does not meet all the French criteria for testing for driving licenses but can detect important visual defects that may impair quality of life. The technical features of VRHs will undoubtedly improve in the future, and it will probably soon be possible to explore a wider visual field with these devices. The adaptation of other language tasks, such as text-reading language tasks, for VRHs combined with eye tracking would be potentially interesting. Furthermore, we recently showed that, even several years after the end of treatment, adult survivors of primary brain tumor may experience daily life behavioral disorders induced by executive impairments with a negative effect on their health-related quality of life [44]. This highlights the potential benefits of adapting VRH combined with eye tracking for tasks exploring executive functions, such as the Stroop test [45].

The DO80 picture naming test and the Esterman test were easy to adapt for the VRH combined with eye tracking, but we encountered more difficulties for the adaptation of neuropsychological tasks evaluating visuospatial and social cognition. Various tests are used to assess these functions during AC. The line bisection test for visuospatial neglect [46] is widely used, because it is simple, rapid and reproducible [20,26,47,48,49,50]. However, even though this task identifies a profound perceptual disorder, it appears to be anatomically and behaviorally independent of the core symptoms of neglect. In 40% of patients with core symptoms of spatial neglect, no impairment is observed in the line bisection task. One possible explanation for this dissociation is that the line bisection task draws on allocentric representation, whereas the core deficit in spatial neglect is egocentric [51,52,53]. Another allocentric test, the target cancelation task, which involves searching for and crossing out target symbols, is more sensitive than the line bisection task for USN detection, but less widely used during AC [16]. For evaluations of social cognition, facial emotion recognition tasks are frequently performed during AC, based on photographs of individuals displaying one of the six primary facial emotions (anger, happiness, fear, surprise, disgust and sadness) extracted from several tests (Ekman’s Faces, the Brief Affect Recognition Test, the Japanese and Caucasian Brief Affective Recognition Test, and the ATR facial expression database) [18,54,55]. Published results suggest that, when presented for 10 sec at the patient’s bedside, the emotion portrayed by each photograph is correctly identified by more than 70% of patients. In our experience, tests involving the recognition of facial emotions from photographs are difficult to perform during AC, with a high rate of error, even in the absence of DES. Dynamic facial expressions are more accurately recognized than static expressions [56,57]. A simplified and adapted “reading the mind in the eyes test” (RMET) based on photographs centered on the eye region is also performed in patients undergoing AC for right-side low-grade gliomas, to explore TOM [28,58]. However, RMET appears to measure emotion recognition rather than TOM ability [59].

Rather than immediately developing a specific application to test and map social cognition, we initially evaluated the potential of the available VR social networks, such as the social VR application vTime^® [35]. This application simulates virtual social interactions with an avatar controlled by the neuropsychologist, who also wears a VRH. However, we experienced several limitations of the use of the social VR application vTime^®, precisely because of the lack of control of all potent non-verbal language cues, including facial expressions and eye gaze. We therefore decided to pursue our efforts to explore visuospatial cognition and non-verbal language during AC by developing an interactive VR application capable of analyzing these functions simultaneously. This project was not an easy one, but the solution was found in Argentinian tango dancing (a hobby of one of the authors). The invitation to dance the tango is highly codified, lasting only a few sec and including spatial exploration of the dance hall, tracking someone who looks at you, social attention fixed on a partner but ready to be reoriented if gazed at by someone else, analysis of faces, interpretations of the emotions of the other person and guessing what he/she thinks from his/her movements, his/her desire to be invited to dance and his/her probable reaction. We adapted this short scenario with VR technology and named this task “TANGO”.

The development of a program of this type was challenging, for several reasons, including the need for the animation to appear natural. A neutral face, capable of producing a smile at the same time as a slight movement of the head, associated with a gaze capable of making eye contact with the patient, must be achieved with professional motion capture tools. We achieved this in collaboration with a group of engineers using techniques from video games and the movie industry, in which facial animation is achieved by filming an actor and transposing his or her movements onto the avatars (“Hardware—Dynamixyz”). We aimed to reproduce a social scene by having several avatars in a virtual world. However, although faces have a spatial advantage for capturing attention, reflecting their particular saliency and their social value, the maximum number of faces that can be analyzed in a visual field of 110° in less than five sec (the maximum time for DES) is five. The scene was therefore designed to include four avatars in the four corners of the visualized field. The patient was asked to search for the avatar trying to make visual contact and to describe the automatically triggered facial emotion of the avatar or their feelings about the desire for communication/social contact as expressed by the avatar. The preliminary results with this application showed that, unlike the line bisection task, the VR-TANGO test was unaffected by homonymous hemianopia [60]. By contrast, the performance of the TANGO task was impaired by USN, with the patient unable to direct attention to the left side. The TANGO task can be seen as a cancelation test with distractors, and is therefore very sensitive for the detection of USN, probably more so than the line bisection test. We observed that the DES of some subcortical areas of the brain affected the response to this test. For example, when the right FAT was transiently disrupted by DES, the patient’s gaze shifted from one face to another, unable to identify which avatar was trying to make eye contact, or remained fixed on the first avatar encountered regardless of emotion or intent to communicate. The involvement of the FAT in the expression and recognition of communicative intentions may account for this behavior [61,62,63]. The right FAT has a putative role in the support of executive function, through inhibitory control and conflict monitoring for action, which may provide an alternative explanation [64,65]. These preliminary data indicate that VR-TANGO may be useful for identifying the various components of prelinguistic social abilities and their neural substrates.

5. Limitations

The main limitation of this study is that it does not provide clinical results for VR tasks during AC in terms of test performance, clinical preservation and extent of resection. Several established bedside tests are currently being performed to evaluate the performance of VR tasks before their use in clinical practice for brain tumor resection in awake patients (ClinicalTrials.gov NCT04288505). For example, the bells test [66], the Ekman test [67] and the RMET [68] are being used to assess the performance of the VR-TANGO task for detecting visuospatial and social cognition impairments. Once this validation is complete, it will be possible to perform prospective studies to assess the benefits of using VR tasks during AC to preserve language, visual field, visuospatial and social cognition or, more globally, the ability to perform daily life activities.

6. Conclusions

VR has a unique advantage over other techniques, in that it provides neurosurgeons with a way of investigating various complex functions, including non-verbal cognition, that would not otherwise be possible in the operating room. The use of virtual environments advocated here should not minimize the contribution of traditional pen-and-paper tasks and/or computer screens/tablets, which have advantages for brain mapping. Eye trackers in VRHs, combining the features of both mobile and remote setups, minimize the risk of calibration loss and improve success rates for the measurement of eye positions and movements. Furthermore, the use of a VRH immerses the patient in the VR task through complete isolation from the surrounding operating room. We acknowledge that the VR-TANGO task we have developed is not deeply immersive. We are currently working on stereoscopic sound and smells, to make the virtual experience more immersive and emotionally engaging. The patient could be immersed in the environment of a coffee bar, for example with the aroma of hot coffee, street sounds and people sitting at neighboring tables. It will soon be possible to use “deep fake” technology to replace the faces of avatars with those of the patient’s friends or family. It will also be possible, in the near future, to map social cognition without the need for verbal participation from the patient. The new VRH technologies are making it possible to track the participant’s behavior and reactions to the VR experience by synchronizing various sources of data, to obtain a holistic, integrated measurement of target social interactions through machine-learning techniques. The most studied data after eye tracking are facial mimicry, pupillometry and electroencephalographic coherence data. Given the speed of the progress in VR technology and the unbelievable sense of immersion given by the latest devices, ethical questions are beginning to arise. Continuous reflection and discussions of ethical and methodological considerations are required concerning the use of these advanced technologies in brain mapping procedures during AC.