Update of fNIRS as an Input to Brain–Computer Interfaces: A Review of Research from the Tufts Human–Computer Interaction Laboratory

Abstract

Over the past decade, the Human–Computer Interaction (HCI) Lab at Tufts University has been developing real-time, implicit Brain–Computer Interfaces (BCIs) using functional near-infrared spectroscopy (fNIRS). This paper reviews the work of the lab; we explore how we have used fNIRS to develop BCIs that are based on a variety of human states, including cognitive workload, multitasking, musical learning applications, and preference detection. Our work indicates that fNIRS is a robust tool for the classification of brain-states in real-time, which can provide programmers with useful information to develop interfaces that are more intuitive and beneficial for the user than are currently possible given today’s human-input (e.g., mouse and keyboard).

1. Introduction

Over the past decade, the HCI (Human–Computer Interaction) laboratory at Tufts University, helmed by Dr. Robert Jacob, has investigated the application of fNIRS (functional near infrared spectroscopy) data as input to dynamic brain–computer interfaces (BCIs). Improvements in technology in fNIRS measurements and in real-time machine learning data analysis have made possible a new generation of passive or implicit brain–computer interfaces (BCIs), which have a different goal and target audience from conventional BCI. Most previous work in BCI has focused on severely paralyzed users. BCI for such people usually requires them to explicitly produce patterns of measurable brain signals, for example by imagining that they are moving their toe or that they are saying a particular word. The resulting user interfaces may be slow and awkward although they are life-changing for those patients with “locked in” syndrome [1]. This review focuses on research conducted in the Tufts HCI lab into passive BCIs, which attempts to bring some of the benefits of BCI to a wider range of “healthy” users, that is, users who have no disability or paralysis. Our work has demonstrated that fNIRS can be used as an effective tool to identify a variety of cognitive responses in real-time, and that this capability can be used to develop and evaluate user interfaces based on fNIRS signals. Our fNIRS based interfaces passively collect data from the user, process that data, and then make helpful adjustments to a user-facing system based on the data. This BCI design paradigm is known as implicit BCI.

2. Related Work

There exists a body of research into the use and design of implicit brain–computer interfaces. First termed by Cutrell and Tan as “passive BCI”, they differentiate direct brain–computer interfaces from passive ones: “We think there is a potential to use brain sensing in a more passive context, looking beyond direct system control to make BCI useful to the general population in a wide range of scenarios” [2]. Zander et al. further discusses implicit interaction, defining it as “... an unconscious action that is integrated in another action, for example mimic and gesture” [3]. The continuing investigation of this paradigm has yielded interfaces which react implicitly to the brain-state of the user, such as cognitive workload, emotion, and attention.

2.1. BCIs and Mental Workload

The field of neuroergonomics subsumes the study of mental workload and neuroimaging. In a review of the field at large, Parasuraman and Wilson state that “real-time neurocognitive assessment of workload can trigger adaptive automation” [4], suggesting the suitability of neuroimaging devices for the detection of cognitive workload. The study of cognitive workload with passive BCIs has been further investigated, notably by C. Herff and T. Schultz. In a research article where they show that workload can be quantified via tests such as n-back tasks, Herff et al. [5] demonstrate that high versus low mental workload states can be differentiated with a 78% accuracy in a single trial. This line of research was further explored by Ayaz et al. [6] through the assessment of participants’ workload states via n-back and more complex mental tasks, and by observation of development of skills for complex multi-modal tasks. Notably, their results suggest that hemodynamic responses detected by fNIRS are related to mental workload. Furthermore, their results showed that expertise level is detectable in the hemodynamic response of the left dorsolateral pre-frontal cortex for some tasks.

2.2. Current State-of-the-Art of fNIRS and Short-Wave Infrared Spectroscopy

Although many current state-of-the-art BCIs rely on fNIRS, there is potential to develop BCIs that rely on other spectroscopy methods. Short-wave infrared spectroscopy (SWIR), a cousin to fNIRS, has the potential to measure concentrations of water, lipids, and collagen, whose absorption coefficients lie within the short-wave infrared spectrum of ~1000–2000 nm [7]. Water and lipids, whose absorption coefficients are higher than that of oxygenated and deoxygenated hemoglobin, are also measurable by fNIRS. However, the near infrared spectral zone detected by NIR devices have distinct advantages: Diminished scattering and absorption, deeper imaging capability, and abundant silicon photo detectors [8]. Unfortunately, an fNIRS device configured to measure lipids forgoes the detection of changes in blood-oxygenation as the metric by which to adapt an interface. Nevertheless, it is possible that the information encoded in the short-wave spectrum could be useful to drive the brain–computer interfaces of the future.

3. Experimental Framework

Our work typically uses fNIRS to measure oxygenated and deoxygenated hemoglobin concentrations in the rostromedial and rostrolateral prefrontal cortices. The overall experimental design for fNIRS BCI research often is composed of two phases—a training phase, where data is collected while the user is engaged two conditions and used to train a machine learning classifier, and a testing phase, where that classifier is used to dynamically adapt an interface in real-time. We typically filter the data, most recently employing near source-detector pair RLS adaptive filtering techniques [9] made popular by Zhang et. al. [10]. As the prefrontal cortex has been shown to activate under high mental workload conditions, the machine learning classifiers we develop are often attempting to distinguish between high and low mental workload conditions. In addition to showing working on-line machine learning classifiers, we also demonstrate measurable performance improvements through the use of our systems.

4. Specifics of fNIRS Configuration

We use a multichannel frequency domain Imagent fNIRS device from ISS (Champain, IL, USA) for data collection. We have two probes, each with four pairs of source-detector light sources; each pair emits near-infrared light at 690 nm and 830 nm, totaling 16 channels, 8 Hbb and 8 HbO. Each probe has three pairs of 3 cm sources, as well as an 8 mm source, which we have recently used as the short-separation channel for adaptive filtering calculations to reduce noise in the signal. We use Boxy Software provided by ISS to extract the raw AC, DC, and Phase values, and then use a mathematical library (also provided by ISS) to perform the Modified Beer–Lambert law calculations to determine the relative oxygenated and deoxygenated hemoglobin concentrations from a specified starting point. Different experiments have used slightly different preprocessing methods, as well as different machine learning approaches to distinguish the two brain-state conditions; relevant results can be found in Appendix A.

5. Advantages and Challenges of fNIRS as an Input to Implicit Brain-Computer Interfaces

fNIRS presents unique advantages and disadvantages in comparison to other BCI inputs. Initially, the device was designed for clinical use, which intrinsically presents limitations for commercial and consumer applications. However, in comparison with EEG and fMRI, fNIRS retains particular advantages. Some such advantages of fNIRS as an input to BCIs compared with fMRI include: Ease of use and setup, lower cost, as well as portability [11]. Excellent spatial resolution further provides another advantage when compared with EEG [12]. However, we will also discuss some important considerations to be made concerning fNIRS as an input to a BCI system.

5.1. Consideration of the Hemodynamic Response in fNRIS-Based BCI

One important issue of consideration for BCIs is the speed of the hemodynamic response: Given that the response is 6–9 s [11], it is infeasible to implement interfaces that have instantaneous reactions akin to what is manageable with EEG; however, the relatively high spatial resolution of fNIRS allows us the ability to be more precise in terms of areas of brain activation [11]. Given that we are inspecting the rostrolateral and rostromedial prefrontal cortices, for instance, we can modify our interfaces to be sensitive to mental workload and other mental states related to these specific prefrontal regions.

5.2. Sources of Signal Interference

fNIRS is also subject to signal degradation and interference from numerous factors. While some sources of signal degradation are benign to classification tasks, others are detrimental and should be mitigated through appropriate methods.

5.2.1. Movement

The most significant physical movements which contribute to signal degradation are associated with head and facial movement [11]. The motion associated with head movement increases blood flow through the scalp and may increase blood pressure [13]. This interference can be especially significant during situations in which the user is non-stationary and mobile. While the electrical activity associated with blinking and eye movement contributes to EEG signal noise, fNIRS, in contrast, is robust to these movements. However, facial expressions, particularly frowning, which can shift the position of the probes on the user’s forehead, contribute noise [11]. We have determined that overt facial expressions should be avoided. However, the effects of both head and facial movements can be minimized using filtering techniques [13].

Other kinds of physical states which contribute to signal noise include movements associated with computer interaction, respiration, and heartbeat. The very nature of the fNIRS device lends itself to affectations of heartbeat and respiration. Therefore, researchers have long utilized filtering techniques to successfully minimize their effects. In an experiment described in [11], activities associated with keyboard input were examined to determine their influence on signal integrity. It was determined that typing has a small, but acceptable contribution to noise, while mouse clicking is also acceptable if the experiment is controlled.

5.2.2. Environmental

Given that the fNIRS device deals with sensitive light intensity measurements, it is particularly sensitive to ambient light, which necessitates the design and development of devices which prevents such exposure. Reasonable ambient noise has not proven to have an effect on signal integrity [11].

5.3. Methods of Filtering Noise

These factors indicate that filtration of the fNIRS signal is an important step in the data collection process; our lab has approached this task by using different methods, including low pass and third-degree polynomial filtering [14], time-domain band-pass filtering [11], and most recently, by using a near source-detector pair LMS adaptive filtration method [9] from the method of Zhang et al. [10].

6. fNIRS-Dependent Brain-State Distinction Methods

After converting raw values into changes in oxygenated and deoxygenated hemoglobin and the filtration of signal noise, there remains a challenging problem of how to distinguish between brain-state-based signals. In particular, the challenge comes when attempting to train models that can work with a particular subject in a real-time in-trial setting. For instance, although averaged results over multiple participant data often clearly demonstrate increases in oxygenated hemoglobin in the prefrontal cortex during high-workload tasks such as n-back or stroop tasks compared with low-workload tasks such as rest, making these distinctions in a real-time setting, participant by participant, is typically far more challenging. In order to accomplish this task, we commonly use machine learning algorithms to generate a model and classify participants’ brain states [1,2,3,4,5].

6.1. Support Vector Machines (SVM)

Our lab has relied most heavily on SVM, a common supervised machine learning technique that generates a hyperplane linearly differentiating two (or more) classes of data by maximizing the distances of the nearest vectors to the hyperplane in each class. The model minimizes the hinge loss function during the training phase, refitting the linear boundary by maximizing the hyperplane between classes for each new training sample. During the test phase, new test samples are plotted and assigned a class label with respect to where they reside with respect to the decision boundary [15]. Regularization is applied via the soft-margin method, which is determined in model selection [16].

6.2. Feature Extraction

As with any machine learning problem, feature extraction is a critical part of the process. In particular, our case relies on streaming time-series data, for which feature extraction becomes duly difficult. Our common practice is to run trials of ~30 s, and extract a variety of features from each channel. The features we commonly used are: Mean, and linear regression slope [1,4,5,6]. Intuitively, these are sound features for the differentiation of brain states where we expect higher vs. lower blood flow to the prefrontal cortex.

7. Brain-Based Augmentation of User-Interfaces

Our work has shown that a filtered fNIRS signal can prove to be a robust source of information for implicit BCIs. Machine learning models trained on tasks of differing types can provide different types of information to the experimenter. For instance, we have used fNIRS to differentiate between high and low cognitive workload as well as levels of multitasking. These tasks are clearly both related, but the key issue is that their relationship stems from their relationship with the prefrontal cortex, an extremely complex area of the brain. That is to say, a wide range of different approaches to even prefrontal-cortex-based implicit fNIRS BCI are possible. Even within the same category (i.e., mental workload), there are innumerably many potential interface applications. Below are some more detailed references to our previous work vis-à-vis both.

7.1. Cognitive Workload

The study of the mental workload domain has yielded many theories. In a review of the field of neuroergonomics, Parasuraman points out that historically, mental workload has been thought of as brain work. He states that the progenitor of this notion has been supported by the observation of increased blood flow to the prefrontal cortex via fMRI signals [7,12]. Furthermore, measurement of these physiological signals via fNIRS have been shown to indicate a spectrum of cognitive workload, from disengagement to high stress. Using these signals as input, an adaptive interface can implicitly respond to a participant’s cognitive workload, thus yielding an improved user experience and optimized user performance [17]. The lab has chosen to define cognitive workload as related to a user’s perceived difficulty of a task, which is quantifiable by the hemodynamic response observed from an fNIRS device.

7.1.1. Dynamic Difficulty Using Brain Metrics of Workload

A domain that is particularly well suited to the inherent workload sensing abilities of fNIRS is in human–robot interaction. In context of this problem space, it has been demonstrated that as a user’s workload increases, an adaptive response from a semi-autonomous agent may increase the productivity, efficiency, and efficacy of the user in certain tasks [18]. This hypothesis is explored further in an experiment by Afergan et al. [17] wherein participants operated UAVs, or unmanned aerial vehicles, in a simulation. As the participant experienced higher levels of cognitive workload, the UAVs would assume a higher degree of autonomy, yielding optimized participant performance. The operator’s perspective of the UAV simulation is depicted Figure 1.

7.1.2. Experiment Overview

The participant was tasked with managing the flight paths of multiple UAVs simultaneously, a task similar to that of an air traffic controller. Their objective was to guide the UAVs to their respective target locations as quickly as possible, and with as few obstacle collisions as possible. To aid in path planning, the participant was given a bird’s eye view map of the UAVs, their destinations, and obstacles. The participant would incur penalties for leaving UAVs idle, collisions, and flying into no-fly zones. They were also told that, as a member of a team of UAV operators, new vehicles would be passed to them and they would relinquish control of vehicles to other operators. However, UAVs would only be removed when there were no obstacles in their path [17].

The authors aimed to answer two research questions in this experiment:

(1)

Can fNIRS signals be used to determine when a participant experiences high or low difficulty as they engaged in flying multiple UAVs?

(2)

Can the use of real-time fNIRS signals processed by an adaptive interface yield improve operator performance in their UAV navigation tasks?

7.1.3. Experimental Design

In order to calibrate the participant’s workload level, each participant underwent a series of n-back trials. During these trials, the fNIRS device was used to sense the participants workload via signals collected by the fNIRS device from the prefrontal cortex, the signals were processed and then classified by an SVM, indicating high workload versus low workload states. The baseline cognitive state was observed during rest periods between each trial. The trained model is referred to as the Brain-Based Dynamic Difficulty Engine.

After calibration, the participant commenced in the path-planning task. Two conditions were tested:

(1)

Adaptive condition: The Brain-Based Dynamic Difficulty Engine added and removed UAVs according to the participant’s brain signals.

(2)

Non-adaptive condition: The participant’s cognitive state is ignored by the Brain-Based Dynamic Difficulty Engine and UAVs are added and removed at a random interval.

7.1.4. Results and Implications

While controlling approximately the same number of UAV’s for both conditions, the number of successful trials remained equal across conditions. However, the adaptive condition yielded fewer failed trials, fewer entries into no-fly zones, fewer collisions with obstacles and fewer UAVs left idle. Furthermore, under the adaptive condition, participants allowed UAVs to travel shorter distances into no-fly zones than the control condition. Additionally, participants had an average of 1.14 neglected UAVs under the adaptive condition, and 1.37 neglected UAVs under the control condition. We interpret these results to mean that in the adaptive condition, participants were more aware of obstacles and attentive of UAVs.

More broadly, the results of this experiment allow us to conclude that brain computer interfaces such as fNIRS can aid in programmatic task allocation wherein a user’s cognitive workload state is dynamically matched with a task based on task complexity or difficulty.

7.1.5. Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy

In this study, the group utilized fNIRS data to explore the contribution of a user interface design to a user’s mental workload [19]. They hypothesized that a mental workload state receives contributions from two sources: The inherent difficulty of a task and the complexity introduced from the user’s interaction with the user interface. They suggest that the introduction of adaptive interfaces can alleviate the user’s high workload state and increase productivity and task success rates.

7.1.6. Novel Experimental Protocol

With the objective of determining the contribution to workload state, a general protocol was devised to differentiate task difficulty from interface complexity. Provided some UI (user interface), a task analysis is conducted on the UI and a task. fNIRS data is collected from the user during a series of benchmark tasks which stimulate high and low workload states. This is conducted for a task associated with UI task difficulty and a task associated task difficulty. The analysis of the fNIRS data allows the researcher to determine distinct characteristics of user interface related difficulty and task related difficulty.

7.1.7. Experiment Overview

We created two interfaces with which they evaluated a participant’s spatial and verbal working memory (WM). The interfaces allowed the participant to navigate through a simulated environment, termed by the authors as hyperspace. This mode of user interaction facilitated the stimulation of spatial and verbal WM through two exercises. One exercise was meant to stimulate high spatial WM (via traversal of hyperspace) and the other to stimulate high verbal WM (via an information retrieval task). The completion of an information retrieval task via a UI is referred to as an interface exercise. Using the novel experimental protocol, task analysis was completed to derive results.

The author’s sought to answer three questions through their experiment:

(1)

Can fNIRs be used to distinguish brain activity associated with experimental conditions from the rest state?

(2)

Can fNIRs detect the difference between a participant’s no, low, and high-engaged working memory?

(3)

Can well established mental tasks developed to measure working memory reveal information about the syntactic workload for high level user interface tasks?

7.1.8. Experimental Design

The experiment presented 3 conditions:

• Cognition exercises which have been shown to instigate low spatial WM load and low verbal WM.

  ○

  Subjects were asked to view a screen displaying a five-digit number. The screen had a circular fixation point in the center, at which the subject was instructed to keep their focus. After 18 s, the subject was asked to recall the number and location of the cell that the number was located in (see Figure 2).

• Cognition exercises which have been shown to instigate high spatial WM load and low verbal WM.

  ○

  Subjects were asked to view a screen displaying a five-digit number, followed by a screen displaying an arrow, and finally by a screen with fixation point. The participant was tasked with updating the spatial location of the numbers based on the direction of the arrow while remembering the number itself.

• Interface exercises that show users their location in hyperspace while they are tasked with searching for verbal content.

  ○

  The subject was asked to remember a five-digit zip code while searching for the zip code in hyperspace. Subjects navigated hyperspace with arrows at the bottom of the browser page, which were inactive for the first 3 s after a new page was loaded. They were provided with a pictorial representation of their current location. After finding the matching zip code in hyperspace, the subject was instructed to write down the zip code and the corresponding spatial location on an answer sheet.

• Interface exercises that do not show users their location in hyperspace while they search for the zip-code.

This condition replicates the previous condition, with the exception of the pictorial representation of the user’s spatial location. Subjects were instructed to update their location in spatial WM while reciting the zip code in verbal WM.

7.1.9. Results and Implications

To address the first question, the authors used ANOVA to determine discernible differences from our experimental conditions and the no workload condition. They found that all but one of the subjects exhibited results in which experimental conditions were differentiable from the no workload condition.

To address their second question, the researchers utilized the ANOVA results derived to answer the first question. The results demonstrated that all cognitive states associated with the conditions were differentiable.

To answer the third and primary question, the authors clustered the data to gather insight into the relational information between classes. They assumed, based on results from the previous questions, that no workload, High Spatial and Low Spatial exercises indicate a benchmark for spatial working memory. In 90% of subjects, exercises in the No Location condition were shown to have clustered more closely with the higher spatial WM exercises than those under the Display Location condition. In 70% of subjects, the No Location condition exercises are clustered with High Spatial condition exercises. In other words, more spatial working memory is utilized in the No Location exercise than in the Display Location exercise, implying that the No Location condition allocates higher syntactic workload.

We suggest that the use of fNIRS data in usability testing for user interfaces may allow researchers the ability to make generalizations about activation patterns related to the experimental conditions described in this experiment.

7.2. Sensing Cognitive Multitasking for a Brain-Based Adaptive User Interface

The relationship between high cognitive workload states and multitasking are intertwined, and transitively, multitasking can be studied through analysis of fNIRS data. In this study, we apply fNIRS to the subcategories of multitasking and assess their differentiability [20]. Through two experiments which facilitate the detection of various modes of multitasking activity in the prefrontal cortex, we construct an adaptive user interface which responds to the multitasking state of the user. As a proof-of-concept, we present a human–robot interaction scenario in which this user interface facilitates improved task switching, interruption management, and multitasking.

7.2.1. Multitasking Scenarios: Branching, Dual Task, and Delay

Multitasking can be partitioned into three distinct categories, which are associated with various types of tasks and the interactions between those tasks [20].

• Branching occurs when the user is required to maintain overarching goals while investigating and pursuing second goals simultaneously. This is a common form of multitasking which burden users.
• Dual task occurs in situations where the user is constantly switching from one task to the next, but information about one task is not relevant or needed to complete the other. An example of this is where a user may be monitoring something while completing another task.
• Delay occurs when one task is more important than another. The user occasionally acknowledges the non-predominant task while maintaining attention on the predominant task. It is called delay because the acknowledgement of the lesser important task delays the engagement with the predominant task.

In this study, authors Solovey et al. sought to test whether fNIRS was well suited for the detection and distinction of these types of multitasking in a human–robot experiment.

7.2.2. Experimental Overview

Human–robot interaction scenarios inherently require multitasking behavior, and thus this kind of scenario is opportune for multitasking activation detection via fNIRS data and an associated adaptive interface. Through two experiments, in which the user required the assistance of the robot to complete a classification task, we investigated different exercises which stimulated the various multitasking categories. The human–robot team was tasked with rock investigation, in which the robot informed the user with status updates regarding newly discovered rocks or relocation to a new area. With the discovery of a new rock, the user was informed of its class, which is assigned by its size.

7.2.3. Experimental Design

The primary task assigned to the user was to sort rocks while the secondary task was to monitor the robot’s location. When provided with status update, (s)he was instructed to respond by typing “S” with the left hand to signify “same”, or “N” with the right hand to signify “new”. An “S” command indicated that the robot should maintain the same transmission, while an “N” command instructed the robot to commence a new transmission.

Experiment 1 had three conditions which corresponded to the types of multitasking previously outlined:

• Delay condition was demonstrated by the following question: Do two successive rock classification updates follow in consecutive order? If the condition is true, the rock is placed into the same bin. Otherwise, it is placed in the new bin.
• Dual-Task condition was demonstrated by the following question: Do two updates of the same type follow in consecutive order? If this condition is satisfied, then the same response is warranted. Otherwise, the new response is appropriate.
• Branching condition required that for the rock classification updates, the user respond as described in the Delay condition. For location updates, the user was to respond as in the Dual Task condition.

Experiment 2 was conducted as a follow up to the first study in which the objective was to determine if different types of branching tasks could be differentiated. This experiment exhibited two conditions:

• Random Branching: the updates provided by the robot were delivered to the user in a random fashion.
• Predictive Branching: the rock classification updates were delivered to the user after every three stimuli.

7.2.4. Experimental Results

During the first experiment, statistical significance was observed in the response times and accuracies of delay and dual, and delay and branching. However, correlations between accuracy and response time for all tasks were not statistically significant. A main effect of condition was determined through ANOVA of the condition means for each subject, in which the branching condition exhibited the highest hemoglobin measures. From these results, we determined that the hemodynamic signature was differentiable across the three conditions, indicating that there exists some differentiability between the types of multitasking.

During the second experiment, it was determined that there was no statistically significant difference in response time or accuracies between the conditions. While the correlation between accuracy and response time for random branching was not statistically significant, we did find a statistically significant correlation in predictive branching. We also found that deoxygenated hemoglobin levels were higher in the random branching condition than in the predictive branching during the first half of the trials. This trend reversed during the second half of the trials. This indicates that it is possible to differentiate between random and predictive conditions using solely the deoxygenated hemoglobin levels observed in the fNIRS data.

7.2.5. A Brain-based Adaptive User Interface Platform

Encouraged by the results which verify that the cognitive multitasking conditions we have outlined are distinguishable through analysis of fNIRS data, a proof-of-concept brain-based adaptive user interface platform was developed to facilitate implicit communication between a human user and a robot. The proof of concept and its components is illustrated in Figure 3.

Using machine-learning methods to identify these brain states, an adaptive user interface can be built to respond appropriately to the user’s multitasking brain state. We designed the platform to have two main components: The Online fNIRS Analysis and Classification (OFAC) system and the Distributed Integrated Affect, Reflection, Cognition Architecture (DIARC). The OFAC is responsible for receiving the fNIRS signals, classifying them using a multi-class machine learning algorithm, and sending the classification results to the DIARC module. We propose that this can be done in real-time. The DIARC, using the results communicated by the OFAC module, is responsible for interfacing with the robot, where the classification may induce higher levels of robot autonomy and functionality to support the multitasking state of the user.

7.2.6. BRAAHMS: A Novel Adaptive Musical Interface Based on Users’ Cognitive State

Previous work in the application of BCIs to the domain of music involve mapping brainwaves to audio signals or require explicit brain signals to control some music device or instrument. We chose to explore the implicit application of BCIs to music-related tasks. The design of our interface does not require explicit input from the user, which may distract the attention of the user while they are playing an instrument [21]. Rather, this real-time BCI “creatively” augments the user’s playing by adapting to their cognitive workload states.

7.2.7. Experimental Overview

The main research goal of this study was to utilize semantically informative fNIRS data to develop this adaptive musical BCI and evaluate the responses of the participant’s under different experimental conditions. We approached this with the following process:

(1)

Through a pilot study, determine whether differences in a participant’s perceived difficulty is detectable in the collected fNIRs data.

(2)

Design and build the adaptive musical interface through an iterative design process and using the participant data collected in the pilot study.

(3)

Evaluate the participant’s reactions to the adaptive musical interface through a series of qualitative interview responses.

This musical adaptation system, named BRAAHMS, added a harmonic of one octave above and below the user’s key (for the left and right hand respectively). Through multiple design iterations and pilot studies, we showed that participants found a harmonic augmentation of their music more enjoyable, while not detracting from the originality of their own playing. An illustration of the harmonic augmentation can be found in Figure 4.

7.2.8. Experimental Design

We selected 15 easy and hard pieces of music and randomly selected a piece for the participants to play in 30 s increments. There was a 30 s rest period between each piece. We collected fNIRS data from the user’s during their periods of play and rest and used that data to train a machine learning model which classified between the cognitive workload states associated with the easy and hard pieces. Using the BRAAHMS system, we devised four conditions, consisting of two BCI conditions and two additional conditions:

-

BCI 1: musical harmonies are added with low cognitive workload and removed when BRAAHMS detects high cognitive workload.

-

BCI 2: musical harmonies are added with low cognitive workload and removed when BRAAHMS detects low cognitive workload.

-

Constant: BRAAHMS provides musical harmonies continuously.

-

Non-Adaptive: There are no musical harmonies present.

The non-adaptive condition preceded the other conditions in order to extract the automated threshold but otherwise, all conditions were selected randomly. The experiment was evaluated with post-experiment interviews during which time, participants watched themselves playing during each condition.

7.2.9. Results

We observed that participant brain signals are differentiable when playing a hard piece from an easy piece. The results of this experiment were collected and analyzed from a series of interview questions. The feedback indicated that the BCI conditions were preferred (BCI 2 condition most notably) over the non-adaptive or constant conditions. Participants reported feeling more creative and that the BCI was responsive and augmented their own music in a pleasing manner. Furthermore, the BCI conditions were perceived by participants to be better sounding than had the harmonics been applied in a random manner. Additionally, players with less experience reportedly had higher and more consistent preference for BCI conditions, whereas more experienced players had less consistent preference for the BCI conditions. Overall, the researchers concluded that, “BRAAHMS enhances the communication bandwidth between user and musical instrument, responding to the user’s cognitive state and providing appropriate musical additions just when they are needed” [21].

7.3. Information Filtering Systems

Recommendation engines, made famous by Netflix and Amazon product recommendation systems, attempt to improve user experience by making optimal suggestions. However, the datasets required for these systems to make highly accurate preference predictions are sparse. We suggest that measures obtained from an fNIRS device can be used to augment recommendation systems and improve user experience [22].

7.3.1. Experimental Overview

We constructed the Brain Recommender, a system that makes suggestions solely on the fNIRS data supplied by the user, and compared its results to a standard recommender system. The Brain Recommender receives the fNIRS data from the user, which is classified according to preference by a machine learning classification algorithm and updates the user’s preference model accordingly. During the training period for the Brain Recommender, the participants were provided with a list of films and were asked to select their three favorite and least favorite films on the list. Using these “labels” and the associated fNIRS data provided by the user, we train the Brain Recommender model. Subsequently, we test the model by allowing it to classify based on real-time fNIRS data, updating the model with each classification.

7.3.2. Experimental Design

We began each iteration of the experiment by training the model. We provided the participants with a subset of films from the IMDB database of 250 best movies and 100 worst and asked them to choose their three favorite and 3 least favorite films. We tasked the participant with watching a slideshow of their selected movie webpages for 25 s per page, at which time we recorded their brainwaves using the fNIRS device. Between each page, we allowed a 10 s rest period. In total, that participants viewed 12 slides of their preferred films and 12 slides of their disliked films. The data gathered was used to train the model and was not altered during the testing phase.

During testing, participants viewed a string of twenty movie websites for 25 s each, followed by an 18 s rest. To evaluate our results, we created two conditions:

• Control condition: a series of pre-selected films with average ratings are used in the string of movie websites.
• Brain recommender condition: fNIRS data from the participant is fed into our Brain Recommender.

Regardless of the condition, we begin with the same start movie, but the following movies are indicated by the condition. If the control condition is selected, an averagely rated film is shown next. If the brain recommender condition is selected, the Brain Recommender will select the next movie based on the participant’s preference of the previously viewed film(s).

7.3.3. Results and Implications

In assessing the results of the experiment, the authors considered three questions:

(1)

How did the participant’s rating of movies differ between the Brain Recommender condition and the control condition?

(2)

Does the Brain Recommender improve over time?

(3)

How well does the Brain Recommender predict a user’s preference for a movie?

During early testing (prior to the 13th film) the Brain Recommender model was still learning about the user’s preference and thus, the results were not significantly better than the vanilla model. However, following the 13th film, the improvement from the Brain Recommender model versus the vanilla model became statistically significant. We attribute this to the limited size and scope of the training set. When the model predicted a high preference for a particular movie, participants were five times more likely to have given that movie a high rating. More precisely, the model predicted the participant’s exact rating 27% of the time, while predicting within a point of the participant’s rating 72% of the time. We conclude from this that the Brain Recommender model continues to improve with use, suggesting that fNIRS data is indeed informative for preference detection. Furthermore, a Mann–Whitner’s U test revealed a significant condition effect. We investigated the possibility that the recommender was not fitting the model to predict generally well-liked films instead of predicting individual preference. We determined that 125 of 280 movie recommendations were unique selections, which supports the validity of the condition effect.

8. Conclusions

In the decade of investigation of fNIRS and implicit brain–computer interfaces, we have determined that the fNIRS can be a valuable tool for distinguishing among differing brain activation patterns for the development of next-generation user-interfaces. As we look to the future, we intend to study more ways that fNIRS and associated interfaces can be used to seamlessly adapt to and improve user experience. We look forward to the advent and development of cheaper and more portable fNIRS devices, which will broaden the horizons for applications of study and bring this device into the forefront of individual use. These new brain–computer interfaces are part of the larger, growing trend in “lightweight”, “passive”, or “noncommand” interfaces, in areas such as physiological computing, neuroadaptive technology, passive BCI, neuroergonomics, affective computing, ambient media, context-aware interaction, and noncommand interfaces. These related trends all attempt to get more information from the user, with no or minimal user effort or attention. The user does not really give explicit commands, but rather the system observes, guesses, infers, and takes hints to augment the interface. Interfaces that combine several such measurements are a promising new direction in human computer interaction.