Update of fNIRS as an Input to Brain–Computer Interfaces: A Review of Research from the Tufts Human–Computer Interaction Laboratory
2. Related Work
2.1. BCIs and Mental Workload
2.2. Current State-of-the-Art of fNIRS and Short-Wave Infrared Spectroscopy
3. Experimental Framework
4. Specifics of fNIRS Configuration
5. Advantages and Challenges of fNIRS as an Input to Implicit Brain-Computer Interfaces
5.1. Consideration of the Hemodynamic Response in fNRIS-Based BCI
5.2. Sources of Signal Interference
5.3. Methods of Filtering Noise
6. fNIRS-Dependent Brain-State Distinction Methods
6.1. Support Vector Machines (SVM)
6.2. Feature Extraction
7. Brain-Based Augmentation of User-Interfaces
7.1. Cognitive Workload
7.1.1. Dynamic Difficulty Using Brain Metrics of Workload
7.1.2. Experiment Overview
- Can fNIRS signals be used to determine when a participant experiences high or low difficulty as they engaged in flying multiple UAVs?
- Can the use of real-time fNIRS signals processed by an adaptive interface yield improve operator performance in their UAV navigation tasks?
7.1.3. Experimental Design
- Adaptive condition: The Brain-Based Dynamic Difficulty Engine added and removed UAVs according to the participant’s brain signals.
- Non-adaptive condition: The participant’s cognitive state is ignored by the Brain-Based Dynamic Difficulty Engine and UAVs are added and removed at a random interval.
7.1.4. Results and Implications
7.1.5. Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy
7.1.6. Novel Experimental Protocol
7.1.7. Experiment Overview
- Can fNIRs be used to distinguish brain activity associated with experimental conditions from the rest state?
- Can fNIRs detect the difference between a participant’s no, low, and high-engaged working memory?
- Can well established mental tasks developed to measure working memory reveal information about the syntactic workload for high level user interface tasks?
7.1.8. Experimental Design
- Cognition exercises which have been shown to instigate low spatial WM load and low verbal WM.
- Subjects were asked to view a screen displaying a five-digit number. The screen had a circular fixation point in the center, at which the subject was instructed to keep their focus. After 18 s, the subject was asked to recall the number and location of the cell that the number was located in (see Figure 2).
- Cognition exercises which have been shown to instigate high spatial WM load and low verbal WM.
- Subjects were asked to view a screen displaying a five-digit number, followed by a screen displaying an arrow, and finally by a screen with fixation point. The participant was tasked with updating the spatial location of the numbers based on the direction of the arrow while remembering the number itself.
- Interface exercises that show users their location in hyperspace while they are tasked with searching for verbal content.
- The subject was asked to remember a five-digit zip code while searching for the zip code in hyperspace. Subjects navigated hyperspace with arrows at the bottom of the browser page, which were inactive for the first 3 s after a new page was loaded. They were provided with a pictorial representation of their current location. After finding the matching zip code in hyperspace, the subject was instructed to write down the zip code and the corresponding spatial location on an answer sheet.
- Interface exercises that do not show users their location in hyperspace while they search for the zip-code.
7.1.9. Results and Implications
7.2. Sensing Cognitive Multitasking for a Brain-Based Adaptive User Interface
7.2.1. Multitasking Scenarios: Branching, Dual Task, and Delay
- Branching occurs when the user is required to maintain overarching goals while investigating and pursuing second goals simultaneously. This is a common form of multitasking which burden users.
- Dual task occurs in situations where the user is constantly switching from one task to the next, but information about one task is not relevant or needed to complete the other. An example of this is where a user may be monitoring something while completing another task.
- Delay occurs when one task is more important than another. The user occasionally acknowledges the non-predominant task while maintaining attention on the predominant task. It is called delay because the acknowledgement of the lesser important task delays the engagement with the predominant task.
7.2.2. Experimental Overview
7.2.3. Experimental Design
- Delay condition was demonstrated by the following question: Do two successive rock classification updates follow in consecutive order? If the condition is true, the rock is placed into the same bin. Otherwise, it is placed in the new bin.
- Dual-Task condition was demonstrated by the following question: Do two updates of the same type follow in consecutive order? If this condition is satisfied, then the same response is warranted. Otherwise, the new response is appropriate.
- Branching condition required that for the rock classification updates, the user respond as described in the Delay condition. For location updates, the user was to respond as in the Dual Task condition.
- Random Branching: the updates provided by the robot were delivered to the user in a random fashion.
- Predictive Branching: the rock classification updates were delivered to the user after every three stimuli.
7.2.4. Experimental Results
7.2.5. A Brain-based Adaptive User Interface Platform
7.2.6. BRAAHMS: A Novel Adaptive Musical Interface Based on Users’ Cognitive State
7.2.7. Experimental Overview
- Through a pilot study, determine whether differences in a participant’s perceived difficulty is detectable in the collected fNIRs data.
- Design and build the adaptive musical interface through an iterative design process and using the participant data collected in the pilot study.
- Evaluate the participant’s reactions to the adaptive musical interface through a series of qualitative interview responses.
7.2.8. Experimental Design
- BCI 1: musical harmonies are added with low cognitive workload and removed when BRAAHMS detects high cognitive workload.
- BCI 2: musical harmonies are added with low cognitive workload and removed when BRAAHMS detects low cognitive workload.
- Constant: BRAAHMS provides musical harmonies continuously.
- Non-Adaptive: There are no musical harmonies present.
7.3. Information Filtering Systems
7.3.1. Experimental Overview
7.3.2. Experimental Design
- Control condition: a series of pre-selected films with average ratings are used in the string of movie websites.
- Brain recommender condition: fNIRS data from the participant is fed into our Brain Recommender.
7.3.3. Results and Implications
- How did the participant’s rating of movies differ between the Brain Recommender condition and the control condition?
- Does the Brain Recommender improve over time?
- How well does the Brain Recommender predict a user’s preference for a movie?
Conflicts of Interest
|Title||Authors||Participants||fNIRS Device||Filtering Methods||Features Selected for Machine Learning Model||Machine Learning Model||Results|
|Dynamic Difficulty Using Brain Metrics of Workload||Daniel Afergan, Evan M. Peck, Erin T. Solovey, Andrew Jenkins, Samuel W. Hincks, Eli T. Brown, Remco Chang, Robert J.K. Jacob||12 participants (5 male, 7 female).||Multichannel frequency domain OxiplexTS from ISS Inc. (Champaign, IL, USA)||Low pass filter (0.025 Hz)||25 s trials, slope and mean of linear regression||SVM with a linear kernel, parameter search for cost and gamma parameters, 10-fold cross validation||Increased performance improvements during UAV flight task|
|Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy||Leanne M. Erin Treacy Audrey Hirshfield * Solovey * Girouard *|
James Kebinger *
Robert J. Angelo Sergio K. Jacob * Sassaroli+ Fantini+
|10 subjects, 6 female, 9 college aged students, one 42-year-old, 9 righthanded||Multichannel frequency domain OxiplexTS from ISS Inc. (Champaign, IL, USA)||Moving average band pass filter.||~18 s trial length.||Folding Average Analysis (ANOVA); Foldering Average with Clustering||Determined difference in conditions with 95% confidence. Using clustering method, distinguished between high, low, and no workload for 90% of participants.|
|Sensing Cognitive Multitasking for a Brain-Based Adaptive User Interface||Erin Treacy Solovey, Francine Lalooses, Krysta Chauncey, Douglas Weaver, Margarita Parasi, Matthias Scheutz, Angelo Sassaroli, Sergio Fantini, Paul Schermerhorn,|
Audrey Girouard, Robert J.K. Jacob
|Exp 1: 12 volunteers (10 male), be- tween the ages of 18 and 34.|
Exp 2: This study included 12 healthy volunteers (5 male), between the ages of 19 and 32.
|Multichannel frequency domain OxiplexTS from ISS Inc. (Champaign, IL, USA)||3rd Degree low-pass elliptical filter; band pass filter.||(both experiments) ~5.57 s trials;||ANOVA comparing mean within subjects||(Over both experiments) Found total hemoglobin measures were higher in cognitive branching condition than in dual-task or delay condition.|
|BRAAHMS: A Novel Adaptive Musical Interface Based on Users’ Cognitive State||Beste F Yuksel, Daniel Afergan, Evan M Peck *, Garth Griffin, Lane Harrison, Nick W.B. Chen, Remco Chang and Robert J.K. Jacob||20 participants||Multichannel frequency domain OxiplexTS from ISS Inc. (Champaign, IL, USA)||“The signals were filtered for heart rate, respiration, and movement artifacts”||30 s trials. Mean and linear regression slope.||Linear SVM||15 out of 20 participants (blindly) chose a brain-controlled condition as their favorite among the conditions.|
|Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty Based on Brain State||Beste Yuksel, Kurt Oleson, Lane Harrison, Evan Peck, Daniel Afergan, Remco Chang, Robert Jacob||16 (8 female, 8 male)||Multichannel frequency domain OxiplexTS from ISS Inc. (Champaign, IL, USA)||Third degree polynomial and low-pass elliptical filter.||30 s trials. Mean and linear regression slope.||Linear SVM||Participants played more correct notes with BACh, and made fewer errors. They also played pieces faster.|
|Investigation of fNIRS Brain Sensing as Input to Information Filtering Systems||Evan M. Peck, Daniel Afergan, Robert J.K. Jacob||6 male and 8 female volunteers aged 19–28||Multichannel frequency domain OxiplexTS from ISS Inc. (Champaign, IL, USA)||3rd Degree low-pass elliptical filter; band pass filter.||25 s trials; used filtered light readings at each point as features||Linear SVM||Predicted preferences precisely with 27% accuracy, and within one point (out of 5), with 72% accuracy. When model predicted 4 or 5, participants were 5 times more likely to rate 4 or 5 rather than 1 or 2.|
- Chaudhary, U.; Birbaumer, N.; Ramos-Murguialday, A. Brain-computer interfaces in the completely locked-in state and chronic stroke. In Progress in Brain Research; Elsevier: Amsterdam, The Netherlands, 2016; Volume 228, pp. 131–161. [Google Scholar]
- Cutrell, E.; Tan, D.S. BCI for passive input in HCI. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems Workshop on Brain-Computer Interfaces for HCI and Games; ACM Press: New York, NY, USA, 2007. [Google Scholar]
- Zander, T.O.; Kothe, C.; Welke, S.; Roetting, M. Utilizing Secondary Input from Passive Brain-Computer Interfaces for Enhancing Human-Machine Interaction. In Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience; Lecture Notes in Computer Science; Springer: Berlin, Germany, 2009. [Google Scholar]
- Parasuraman, R.; Glenn, F.W. Putting the Brain to Work: Neuroergonomics Past, Present, and Future. Hum. Factors 2008, 50, 468–474. [Google Scholar] [CrossRef] [PubMed]
- Herff, C.; Heger, D.; Fortmann, O.; Hennrich, J.; Putze, F.; Schultz, T. Mental workload during n-back task—Quantified in the prefrontal cortex using fNIRS. Front. Hum. Neurosci. 2014, 7, 935. [Google Scholar] [CrossRef] [PubMed]
- Ayaz, H.; Shewokis, P.A.; Bunce, S.; Izzetoglu, K.; Willems, B.; Onaral, B. Optical brain monitoring for operator training and mental workload assessment. NeuroImage 2012, 59, 36–47. [Google Scholar] [CrossRef] [PubMed]
- Wilson, R.H.; Nadeau, K.P.; Jaworski, F.B.; Tromberg, B.J.; Durkin, A.J. Review of short-wave infrared spectroscopy and imaging methods for biological tissue characterization. J. Biomed. Opt. 2015, 20, 030901. [Google Scholar] [CrossRef] [PubMed]
- Shi, L.Y.; Sordillo, L.; Rodriguez-Contreras, A.; Alfano, R. Transmission in Near-Infrared Optical Windows for Deep Brain Imaging. J. Biophotonics 2016. [Google Scholar] [CrossRef] [PubMed]
- Hincks, S.; Afergan, D.; Jacob, R. Using fNIRS for Real-Time Cognitive Workload Assessment. In International Conference on Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience; Springer: New York, NY, USA, 2016; Volume 9743, pp. 198–208. [Google Scholar]
- Zhang, Q.; Brown, E.; Strangman, G. Adaptive filtering for global interference cancellation and real-time recovery of evoked brain activity: A Monte Carlo simulation study. J. Biomed. 2007, 12, 044014. [Google Scholar] [CrossRef] [PubMed]
- Solovey, E.; Girouard, A.; Chauncey, K.; Hirshfield, L.; Sassaroli, A.; Zheng, F.; Fantini, S.; Jacob, R. Using fNIRS brain sensing in realistic HCI settings: Experiments and guidelines. In 22nd Annual ACM Symposium on User Interface Software and Technology; ACM: New York, NY, USA, 2009; pp. 157–166. [Google Scholar]
- Takeda, K.; Okazaki, S. Special Issues No.3: Measurement Technique for Ergonomics, Section 4: Measurements and Analyses of Bioelectric Phenomena and Others (3): Noninvasive Techniques for Measuring Brain Activity: EEG, fMRI, and fNIRS. Jpn. J. Ergon. 2016, 51, 411–419. [Google Scholar] [CrossRef]
- Matthews, F.; Pearlmutter, B.A.; Ward, T.E.; Soraghan, C.; Markham, C. Hemodynamics for BrainComputer Interfaces. Signal Process. Mag. 2008, 25, 87–94. [Google Scholar] [CrossRef]
- Yuksel, B.; Oleson, L.; Harrison, M.; Peck, E.; Afergan, D.; Chang, R.; Jacob, R. Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty based on Brain State. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems Conference; ACM Press: New York, NY, USA, 2016. [Google Scholar]
- Hearst, M.; Dumais, S.T.; Osman, E.; Platt, J.; Scholkopf, B. Support vector machines. Intell. Syst. Appl. IEEE 1998, 13, 18–28. [Google Scholar] [CrossRef][Green Version]
- Hastie, T.; Rosset, S.; Tibshirani, R.; Zhu, J. The Entire Regularization Path for the Support Vector Machine. J. Mach. Learn. Res. 2004, 5, 1391–1415. [Google Scholar]
- Afergan, D.; Peck, M.; Solovey, E.; Jenkins, A.; Hincks, S.; Brown, E.; Chang, R.; Jacob, R. Dynamic difficulty using brain metrics of workload. In SIGCHI Conference on Human Factors in Computing Systems; ACM: New York, NY, USA, 2014; pp. 3797–3806. [Google Scholar][Green Version]
- Hirshfield, L.; Solovey, E.; Girouard, A.; Kebinger, J.; Jacob, R.; Sassaroli, A.; Fantini, S. Brain measurement for usability testing and adaptive interfaces: An example of uncovering syntactic workload with functional near infrared spectroscopy. In SIGCHI Conference on Human Factors in Computing Systems (CHI ‘09); ACM: New York, NY, USA, 2009; pp. 2185–2194. [Google Scholar]
- Solovey, E.; Lalooses, F.; Chauncey, K.; Weaver, D.; Parasi, M.; Scheutz, M.; Sassaroli, A.; Fantini, S.; Schermerhorn, P.; Girouard, A.; et al. Sensing cognitive multitasking for a brain-based adaptive user interface. In SIGCHI Conference on Human Factors in Computing Systems (CHI ‘11); ACM: New York, NY, USA, 2011; pp. 383–392. [Google Scholar]
- Yuksel, B.; Afergan, D.; Peck, E.; Griffin, G.; Harrison, L.; Chen, N.; Chang, R.; Jacob, R. BRAAHMS: A Novel Adaptive Musical Interface Based on Users Cognitive State. In International Conference on New Interfaces for Musical Expression; Louisiana State University: Baton Rouge, LA, USA, 2015; pp. 136–139. [Google Scholar]
- Peck, E.; Afergan, D.; Jacob, R. Investigation of fNIRS brain sensing as input to information filtering systems. In Proceedings of the 4th Augmented Human International Conference (AH ‘13), 7–8 March 2013, Stuttgart, Germany; ACM: New York, NY, USA, 2013; pp. 142–149. [Google Scholar] [CrossRef][Green Version]
- Curtin, A.; Ayaz, H. The Age of Neuroergonomics: Towards Ubiquitous and Continuous Measurement of Brain Function with fNIRS. Jpn. Psychol. Res. 2018, 60, 374–386. [Google Scholar] [CrossRef]
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Bosworth, A.; Russell, M.; Jacob, R.J.K. Update of fNIRS as an Input to Brain–Computer Interfaces: A Review of Research from the Tufts Human–Computer Interaction Laboratory. Photonics 2019, 6, 90. https://doi.org/10.3390/photonics6030090
Bosworth A, Russell M, Jacob RJK. Update of fNIRS as an Input to Brain–Computer Interfaces: A Review of Research from the Tufts Human–Computer Interaction Laboratory. Photonics. 2019; 6(3):90. https://doi.org/10.3390/photonics6030090Chicago/Turabian Style
Bosworth, Alexa, Matthew Russell, and Robert J. K. Jacob. 2019. "Update of fNIRS as an Input to Brain–Computer Interfaces: A Review of Research from the Tufts Human–Computer Interaction Laboratory" Photonics 6, no. 3: 90. https://doi.org/10.3390/photonics6030090