Next Article in Journal
Profile-Splitting Linearized Bregman Iterations for Trend Break Detection Applications
Next Article in Special Issue
Use of Both Eyes-Open and Eyes-Closed Resting States May Yield a More Robust Predictor of Motor Imagery BCI Performance
Previous Article in Journal
Time-Domain Electromagnetic Scattering by Buried Dielectric Objects with the Cylindrical-Wave Approach for GPR Modelling
Previous Article in Special Issue
Motor Imagery Based Continuous Teleoperation Robot Control with Tactile Feedback
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 9
Issue 3
10.3390/electronics9030422
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Brain-Switches for Asynchronous Brain–Computer Interfaces: A Systematic Review

by
Chang-Hee Han
1,
Klaus-Robert Müller
1,2,3 and
Han-Jeong Hwang
4,*
1
Machine Learning Group, Technical University of Berlin, 10587 Berlin, Germany
2
Department of Brain and Cognitive Engineering, Korea University, Seoul 136-713, Korea
3
Max Planck Institute for Informatics, 66123 Saarbrücken, Germany
4
Department of Electronics and Information Engineering, Korea University, Sejong 30019, Korea
*
Author to whom correspondence should be addressed.
Electronics 2020, 9(3), 422; https://doi.org/10.3390/electronics9030422
Submission received: 28 November 2019 / Revised: 29 December 2019 / Accepted: 9 January 2020 / Published: 2 March 2020
(This article belongs to the Special Issue The Challenges in Brain-Computer Interface (BCI) - toward Practical BCI)
Browse Figures
Versions Notes

Abstract

:
A brain–computer interface (BCI) has been extensively studied to develop a novel communication system for disabled people using their brain activities. An asynchronous BCI system is more realistic and practical than a synchronous BCI system, in that, BCI commands can be generated whenever the user wants. However, the relatively low performance of an asynchronous BCI system is problematic because redundant BCI commands are required to correct false-positive operations. To significantly reduce the number of false-positive operations of an asynchronous BCI system, a two-step approach has been proposed using a brain-switch that first determines whether the user wants to use an asynchronous BCI system before the operation of the asynchronous BCI system. This study presents a systematic review of the state-of-the-art brain-switch techniques and future research directions. To this end, we reviewed brain-switch research articles published from 2000 to 2019 in terms of their (a) neuroimaging modality, (b) paradigm, (c) operation algorithm, and (d) performance.

1. Introduction

Brain–computer interfaces (BCIs) are a technology used to provide patients with locked-in syndrome (LIS) caused by a neurodegenerative disease with an alternative communication channel [1]. A BCI decodes voluntarily modulated neural signals to control external devices or generate communication messages [2,3,4,5,6]. BCI systems have been developed using various neuroimaging modalities, such as electroencephalography (EEG) [7,8,9,10,11,12], magnetoencephalography (MEG) [13], functional near-infrared spectroscopy (fNIRS) [14,15], functional magnetic resonance imaging (fMRI) [16], and electrocorticography (ECoG) [17,18]. Among these neuroimaging modalities, EEG has been widely used as the most popular neuroimaging modality owing to its portability, non-invasiveness, high temporal resolution, and reasonable cost [19,20]. In addition, fNIRS has also been frequently used as an alternative to EEG because it is relatively more robust to physiological artifacts [21,22].
Over the last couple of decades, promising and successful BCI applications have been proposed [23], and their feasibility has been proved through validation experiments [24]. A mental speller is one of the most popular BCI applications, which allows LIS patients to communicate a message [7,25,26,27,28]. In addition to a mental speller, different BCI applications have been developed, such as a robotic arm [29,30,31], mobile robot [32], wheelchair [33,34,35,36,37,38], mouse [39,40], smart home system [41], mobile phone application [42], and various games [43,44,45].
BCI applications can be realized through two different paradigms: (i) synchronous or (ii) asynchronous (or self-paced) [46]. A synchronous paradigm has remained the most general control mode for BCIs because synchronous BCI systems have shown excellent performance (e.g., >90% for binary communication) [47]. Synchronous BCI systems are operated within prefixed time-periods, meaning that the users cannot use a synchronous BCI system whenever they want. Although a synchronous BCI system cannot be used freely, such systems have been widely applied owing to their high performance compared to the asynchronous paradigm [48,49]. By contrast, asynchronous BCI systems are not restricted in terms of the operation period; an asynchronous BCI system can be freely used without time constraints. Considering this advantage of asynchronous BCI systems, the asynchronous paradigm has been regarded as more promising and practical. However, asynchronous BCI systems have shown relatively inferior performances compared to those of synchronous BCI systems, such as a low true-positive rate (TPR) and high false-positive rate (FPR) [48,49]. In particular, a high FPR is an extremely serious problem for asynchronous BCI systems because the user has to make additional commands to modify a false operation [50,51] whenever a false positive occurs [52]. The additional command generation is time consuming and impractical under real BCI scenarios. It was reported that LIS patients wanted to use a BCI system without false operation for about 2–4 h [53], but the state-of-the-art BCI system does not meet the requirement; one false operation occurs for every 10 min. [54]. In addition, there has been no standard criterion of the time duration for acceptable false operation, thus only a relative comparison between BCI studies is available in terms of the FPR performance.
In some BCI studies, alternative solutions have been proposed to overcome the relatively poor performance of asynchronous BCI systems, mainly caused by the FPR, namely, a two-step approach that consists of control and no-control states by introducing a brain-switch to an asynchronous BCI system [8,33,55,56,57,58,59,60,61]. A brain-switch is a means of turning on or off the control and no-control states in an asynchronous BCI system based on a two-step approach. In the first step of an asynchronous BCI system, the brain-switch of an asynchronous BCI system continuously monitors a user’s intention to turn on a control state from a no-control state. If the user’s control intention is detected by the brain-switch, the no-control state is changed into a control state, thereby initiating the second step of an asynchronous BCI system, where the user can generate commands to either operate an asynchronous BCI system or turn back to a no-control state from a control state (second step → first step). If the user’s control intention is not detected in the first step, the no-control state is retained and the second step is not initiated, and thereby the FPR of an asynchronous BCI system can be significantly prevented.
The brain-switch is a key aspect in the two-step approach and has been developed using various types of brain patterns, such as the event-related potential (ERP) [33,62], steady-state visual evoked potential (SSVEP) [8,59], sensorimotor rhythm (SMR) [40,51,58,61], and hemodynamic response [63,64,65]. To date, promising and successful results have been reported for brain-switches; however, despite the feasible potential of brain-switches, few systematic reviews of brain-switches have been reported. In this review study, to investigate the state-of-the-art technologies of brain-switches for asynchronous BCI systems, we collected articles related to brain-switches published during the last 20 years (2000–2019), thereby providing insight into the development of brain-switches and future directions. In this review, we first introduce the operation principle of a brain-switch, experimental paradigms to evaluate the performance of a brain-switch, its operation algorithms, and its taxonomy. We then review the collected articles from the following four aspects: neuroimaging modality, experimental paradigms, features, and performance.
The remainder of this review is organized as follows: Section 2 introduces the procedure used for collecting the articles, and Section 3 summarizes the review results. Finally, we discuss the review results in Section 4.

2. Methods

2.1. Procedure for Selecting Target Papers

Google Scholar was used to collect the candidate papers for this review based on following 14 keywords: “brain switch,” “brain-switch,” “mental switch,” “mental-switch,” “electroencephalography,” “EEG,”, “near-infrared spectroscopy,” “NIRS,” “brain–computer interface,” “BCI,” “asynchronous,” “self-paced,” “brain–machine interface,” and “direct brain interface.” The advance search function of Google Scholar was used to find candidate papers; the two keywords, “brain–computer interface” and “brain switch”, were set as necessary keywords that had to be included in a candidate paper, while at least one of the other keywords had to be contained in their title, abstract, or keyword list. A total of 977 papers were found by this method. We first read their titles and abstracts to check whether each of the 977 papers was related to asynchronous BCI and brain-switch studies, and made an initial list for this review; 116 papers were included in our initial list. The following article types were included in the initial list for this review: research articles, review papers, case reports, and conference proceedings (no specific article types were defined when searching candidate papers in Google Scholar). To assess the eligibility of the candidate papers for a further detailed review, the papers in the initial list were carefully read to clearly determine whether we should include them in our final review list. We carefully selected the brain-switch articles developed for asynchronous BCIs while excluding articles irrelevant to asynchronous BCI controls. We included 38 brain-switch studies that used a single task (e.g., foot motor imagery) to turn on or off a control state and a no-control state. We also included 5 brain-switch studies that attempted to detect the user’s control intention regardless of what type of movement task is performed, even though they used multiple tasks (e.g., right and left hand movement task) [52,66,67,68,69]; all tasks were equally regarded as a means to change a control and no-control state without discriminating them to each other.
To find more candidate papers that might be missed in Google Scholar, we also searched candidate papers in Web of Science (WoS) using the same 14 keywords. We found 182 candidate papers that mostly overlapped with the 977 papers found in Google Scholar, among which 6 papers were met in our selection criteria, but not searched in Google Scholar. Thus, we added the 6 papers, and a total of 49 papers were finally selected in the end for this review. The documents included in the final list were published between 2000 and 2019. Note that we excluded asynchronous BCI studies which directly classify multiple control tasks without using a brain-switch; it is assumed in the BCI studies that a control-state was already turned on from a no-control state, thus a no-control state was not considered at all.

2.2. Categories of Selected Papers

The papers selected in our final list were categorized into three different groups based on the neuroimaging modality: (i) EEG-based, (ii) NIRS-based, and (iii) ECoG-based brain-switches. Next, EEG-based brain-switch studies were divided into three subgroups: (i) endogenous, (ii) exogenous, and (iii) hybrid brain-switches. An endogenous brain switch indicates the use of self-modulated brain activities through mental imagery tasks, whereas an exogenous brain switch indicates the use of brain activity evoked by an external visual/auditory/tactile stimulus. In addition, a hybrid brain switch means the use of two BCI paradigms simultaneously, such as a combination of SSVEP and motor imagery (MI). NIRS-based brain-switch studies were divided into two subgroups based on their paradigm (i.e., (i) endogenous and (ii) exogenous brain-switches), but we did not create subsections for each subgroup because of a small number of NIRS-based brain-switch studies (n = 3). ECoG-based brain-switch studies were based on only the endogenous paradigm.

2.3. Description of BCI-Related Terminologies Used in This Review

In Table 1, we provide a summary of the detailed descriptions of BCI-related terminologies used in this review to better help understand our results.

3. Results

In this section, our review results are provided according to the following seven topics: (i) the operational principle of a brain-switch, (ii) experimental paradigms to evaluate the performance of brain-switches, (iii) detection algorithms used in brain-switches, (iv) the taxonomy of brain-switches, (v) EEG-based brain-switches, (vi) NIRS-based brain-switches, and (vii) ECoG-based brain-switches.

3.1. Operational Principle of a Brain-Switch

Figure 1 shows a schematic representation of an asynchronous BCI system operated based on a two-step approach realized using a brain-switch; we assume that a foot MI is used to produce a brain-switch command that turns into a control state from a no-control state, or vice versa (e.g., start or stop control of a BCI wheelchair), whereas left- and right-hand MI are used to produce BCI commands (e.g., turning a BCI wheelchair left or right). In a no-control state, an intention (e.g., foot MI) to turn into a control-state is only monitored by a brain-switch while ignoring all BCI commands (e.g., left- and right-hand MI), which can significantly prevent unwanted FPRs; the first step of an asynchronous BCI system. Once the user’s intention is detected by a brain-switch, a no-control state is changed into a control state (first step → second step), and the BCI system starts to monitor the user’s intentions (e.g., left- and right-hand MI) for generating BCI commands as well as another intention (e.g., a foot MI) to turn back to a no-control state from a control state (second step → first step). The mentioned procedure is repeated while the BCI system is turned on.

3.2. Experimental Paradigms to Evaluate the Performance of Brain-Switches

Two different experimental paradigms have been used to study brain-switches: (i) synchronous and (ii) asynchronous experimental paradigm. Under a synchronous experimental paradigm [61,64,65,69], the duration of a no-control state is fixed (e.g., 10 s), during which a subject takes a rest without any thoughts to stay in a no-control state. After a no-control state, another time period (e.g., 10 s) is sequentially given with a synchronous cue, indicating that a subject can start to perform a given task (e.g., foot MI) to turn into a control state from a no-control state. On the other hand, under an asynchronous experimental paradigm, the durations of a no-control and control state are not fixed; a subject stays in a no-control state as long as the subject wants or freely converts a no-control state into a control state whenever the user wants [8,58,60]. In general, signal processing methods using sliding windows with specific sizes (e.g., 1 s) have been used to estimate the performance of a brain-switch regardless of the types of experimental paradigms [8,58,60,61,64,65,69]. For example, when using a threshold-based method to operate a brain-switch [61], the threshold is defined using a training dataset before testing a brain-switch, and then sequential outputs of a classifier are monitored for each time window to check whether the output exceeds the threshold or not. If an output exceeds the predefined threshold, a decision is made as a result.
Various performance measures have been used for evaluating the performance of a brain-switch. TPR, FPR, and ODT have been the representative performance measures for evaluating the performance of a brain-switch. These performance measures can be calculated under both of the synchronous and asynchronous experimental paradigms. However, the different units of TPR and FPR have been used according to the type of the used experimental paradigm. As we mentioned above, the synchronous experimental paradigm generally uses the predefined durations for a no-control and control state. In this case, even though occurrences (e.g., FPs/min or TPs/min) of TPR and FPR can be calculated, TPR and FPR have been generally reported as percentages because the occurrences of TPR and FPR are meaningless when the fixed durations are used for a no-control and control state. For example, it is impossible to exactly estimate how many and how often TPs and FPs are generated when longer durations (e.g., >10 s) are used for a no-control and control state. On the other hand, the duration of a no-control and control state is not fixed when the asynchronous experimental paradigm is used, and thus the occurrences of TPR and FPR (e.g., FPs/min or TPs/min) can be meaningfully interpreted more than those of the synchronous experimental paradigm.

3.3. Detection Algorithms Used in Brain-Switches

Table 2 shows a list of detection algorithms used in previous brain-switch studies. There are four different types of detection algorithms used: (i) a simple threshold method, (ii) a classifier-based threshold method, (iii) a classifier-based template matching method, and (iv) a classifier-based threshold-free method. Most of the studies in this area have applied a simple threshold method or a classifier-based threshold method. Simple and classifier-based threshold methods detect a user’s intention to operate a brain-switch when a pre-defined feature and a classifier output exceed a specific threshold pre-defined based on the training data. In particular, the detection performance of the classifier-based threshold method is closely related to the classification performance of a classifier. In addition to threshold-based methods, a classifier-based template-matching [8,70] and a threshold-free method [33] have also been introduced. A classifier-based template-matching algorithm [8,70] detects the user’s intention to operate a brain-switch when a classifier provides the same class outputs consecutively at a predefined template size. For example, if the predefined template size is 3, the algorithm detects the user’s intention when a classifier produces the same class labels consecutively three times in a row (e.g., 010111, 01100111, and 0111, where 0 = resting state and 1 = foot MI). The classifier-based template-matching algorithm is similar to the classifier-based threshold method in that both algorithms use the outputs of a classifier. However, the classifier-based template-matching algorithm uses the final output values (i.e., discrete class labels) of a classifier, whereas the classifier-based threshold method uses the intermediate output values (i.e., continuous values) of a classifier, which are used to determine the final discrete outputs. A classifier-based threshold-free method was also proposed [33], which is based on two different support vector machines (SVMs) to detect the user’s intention to control a BCI system. The first SVM is used to check whether a user wants to use a BCI system in a no-control state, and the second SVM discriminates between the control and no-control states when the first SVM decides that the user wants to use a BCI system to make a BCI operational command.

3.4. Taxonomy of Brain-Switches

Figure 2 presents the taxonomy of different brain-switches. As mentioned in the Methods section, a total of 49 articles were selected for this review. Among these articles, forty-four, three, and two articles introduced EEG-, NIRS-, and ECoG-based brain-switches, respectively. Most studies have used EEG for developing a brain-switch [8,33,35,40,51,52,54,55,58,59,60,61,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,86,87,88,89,90,91,92,93,94,95,96,97,98]. EEG-based brain-switches have been mostly developed based on endogenous paradigms, which use self-regulated brain activity through mental imagery tasks without the use of external stimuli (n = 35) [40,51,52,55,58,60,61,66,67,68,69,70,71,72,73,75,76,77,79,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,97]. Five EEG-based brain-switches have been developed based on exogenous paradigms that apply external stimuli to generate discriminative brain patterns, such as ERP and SSVEP (n = 5) [8,33,54,59,74]. Four EEG-based brain-switch studies introduced hybrid brain-switches that use two different BCI paradigms simultaneously (i.e., MI + SSVEP [78] and SSVEP+ERP [35,80,96]). Three studies introduced NIRS-based brain-switches based on endogenous (n = 2) [63,65] and exogenous (n = 1) [64] paradigms. Please note that, although two studies [64,65] used both EEG and NIRS simultaneously, we categorized them as NIRS-based brain-switch studies because only the NIRS was used for a brain-switch. In addition, two studies developed ECoG-based brain-switches based on endogenous paradigms [17,18].

3.5. EEG-Based Brain-Switches

3.5.1. Endogenous Brain-Switches

Table 3 shows a list of EEG-based endogenous brain-switch studies. Most of the endogenous brain-switches used sensorimotor brain activity modulated by MI or ME. Only three studies performed by Faradji et al. [89,90,93] have used different mental tasks, such as mental multiplication, letter composing, object rotation, and number counting, as a control task for an EEG-based endogenous brain-switch. Early studies mainly used ME to verify the possibility of using sensorimotor brain activity on the development of a brain-switch [51,52,66,67,68,86,87,88,91], after which MI has been mostly applied to develop a brain-switch because MI is a more realistic paradigm for the main targets of BCI technology (e.g., LIS patients, who cannot freely move their body). The following studies proposed novel features to improve the performance of a brain-switch [55,77,81,83]. MRCP and ERD/ERS are representative features used to detect a user’s intention regarding an EEG-based endogenous brain-switch, because these two features represent motor-related brain activities and can be classified from the brain activity measured during a resting state based on machine learning/pattern recognition algorithms.
Early studies on the development of EEG-based endogenous brain-switches mainly focused on voluntary body movements (e.g., index finger movements) [51,52,67,68,86,87,88]. One of the EEG-based endogenous brain-switches proposed a low-frequency asynchronous switch design (LF-ASD) [51,66,84,85,86,87,88], which used an increase in the frequency band power of 1–4 Hz (SCP) as a feature for intention detection. An LF-ASD generally uses a motor task as a control task (e.g., an index finger flexion). If a subject successfully performs a given motor task, the LF-ASD converts a no-control into a control state. In 2002, an LF-ASD study demonstrated its feasibility with able-bodied and spinal cord-injured subjects [85], who were able to control an LF-ASD with mean classification accuracies of over 96%. In 2007, Bashashati et al. developed an LF-ASD consecutively using two different classifiers [66]; the first classifier was used as a brain-switch that determines whether a hand movement is performed regardless of the type of the movement for the change between a no-control and control state (left- and right-hand movement), and the second classifier decided whether the detected movement is a right-hand or a left-hand movement for the classification of two different BCI outputs. From their experimental results, they validated the proposed LF-ASD can perform almost the same as an LF-ASD which only detects a no-control and a BCI output. Despite such promising results of LF-ASDs, a high FPR during a no-control state limits the usefulness of an LF-ASD. Therefore, the following studies attempted to reduce the FPR of an LF-ASD using advanced algorithms based on hybrid genetic algorithms [86,88], and demonstrated the usefulness of the suggested algorithms to effectively reduce the FPR of an LF-ASD. In addition to the LF-ASD, several EEG-based brain-switches have been developed using voluntary body movements. In 2002, the Berlin BCI (BBCI) group developed an EEG-based endogenous brain-switch using a left or right index finger movement [52,67,68], which used a readiness potential generated between −400 and 0 ms before real movements (an early component of MRCP) as a feature to detect movement intentions and predict the type of movement (left vs. right). After preprocessing of EEG data, two classifiers were trained for two different objectives; the first classifier only distinguishes movement events in a no-control state, and then the second classifier recognizes the type of finger movements (left or right). A pseudo-online experiment confirmed the feasibility of the proposed BBCI brain-switch, showing a reliable classification result (around 10% error rate, but TPR and FPR were not reported). In 2012, Mohammadi et al. proposed a combination of pre- and post-signal processing techniques to enhance the onset detection performance of an EEG-based endogenous brain-switch based on body movements [91]. In their experimental paradigm, a subject was asked to perform a voluntary foot movement to interchange a no-control and control state. As a new preprocessing method for a brain-switch, they used a frequency decomposition method of EEG signals by constant-Q filters instead of constant bandwidth filters. The constant-Q frequency decomposition method using varying bandwidth sizes depending on the center frequency has shown better classification accuracies in determining different motor imagery tasks. Furthermore, as a postprocessing method for a brain-switch, for each subject, they optimized a threshold level for operating a brain-switch and a dwell time. The dwell time means the amount of time that the output signal of a classifier must exceed a threshold to be considered as a control event to turn on a control-state. They compared the performances of brain-switches with or without the pre- and post-processing methods and demonstrated that the proposed combination of pre- and post-processing methods could improve the overall performance of a brain-switch; a mean TPR significantly increased by 10% and FPR decreased by 1% compared to a conventional method without using the two methods. Although the EEG-based endogenous brain-switches using ME have shown promising results, this remains a critical limitation in using the actual body movement (ME) for a brain-switch instead of MI, because LIS patients who are the main target of BCI technology cannot freely generate actual body movements.
After demonstrating the feasibility of using sensorimotor brain activity induced by ME on the development of a brain-switch, MI-based brain-switches were developed to overcome the limitation of the early brain-switches based on ME. In 2007, Leeb et al. developed an endogenous brain-switch based on an imagination of foot movements to control a wheelchair in a virtual environment [73]. The beta ERS induced by a foot MI was used as a feature to detect the user’s control intention. A tetraplegic patient in his wheelchair was placed inside a virtual street populated with avatars using a virtual reality system. If the beta ERS was successfully produced, the patient could feel as if he approaches a target avatar by moving his wheelchair in a virtual environment. According to Leeb et al.’s report, the tetraplegic patient recruited in their study successfully controlled a wheelchair with a high classification accuracy of more than 90% (TPR and FPR were not reported). In 2007, Scherer et al. introduced another MI-based brain-switch where four different motor imagery tasks were used (i.e., left-hand, right-hand, foot, and tongue MI), and three of which were selected to control an avatar in a virtual environment (i.e., rotate left, rotate right, and move forward) [69]. They used two consecutive classifiers based on ERD/ERS features to control the avatar; The first classifier only detected the presence of MIs from a no-control state as a brain-switch, and the second classifier classified three different MIs selected in the training session for each subject. By using the consecutive strategy of two different classifiers, subjects could control an avatar in a virtual environment with TPR of 28.40% and FPR of 16.90%.
In 2009, Pfurtscheller et al. proposed an EEG-based endogenous brain-switch that uses a peri-imagery ERD observed around the onset of the imagery task and post-imagery beta rebound (ERS) induced by a cue-based brisk foot MI as a feature [61], showing a TPR of 78.6% and an FPR of less than 10%. From the same BCI research group, Müller-Putz et al. proposed an EEG-based endogenous brain-switch using only a single channel EEG [58], and also used a post-imagery beta rebound observed after a foot MI to control a brain-switch; the proposed brain-switch converted a no-control state into a control state when the beta rebound was detected. Interestingly, they built a classifier using the training data obtained during a foot ME, and the EEG data recorded during the foot MI were then used as test data. The proposed brain-switch achieved a TPR of 79% and an FPR of less than 0.67 FPs/min. Because the ME was used to collect the training data, this paradigm can be applied to those who have at least partially intact body functions. In 2010, Pfurtscheller and Müller-Putz integrated a foot-MI-based brain-switch into an asynchronous SSVEP-based orthosis BCI system [60], and reported that the foot-MI-based brain-switch can reduce the FPR of an asynchronous SSVEP-based orthosis BCI system, as compared to that without a brain-switch (i.e., from 5.40 to 1.46 FPs/min). This is a representative example of applying an EEG-based endogenous brain-switch to an asynchronous BCI system. Qian et al. extended the EEG-based endogenous brain-switch developed by Pfurtscheller and Müller-Putz [71]. They designed a novel brain-switch based on ERDs following external sync signals; external auditory beeps with a high or low pitch were provided to the subjects, who were requested to perform a hand MI synchronously with the beeps when they wanted to enter a control state from a no-control state. This experimental design made it possible to extract ERDs time-locked to external sync signals, and the performance of the brain-switch was; therefore, improved, reaching an FPR of 0.8%, although their brain-switch showed a slow response time of 36.9 s when changing from a no-control state into a control state. In 2012, Wang et al. developed an endogenous brain-switch for control of ambulation in a virtual reality environment [92]. If a subject successfully performs a given MI related to walking, the subject’s avatar ambulated in a virtual reality environment. The offline classification accuracy was 77.20%, and all subjects successfully controlled their avatar in an online test session with an accuracy of 85.00%.
Most EEG-based endogenous brain-switch studies have used MRCP and ERD/ERS as features to change a control and no-control state. Novel feature extraction and optimization methods have also been proposed to improve the performance of EEG-based endogenous brain-switches using MI tasks [55,77,81,83]. In 2011, Barachant et al. proposed an EEG-based endogenous brain-switch using Riemannian geometry, which is based on the fact that spatial covariance matrices obtained from brief EEG segments during MI tasks contain all task-related information [55]. Through an experiment conducted on six healthy subjects, they showed a high TPR of 91.3% and a low FPR of 0.4 FPs/min. More recently, common spatial pattern (CSP) algorithms have been used to develop EEG-based endogenous brain-switches [40,77,97]. A CSP algorithm uses EEG signals filtered between 8 and 30 Hz containing MI-related brain activity information. In a CSP, covariance matrices corresponding to each MI task are extracted from the EEG signals, and log-variances of the CSP components are generally used as features. In 2013, Choi et al. used the CSP for an MI-based brain-switch integrated into an SSVEP and a P300 BCI based on a low-cost EEG system for asynchronous humanoid robot control, although the detailed quantitative performance of their brain-switch was not reported [77]. In 2013, Xia et al. also used a CSP algorithm to implement an EEG-based endogenous brain-switch operated in a two-dimensional cursor control BCI system [40]. In 2017, Yu et al. also used a CSP algorithm to change a control and no-control state in a self-paced BCI system combining MI and P300 for wheelchair control [97]. For task optimization of an EEG-based endogenous brain-switch, Xu et al. in 2016 conducted a comprehensive study to find an optimal condition to develop an EEG-based endogenous brain-switch in terms of the type of MI, the frequency band used for feature extraction, and the processing algorithm. The highest detection accuracy was attained when using a ballistic MI, an MRCP band between 0.05 and 3 Hz, and a time-series analysis (MRCP morphology analysis) [81]. More recently, in 2019, Liu et al. developed a novel MI-based brain-switch using a sample entropy as a feature [83]. They tested the performance of the proposed brain-switch on five patients with amyotrophic lateral sclerosis and demonstrated its feasibility (TPR of 89.50% and FPR of 7.00%). In addition to feature extraction methods based on covariance matrices, such as Riemannian geometry and CSP, autoregressive coefficients have been also used as a feature for an EEG-based endogenous brain-switches [89,90,93]. In 2009, Faradji et al. performed a study to assess the plausibility of an endogenous brain-switch based on a mental imagery task [89]. Their endogenous brain-switch was only activated when a subject performed a specific MI task and remained otherwise. They optimized several factors for each subject, such as the type of mental tasks (mental multiplication, letter composing, object rotation, and number counting), autoregressive coefficients, and classifiers (linear discriminant analysis, quadratic discriminant analysis, Mahalanobis discriminant analysis, support vector machine, and radial basis function neural network). They reported that the TPR and FPR of the optimized best classification model were 78.70% and 0.00%, respectively. In 2019, the same group tested whether an endogenous brain-switch can operate with a small number of EEG channels [93]. Most of the methods were equal to the previous study [89], such as the use of the four mental tasks and autoregressive coefficients. They showed that the two-state endogenous brain-switch can show reasonable TPR of 54.60% and FPR of 0.00%, even though only five electrodes were used for classification of a no-control state and a specific mental task.

3.5.2. Exogenous Brain-Switches

Table 4 shows a list of EEG-based exogenous brain-switch studies. As mentioned in the Methods section, a few studies have developed EEG-based exogenous brain-switches using visual stimuli, such as visual ERP [33,54] or SSVEP [8,59,74]. No other sensory-based stimuli have been used to develop EEG-based exogenous brain-switches. A main concern in EEG-based exogenous brain switch studies has been minimizing the brain responses evoked during a no-control state because visual stimuli are continuously presented to subjects even in a no-control state. Another concern was to minimize the user fatigue induced through the repetitive presentation of visual stimuli. Therefore, in EEG-based exogenous brain-switch studies, some solutions have been proposed to overcome the two mentioned concerns by introducing a pseudo-key-based approach [33,59] and a chromatic visual stimulus [8].
In 2011, Panicker et al. proposed an asynchronous P300-based speller with a brain-switch based on SSVEP [74]. The SSVEP-based brain-switch was used for control state detection of the P300-based speller. In the experiment, the whole display of a screen was flickered at one frequency around 18 Hz to elicit SSVEP responses. If a subject wants to control the P300-based speller, she/he just gazed at the screen to evoke SSVEP responses. Once the desired SSVEP response was detected, the user could type characters using the P300-based speller. The SSVEP-based brain-switch showed the TPR of 96.60% and the FPR of 0.03%. The limitation of this study was that a subject has to close their eyes during a no-control state to prevent the generation of SSVEP responses. In 2013, Pan et al. developed an EEG-based exogenous brain-switch that uses SSVEP responses [59]. Strong SSVEP responses can be produced even in a no-control state because visual stimuli are continuously presented not only in a control state but also in a no-control state. To solve this problem, Pan et al. introduced a pseudo-key-based approach. A predefined target key used for a brain-switch was presented with three pseudo-keys, and the SSVEP amplitudes of each key were simultaneously considered to prevent FPRs [59]. They showed that the proposed SSVEP-based brain switch can dramatically reduce false operations, with a TPR of 78.75% and an FPR of 4.17%. More recently, in 2017, Lim et al. developed an SSVEP-based brain-switch for an emergency call system [8] and designed a chromatic visual stimulus, which was proved to be visually less fatiguing than conventional visual stimuli for SSVEP (e.g., a checkerboard pattern stimulus). The chromatic visual stimulus was located at the right-bottom corner of a screen, and an SSVEP-based brain-switch monitored the ongoing SSVEP responses to check whether the SSVEP response evoked by the chromatic visual stimulus meets the criterion predefined based on a template matching algorithm. The authors demonstrated for the first time the clinical feasibility of the proposed SSVEP-based brain-switch with LIS patients; their brain-switch showed a TPR of 100%, an FPR of 0.38 FPs/min, and an ODT of 9.90 s.
ERP-based brain-switches have been also developed [33,54]. Fedorova et al. developed an EEG-based exogenous brain-switch based on ERP responses evoked by a single visual stimulus [54]. In their paradigm, visual stimuli (animal faces) were sequentially presented to a subject, and the subject silently counted the number of target stimuli to control the ERP-based brain-switch. The proposed brain-switch was operated when the ERP feature exceeded a threshold predefined through a calibration session. The developed ERP-based brain-switch was successfully applied to produce a “stop” command for a robot arm control with a low FPR of 0.1/min and an ODT of approximately 4 s. In addition, in 2013, He et al. developed a P300-based brain-switch to control a wheelchair, in which a brain-switch was used to produce “start” and “stop” control commands [33]. In their paradigm, four buttons were located at the four corners of a monitor screen, and a key presented at the right-bottom corner was used as a target key for a brain-switch, whereas the others were pseudo keys, similar to Pan et al.’s study on SSVEP [59]. A threshold-free algorithm based on two SVMs was used to detect the user’s intention to turn on a wheelchair system. The first SVM was used to check whether a target key was focused on, and the second SVM then discriminated the control and no-control states when the first SVM determined that the user had focused on the target key. The proposed ERP-based brain switch was successfully used to produce “start” and “stop” commands for the control of an intelligent wheelchair, where the TPR and FPR during an offline experiment were 86.30% and 0.94%, respectively.

3.5.3. Hybrid Brain-Switches

Table 5 shows a list of EEG-based hybrid brain-switch studies that used two different BCI paradigms simultaneously (i.e., SSVEP + ERP [35,80,96] or SSVEP + MI [78]). In 2013, Li et al. developed a hybrid brain-switch combining P300 and SSVEP for wheelchair control [35]. They arranged four different visual stimuli at center-left, center-right, top, and bottom on a screen. Each stimulus could simultaneously elicit an SSVEP and a P300 response, and a subject has to gaze at a stimulus if he/she wants to control a wheelchair. Once the user gazes at a stimulus, a no-control state was changed to a control state regardless of the type of stimuli. During the no-control state, the subject was asked to rest without paying attention to any stimuli. They compared the performance of P300-, SSVEP-, and P300/SSVEP-based brain-switches and reported that the P300/SSVEP-based brain-switch outperforms the P300- or SSVEP-based brain-switch. The hybrid brain-switch showed the FPR of 0.49 FPs/min and the ODT of 4.30 s. In 2014, Cao et al. proposed a hybrid BCI system that combines MI and SSVEP to control an intelligent wheelchair [78], where turning on or off the hybrid BCI system is achieved by focusing on flickering buttons and hand motor imagery concurrently; the user focused on a button flickering at 8 Hz and performed a left-handed MI simultaneously to turn on a BCI wheelchair while focusing on a button flickering at 7 Hz and performing a right-handed MI to turn off the BCI wheelchair. All subjects in this study successfully completed a goal-directed wheelchair control task, validating the reliability of the proposed hybrid brain switch. In 2015, Pinegger et al. evaluated the performance of a hybrid P300/SSVEP-based brain-switch by comparing a P300- and an SSVEP-based brain-switches [96]. They used two screens for the experiments; a P300-based speller was displayed on the left screen while a video clip or a frozen screen was displayed on the right screen. An SSVEP response and a P300 response were simultaneously evoked when the subject gazed at a stimulus of the P300-based speller. A user had to use the P300-based speller during a control state while watch a video clip or a frozen screen to prevent the generation of P300 and SSVEP responses during a no-control state. In an offline analysis, the P300/SSVEP hybrid brain-switch showed the highest TPR of 99.00%. More recently, in 2016, Peng et al. proposed an intelligent nursing bed system controlled by a P300-based BCI [80]. In this system, a brain-switch was designed to turn on/off the P300-based BCI system based on the simultaneous use of SSVEP and P300; the user gazed at a flashing button, which simultaneously evoked SSVEP and P300 to operate the brain-switch. If significant SSVEP and P300 responses were detected, the brain-switch turned on or off the P300-based BCI system for the nursing bed control. This hybrid brain-switch showed a reliable FPR of 0.15 FPs/min with an ODT of 9.31 s.

3.6. NIRS-Based Brain-Switches

Table 6 shows a list of NIRS-based brain-switches. According to our investigation, there have been only three studies conducted on developing NIRS-based brain-switches [63,64,65], in which mental subtraction (MS), MI, and SSVEP were used as control tasks for the NIRS-based brain-switches, respectively. All studies have mainly used changes in oxygenated or deoxygenated hemoglobin as features for NIRS-based brain-switches.
In 2012, a study introduced by Sagara et al. described the development of a NIRS-based endogenous brain switch [63], where MS was used as a control task. The NIRS-based endogenous brain-switch was operated when the blood volume exceeded a predefined threshold. Eight healthy subjects were able to successfully change a television channel or the movement of a toy robot with an average switching time of 11.50 s and a TPR of 83.30%. This study was meaningful in terms of its practicality because only two NIRS channels were used to develop a NIRS-based brain-switch. More recently, in 2015, Koo et al. developed an asynchronous MI-based BCI based on the simultaneous use of EEG and NIRS [65]. In their system, NIRS was used to detect the MI intention as a brain-switch, whereas EEG was used to classify left-hand and right-hand MI during a control state. The NIRS-based endogenous brain-switch was operated using a threshold-based algorithm. If a subject performed an MI task regardless of the MI type, HbO significantly increased during the task period. The NIRS-based brain-switch could turn on or off when the HbO exceeded a predefined threshold and showed a TPR of 98% and an FPR of 3% with an ODT of 10.36 s.
In 2014, Tomita et al. proposed a NIRS-based exogenous brain switch based on SSVEP [64]; NIRS was used to detect active (control) and idle (no-control) states as a brain-switch, whereas EEG was used to select one of the eight BCI commands. In their experiment, EEG and NIRS sensors were attached to the occipital area to measure the brain activity evoked through a visual stimulation. The NIRS-based brain-switch determined either a control or no-control state based on the fact that HbO gradually increases while a visual stimulus is presented regardless of its reversing frequency. Both a NIRS-based brain switch and an EEG-based BCI system were operated based on threshold-based algorithms. The authors reported that the control and no-control states could be detected with an accuracy of approximately 80%.

3.7. ECoG-Based Brain-Switches

Table 7 shows a list of ECoG-based brain-switches. Only two studies have used ECoG to develop brain-switches based on ME [17] and MI [18]. The two studies all used ERD/ERS as a feature for ECoG-based brain-switches.
In 2013, Williams et al. proposed an ECoG-based brain-switch [17]. Their objective was to detect a control intention triggered by a hand ME from a no-control state. Two monkeys were implanted bilaterally with ECoG arrays in epidural space and trained to perform a well-designed hand movement task that moves a cursor to peripheral targets. During a no-control state, the monkeys did not conduct any movement tasks. The authors examined the spectral differences in epidural micro-ECoG signals between the control and no-control state and used ERD/ERS values as a feature for identification of the control and no-control state. The proposed ECoG-based brain-switch showed a TPR of 100.00% and an FPR of 0.08%.
In 2019, Benabid et al. developed an ECoG-based BCI system to control an exoskeleton [18]. A patient with tetraplegia was implanted with wireless epidural recorders and performed various mental tasks to progressively increase the number of degrees of freedom (DoF) for the control of an exoskeleton for 24 months. The mental tasks included mono-dimensional (1D) for a linear translation or a rotation around an axis, bidimensional (2D) for movements in a plane, and three-dimensional (3D) for movements in a volume. The patient could perform tasks with eight DoF at the end of the study. The first step of the study was to develop an ECoG-based brain-switch based on a 1D task. After movement instruction, epidural ECoG data were recorded while the patient performed a given 1D task. The ECoG data were converted into time-frequency pattern maps and they were used to train a Markov switching linear model. The ECoG-based brain-switch showed a TPR of 92.10% and an FPR of 4.90 FPs/min.

4. Discussions

During the past decades, various types of brain-switches have been introduced to develop practical asynchronous BCI systems with a low FPR. According to previous studies, brain-switches have effectively improved the performance of asynchronous BCI systems, and the FPR of asynchronous BCIs has been dramatically reduced when a brain-switch is combined with an asynchronous BCI. In this study, we conducted a systematic review to investigate the state-of-the-art development of technologies used in brain-switches, thereby providing profound insight into future directions for the development of practical BCI systems. A total of 49 papers published between 2000 and 2019 were collected for this review, and we provided a thorough summary of such studies regarding the development of brain-switches.
Brain-switches have been developed using EEG, NIRS, or ECoG. Most brain-switches have been developed using EEG (about 90%, 44 of 49), which is related to its several advantages. EEG has a high-temporal resolution compared to other neuroimaging modalities, such as NIRS, which can lead to a fast detection speed of brain-switches. In addition, EEG has high portability and its cost is reasonable compared to other neuroimaging modalities, which are why most researchers have used the approach in the development of various brain-switches. By contrast, NIRS and ECoG have been used relatively less frequently in the development of brain-switches (NIRS—about 6%, 3 of 49; and ECoG—about 4%, 2 of 49), which might be related to the poorer temporal resolution of NIRS and the high risk of ECoG to mount electrodes on the brain than those of EEG. However, because NIRS is less susceptible to physiological artifacts and ECoG has faster temporal and higher spatial resolution as compared to EEG, NIRS and ECoG can also be usefully applied to the development of a brain-switch if their limitations are overcome. Recent NIRS studies have introduced a solution to tackle the poor temporal resolution of NIRS, which used task-related initial dip features extracted at approximately 0–2.5 s from the task onset [100,101]. In fact, one NIRS study showed that a BCI command generation time can be significantly reduced from 7 to 2.5 s even though a slight loss in classification accuracy occurred [102]. Therefore, it is expected that NIRS might be more frequently used to develop brain-switches in future studies. In addition, recent ECoG studies have developed ECoG-electrode arrays with flexibility [103,104] and proposed minimally-invasive methods of ECoG recording [105,106,107]. The state-of-the-art technologies for ECoG recording can be used to overcome its limitations, such as the high risk of ECoG surgeries and short survival time of ECoG electrodes, and thereby ECoG-based brain-switches might be actively developed in the near future.
Endogenous paradigms have been used most frequently in the development of brain-switches (about 80%, 40 of 49). Considering that LIS patients who are one of the main target groups of BCI technology cannot freely move their body, the use of endogenous paradigms is reasonable because endogenous brain-switches can be operated without any body movement; it is only necessary for the user to conduct a pre-defined mental task to control a brain-switch. However, brain-switches based on exogenous paradigms, such as SSVEP and ERP, have also been developed (about 20%, 10 of 49). It would be a disadvantage for potential BCI users to use visual stimuli to evoke specific brain patterns because eye movement is generally required to obtain a high performance when using such visual-stimuli-based exogenous BCIs [108]. However, exogenous BCI systems generally show higher performance than that of endogenous BCI systems [8,109], which was also confirmed for brain-switches (Table 8). Thus, exogenous brain-switches can be usefully applied to LIS patients who have (partial) intact visual functions, achieving high performance. Because gaze-independent visual paradigms based on covert attention have been successfully employed in the development of BCI systems [110,111], they can be used as an alternative to the current vision-based exogenous paradigm of brain-switches; no eye movement is required in gaze-independent BCIs. Other than vision, no other-sensory organs have been utilized for developing an exogenous brain-switch, and thus other sensory approaches, such as hearing [112] and the use of somatic sensations [113], can be alternatives to the vision-based exogenous paradigm in the development of exogenous brain-switches. In general, auditory and tactile BCIs do not require any body movements.
A standard deviation is not reported when only one study was used to calculate statistics.
The overall performances of all brain-switch approaches considered are compared in Table 8. The mean performances summarized in Table 8 were obtained by averaging the performance values listed in Table 3, Table 4, Table 5, Table 6 and Table 7. Because each study reported different performance metrics, the mean performance of each brain-switch approach cannot be fairly compared or generalized. However, the results shown in Table 8 can be used as a helpful reference for future studies. For EEG-based brain-switches, exogenous brain-switches showed a higher performance than endogenous brain-switches. EEG-based hybrid brain-switches showed the most comparable results, and in particular demonstrated a significantly low FPR. However, note that because only three hybrid brain-switch studies reported its performance, it should not be generalized, and more hybrid brain-switch studies should be conducted to confirm its superior performance over other approaches. Interestingly, NIRS- and ECoG-based brain-switches generally showed performances quite comparable to those of EEG-based brain-switches; however, owing to the small number of NIRS- and ECoG-based brain-switch studies (NIRS = 3 and ECoG = 2), more NIRS- and ECoG-based brain-switch studies should be also conducted to accurately confirm their performance.
What will be the next steps for the advancement of brain-switches? The most important point will be to improve the overall performance of identifying different brain patterns used for brain-switches, which will naturally result in a performance increase. We comment on four different categories for future directions: (i) advanced classification algorithms, (ii) robust feature extraction methods, (iii) multimodal approaches, and (iv) new stimulation methods for exogenous brain-switches.
To improve the performance of a brain-switch in terms of the classification algorithm, deep learning algorithms can be applied to the classification of different brain patterns. Deep learning has been successfully used in other research fields, such as speech [114] and image recognition [115], and has also been applied to the BCI research field, showing promising results [116,117]. However, deep learning has not been applied in the brain-switch context. It is expected that novel algorithms based on deep learning might be a good solution to improving the performance of brain-switches [118].
In addition, the introduction of novel feature extraction methods is required to improve the performance of brain-switches, such as RG and CSP, which have generally shown a higher performance than traditional ERD/ERS and MRCP (Table 3). As mentioned, the initial dip might be a useful feature to enhance the performance of NIRS-based brain-switches.
A multimodal approach can also be a good solution to improve their performance [119]. Recently, hybrid BCI systems, which simultaneously use EEG and NIRS, have reported a higher performance compared to a unimodal EEG- or NIRS-based BCI system [120,121,122]. The performance improvement of hybrid EEG–NIRS BCI systems is based on utilizing complementary EEG and NIRS features showing more separability compared to unimodal EEG or NIRS features. Two brain-switch studies used EEG and NIRS to measure the brain activity at the same time, but they applied each modality independently; NIRS and EEG were used to implement a brain-switch and a BCI system, respectively [64,65]. Therefore, a multimodal-based approach can be a solution to improve the performance of brain-switches.
For exogenous brain-switches, the development of new stimulation methods is required to reduce user fatigue induced by external stimulation such as the use of a chromatic visual stimulus [8], thereby fundamentally improving the performance of a brain-switch. If an auditory paradigm is used, natural sounds (e.g., syllables) can be used instead of artificial sounds (e.g., beeps) to improve the ergonomic factors and BCI performance [123].
A way to improve the performance of brain-switches has been discussed in terms of external stimulation and neuroimaging modality, but they should be well integrated to produce a synergy along with optimal feature extraction and classification methods. Thus, an investigation into their optimal combinations should be conducted to maximally increase the performance of brain-switches, as in a previous study investigating an optimal combination of the type of MI, analysis frequency band, and processing method [81].

Author Contributions

C.-H.H. investigated all previous studies and wrote first draft. K.-R.M. reviewed and wrote this manuscript. H.-J.H. supervised this study and wrote this manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by an Institute for Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (No. 2017-0-00451; Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning), by a National Research Foundation of Korea (NRF) grant funded by the Korea Ministry of Science, ICT and Future Planning (MSIP) (No. 2017R1C1B5017909), and by the Ministry of Trade Industry and Energy(MOTIE, Korea), Ministry of Science and ICT(MSIT, Korea), and Ministry of Health and Welfare(MOHW, Korea) under Technology Development Program for AI-Bio-Robot-Medicine Convergence (20001650).

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

BCIBrain–computer interface
LISLocked-in syndrome
EEGElectroencephalography
MEGMagnetoencephalography
fNIRSfunctional near-infrared spectroscopy
fMRIfunctional magnetic-resonance imaging
ECoGElectrocorticography
ITRInformation transfer rate
TPRTrue positive rate
FPRFalse positive rate
ERPEvent-related potential
SSVEPSteady-state visual evoked potential
SMRSensorimotor rhythm
MEMotor execution
MIMotor imagery
RGRiemannian geometry
CSPCommon spatial pattern
MRCPMotor-related cortical potential
ERDEvent-related desynchronization
ERSEvent-related synchronization
LF-ASDLow frequency asynchronous switch design
PSDPower spectrum density
CCACanonical correlation analysis
SVMSupport vector machine

References

  1. Kübler, A.; Kotchoubey, B.; Kaiser, J.; Wolpaw, J.R.; Birbaumer, N. Brain–computer communication: Unlocking the locked in. Psychol. Bull. 2001, 127, 358. [Google Scholar] [CrossRef] [PubMed]
  2. Wolpaw, J.R.; Birbaumer, N.; Heetderks, W.J.; McFarland, D.J.; Peckham, P.H.; Schalk, G.; Donchin, E.; Quatrano, L.A.; Robinson, C.J.; Vaughan, T.M. Brain-computer interface technology: A review of the first international meeting. IEEE Trans. Rehabil. Eng. 2000, 8, 164–173. [Google Scholar] [CrossRef] [PubMed]
  3. Wolpaw, J.R.; Birbaumer, N.; McFarland, D.J.; Pfurtscheller, G.; Vaughan, T.M. Brain–computer interfaces for communication and control. Clin. Neurophysiol. 2002, 113, 767–791. [Google Scholar] [CrossRef]
  4. Dornhege, G.; Millan, J.d.R.; Hinterberger, T.; McFarland, D.J.; Müller, K.-R. Toward Brain-Computer Interfacing; MIT press: Cambridge, MA, USA, 2007. [Google Scholar]
  5. Wolpaw, J.; Wolpaw, E.W. Brain-Computer Interfaces: Principles and Practice; OUP USA: Oxford, UK, 2012. [Google Scholar]
  6. Brunner, C.; Birbaumer, N.; Blankertz, B.; Guger, C.; Kübler, A.; Mattia, D.; Millán, J.d.R.; Miralles, F.; Nijholt, A.; Opisso, E. BNCI Horizon 2020: Towards a roadmap for the BCI community. Brain-Comput. Interfaces 2015, 2, 1–10. [Google Scholar] [CrossRef] [Green Version]
  7. Hwang, H.-J.; Lim, J.-H.; Jung, Y.-J.; Choi, H.; Lee, S.W.; Im, C.-H. Development of an SSVEP-based BCI spelling system adopting a QWERTY-style LED keyboard. J. Neurosci. Methods 2012, 208, 59–65. [Google Scholar] [CrossRef] [PubMed]
  8. Lim, J.H.; Kim, Y.W.; Lee, J.H.; An, K.O.; Hwang, H.J.; Cha, H.S.; Han, C.H.; Im, C.H. An emergency call system for patients in locked-in state using an SSVEP-based brain switch. Psychophysiology 2017, 54, 1632–1643. [Google Scholar] [CrossRef]
  9. Han, C.-H.; Kim, Y.-W.; Kim, S.H.; Nenadic, Z.; Im, C.-H. Electroencephalography-based endogenous brain–computer interface for online communication with a completely locked-in patient. J. Neuroeng. Rehabil. 2019, 16, 18. [Google Scholar] [CrossRef]
  10. Blankertz, B.; Tomioka, R.; Lemm, S.; Kawanabe, M.; Muller, K.-R. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process. Mag. 2007, 25, 41–56. [Google Scholar] [CrossRef]
  11. Blankertz, B.; Acqualagna, L.; Dähne, S.; Haufe, S.; Schultze-Kraft, M.; Sturm, I.; Ušćumlic, M.; Wenzel, M.A.; Curio, G.; Müller, K.-R. The Berlin brain-computer interface: Progress beyond communication and control. Front. Neurosci. 2016, 10, 530. [Google Scholar] [CrossRef] [Green Version]
  12. Blankertz, B.; Lemm, S.; Treder, M.; Haufe, S.; Müller, K.-R. Single-trial analysis and classification of ERP components—a tutorial. NeuroImage 2011, 56, 814–825. [Google Scholar] [CrossRef]
  13. Lin, P.T.; Sharma, K.; Holroyd, T.; Battapady, H.; Fei, D.-Y.; Bai, O. A high performance MEG based BCI using single trial detection of human movement intention. In Functional Brain Mapping and the Endeavor to Understand the Working Brain; IntechOpen: London, UK, 2013. [Google Scholar]
  14. Hwang, H.-J.; Lim, J.-H.; Kim, D.-W.; Im, C.-H. Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces. J. Biomed. Opt. 2014, 19, 077005. [Google Scholar] [CrossRef] [PubMed]
  15. Shin, J.; Müller, K.-R.; Hwang, H.-J. Near-infrared spectroscopy (NIRS)-based eyes-closed brain-computer interface (BCI) using prefrontal cortex activation due to mental arithmetic. Sci. Rep. 2016, 6, 36203. [Google Scholar] [CrossRef]
  16. Weiskopf, N.; Mathiak, K.; Bock, S.W.; Scharnowski, F.; Veit, R.; Grodd, W.; Goebel, R.; Birbaumer, N. Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI). IEEE Trans. Biomed. Eng. 2004, 51, 966–970. [Google Scholar] [CrossRef]
  17. Williams, J.J.; Rouse, A.G.; Thongpang, S.; Williams, J.C.; Moran, D.W. Differentiating closed-loop cortical intention from rest: Building an asynchronous electrocorticographic BCI. J. Neural Eng. 2013, 10, 046001. [Google Scholar] [CrossRef] [PubMed]
  18. Benabid, A.L.; Costecalde, T.; Eliseyev, A.; Charvet, G.; Verney, A.; Karakas, S.; Foerster, M.; Lambert, A.; Morinière, B.; Abroug, N. An exoskeleton controlled by an epidural wireless brain–machine interface in a tetraplegic patient: A proof-of-concept demonstration. Lancet Neurol. 2019, 18, 1112–1122. [Google Scholar] [CrossRef]
  19. Lakshmi, M.R.; Prasad, T.; Prakash, D.V.C. Survey on EEG signal processing methods. Int. J. Adv. Research Comput. Sci. Softw. Eng. 2014, 4, 84–91. [Google Scholar]
  20. Han, C.-H.; Lim, J.-H.; Lee, J.-H.; Kim, K.; Im, C.-H. Data-driven user feedback: An improved neurofeedback strategy considering the interindividual variability of EEG features. BioMed Res. Int. 2016, 2016, 1–7. [Google Scholar] [CrossRef] [Green Version]
  21. Fazli, S.; Mehnert, J.; Steinbrink, J.; Curio, G.; Villringer, A.; Müller, K.-R.; Blankertz, B. Enhanced performance by a hybrid NIRS–EEG brain computer interface. Neuroimage 2012, 59, 519–529. [Google Scholar] [CrossRef]
  22. Fazli, S.; Dähne, S.; Samek, W.; Bießmann, F.; Mueller, K.-R. Learning from more than one data source: Data fusion techniques for sensorimotor rhythm-based brain–computer interfaces. Proc. IEEE 2015, 103, 891–906. [Google Scholar] [CrossRef]
  23. Millán, J.d.R.; Rupp, R.; Müller-Putz, G.; Murray-Smith, R.; Giugliemma, C.; Tangermann, M.; Vidaurre, C.; Cincotti, F.; Kubler, A.; Leeb, R. Combining brain–computer interfaces and assistive technologies: state-of-the-art and challenges. Front. Neurosci. 2010, 4, 161. [Google Scholar] [CrossRef]
  24. Hwang, H.-J.; Kim, S.; Choi, S.; Im, C.-H. EEG-based brain-computer interfaces: A thorough literature survey. Int. J. Hum.-Comput. Interact. 2013, 29, 814–826. [Google Scholar] [CrossRef]
  25. Salvaris, M.; Sepulveda, F. Visual modifications on the P300 speller BCI paradigm. J. Neural Eng. 2009, 6, 046011. [Google Scholar] [CrossRef] [PubMed]
  26. Yin, E.; Zhou, Z.; Jiang, J.; Chen, F.; Liu, Y.; Hu, D. A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. J. Neural Eng. 2013, 10, 026012. [Google Scholar] [CrossRef] [PubMed]
  27. Lim, J.-H.; Lee, J.-H.; Hwang, H.-J.; Kim, D.H.; Im, C.-H. Development of a hybrid mental spelling system combining SSVEP-based brain–computer interface and webcam-based eye tracking. Biomed. Signal Process. Control 2015, 21, 99–104. [Google Scholar] [CrossRef]
  28. Birbaumer, N.; Ghanayim, N.; Hinterberger, T.; Iversen, I.; Kotchoubey, B.; Kübler, A.; Perelmouter, J.; Taub, E.; Flor, H. A spelling device for the paralysed. Nature 1999, 398, 297. [Google Scholar] [CrossRef] [PubMed]
  29. Ortner, R.; Allison, B.Z.; Korisek, G.; Gaggl, H.; Pfurtscheller, G. An SSVEP BCI to control a hand orthosis for persons with tetraplegia. IEEE Trans. Neural Syst. Rehabil. Eng. 2010, 19, 1–5. [Google Scholar] [CrossRef]
  30. Inoue, S.; Akiyama, Y.; Izumi, Y.; Nishijima, S. The development of BCI using alpha waves for controlling the robot arm. IEICE Trans. Commun. 2008, 91, 2125–2132. [Google Scholar] [CrossRef]
  31. McFarland, D.J.; Wolpaw, J.R. Brain-computer interface operation of robotic and prosthetic devices. Computer 2008, 41, 52–56. [Google Scholar] [CrossRef]
  32. Gandhi, V.; Prasad, G.; Coyle, D.; Behera, L.; McGinnity, T.M. EEG-based mobile robot control through an adaptive brain–robot interface. IEEE Trans. Syst. Man Cybern. Syst. 2014, 44, 1278–1285. [Google Scholar] [CrossRef]
  33. He, S.; Zhang, R.; Wang, Q.; Chen, Y.; Yang, T.; Feng, Z.; Zhang, Y.; Shao, M.; Li, Y. A P300-based threshold-free brain switch and its application in wheelchair control. IEEE Trans. Neural Syst. Rehabilitation Eng. 2016, 25, 715–725. [Google Scholar] [CrossRef]
  34. Huang, D.; Qian, K.; Fei, D.-Y.; Jia, W.; Chen, X.; Bai, O. Electroencephalography (EEG)-based brain–computer interface (BCI): A 2-D virtual wheelchair control based on event-related desynchronization/synchronization and state control. IEEE Trans. Neural Syst. Rehabil. Eng. 2012, 20, 379–388. [Google Scholar] [CrossRef] [PubMed]
  35. Li, Y.; Pan, J.; Wang, F.; Yu, Z. A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control. IEEE Trans. Biomed. Eng. 2013, 60, 3156–3166. [Google Scholar] [PubMed]
  36. Tsui, C.S.L.; Gan, J.Q.; Hu, H. A self-paced motor imagery based brain-computer interface for robotic wheelchair control. Clin. EEG Neurosci. 2011, 42, 225–229. [Google Scholar] [CrossRef] [PubMed]
  37. Choi, K.; Cichocki, A. Control of a wheelchair by motor imagery in real time. In Proceedings of the International conference on intelligent data engineering and automated learning, Daejeon, Korea, 2–5 November 2018; pp. 330–337. [Google Scholar]
  38. Rebsamen, B.; Guan, C.; Zhang, H.; Wang, C.; Teo, C.; Ang, M.H.; Burdet, E. A brain controlled wheelchair to navigate in familiar environments. IEEE Trans. Neural Syst. Rehabil. Eng. 2010, 18, 590–598. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  39. Li, Y.; Long, J.; Yu, T.; Yu, Z.; Wang, C.; Zhang, H.; Guan, C. An EEG-based BCI system for 2-D cursor control by combining Mu/Beta rhythm and P300 potential. IEEE Trans. Biomed. Eng. 2010, 57, 2495–2505. [Google Scholar] [CrossRef] [PubMed]
  40. Xia, B.; An, D.; Chen, C.; Xie, H.; Li, J. A mental switch-based asynchronous brain-computer interface for 2D cursor control. In Proceedings of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan, 3–7 July 2013; pp. 3101–3104. [Google Scholar] [CrossRef]
  41. Lin, C.-T.; Lin, F.-C.; Chen, S.-A.; Lu, S.-W.; Chen, T.-C.; Ko, L.-W. EEG-based brain-computer interface for smart living environmental auto-adjustment. J. Med. Biol. Eng. 2010, 30, 237–245. [Google Scholar] [CrossRef]
  42. Wang, Y.-T.; Wang, Y.; Jung, T.-P. A cell-phone-based brain–computer interface for communication in daily life. J. Neural Eng. 2011, 8, 025018. [Google Scholar] [CrossRef]
  43. Coyle, D.; Garcia, J.; Satti, A.R.; McGinnity, T.M. EEG-based continuous control of a game using a 3 channel motor imagery BCI: BCI game. In Proceedings of the 2011 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), Paris, France, 11–15 April 2011; pp. 1–7. [Google Scholar]
  44. Yoh, M.-S.; Kwon, J.; Kim, S. NeuroWander: A BCI game in the form of interactive fairy tale. In Proceedings of the 12th ACM international conference adjunct papers on Ubiquitous computing-Adjunct, Copenhagen, Denmark, 26–29 September 2019; pp. 389–390. [Google Scholar]
  45. Tangermann, M.W.; Krauledat, M.; Grzeska, K.; Sagebaum, M.; Vidaurre, C.; Blankertz, B.; Müller, K.-R. Playing pinball with non-invasive BCI. In Proceedings of the 21st International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–11 December 2008; pp. 1641–1648. [Google Scholar]
  46. Diez, P.F.; Mut, V.A.; Perona, E.M.A.; Leber, E.L. Asynchronous BCI control using high-frequency SSVEP. J. Neuroeng. Rehabil. 2011, 8, 39. [Google Scholar] [CrossRef] [Green Version]
  47. Lim, J.-H.; Hwang, H.-J.; Han, C.-H.; Jung, K.-Y.; Im, C.-H. Classification of binary intentions for individuals with impaired oculomotor function:‘eyes-closed’SSVEP-based brain–computer interface (BCI). J. Neural Eng. 2013, 10, 026021. [Google Scholar] [CrossRef]
  48. Xia, B.; Li, X.; Xie, H.; Yang, W.; Li, J.; He, L. Asynchronous brain–computer interface based on steady-state visual-evoked potential. Cogn. Comput. 2013, 5, 243–251. [Google Scholar] [CrossRef]
  49. Liu, R.; Wang, Y.-X.; Zhang, L. An FDES-based shared control method for asynchronous brain-actuated robot. IEEE Trans. Cybern. 2015, 46, 1452–1462. [Google Scholar] [CrossRef] [PubMed]
  50. Zhang, R.; Li, Y.; Yan, Y.; Zhang, H.; Wu, S.; Yu, T.; Gu, Z. Control of a wheelchair in an indoor environment based on a brain–computer interface and automated navigation. IEEE Trans. Neural Syst. Rehabil. Eng. 2015, 24, 128–139. [Google Scholar] [CrossRef] [PubMed]
  51. Borisoff, J.F.; Mason, S.G.; Bashashati, A.; Birch, G.E. Brain-computer interface design for asynchronous control applications: Improvements to the LF-ASD asynchronous brain switch. IEEE Trans. Biomed. Eng. 2004, 51, 985–992. [Google Scholar] [CrossRef] [PubMed]
  52. Blankertz, B.; Dornhege, G.; Schafer, C.; Krepki, R.; Kohlmorgen, J.; Muller, K.-R.; Kunzmann, V.; Losch, F.; Curio, G. Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG analysis. IEEE Trans. Neural Syst. Rehabil. Eng. 2003, 11, 127–131. [Google Scholar] [CrossRef]
  53. Huggins, J.E.; Wren, P.A.; Gruis, K.L. What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotroph. Lateral Scler. 2011, 12, 318–324. [Google Scholar] [CrossRef]
  54. Fedorova, A.A.; Shishkin, S.L.; Nuzhdin, Y.O.; Faskhiev, M.N.; Vasilyevskaya, A.M.; Ossadtchi, A.E.; Kaplan, A.Y.; Velichkovsky, B.M. A fast “single-stimulus” brain switch. In Proceedings of the Proc. 6th Int. Brain-Computer Interface Conf., Graz, Austria, 16–19 September 2014. [Google Scholar] [CrossRef]
  55. Barachant, A.; Bonnet, S.; Congedo, M.; Jutten, C. A Brain-Switch Using Riemannian Geometry; HAL Archive Ouvertes: Lyon, France, 2011; Available online: http://hal.archives-ouvertes.fr/hal-00629110 (accessed on 5 October 2011).
  56. Ding, S.; Zhang, H.; Li, J.; Li, C.; Sun, L. Study of A Brain-Controlled Switch during Motor Imagery. In Proceedings of the 2018 3rd International Conference on Advanced Robotics and Mechatronics (ICARM), Singapore, 18–20 July 2018; pp. 152–157. [Google Scholar]
  57. Jiang, N.; Mrachacz-Kersting, N.; Xu, R.; Dremstrup, K.; Farina, D. An accurate, versatile, and robust brain switch for neurorehabilitation. In Brain-Computer Interface Research; Springer: Berlin/Heidelberg, Germany, 2014; pp. 47–61. [Google Scholar]
  58. Müller-Putz, G.R.; Kaiser, V.; Solis-Escalante, T.; Pfurtscheller, G. Fast set-up asynchronous brain-switch based on detection of foot motor imagery in 1-channel EEG. Med Biol. Eng. Comput. 2010, 48, 229–233. [Google Scholar] [CrossRef]
  59. Pan, J.; Li, Y.; Zhang, R.; Gu, Z.; Li, F. Discrimination between control and idle states in asynchronous SSVEP-based brain switches: A pseudo-key-based approach. IEEE Trans. Neural Syst. Rehabil. Eng. 2013, 21, 435–443. [Google Scholar]
  60. Pfurtscheller, G.; Solis-Escalante, T.; Ortner, R.; Linortner, P.; Muller-Putz, G.R. Self-paced operation of an SSVEP-Based orthosis with and without an imagery-based “brain switch”: A feasibility study towards a hybrid BCI. IEEE Trans. Neural Syst. Rehabil. Eng. 2010, 18, 409–414. [Google Scholar] [CrossRef]
  61. Pfurtscheller, G.; Solis-Escalante, T. Could the beta rebound in the EEG be suitable to realize a “brain switch”? Clin. Neurophysiol. 2009, 120, 24–29. [Google Scholar] [CrossRef]
  62. Nikulin, V.V.; Linkenkaer-Hansen, K.; Nolte, G.; Lemm, S.; Müller, K.R.; Ilmoniemi, R.J.; Curio, G. A novel mechanism for evoked responses in the human brain. Eur. J. Neurosci. 2007, 25, 3146–3154. [Google Scholar] [CrossRef]
  63. Sagara, K.; Kido, K. Evaluation of a 2-channel NIRS-based optical brain switch for motor disabilities’ communication tools. IEICE Trans. Inf. Syst. 2012, 95, 829–834. [Google Scholar] [CrossRef] [Green Version]
  64. Tomita, Y.; Vialatte, F.-B.; Dreyfus, G.; Mitsukura, Y.; Bakardjian, H.; Cichocki, A. Bimodal BCI using simultaneously NIRS and EEG. IEEE Trans. Biomed. Eng. 2014, 61, 1274–1284. [Google Scholar] [CrossRef] [PubMed]
  65. Koo, B.; Lee, H.-G.; Nam, Y.; Kang, H.; Koh, C.S.; Shin, H.-C.; Choi, S. A hybrid NIRS-EEG system for self-paced brain computer interface with online motor imagery. J. Neurosci. Methods 2015, 244, 26–32. [Google Scholar] [CrossRef]
  66. Bashashati, A.; Ward, R.K.; Birch, G.E. Towards development of a 3-state self-paced brain-computer interface. Comput. Intell. Neurosci. 2007, 2007, 84386. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  67. Blankertz, B.; Curio, G.; Müller, K.-R. Classifying single trial EEG: Towards brain computer interfacing. Proceedings of Advances in neural information processing systems, Vancouver, BC, Canada, 3–8 December 2001; pp. 157–164. [Google Scholar]
  68. Blankertz, B.; Dornhege, G.; Lemm, S.; Krauledat, M.; Curio, G.; Müller, K.-R. The Berlin brain-computer interface: Machine learning based detection of user specific brain states. J. UCS 2006, 12, 581–607. [Google Scholar]
  69. Scherer, R.; Lee, F.; Schlögl, A.; Leeb, R.; Bischof, H.; Pfurtscheller, G. Towards self-paced (asynchronous) Brain-Computer Communication: Navigation through virtual worlds. IEEE Trans. Biomed. Eng. 2007, 55, 675–682. [Google Scholar] [CrossRef]
  70. Xu, R.; Jiang, N.; Dosen, S.; Lin, C.; Mrachacz-Kersting, N.; Dremstrup, K.; Farina, D. Endogenous sensory discrimination and selection by a fast brain switch for a high transfer rate brain-computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 2016, 24, 901–910. [Google Scholar] [CrossRef]
  71. Qian, K.; Nikolov, P.; Huang, D.; Fei, D.-Y.; Chen, X.; Bai, O. A motor imagery-based online interactive brain-controlled switch: Paradigm development and preliminary test. Clin. Neurophysiol. 2010, 121, 1304–1313. [Google Scholar] [CrossRef]
  72. Marquez-Chin, C.; Marquis, A.; Popovic, M.R. EEG-triggered functional electrical stimulation therapy for restoring upper limb function in chronic stroke with severe hemiplegia. Case Rep. Neurol. Med. 2016, 2016. [Google Scholar] [CrossRef] [Green Version]
  73. Leeb, R.; Friedman, D.; Müller-Putz, G.R.; Scherer, R.; Slater, M.; Pfurtscheller, G. Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: A case study with a tetraplegic. Comput. Intell. Neurosci. 2007, 2007, 79642. [Google Scholar] [CrossRef] [Green Version]
  74. Panicker, R.C.; Puthusserypady, S.; Sun, Y. An asynchronous P300 BCI with SSVEP-based control state detection. IEEE Trans. Biomed. Eng. 2011, 58, 1781–1788. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  75. Mueller-Putz, G.; Scherer, R.; Pfurtscheller, G.; Neuper, C. Temporal coding of brain patterns for direct limb control in humans. Front. Neurosci. 2010, 4, 34. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  76. Solis-Escalante, T.; Müller-Putz, G.; Brunner, C.; Kaiser, V.; Pfurtscheller, G. Analysis of sensorimotor rhythms for the implementation of a brain switch for healthy subjects. Biomed. Signal Process. Control 2010, 5, 15–20. [Google Scholar] [CrossRef]
  77. Choi, B.; Jo, S. A low-cost EEG system-based hybrid brain-computer interface for humanoid robot navigation and recognition. PLoS ONE 2013, 8, e74583. [Google Scholar] [CrossRef]
  78. Cao, L.; Li, J.; Ji, H.; Jiang, C. A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control. J. Neurosci. Methods 2014, 229, 33–43. [Google Scholar] [CrossRef]
  79. Spyrou, L.; Blokland, Y.; Farquhar, J.; Bruhn, J. Optimal multitrial prediction combination and subject-specific adaptation for minimal training brain switch designs. IEEE Trans. Neural Syst. Rehabil. Eng. 2015, 24, 700–709. [Google Scholar] [CrossRef]
  80. Peng, N.; Zhang, R.; Zeng, H.; Wang, F.; Li, K.; Li, Y.; Zhuang, X. Control of a nursing bed based on a hybrid brain-computer interface. In Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 16–20 August 2016; pp. 1556–1559. [Google Scholar]
  81. Xu, R.; Jiang, N.; Mrachacz-Kersting, N.; Dremstrup, K.; Farina, D. Factors of influence on the performance of a short-latency non-invasive brain switch: Evidence in healthy individuals and implication for motor function rehabilitation. Front. Neurosci. 2016, 9, 527. [Google Scholar] [CrossRef] [Green Version]
  82. Yu, Y.; Zhou, Z.; Jiang, J.; Yin, E.; Liu, K.; Wang, J.; Liu, Y.; Hu, D. Toward a Hybrid BCI: Self-Paced Operation of a P300-based Speller by Merging a Motor Imagery-Based “Brain Switch” into a P300 Spelling Approach. Int. J. Hum.–Comput. Interact. 2017, 33, 623–632. [Google Scholar]
  83. Liu, Y.-H.; Huang, S.; Huang, H.-C.; Peng, W.-H. Novel Motor Imagery-Based Brain Switch for Patients With Amyotrophic Lateral Sclerosis: A Case Study Using Two-Channel Electroencephalography. IEEE Consum. Electron. Mag. 2019, 8, 72–77. [Google Scholar] [CrossRef]
  84. Bozorgzadeh, Z.; Birch, G.E.; Mason, S.G. The LF-ASD brain computer interface: On-line identification of imagined finger flexions in the spontaneous EEG of able-bodied subjects. In Proceedings of the 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing, Proceedings (Cat. No. 00CH37100), Istanbul, Turkey, 5–9 June 2000; pp. 2385–2388. [Google Scholar]
  85. Birch, G.E.; Bozorgzadeh, Z.; Mason, S.G. Initial on-line evaluations of the LF-ASD brain-computer interface with able-bodied and spinal-cord subjects using imagined voluntary motor potentials. IEEE Trans. Neural Syst. Rehabil. Eng. 2002, 10, 219–224. [Google Scholar] [CrossRef]
  86. Fatourechi, M.; Bashashati, A.; Ward, R.K.; Birch, G.E. A hybrid genetic algorithm approach for improving the performance of the LF-ASD brain computer interface. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, PA, USA, 23 March 2005; Volume 345, pp. v/345–v/348. [Google Scholar]
  87. Borisoff, J.F.; Mason, S.G.; Birch, G.E. Brain interface research for asynchronous control applications. IEEE Trans. Neural Syst. Rehabil. Eng. 2006, 14, 160–164. [Google Scholar] [CrossRef] [PubMed]
  88. Fatourechi, M.; Ward, R.; Birch, G. A self-paced brain–computer interface system with a low false positive rate. J. Neural Eng. 2007, 5, 9. [Google Scholar] [CrossRef] [PubMed]
  89. Faradji, F.; Ward, R.K.; Birch, G.E. Plausibility assessment of a 2-state self-paced mental task-based BCI using the no-control performance analysis. J. Neurosci. Methods 2009, 180, 330–339. [Google Scholar] [CrossRef] [PubMed]
  90. Faradji, F.; Ward, R.K.; Birch, G.E. Toward development of a two-state brain–computer interface based on mental tasks. J. Neural Eng. 2011, 8, 046014. [Google Scholar] [CrossRef] [PubMed]
  91. Mohammadi, R.; Mahloojifar, A.; Coyle, D. A combination of pre-and postprocessing techniques to enhance self-paced BCIs. Adv. Hum.-Comput. Interact. 2012, 2012, 3. [Google Scholar] [CrossRef] [Green Version]
  92. Wang, P.T.; King, C.E.; Chui, L.A.; Do, A.H.; Nenadic, Z. Self-paced brain–computer interface control of ambulation in a virtual reality environment. J. Neural Eng. 2012, 9, 056016. [Google Scholar] [CrossRef]
  93. Faradji, F.; Ward, R.K.; Birch, G.E. A Self-Paced Two-State Mental Task-Based Brain-Computer Interface with Few EEG Channels. In New Frontiers in Brain-Computer Interfaces; IntechOpen: London, UK, 2019. [Google Scholar]
  94. Müller-Putz, G.R.; Pokorny, C.; Klobassa, D.S.; Horki, P. A single-switch BCI based on passive and imagined movements: Toward restoring communication in minimally conscious patients. Int. J. Neural Syst. 2013, 23, 1250037. [Google Scholar] [CrossRef]
  95. Xu, R.; Dosen, S.; Jiang, N.; Yao, L.; Farooq, A.; Jochumsen, M.; Mrachacz-Kersting, N.; Dremstrup, K.; Farina, D. Continuous 2-D control via state-machine triggered by endogenous sensory discrimination and a fast brain switch. J. Neural Eng. 2019, 16, 056001. [Google Scholar] [CrossRef]
  96. Pinegger, A.; Faller, J.; Halder, S.; Wriessnegger, S.C.; Müller-Putz, G.R. Control or non-control state: That is the question! An asynchronous visual P300-based BCI approach. J. Neural Eng. 2015, 12, 014001. [Google Scholar] [CrossRef]
  97. Yu, Y.; Zhou, Z.; Liu, Y.; Jiang, J.; Yin, E.; Zhang, N.; Wang, Z.; Liu, Y.; Wu, X.; Hu, D. Self-paced operation of a wheelchair based on a hybrid brain-computer interface combining motor imagery and P300 potential. IEEE Trans. Neural Syst. Rehabil. Eng. 2017, 25, 2516–2526. [Google Scholar] [CrossRef]
  98. Blokland, Y.; Vlek, R.; Karaman, B.; Özin, F.; Thijssen, D.; Eijsvogels, T.; Colier, W.; Floor-Westerdijk, M.; Bruhn, J.; Farquhar, J. Detection of event-related desynchronization during attempted and imagined movements in tetraplegics for brain switch control. In Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA, 28 August–1 September 2012; pp. 3967–3969. [Google Scholar] [CrossRef]
  99. Yu, Y.; Jiang, J.; Zhou, Z.; Yin, E.; Liu, Y.; Wang, J.; Zhang, N.; Hu, D. A self-paced brain-computer interface speller by combining motor imagery and P300 potential. In Proceedings of the 2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC), Hangzhou, China, 27–28 August 2016; pp. 160–163. [Google Scholar]
  100. Yoshino, K.; Kato, T. Vector-based phase classification of initial dips during word listening using near-infrared spectroscopy. Neuroreport 2012, 23, 947–951. [Google Scholar] [CrossRef] [PubMed]
  101. Hong, K.-S.; Naseer, N. Reduction of delay in detecting initial dips from functional near-infrared spectroscopy signals using vector-based phase analysis. Int. J. Neural Syst. 2016, 26, 1650012. [Google Scholar] [CrossRef] [PubMed]
  102. Zafar, A.; Hong, K.-S. Detection and classification of three-class initial dips from prefrontal cortex. Biomed. Opt. Express 2017, 8, 367–383. [Google Scholar] [CrossRef]
  103. Rubehn, B.; Bosman, C.; Oostenveld, R.; Fries, P.; Stieglitz, T. A MEMS-based flexible multichannel ECoG-electrode array. J. Neural Eng. 2009, 6, 036003. [Google Scholar] [CrossRef] [PubMed]
  104. Tolstosheeva, E.; Gordillo-González, V.; Hertzberg, T.; Kempen, L.; Michels, I.; Kreiter, A.; Lang, W. A novel flex-rigid and soft-release ECoG array. In Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA, 30 August–3 September 2011; pp. 2973–2976. [Google Scholar]
  105. Yamakawa, T.; Yamakawa, T.; Aou, S.; Ishizuka, S.; Suzuki, M.; Fujii, M.; Aoki, T. Subdural electrocorticogram measurement with a minimally-invasive procedure using an SMA-manipulated microelectrode array. Adv. Mater. Res. 2011, 222, 313–317. [Google Scholar] [CrossRef]
  106. Leuthardt, E.C.; Freudenberg, Z.; Bundy, D.; Roland, J. Microscale recording from human motor cortex: implications for minimally invasive electrocorticographic brain-computer interfaces. Neurosurg. Focus 2009, 27, E10. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  107. Zhang, D.; Song, H.; Xu, R.; Zhou, W.; Ling, Z.; Hong, B. Toward a minimally invasive brain–computer interface using a single subdural channel: A visual speller study. Neuroimage 2013, 71, 30–41. [Google Scholar] [CrossRef]
  108. Marchetti, M.; Piccione, F.; Silvoni, S.; Gamberini, L.; Priftis, K. Covert visuospatial attention orienting in a brain-computer interface for amyotrophic lateral sclerosis patients. Neurorehabilit. Neural Repair 2013, 27, 430–438. [Google Scholar] [CrossRef]
  109. Vialatte, F.-B.; Maurice, M.; Dauwels, J.; Cichocki, A. Steady-state visually evoked potentials: Focus on essential paradigms and future perspectives. Prog. Neurobiol. 2010, 90, 418–438. [Google Scholar] [CrossRef]
  110. Hwang, H.-J.; Ferreria, V.Y.; Ulrich, D.; Kilic, T.; Chatziliadis, X.; Blankertz, B.; Treder, M. A gaze independent brain-computer interface based on visual stimulation through closed eyelids. Sci. Rep. 2015, 5, 15890. [Google Scholar] [CrossRef]
  111. Won, D.-O.; Hwang, H.-J.; Kim, D.-M.; Müller, K.-R.; Lee, S.-W. Motion-based rapid serial visual presentation for gaze-independent brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 2017, 26, 334–343. [Google Scholar] [CrossRef] [PubMed]
  112. Kim, D.-W.; Hwang, H.-J.; Lim, J.-H.; Lee, Y.-H.; Jung, K.-Y.; Im, C.-H. Classification of selective attention to auditory stimuli: Toward vision-free brain–computer interfacing. J. Neurosci. Methods 2011, 197, 180–185. [Google Scholar] [CrossRef] [PubMed]
  113. Brouwer, A.-M.; Van Erp, J.B. A tactile P300 brain-computer interface. Front. Neurosci. 2010, 4, 19. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  114. Deng, L.; Hinton, G.; Kingsbury, B. New types of deep neural network learning for speech recognition and related applications: An overview. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 26–31 May 2013; pp. 8599–8603. [Google Scholar]
  115. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
  116. Schirrmeister, R.T.; Springenberg, J.T.; Fiederer, L.D.J.; Glasstetter, M.; Eggensperger, K.; Tangermann, M.; Hutter, F.; Burgard, W.; Ball, T. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 2017, 38, 5391–5420. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  117. Lawhern, V.J.; Solon, A.J.; Waytowich, N.R.; Gordon, S.M.; Hung, C.P.; Lance, B.J. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces. J. Neural Eng. 2018, 15, 056013. [Google Scholar] [CrossRef] [Green Version]
  118. Sturm, I.; Lapuschkin, S.; Samek, W.; Müller, K.-R. Interpretable deep neural networks for single-trial EEG classification. J. Neurosci. Methods 2016, 274, 141–145. [Google Scholar] [CrossRef] [Green Version]
  119. Müller-Putz, G.; Leeb, R.; Tangermann, M.; Höhne, J.; Kübler, A.; Cincotti, F.; Mattia, D.; Rupp, R.; Müller, K.-R.; Millán, J.d.R. Towards noninvasive hybrid brain–computer interfaces: Framework, practice, clinical application, and beyond. Proc. IEEE 2015, 103, 926–943. [Google Scholar] [CrossRef]
  120. Shin, J.; von Lühmann, A.; Blankertz, B.; Kim, D.-W.; Jeong, J.; Hwang, H.-J.; Müller, K.-R. Open access dataset for EEG+ NIRS single-trial classification. IEEE Trans. Neural Syst. Rehabil. Eng. 2016, 25, 1735–1745. [Google Scholar] [CrossRef]
  121. Shin, J.; Kim, D.-W.; Müller, K.-R.; Hwang, H.-J. Improvement of information transfer rates using a hybrid EEG-NIRS brain-computer interface with a short trial length: Offline and pseudo-online analyses. Sensors 2018, 18, 1827. [Google Scholar] [CrossRef] [Green Version]
  122. Shin, J.; Müller, K.-R.; Schmitz, C.H.; Kim, D.-W.; Hwang, H.-J. Evaluation of a compact hybrid brain-computer interface system. BioMed Res. Int. 2017, 2017, 6820482. [Google Scholar] [CrossRef]
  123. Höhne, J.; Krenzlin, K.; Dähne, S.; Tangermann, M. Natural stimuli improve auditory BCIs with respect to ergonomics and performance. J. Neural Eng. 2012, 9, 045003. [Google Scholar] [CrossRef] [PubMed]
Electronics 09 00422 g001 550
Figure 1. Schematic of a two-step approach operated by a brain-switch. Red and blue vertical arrows indicate a brain-switch changing a no-control state into a control state, and vice versa, respectively.
Figure 1. Schematic of a two-step approach operated by a brain-switch. Red and blue vertical arrows indicate a brain-switch changing a no-control state into a control state, and vice versa, respectively.
Electronics 09 00422 g001
Electronics 09 00422 g002 550
Figure 2. Taxonomy of brain-switches. N indicates the number of articles included in each category. MT indicates mental task except for motor imagery (MI) or motor execution (ME), such as mental multiplication, letter composing, object rotation, and number counting.
Figure 2. Taxonomy of brain-switches. N indicates the number of articles included in each category. MT indicates mental task except for motor imagery (MI) or motor execution (ME), such as mental multiplication, letter composing, object rotation, and number counting.
Electronics 09 00422 g002
Table
Table 1. Description of brain-computer interface (BCI)-related terminologies used in this review.
Table 1. Description of brain-computer interface (BCI)-related terminologies used in this review.
TermDescription
Motor imagery (MI)Motor imagery (MI) is defined as imagination of a kinesthetic body movement, such as foot, tongue, left hand, right hand, and so on.
Motor execution (ME)Motor execution (ME) is a voluntary body movement, such as foot, tongue, left hand, right hand, and so on.
Slow cortical potential (SCP)Slow cortical potential (SCP) is brain activity related to motor and cognitive preparation, which lasts for from 300 ms to several seconds.
Event-related (de)synchronization (ERD/ERS)Event-related (de)synchronization (ERD/ERS) is the decrease/increase in frequency power induced by an internally or externally paced event regardless of the event type, and thus ERD/ERS is induced during sensory, cognitive, or motor tasks. In particular, mu and beta ERD/ERS are generally shown around the sensorimotor cortex during MI or ME.
Event-related potential (ERP)Event-related potential (ERP) is a time-locked brain response evoked by specific visual, auditory, and tactile stimuli.
Steady-state visual evoked potential (SSVEP)Steady-state visual evoked potential (SSVEP) is a periodic brain response evoked by repetitive presentation of a visual stimulus, flickering or reversing at a specific frequency.
Movement-related cortical potential (MRCP)Movement-related cortical potential (MRCP) is brain activity observed before and after ME and MI, which has a slow negative potential occurring from about 2 s prior to the onset of ME and MI.
Sensorimotor Rhythm (SMR)Sensorimotor rhythm (SMR) is a specific brain wave over the sensorimotor cortex generated after MI or ME. The frequency of the SMR is in range of 13 to 15 Hz.
True positive rate (TPR)True positive rate (TPR) means the ratio of the number of correctly detected trials to the number of all trials. In TPR, a trial means an intention to turn into the control state from the no-control state.
False positive rate (FPR)False positive rate (FPR) means the ratio of the number of incorrectly detected trials to the number of all trials. In FPR, a trial means no intention to turn into the control state from the no-control state. Two metrics are used to quantify FPR, which are percentage and the number of FPs/minute. The percentage does not have any time information and only shows the rate of incorrectly detected trials while FPs/minute has the time information how long a brain-switch is operated without false positives.
Onset detection time (ODT)Onset detection time (ODT) means the time taken to detect the user’s intention to turn on a control state from a no-control state using a brain-switch. Note that ODT was reported for 8 out of 49 studies using asynchronous paradigms that wait until onset intention was detected. This is because other studies used a fixed time for onset detection (e.g., 5 s), thus it is impossible to estimate when exactly onset intention is detected during the fixed time.
Table
Table 2. Detection algorithms used in brain-switches.
Table 2. Detection algorithms used in brain-switches.
Detection AlgorithmPaper #
Simple threshold method[59,63,64,65,71,72,73,74]
Classifier-based threshold method[17,18,35,40,51,52,54,55,58,60,61,66,67,68,69,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97]
Classifier-based template matching method[8,70]
Classifier-based threshold-free method[33]
Table
Table 3. List of electroencephalography (EEG)-based endogenous brain-switches. Note that eight papers [52,67,68,72,73,77,92,95] did not report true positive rate (TPR) and false positive rate (FPR) at all. MM, mental multiplication; LC, letter composing; OR, object rotation; NC, number counting; FBP, frequency band power; AR, autoregressive coefficients; RG, Riemannian geometry; CSP, common spatial pattern.
Table 3. List of electroencephalography (EEG)-based endogenous brain-switches. Note that eight papers [52,67,68,72,73,77,92,95] did not report true positive rate (TPR) and false positive rate (FPR) at all. MM, mental multiplication; LC, letter composing; OR, object rotation; NC, number counting; FBP, frequency band power; AR, autoregressive coefficients; RG, Riemannian geometry; CSP, common spatial pattern.
Paper #TaskFeature TypeTPR (%)FPRODT
(%)(FPs/min)(s)
[51]Finger MEERD/ERS81.002.00--
[84]Finger MIERD/ERS≥70.00≤3.00--
[85]Finger MIERD/ERS≥70.00≤3.00--
[86]Finger MEERD/ERS76.902.00--
[87]Finger MEERD/ERS73.002.00--
[66]Foot MEERD/ERS58.001.00
[69]Hand, foot, or tongue MIFBP28.4016.90
[88]Finger MEERD/ERS56.200.10--
[61]Foot MIERD/ERS59.20≤10.00--
[89]MM, LC, OR, NCAR78.700.00--
[58]Foot MIERD/ERS79.00≤10.00--
[75]Hand or foot MIERD/ERS70.2036.00--
[60]Foot MIERD/ERS--1.46-
[71]Hand MIERD/ERS-0.78-36.90
[76]Foot MIERD/ERS46.0014.00--
[90]MM, LC, OR, NCAR67.260.00
[55]Hand MIRG91.30-0.40-
[91]Foot MEFBP85.005.00
[94]Hand or foot MIERD/ERS58.4427.50--
[40]Hand or foot MICSP≥80.00---
[79]Hand MEERD/ERS64.0033.00--
[70]Foot MIMRCP72.00-1.90-
[81]Foot MIMRCP70.00---
[99]Hand MICSP86.46---
[83]Foot MISample entropy89.507.00--
[93]MM, LC, OR, or hand MIAR54.600.00--
[97]Hand MICSP85.704.43-2.05
[52]Hand MEERP----
[72]Hand MEERD/ERS----
[77]Hand or foot MIERD/ERS----
[67]Hand MEERP----
[68]Hand MEERP----
[73]Foot MIFBP----
[92]Foot MIFBP----
[95]Foot MIMRCP----
Table
Table 4. A list of electroencephalography (EEG)-based exogenous brain-switches. PSD, power spectral density.
Table 4. A list of electroencephalography (EEG)-based exogenous brain-switches. PSD, power spectral density.
Paper #TaskFeature TypeTPR (%)FPRODT
(%)(FPs/min)(s)
[59]SSVEPPSD78.754.170.98-
[54]ERPP300--0.10≒4.00
[33]ERPP30086.300.94--
[8]SSVEPPSD100.00 0.389.90
[74]SSVEPPSD96.600.03--
Table
Table 5. List of electroencephalography (EEG)-based hybrid brain-switches. Note that one study [78] did not report any performance metrics. CCA, canonical correlation analysis.
Table 5. List of electroencephalography (EEG)-based hybrid brain-switches. Note that one study [78] did not report any performance metrics. CCA, canonical correlation analysis.
Paper #TaskFeature TypeTPR (%)FPRODT
(%)(FPs/min)(s)
[35]SSVEP + ERPPSD + P300--0.494.30
[80]SSVEP + ERPPSD + P30093.75-0.159.31
[96]SSVEP + ERPPSD + P30099.00---
[78]SSVEP + MICCA + CSP----
Table
Table 6. List of near-infrared spectroscopy (NIRS)-based brain-switch studies. Δ HbO, the change in oxygenated hemoglobin; △ HbR, the change in deoxygenated hemoglobin; MS, mental subtraction.
Table 6. List of near-infrared spectroscopy (NIRS)-based brain-switch studies. Δ HbO, the change in oxygenated hemoglobin; △ HbR, the change in deoxygenated hemoglobin; MS, mental subtraction.
Paper #TaskFeature TypeTPR (%)FPRODT
(%)(FPs/min)(s)
[63]MSΔ HbO83.30--11.50
[64]SSVEP2nd derivatives of Δ HbO and Δ HbR≒80.00--
[65]Hand MIΔ HbO98.003.00-10.36
Table
Table 7. List of electrocorticography (ECoG)-based brain-switch studies.
Table 7. List of electrocorticography (ECoG)-based brain-switch studies.
Paper #TaskFeature TypeTPRFPRODT
(%)(%)(FPs/min)(s)
[17]Hand MEERD/ERS100.000.08--
[18]Hand MIERD/ERS92.10-4.90-
Table
Table 8. Overall performances of each brain-switch approach. Note that the means and standard deviations are calculated based on studies reporting corresponding performance metrics, and the numbers of studies used to calculate each performance metric are provided in parentheses.
Table 8. Overall performances of each brain-switch approach. Note that the means and standard deviations are calculated based on studies reporting corresponding performance metrics, and the numbers of studies used to calculate each performance metric are provided in parentheses.
The Type of Brain-SwitchesNumber of StudiesTPRFPRODT
(%)(%)(FPs/min)(s)
EEG-based endogenous brain-switches3570.03 ± 14.70
(25)		08.46 ± 11.05
(21)		1.25 ± 0.77
(3)		19.48 ± 24.64
(2)
EEG-based exogenous brain-switches590.41 ± 9.710
(4)		1.71 ± 2.18
(3)		0.49 ± 0.45
(3)		06.95 ± 4.17
(2)
EEG-based hybrid brain-switches496.38 ± 3.710
(2)		-0.32 ± 0.24
(2)		06.81 ± 3.54
(2)
NIRS-based endogenous brain-switches290.65 ± 10.39
(2)		3.00
(1)		-10.93 ± 0.81
(2)
NIRS-based exogenous Brain-switches1≒80.00
(1)		---
ECoG-based endogenous brain-switches296.05 ± 5.59
(2)		0.08
(1)		4.90
(1)		-
A standard deviation is not reported when only one study was used to calculate statistics.

Share and Cite

MDPI and ACS Style

Han, C.-H.; Müller, K.-R.; Hwang, H.-J. Brain-Switches for Asynchronous Brain–Computer Interfaces: A Systematic Review. Electronics 2020, 9, 422. https://doi.org/10.3390/electronics9030422

AMA Style

Han C-H, Müller K-R, Hwang H-J. Brain-Switches for Asynchronous Brain–Computer Interfaces: A Systematic Review. Electronics. 2020; 9(3):422. https://doi.org/10.3390/electronics9030422

Chicago/Turabian Style

Han, Chang-Hee, Klaus-Robert Müller, and Han-Jeong Hwang. 2020. "Brain-Switches for Asynchronous Brain–Computer Interfaces: A Systematic Review" Electronics 9, no. 3: 422. https://doi.org/10.3390/electronics9030422

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop