Brain–computer interfaces (BCIs) allow users to communicate with the external devices by converting brain signals into commands [1
]. BCIs can help people with neuromuscular diseases to improve the life quality [3
] or help special appliance operators like astronauts whose movements were restricted by the environment to work more efficiently [4
]. As a kind of brain signal that owns high temporal resolution and convenience of acquisition, electroencephalogram (EEG) is welcomed by BCI researchers. Event-related potentials (ERPs) [6
], steady-state visual evoked potentials (SSVEPs) [8
], and event-related desynchronization/synchronization (ERD/ERS) [10
] are typical EEG features used in BCI researches.
Of these features SSVEPs that were induced by repetitive stimuli are widely employed in cognitive research and high-speed BCI systems for their high stability and signal-to-noise ratio (SNR) [12
]. Researchers payed great efforts on the performance improvement of SSVEP-based BCIs in recent years, which mainly focused on the number of targets and target recognition algorithms. In order to increase the number of targets, a variety of novel coding methods were proposed, e.g., frequency shift keying (FSK) method that encodes commands into binary digits with two frequencies [14
], intermodulation frequencies method that uses additional modulation frequencies [15
], and hybrid coding methods that combine other EEG features such as P300 [16
], etc. In particular, joint frequency-phase modulation (JFPM) method has been proved to improve the separability between targets and achieve high-speed SSVEP-BCI systems [18
]. With regard to algorithms, various kinds of identification algorithms were applied in the SSVEPs-based BCIs [19
], e.g., canonical correlation analysis (CCA) [20
] and its various optimizations [21
], multivariate synchronization index (MSI) [23
], maximal-phase-locking value and minimal-distance (MP and MD, respectively) [24
], and task-related component analysis (TRCA) [25
], etc. Thanks to the endeavor in the encoding and decoding methods, the SSVEP-based BCIs have achieved the highest information transfer rate (ITR) among the noninvasive BCI paradigms.
Although many high-speed systems based on SSVEPs have been established in previous studies, it is meaningful to enhance the practicability of the systems for spelling in real life. For example, most of these studies used EEG devices designed for research in the laboratory [9
], such as Neuroscan Synamps2 system. The research-grade devices possess excellent signal amplification performance and diverse functions, whereas most of the functions are superfluous for a practical BCI system and push up the cost. In addition, the 5 × 8 matrix layout was popular in previous SSVEP-based spellers and the users needed to remember the location of each command before the experiment, which increases the workload and slows down the spelling. Furthermore, these systems used fixed stimulating time, i.e., the fixed stopping (FS) strategy. As is known in the P300-based BCIs, the dynamic stopping (DS) strategy enables the self-check of recognition confidence for a BCI system so as to quicken the output when it is confident about the result, whereas keep acquiring data when the correctness of the decision is not sure [26
]. A few studies have tested the performance of DS strategy in an SSVEP-based BCI [28
], but it was evaluated by offline analyzing the offline collected EEG data like an online experiment. The feasibility of real-time DS strategy in a practical online system needs further verification.
The goal of this study was to design a high-speed SSVEP-based BCI system for practical use. Figure 1
is the block diagram of our system. The novelty of our system was reflected from several aspects. Firstly, we simplified the acquisition by developing a dedicated low-cost EEG amplifier with self-designed circuit and optimized the stimulation by arranging the instructions like a keyboard that would be familiar to users. Secondly, in order to extract the SSVEPs more effectively for a high-speed system, the standard forward filtering was applied instead of the frequently-used zero-phase filtering for reliable online noise reduction and the TRCA spatial filter was modified (named mTRCA) to enhance the target recognition. Last, but not the least, the DS strategy based on Bayesian posterior probability was incorporated into the system to obtain flexible stimulating time and improve the ITRs.
As the trial duration of a SSVEP-BCI is much shorter than that of a P300-BCI due to the different coding schemes, three issues were concerned for real-time DS in an online SSVEP-BCI. The first concern is the unfixed stimulating time caused by the immediate stopping of stimulus after satisfying the output condition in DS strategy. As most users have been used to fixed stimulating time in normal FS BCIs, the sudden stopping of stimulating might distract the attention of users and delay the shifting to the next target; thus, leading to the performance decline. Therefore, this study used unfixed stimulating time in the offline calibration experiment to imitate the DS in online operation so that the subjects could adjust to the unfixed timing, and the variation of EEGs could also be covered by the calibration data. The second concern focused on the output condition of DS due to the fact that the probability distribution might change with the stimulus frequency and data length. We raised an adaptive threshold generating method that was easy to implement so as to fit the variation of probability distribution. Another concern is that the real-time DS requires the system to perform the recognition algorithm in very short time. To this end, the programs of some key processes for recognition were ported to C Mex from MATLAB for accelerating execution to ensure the real-time performance.
The high-speed BCI systems based on SSVEPs attracted growing attention and stronger demand of applying this paradigm in daily life in recent years. This study optimized a high-speed SSVEP-based speller towards practical application. The speller was designed according to the layout of a keyboard so that it is convenient for users to find the intended character. The EEG data was acquired using a dedicated amplifier developed in our laboratory instead of the research-grade system in previous high-speed BCI studies. In order to provide flexible stimulating time, we incorporated the DS strategy based on Bayesian posterior probability into the online SSVEP-BCI. The introducing of above measures brought new problems to the BCI system, which need optimization from several perspectives.
The filtering process is our first concern for a high-speed BCI system. Previous studies have barely discussed the details of online filtering in a SSVEP-based system. As is known, the standard digital filtering is a convolution of the input signal with the impulse response of a digital filter. The IIR filters are the digital form of analog filters and welcomed in real-time applications for their low time delay. However, they cannot realize exact linear phase like finite impulse response (FIR) filters; thus, distorting the EEG signal and degrading the BCI performance. The forward and reverse filtering is a commonly used method to achieve the zero-phase filtering [39
]. However, it is a noncausal filtering process that we need the signal prior and posterior to the current time for filtering and take out the useful signal afterwards. This will cause the problem that no signal exists posterior to the current time in a real-time system, leading to the distortion of the signals near the current time owing to the transient response at the beginning of filtering. In the filtfilt() function of MATLAB, the initial condition of the filter and the signal extending method were employed to mitigate the distortion [40
]. Nevertheless, the comparison in Section 3.4
indicates that these methods could not compensate the loss of accuracy caused by the transient response (the Filtfilt(−) in Figure 9
). If we use the posterior signal as the Filtfilt(−)(+) in Figure 9
, the accuracy would be higher whereas the longer data length declined the ITR. Hence, the classical forward filtering method (Filter in Figure 9
) is more suitable for the high-speed operation.
Another important part of work focused on the implementation of DS strategy. Although it has been applied in the P300-based BCI systems and significantly reduced the number of stimulating rounds for the output [34
], the implementation of DS faces new challenges in an online SSVEP-based system. First, as illustrated in the introduction, the unfixed stimulating time in the DS situation results in different feelings for users compared with those in FS situation. Therefore, the offline experiment of this study was designed using three kinds of alternant trial length to emulate the unfixed stimulating time in DS situation. This strategy also increased the training samples corresponding to short data length so that guaranteed the accuracies. Second, the online DS requires the system to process the data as quickly as possible. In other words, the system needs higher “temporal resolution” to provide the real-time performance. The resolution Δt
in this study was set at 20 ms, which meant the data transmission and recognition process should be completed in 20 ms. The self-developed amplifier could send the data packet every 4 ms, i.e., the sample point could be obtained by the online program immediately after being collected, while the Neuroscan system sends the data packet every 40 ms according to our previous testing. Hence, the dedicated amplifier provided flexible temporal resolution to enhance the real-time performance. As for the data processing, it is of great importance to assess the execution time of the program before experiment. It is known that the MATLAB used in many BCI applications is a kind of interpreted language. The code execution efficiency is limited to some degree compared with the compiled language, especially for the looping structure. The recognition of SSVEP-based BCIs contains lots of loops, e.g., the calculation of correlation coefficients contains 40 (frequencies) × 8 (bands) = 320 loops in this study. The three key procedures of recognition consumed nearly 20 ms in total using the MATLAB program (Figure 8
). If considering the other procedures, the run time would exceed Δt
when executed in MATLAB, and the accumulation of the delay might collapse the system. This study reprogrammed the key procedures with C++ and compiled them into MATLAB executable files. The run time was reduced to 4.76 ms on average, which was far less than the MATLAB program. In future works, the calculation could be further accelerated with multiple threads or dedicated processors such as field programmable gate arrays (FPGAs); thus, improving the temporal resolution for higher real-time performance.
Considering that the lab-assembled EEG device and the non-zero-phase filtering might put an impact on the classification accuracy, we modified the TRCA spatial filter by subtracting covariance of other frequencies from the current frequency, which could reduce the interference of irrelevant information. This modification obtained significantly higher ITRs than those of TRCA and S8 achieved a highest ITR of 420.2 bits/min, which is the highest online ITR for SSVEP-based BCI to our knowledge.
Despite the various measures towards a practical SSVEP-BCI conducted in this study, it should be noted that more aspects need to be considered in future work. One of the most important issues is about analyzing the experience of the users from the perspective of psychology. For example, could a patient with physical impairments accept such a system [44
]? Could the users adapt to this system as it works at a very high speed? If the system output several incorrect results, could the users avoid negative emotion and continue focusing on the intended command [46
]? If not, there might be more and more wrong outputs and the human–computer system would collapse. As the AI has caused some ethical problems [47
], the BCI might face similar situation someday. Such problems have not got enough attention to our knowledge. In future developments, it might be better to design some psychological programs for users such as a questionnaire or an interview about their feeling about the system, and to personalized optimize the system configuration so as to make the users work with the BCIs comfortably and efficiently.