The BciAi4SLA Project: Towards a User-Centered BCI

Abstract

The brain–computer interfaces (BCI) are interfaces that put the user in communication with an electronic device based on signals originating from the brain. In this paper, we describe a proof of concept that took place within the context of BciAi4Sla, a multidisciplinary project involving computer scientists, physiologists, biomedical engineers, neurologists, and psychologists with the aim of designing and developing a BCI system following a user-centered approach, involving domain experts and users since initial prototyping steps in a design–test–redesign development cycle. The project intends to develop a software platform able to restore a communication channel in patients who have compromised their communication possibilities due to illness or accidents. The most common case is the patients with amyotrophic lateral sclerosis (ALS). In this paper, we describe the background and the main development steps of the project, also reporting some initial and promising user evaluation results, including real-time performance classification and a proof-of-concept prototype.

1. Introduction

The brain–computer interfaces (BCIs) are interfaces that put the user in communication with an electronic device based on signals originating from the brain. There are two general classes of brain imaging technologies: invasive technologies, in which sensors are implanted directly on or in the brain, and non-invasive technologies, which measure brain activity using external sensors [1]. Non-invasive BCIs are mainly based on electroencephalographic (EEG) signals. In these systems, users are enabled to manipulate their own brain activity to produce signals that will then be used to control computers or communication devices without the aid of motor movements. EEG-based interfaces use an alteration of the brain’s electrical activity as an input signal, which is defined as event-related synchronization or desynchronization (for a review of EEG-based BCI paradigms, see [2]). The change in brain activity may be caused by stimuli triggered by external events (exogenous) or stimuli voluntarily produced by the user, for instance, while imagining something (endogenous) [3]. In addition to voluntary signals, involuntary physiological or passive signals can also be used in BCIs (for a review, see [4]). In this case, brain activity is an expression of the spontaneous cognitive and affective states of the user: spontaneous brain activity represents the (implicit) input from which the system derives its output to assist the user in a given task [5].

Endogenous stimuli can be used by a larger sample of subjects that could benefit from this technology since they do not require to move body parts or perform physical activities. Cognitive tasks are, indeed, the most used, and they consist of the effort produced through a mental task where the user is asked to imagine or do something.

BCIs can offer promising applications for assistance to patients with reduced or absent mobility, as in the case of patients with amyotrophic lateral sclerosis (ALS). EEG-based BCIs use sensor-equipped headsets positioned on the scalp and/or forehead, which are able to monitor the alterations of the brain waves that correspond to certain forms of thought. For this purpose, the users are explicitly invited to perform some mental tasks (i.e., endogenous tasks), such as motor imagery (MI), concentration, relaxation, induction of emotions, etc., thus voluntarily manipulating their brain activity to produce signals which can then be used to control an electronic device. It is still a matter of debate and active research on which mental task should be adopted to maximize accuracy, although motor imagery tasks in BCIs (MI-BCI) are reported among the most promising ones [6]. Patients with ALS may retain full control of their cognitive abilities; therefore, this capacity needs to be fully exploited in the design of a BCI in order to enable these patients to communicate.

In the present paper, we describe a proof of concept that took place within the context of the BciAi4Sla (brain–computer interfaces and artificial intelligence for amyotrophic lateral sclerosis) project (https://bciai4sla.di.unito.it/) (accessed on 23 December 2022), a multidisciplinary project involving computer scientists, physiologists, biomedical engineers, neurologists, and psychologists, with the aim of designing and developing a BCI system following a user-centered approach by involving domain experts and users since initial prototyping steps in a design–test–redesign development cycle.

The project intends to develop a software platform capable of restoring a communication channel in patients with impaired communication possibilities due to illness or accidents. The most common case is the patients with ALS: a neurodegenerative disease of adulthood that causes a rapid and progressive motor disability, up to the complete loss of motility, swallowing, speech and speech articulation, and respiratory function. ALS has a peak incidence of 3.1 cases per 100,000 people/year and a prevalence of about 10 cases out of 100,000 inhabitants in the Piedmont Region [7]. However, it is expected that the number of cases will increase significantly in the coming years due to the aging of the population and the improvement of therapeutic treatments [8]. On the other hand, the state of consciousness is never altered, and although only 10–15% of cases can develop frontotemporal dementia, most patients do not develop significant cognitive deficits. Therefore, in long-surviving patients, the need for tools that allow effective communication is vital.

In ALS, the extraocular muscles are often the last to be affected by paralysis, and they represent the last expressive mean available to the patient to communicate with the outside world (locked-in syndrome, LIS). As the disease progresses, paralysis also affects the extraocular muscles, leaving the patient completely isolated (complete locked-in syndrome, CLIS) even if still able to hear and perceive stimuli from the outside. Although it has proved extremely difficult to re-establish a communicative channel in patients who entered CLIS for a long time [9,10], trials with LIS patients have given, in many cases, comforting results [11]. For patients in LIS or CLIS conditions, it is vital to maintain or restore communication with the external world.

Therefore, research on BCIs that does not require oculomotor skills (required instead in eye-tracking systems used by patients in LIS) is also particularly interesting for experiments on patients with LIS condition before entering the CLIS phase because there is time to develop the necessary training. A study has shown the reliability of P300-based BCIs (P300 event-related potentials are EEG-based signals that can be spontaneously produced or cortical responses to external/internal events and occur 200–700 msec after stimulus) regardless of the severity of the disease and the level of physical decline [12].

The aim of the BciAi4Sla research project is to study and test BCI-based approaches on patients with ALS in LIS and CLIS conditions using brain signals acquired non-invasively with high-end commercial-grade BCI devices, which are cheaper, portable, and safer compared to more invasive solutions. The raw data (brain waves) collected via these devices via EEG will then be analyzed with machine learning and classification algorithms in order to extrapolate in a more precise way the type of thought required by the user. Our goal is to design and develop a user-centered and intelligent software platform (BciAI4Sla) that allows, via BCI devices and the possible use of wearable sensors, direct communication between the brain and the surrounding digital environment (made of smartphones, tablets, robots, and smart objects, etc.) in an easy, usable, ergonomic, and adaptive way to the end user.

In the long run, the goal is to obtain a non-invasive system, able to quickly learn and adapt to the specific user to enhance his/her cognitive abilities, thanks to feedback. Such a system will increase the inclusion of users with ALS and improve their quality of life and safety. According to estimates by Allied Market Research, the global brain–computer interface market size was valued at $1488.00 million in 2020 and is projected to reach $5463.00 million by 2030 (https://www.alliedmarketresearch.com/brain-computer-interfaces-market) (accessed on 23 December 2022), and Healthcare, Communication, and Control are the driving applications. The expected high-level benefits deriving from the diffusion of BCI devices are the facilitation and increase of the potential and ability of users with motor disabilities, the removal of physical and cognitive barriers and related difficulties that characterize individuals with special needs, the consequent increase in social inclusion, and a potential and desirable autonomy.

This paper has been organized as follows: Section 2 overviews the related state of the art; Section 3 introduces the background and the BciAi4Sla project; Section 4 details the study and implementation of cognitive tests selected for the research; Section 5 describes the implementation of the motor imagery task in our study; Section 6 reports preliminary results obtained in healthy subjects; Section 7 introduces a proof of concept prototype for the BciAi4Sla user experience; while Section 8 concludes the paper and present the future work.

2. State-of-the-Art

BCIs allow the interaction between the cerebral activity of a user and an electronic device [13], and they can be based on different neuroscientific techniques (e.g., electrocorticography, functional magnetic resonance imaging, positron emission tomography, etc.). The electroencephalography-based (EEG) method is less invasive and expensive. EEG-based interfaces use an alteration of the brain’s electrical activity that can be caused by exogenous (external events) or endogenous stimuli (voluntarily produced by the user) as an input signal (see the reviews by [2,3]). For the purpose of our research, we will focus on endogenous stimuli that are more suitable for subjects with limited abilities to interact with the outside environment. For instance, persons with CLIS could not perform a visual task since their sight is compromised but instead would use a somatosensory paradigm, such as the vibrotactile or auditory paradigms [14,15,16]. Here, users are required to imagine or do something. However, cognitive tasks are highly influenced by individual differences in responsiveness [17]. It is, therefore, vital to consider some elements while choosing or designing these mental tasks. For example, based on their personality and experiences, users may prefer to perform different tasks than others [18,19]. Additionally, the residual cognitive abilities (in terms of attention, motivation, memory, and so on) possessed by participants should be taken into consideration [14,20]. In addition, several pathologies, e.g., Alzheimer’s disease, impact cognitive functioning, thus limiting patients’ capacity to perform mental tasks [21]. The ability to concentrate and to direct attention toward some activities by repressing other stimuli are some of the capacities that can be highly impaired in pathological conditions [18]. Furthermore, fatigue and frustration coming from the effort to perform the task could affect mood and motivation and have an impact on the final outcomes [18,22]. A way to limit all the aforementioned difficulties could be by adding a training session preparatory to the recording sessions [19]. Patients with serious pathological conditions, such as ALS, may obtain some benefits since they might train themselves to perform the task while they are in a LIS condition. After the worsening of the disease that may move to a CLIS condition where muscular abilities are permanently impaired [23], patients can use the previously acquired competence to perform the cognitive task for the BCI activation.

Concerning example applications, although, initially, the goal of BCI research was to provide assistive technologies for people with severe disabilities, the evolution of technologies and computing power has made it possible to expand BCI applications.

BCI-applied research has focused more on restoring the ability to communicate, control the environment and provide mobility for people with severe physical disabilities. One of their greatest needs is the opportunity to be able to communicate because this improves the quality of their life. Basic communication is, for instance, the ability to say yes/no. Many of the early BCI systems relied on training users to provoke brain rhythms by performing mental tasks. For communication, one of the first BCIs was the “Right Justified Box” of the Wadsworth Center [24], where users learned to modulate the brain signal by imagining motor movements to select yes or no answers. In the one-dimensional case, the cursor moves across the screen at a constant rate of speed. Target areas are discretized regions of the right edge of the screen, each representing a selection alternative, for instance, yes or no. The BCI user either performs or imagines movements, such as finger tapping, which influences the y-position (height) of the cursor on the screen. The trial concludes when the cursor reaches the end of the screen, completing the selection based on the y-position of the cursor. Another important aspect covered by the “Right Justified Box” approach is the one of spelling. A way to perform it is to use a binary spelling using the Right Justified Box to communicate yes/no, progressively dividing the alphabet in half to speed up the selection of the desired letter, such as in the case of odd binary search, as the famous speller devised by Farwell and Donchin [25].

As well as helping in communication, another facility offered by BCIs is to empower users to control devices, such as the television or the thermostat. Cheng et al. [26] describe an SSVEP-based BCI that allows users to employ the discrete selection capabilities of an SSVEP (steady-state visual evoked potentials (steady-state evoked potentials (SSVEPs) is another evoked-response approach for BCIs. The SSVEP response is measured over the visual cortex in response to steadily flashing stimuli [27])) control interface to dial numbers to place a phone call. Adams et al. [28] described the Aware Chair project, which focused on integrating environmental control, such as radio, lights, and television, into a communication device mounted on a wheelchair.

Another important aid to users with motor disabilities is the restoration of their motor functions. BCI research studied different ways to control accessories that can assist the person with motor difficulties to help him/her in his/her movements. It has demonstrated the possibility of being able to control movements by obtaining EEG signals. To date, wheelchairs controlled with brain signals have already been designed and implemented [29], and smart home prototypes controlled by BCI systems have been realized and evaluated [27].

Robots can also be controlled through brain signals (see, for instance, [30]). Currently, applications for controlling robots are particularly centered on assistive technology, but research for military and industrial applications is also underway. Several experiments were performed, one of them was to combine the movement of the robot with a P300-based BCI, and the robot was configured to perform the various steps to make coffee, as reported in [27]. Furthermore, telepresence robots have been combined with BCIs to enable impaired users to interact through a telepresence system in remote places being able to control a robotic alter ego [31].

Other interesting applications are related to the possibility of being able to recover motor control through the control of motor robots, especially for people whose paralysis is very advanced. Pfurtscheller and colleagues [32] found that it is possible to regulate sensorimotor rhythms by imagining a movement, to control functional electrical stimulation (FES) of the muscles and, thus, allow the limbs to move. Indeed, a subject paralyzed from spinal cord injury learned to regulate sensorimotor rhythms to control functional electrical stimulation (FES) of arm and hand muscles to perform simple tasks, such as grasping a glass. Birbaumer and Cohen [21] instead devised a system based on magnetoencephalography (MEG) that allowed a user to imagine the movement of a hand, increasing or decreasing sensorimotor rhythm amplitudes. Depending on the amplitude of the mu rhythm (the Mu (μ) rhythm is a type of emitted brain wave that can be measured via electroencephalography (EEG). The mu rhythm frequency band is defined by activity falling between 8 and 13 Hz and recorded by scalp electrodes over the sensorimotor cortex during waking neural activity. The mu rhythm band is posited to reflect the conductance of synchronized activity in large groupings of pyramidal neurons in the brain’s motor cortex but has also been proposed to reflect the activity of the mirror neuron system [27]), patients can open or close the prosthesis. As the amplitude of the mu rhythm increases, the hand opens, while to close it, they must decrease the rhythm.

In the past, there have been relevant projects facing the use of BCIs for ALS. Smart hoMes for All (SM4All) [33,34] was an international scientific research project funded by the European Community, which ended in 2011. The SM4All project developed an innovative middleware platform for the interworking of smart embedded services in immersive and user-centered environments. In one of the project scenarios, a BCI was connected with a virtual reality system to control a smart home application [35]. Special control masks were developed, which allowed using the P300 component of the EEG as an input signal for the BCI system. Control commands for switching TV channels, opening and closing doors and windows, navigation, and conversation were realized. Experiments with 12 healthy subjects were made to investigate the speed and accuracy that could be achieved if several hundred commands were used to control the smart home environment. The study demonstrated that such BCI systems could be used for smart home control [36].

The European ICT Programme Project TOBI [37] developed practical technologies for brain–computer interaction, namely non-invasive BCI prototypes combined with other assistive technologies (AT) for improving the quality of life of disabled people. The goal of the project was the widespread use of BCI-assistive technology endowed with adaptive capabilities that augment those other ATs they are combined with. In such a hybrid approach, users can fuse brain interaction and muscle-based interaction or can switch between different channels naturally (based on monitoring of physiological parameters or mental states). There have been four application areas in the project: communication and control; motor substitution; entertainment; and motor recovery. Scientific results using the multimodal fusion approach yielded a more accurate and stable control compared to a single signal use [38]. It has shown, for instance, a telepresence robot controlled by applying the proposed multimodal fusion approach [31]. Two patients with no previous experience achieved levels of performance similar to those of a group of healthy users who were familiar with the task.

Finally, we mention the ongoing project called BRAINTEASER [39,40], a data science project that seeks to exploit the value of big data, including those related to health, lifestyle habits, and environment, to support patients with amyotrophic lateral sclerosis and multiple sclerosis, and their clinicians. Taking advantage of sensors and apps, BRAINTEASER will integrate large clinical datasets that host both patient-generated and environmental data. The goals of the project are to integrate societal, environmental, and human health data to develop patient stratification and disease progression models for amyotrophic lateral sclerosis and multiple sclerosis able to address the needs of personalized medicine. The system developed in the project will try to provide quantitative evidence of the benefits and effectiveness of artificial intelligence (AI) tools in healthcare pathways to present a proof-of-concept of their use in real clinical settings.

3. Background and Research Project

The condition of ALS patients in the Italian Piedmont area, wherein the project is being carried out, is specifically controlled by CRESLA (https://neuroen.campusnet.unito.it/do/gruppi.pl/Show?_id=1zhm) (accessed on 23 December 2022), the Regional Expert Center for ALS (DGR 30 December 2009, n.27-12969). The research activities of CRESLA are aimed at studying ALS on the epidemiological, clinical, cognitive, genetic, neuropathological, and neurobiological slopes. From the neuroepidemiological point of view, the Piemonte and Valle d’Aosta Register for ALS belongs to CRESLA, recognized as a Regional Register of Relevant Health Interest (Piedmont Region BU 19/04/2012, Regional Law 11 April 2012, n. 4). The Registry has been operational since 1995 and is the most extensive record for worldwide ALS.

As specified above, ALS has a peak incidence of 3.1 cases per 100,000 inhab/year and a prevalence of about 10 cases out of 100,000 in the Piedmont Region [41]. The number of these cases is not insignificant and is destined to increase over time [8]; therefore, projects like this have a fundamental current and future impact on clinical management and the daily life of patients suffering from ALS and other pathologies that lead to LIS or CLIS conditions.

The BciAi4Sla project has been divided into four phases for a total duration of 21 months:

• PHASE 1: STUDY AND IMPLEMENTATION OF COGNITIVE TESTS BASED ON BCI. The main goals of this initial phase were to study, design, and implement a set of cognitive tests aimed at evaluating the best mental activities detectable through BCIs (Section 4);
• PHASE 2: TESTING. The main goals of the second phase were to test the above-selected and implemented tests, first, with neuro-typical users and then with patients suffering from ALS in LIS condition with the support of the University Hospital “Città della Salute e della Scienza” of Turin (https://www.cittadellasalute.to.it/, accessed on 23 December 2022) and of the CRESLA, partners of the project (Section 6);
• PHASE 3: ANALYSIS OF COLLECTED DATA. Raw data (brain waves) collected through the BCI headsets have been first analyzed via filtering and then via classification algorithms to extrapolate the input data as accurately as possible. The processed and classified signals have been then correlated with the right requests in order to evaluate the percentage of correctly classified data (accuracy) (Section 5 and Section 6);
• PHASE 4: BCI PLATFORM DEVELOPMENT. Following the results of data analysis, we are developing, from a user-centered perspective, a software platform (BciAi4Sla) that will allow, through the BCI devices, online interaction between the subject with ALS and a dedicated communication system (Section 7).

The details of these four project phases will be described in the following Sections.

4. Study and Implementation of Cognitive Tests

4.1. Study of Cognitive Tests Based on BCI

First, we have extensively reviewed the literature on cognitive endogenous tasks used in previous research in order to present a list of revised tasks suitable for empirical investigation of this domain. The goal of this study, more extensively described in [42], was to select the best detectable mental activities through BCIs in order to be able to use them with patients in our BCI platform.

There are six main mental tasks that have been extensively used in the BCI literature: motor imagery; spatial navigation imagery; auditory imagery; familiar face imagery; geometric figure rotation; and math imagery.

Motor imagery task

During this task, users imagine repeatedly moving a body part (e.g., a foot, arm, etc.) in order to elicit cerebral effort. Imaginative use of an object may also be included, such as imagining squeezing a ball. The cerebral localization of this task [43], which is well-defined in the scientific literature (primary motor cortex, supplementary motor cortex, and premotor cortex), has made this task one of the most reliable in terms of producing a detectable signal [44,45,46,47,48].

Spatial navigation imagery task

Users performing this task must imagine a place familiar to them. Specifically, they have to focus on details of the surroundings or orient themselves in the place [44,45,49,50,51]. Users should avoid focusing on movements since an overlapping with motor brain areas will be present. As a matter of fact, this task activates several brain areas, i.e., the dorsal frontoparietal regions, pre-supplementary motor area, anterior insula, and frontal operculum [51].

Auditory imagery task

In this task, users are required to imagine a song as if they were singing it [44,49,52]. However, they should avoid moving their mouths in order to exclude brain area activations that are not strictly connected to the imagination of the task. This task activated the auditory cortex (i.e., the superior temporal gyrus in the temporal lobe) [53,54].

Familiar face imagery task

The imagination of a familiar face has also been reported as a used task in the BCI literature. The familiar faces can be identified as relatives or friends, but also famous persons, i.e., celebrities [17,55,56]. The neural activations related to this task, however, are not very defined as they depend on the type of stimuli, namely, whether the user imagines the face of a friend or a relative (fusiform gyrus, parahippocampal gyrus, middle superior temporal gyri, middle frontal gyrus) [57].

Geometric figure rotation task

In this task, users should manipulate a figure (e.g., a cube) or an object and imagine it repeatedly rotating on an axe [58,59,60,61]. This effort is usually located in the posterior parietal cortex and posterior-occipital cortex [62].

Math imagery task

This task requires users to perform some math operations in the form of additions or subtractions or, alternatively, to repeatedly imagine a number written on a blackboard, to mentally erase this number and write the subsequent one [59,61,63,64]. Additionally, in this task, users should avoid any movement in order to not activate other brain areas except those required in the task (frontal and parietal areas) [65].

In the present research project, the tasks collected through the review of the literature have been modified in order to optimize their capacity to produce a defined cerebral change and to avoid difficulties connected to the users’ performance, such as task-related stress. Specifically, all tasks are preceded by training sessions that help the user exercise and control their cerebral effort. Although there is no specific recommendation on the duration of training in the BCI literature [66], we believe, based on the preliminary trials with users (described in Section 6), that at least three sessions of training should be sufficient for users to feel confident in cognitively executing the task.

4.1.1. Motor Imagery Task

The user must imagine a hand movement without producing any actual muscular movements. In the training session, the user is seated and sees two hands from a self-centered perspective, on a computer screen, with palms down. The user sees an indication (left or right arrow) on the screen and must imagine the respective hand movement; namely, the right hand will move to the right and the opposite to the left. The indication of the direction (arrow) with the image of the hand will last 1 s, and then, the user must imagine the movement of his/her own hand for 3 s. To facilitate the imagination process, the participant had previously been presented with an example of the movement on the screen for 6 s. Each trial of the training lasts 10 s, and the complete training includes 8 runs with 18 trials each.

4.1.2. Spatial Navigation Imagery Task

The spatial navigation task is divided into two subtasks: navigation with an egocentric perspective and with allocentric perspective [67]. The egocentric perspective (route) is the participant’s point of view, and he/she has to move to the left or the right of the environment. Users have to imagine themselves while moving within a familiar space, e.g., home, from an egocentric view. In the training session, users look at some videos on a computer screen. Videos show a still image of an environment for a few seconds, and at this time, participants have to imagine moving to the left or the right of the room according to some architectural affordances (e.g., the image shows a room with only one door to the right). After that, the video explicitly shows the movement (e.g., in this case, entering through the door to the right).

The allocentric condition refers to the view from above, such as when looking at a map. Users have to imagine a cursor moving on a map where only a path is viable. Training consist of the visualization of some videos showing a map with a cursor moving. The user is asked to imagine the movement of the cursor. The training of both sub-tasks comprises 8 runs with 10 trials each. Each trial is characterized by 6 s of movement imagination and 3 s of movement visualization. Therefore, each run lasts 90 s. Turns in paths (left and right) are balanced to avoid habituation or biases.

4.1.3. Music Tune Imagery Task

Since this task can be subject to personal experiences, a list of very famous songs based on the cultural context (e.g., in Italy, we propose “Volare” by Domenico Modugno or “Azzurro” by Adriano Celentano) is presented to the user, among which he/her can choose the preferred one. In the training session, the user can familiarize themselves with the chosen song by hearing it three times (with the lyrics). Then, the user has to imagine the tune without any muscle movements or verbalizations.

4.1.4. Face Imagery Task

Participants have to imagine the face of a celebrity with particular attention to the eyes, mouth, nose, etc. The celebrity is selected from a list of famous people presented to the subject or chosen by a relative if the user is unable to communicate. We have selected six (three females and three males) national and international persons (such as Roberto Benigni, Lady Diana, etc.). In the training session, the user visualizes images on a computer screen for 10 s each and then has to recall the face of the celebrity during the recording session. The task is repeated six times. It would be helpful to propose a list of celebrities in line with the age of the participant since young celebrities may not be familiar to older persons and vice-versa.

4.1.5. Object Rotation Imagery Task

In this task, users have to imagine a familiar object rotating. Unlike similar tasks retrievable in the literature, we suggest using a real object in the training session, i.e., an hourglass, since we believe that movement of the sand can help the user to better mentally visualize the rotation. The user is seated in front of a computer screen and sees a clockwise or counter clockwise rotation of the hourglass. The training session comprises a series of images with an indication of the subsequent direction of the rotation is present. The user has to imagine the movement in 3 s, and the rotation takes place during a time span of 6 s. The training comprises 8 runs with 18 trials (rotations) each.

4.1.6. Math Counting Task

Here users can imagine a number and then subtract or add a specific number as many times as requested for the recording process. The task is preceded by training where the user visualizes very simple math operations while sitting in front of a computer screen: starting from a specific number presented on the screen, the user has to add or subtract repeatedly a suggested unit (such as 3), e.g., 9 + 3 = 12 + 3 = 15, etc. The type of operation (subtraction or addition) and suggested unit to subtract or add can change and are indicated before the beginning of the training session. After carrying out the operation mentally in 4 s, the user sees the result on the screen. The training session is made of 6 runs, with 20 trials (calculations) each.

5. Processing and Implementations

From the review of the literature on cognitive tasks described in Section 4, a high individual discrepancy with respect to responsiveness to tasks emerged. Among the six tasks identified above, the motor imagery task, one of the most consolidated, was chosen as the basis for the first implementation.

The first release of the BCIai4SLA platform designed for the project was based on the analysis of non-invasive EEG signals obtained from two low-cost headsets, Emotiv Epoc+ (https://www.emotiv.com/epoc/) (accessed on 23 December 2022) and OpenBCI Ultracortex Mark IV (https://docs.OpenBCI.com/AddOns/Headwear/MarkIV/) (accessed on 23 December 2022).

The former is a 14-Channel Wireless EEG Headset capable of 128 or 256 Hz sample frequency for each channel, using saline-based wet EEG sensors. It is well known in research [68] for its accuracy and easy-to-use design that permits positioning the electrodes in a few minutes to be ready to start EEG acquisition. It uses Bluetooth communication to transmit recorded data to a personal computer, and EEG raw data are accessible under a paid license. According to the review by Balart-Sánchez et al. [68], Emotiv Epoc+ can acquire EEG data well enough for research studies in comparison with professional EEGs for medical use, even if, due to inherent properties of the Emotiv system, a consistent delay in the transmission of markers has been reported while using the Emotiv Epoc+, as also reported in other studies [69].

The latter is a 16-Channel Wireless EEG Headset capable of 125 or 250 Hz sample frequency for each channel, using dry EEG electrodes, equipped with Cyton/Daisy Biosensing Boards communicating wirelessly with a computer via a USB dongle using RFDuino radio modules. The Ultracortex Mark IV EEG headset is a scientifically-validated research tool [70,71], and it is appreciated for its open-source, modular design in software and hardware that lets users buy or 3D-print useful parts. One of the main problems of OpenBCI lies in its ergonomics, as also reported in [71], which makes it impractical for prolonged use.

Independently of the hardware used, the raw EEG signals were used to train a classification machine learning algorithm, namely support vector machine (SVM), as will be described below. SVM is one of the most used machine-learning methods for distinguishing brain patterns generated from motor imagery (see, for instance, [72,73,74]). Specifically, we aimed to search for sensorimotor rhythms (SMRs). SMRs are generated when the subject imagines and tries to perform a muscle movement voluntarily. The hand, however, generates a stronger signal than other movements detectable in the so-called cortical homunculus, namely, regions along the primary sensorimotor cortex corresponding to different body parts (see [75] for details).

To expand and improve each piece in the future, the analysis protocol has been made in distinct blocks. The procedure was initially developed in Matlab and, subsequently, in Python 3 (with the help of Numpy v. 1.16/Scipy v. 0.9 libraries) for real-time acquisition for both the Emotiv and OpenBCI helmets.

To summarize the workflow, whose schematic representation of each fundamental block is shown in Figure 1 and Figure 2, the acquired brain wave signals were first suitably filtered, then modified with the common spatial pattern (CSP) algorithm, which in the literature is considered very effective in discriminating different brain states [76,77,78,79], maximizing the variance between the imagined movement signal of the left hand, compared to that of the right hand. We created the whitening matrix W by feeding as input to the CSP algorithm a matrix representing the first-class trials (left-hand imagined movement) and the second-class trials (right-hand imagined movement).

From this W matrix, we took the first and last four columns representing the eigenvalues of the spatial filter, simply called filters. With these filters, we transformed each trial.

At last, we extracted the features related to the trials (which will be used by the SVM) by taking for each of the eight rows of the matrix, their logarithm of the variance, and obtaining eight feature numbers. Then, with the feature extraction algorithm, a few numbers were extracted that could be exhaustive to describe the imagined motion signal, and a support vector machine (SVM) was trained to find the differences between the two signals in real time.

In the following Subsections, an exhaustive description of the general workflow for data acquisition and training for the Emotiv Epoc+ and OpenBCI Mark IV headsets will be provided.

5.1. Connection to Emotiv Epoc+

In the following, we describe the sequence of steps to obtain the CSV files containing the row data acquired with the subjects wearing the Emotiv Epoc+ headset:

• Creation of the connection with the WebSocket protocol;
• Authorization of communication by passing information, such as license, client ID, and client secret (all obtainable from the purchased license);
• Placing the headset on the subject;
• Creation of the recording session;
• Start recording with the insertion of markers at the beginning of each task;
• End registration with the disconnection of the headset and request to send the CSV file with the raw data.

5.2. Connection to OpenBCI Mark IV

In the following, we list the sequence of steps to obtain the CSV files containing the row data acquired with the subjects wearing the OpenBCI Mark IV headset:

• Creation of the connection with the headset via USB dongle;
• Acquisition of the information of the headset in use (such as headset id, serial port, IP address, and IP port);
• Placing the headset on the subject;
• Creation of the recording session;
• Start recording with the insertion of markers at the beginning of each trial;
• End registration; the CSV file is automatically parsed and saved.

5.3. General Procedure for Data Acquisition, Processing, and Training

In this section, we will describe the general workflow for data acquisition and training. This procedure works with both the Emotiv Epoc + and with OpenBCI Mark IV headsets, as schematized in Figure 1 and Figure 2.

5.3.1. Experimental Protocol

The subject sits in front of a monitor with the arms resting on the legs.

The implemented motor imagery task is a finger touch with the thumb of the same hand, which the user starts and stops according to a visual cue (see Figure 1 and Figure 2). A single trial lasts 7 s plus a random time between 0.1 and 0.8 s: after 2 s of rest, the visual cue, consisting of a drawing of the left or of the right hand, is displayed for 3 s. During this time, the subject is invited to imagine the finger movement of the corresponding hand. The right/left sequence is randomized, with the constraint of an equal number of left and right tests in the session. This continues until the preset task number is reached. The monitor turns black during resting intervals while always presenting a central cross at the center of the screen, which acts as a reference for the subject to look at. The slideshow was implemented with the Psychopy v.3 (https://www.psychopy.org/) (accessed on 23 December 2022) library.

5.3.2. Data Filtering

The filtering of the data acquired by the headset is a fundamental part of the proper functioning of the BCI system. Scientific research [75] has shown that sensory-motor rhythms are usually detected in a quite broad frequency bandwidth: mainly in the range of mu-rhythm (8–12 Hz), but often mixed with beta components (around 20 Hz). Since in the real-time application the filter was applied to relatively small pieces of data, a low filter order was fundamental. For this reason, an 18th-order IIR Chebyschev band-pass filter was used, with a frequency range of 8–30 Hz [80]. To avoid phase distortions, a zero-phase filtering approach was used.

5.3.3. Extraction of Central Sequences of Imagined Movement

For each trial lasting 7 s (2-s rest, 3-s motor imagery, 2-s rest), the central 3 s, in which the subject performed the motor imagery task, were extracted, discarding the first 0.5 s after the cue. Thus, an epoch of 2.5 s was selected for the analysis.

5.3.4. Signal Processing

The CSP (common spatial pattern) algorithm has been used to extract the features of the signal coming from an imagined movement with the left hand compared to one with the right hand. Given two input signal S_i with i ∈ 1, 2 representing two distinct classes, each S_i is an NxT matrix with N EEG channels and T samples per channel. The CSP algorithm generates a transformation matrix W that allows to transform the original matrix of signals of imagined movement into a new one that contains signals whose variances are optimal for discriminating the two classes (for more details, see [76,77,78,79]). The construction of the transformation matrix requires a device calibration phase (see Figure 1). At least 10 sequences of left-imagined movement and 10 sequences of right-imagined movement are then used to form this matrix. Once the matrix has been created, some initial and final columns are selected (called filters), which will be used for the transformation of movement trials not yet classified. Currently, the best results are obtained by selecting the first and the last 4–5 columns; thus, several filters range between 8 and 10, as will be described in Section 6.

5.3.5. Feature Extraction

Once the signal has been transformed, features are extracted to train the supervised classifier, see Figure 2. The logarithm of the sum of variances for each filter was chosen as the extracting function. In total, therefore, for each trial, 8–10 numbers are obtained, summarizing the relevant characteristics of the right or left movement.

5.3.6. Classification

A support vector machine (SVM), a supervised learning method used for classification, regression, and outliers’ detection, was chosen as the classification algorithm. For the training, the features extracted from the trials used for the formation of the W matrix are used. The SVM, trained with the above-described calibration sequences (see Figure 1 and Figure 2), allows to classify any trial in real time with an estimated accuracy, namely, the probability (confidence) of how certain the prediction is, as will be detailed below.

In the following, we will explain the steps for training the classifier:

• The training dataset is parsed, and a CSV file with the union of all the training data is generated as output;
• This CSV file is processed by the signal processing module. The data are cleaned with the various filters described above, and features are extracted from them. Tasks are also divided into imaged-left and imagined-right movements;
• The classifier is trained by the SVM algorithm. To tune the hyperparameters of the training set, we used the Python Optunity library (https://pypi.org/project/Optunity/) (accessed on 23 December 2022), which, having a given dataset as input, finds the best possible range of C and gamma parameters of the Radial Basis Function (RBF) kernel SVM (Intuitively, the gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’. The gamma parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors. The C parameter trades off the correct classification of training examples against the maximization of the decision function’s margin. For larger values of C, a smaller margin will be accepted if the decision function is better at classifying all training points correctly. A lower C will encourage a larger margin, therefore encouraging a simpler decision function at the cost of training accuracy. In other words, C behaves as a regularization parameter in the SVM. Source: https://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html) (accessed on 23 December 2022).
• The result of steps 1 to 3 are two persistent Python objects: the classifier; and the W matrix, created by the CSP algorithm;
• Finally, the trained classifier is loaded, and the accuracy test is performed on the test dataset, parsed and filtered with the same filters used for the training set, with the difference that the previously-created W matrix is used.

6. Processing Settings and Assessment of User’s Performance

All the acquisition sessions described below were held as follows:

• A brief explanation of the test objective;
• An explanation with the demonstration of the movement to be made (both real and then imagined);
• A few minutes of free practice of the movement (both real and then imagined);
  at this point, the subject started the protocol described above;
• The protocol starts with a slideshow of pre-established images, as described above.

We performed several acquisitions for the following:

• Tuning of training set/test set;
• Tuning of weights and filters;
• Real-time classification.

All the subjects involved in the experiments described below gave their informed consent for inclusion before they participated in this study. This study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of the University of Turin (Prot. N. 256076).

In the following, we will report the best results obtained in the acquisition of points 1, 2, and 3, using both OpenBCI Mark IV and Emotiv Epoch+. The pre-processing and processing algorithms are platform-independent since they deal with the EEG signals, regardless of which device performed the acquisition; the only thing changing is the interface to the two devices (connection, protocols, the definition of electrodes, etc.). Thus, we reported the OpenBCI Mark IV’s results for tuning of training set/test set, the Emotiv Epoch+’s results for tuning weights and filters, and the real-time classification performed with OpenBCI Mark IV. Regarding the last evaluation, we preferred to initially concentrate on this headset since the literature reported the best accuracy results compared to the ones of Emotiv Epoch+ [71], and delays in the transmission of markers have been reported while using the last one, especially in real-time classifications [69].

6.1. Training Set/Test Set

Design. All the acquisitions were made following 1 to 4 steps described at the beginning of Section 6. On two different days, we recorded 5 sessions of 60 tasks (30 left and 30 right), for a total of 600 tasks per subject. On the first day, we performed two sessions per subject, while on the second day, we performed three sessions per subject. Each session was recorded by pausing and always removing the headset.

Participants. We involved three subjects—two males and one female. They all were young neuroscience researchers aged between 24 and 30, with 18 years of education.

Apparatus. OpenBCI Mark IV

Procedure. In order to discover the better ratio between the training size and the test size, we calculated the accuracy of classification for the following percentage of training/test set, respectively: 80–20%; 60–40%; 40–60%; and 20–80%.

Tests for scores were performed on 5000 divisions of the dataset, namely 5000 random assignments of single trials to the training and test sets. The function dividing the dataset kept the number of examples in the test set in the balance between the classes (the same number of examples for the right and for the left). The SVM hyperparameters were tuned and optimized on the training dataset, and accuracy values were calculated to assess the performance of the test set. Average accuracy, calculated over the 5000 simulations, for each subject and each condition is reported in

Table 1. Accuracy of training/test set trials.

Subjects	80–20%	60–40%	40–60%	20–80%
subject 1	0.812	0.729	0.699	0.633
subject 2	0.645	0.614	0.573	0.565
subject 3	0.729	0.666	0.583	0.552
average	0.73	0.67	0.62	0.58

.

Results. After having analyzed the collected data offline, we obtained the results shown in Table 1:

The best results were obtained with 80% of training and 20% of the test set, see

Table 2. Results of training/test set 80–20% with 2000 divisions of the dataset.

Subjects	Accuracy
subject 1	0.83
subject 2	0.642
subject 3	0.72
average	0.73

. Keeping this ratio, we also performed the same trials with 2000 divisions of the dataset, and we obtained comparable results.

6.2. Tuning of Weights and Filters

Design. All the acquisitions were made following steps 1 to 4, described at the beginning of Section 6. Each session consisted of two acquisitions of 60 tasks each (30 left and 30 right), with a pause of a few minutes between the two acquisitions. Every subject performed a total of 120 tasks.

Participants. We involved five subjects—four males and one female. They all were neuroscience/computer science students aged between 22 and 25.

Apparatus. Emotiv Epoc+.

Procedure. The data were previously filtered and processed as described in Section 5. Repeating steps 1 to 5, as described in Section 5.3.6, with a training/test set ration of 80–20%, with 2000 divisions of the dataset, several classifiers (with respective W matrix) were created by changing some parameters, both in training and in the signal processing phase. The parameters were:

• The classifier’s weights, which were uniform weight, as if it had no weight, and logarithmic weight calculated as the Napierian logarithm of each input value;
• The number of filters, namely, the column’s W matrix, as described in Section 5.3.4, when the signal processing approach with CSP is described, were, respectively, 6, 8, 9, 10, 11 and 12.

Once all the tests had been completed, the classifier that obtained the best result was kept, and the average accuracy of all the tests performed was obtained.

Results. We analyzed the collected data offline. Regarding the comparison between uniform and logarithmic weights, the last ones obtained lower accuracy, as shown in

Table 3. Comparison between average accuracies of classifier’s weights.

Subjects	Logarithmic Weight	Uniform Weight
subject 1	0.738	0.812
subject 2	0.371	0.454
subject 3	0.529	0.563
subject 4	0.408	0.450
subject 5	0.796	0.892
average	0.569	0.634

.

The average accuracy values obtained by the classifier with logarithmic weight are statistically significantly lower than the one obtained using uniform weights (av. 0.569 vs. 0.634), as witnessed by a paired t-test, p = 0.002.

Regarding the number of filters, we considered the data obtained by the classifier with no weight. As described above, the CSP algorithm for signal processing performs a transformation W matrix that allows to transform the matrix of signals of imagined movement. Once the matrix has been created, some initial and final columns are selected (called filters), which will be used for the transformation of movement trials not yet classified. Currently, the best results are obtained by a number of filters around 8–10, as shown in

Table 4. Number of W matrix filters and obtained average accuracies.

Number of Filters	Accuracy
6	0.609
8	0.665
9	0.665
10	0.626
11	0.624
12	0.599

.

6.3. Real-Time Evaluations

6.3.1. Real-Time Evaluation with OpenBCI Mark IV

Design. All the acquisitions were made following steps 1 to 4, described at the beginning of Section 6. We recorded 1 sessions of 60 tasks (30 left and 30 right) for training the classifier, and then each subject performed a real-time trial (30 left and 30 right). Between the training and a real-time session, each subject took a break of 5–10 min and in the meantime, we trained the classifier. Results of real-time classification were given at the end of the session, in order to not influence the subject’s performance.

Participants. We involved eight subjects—four males and four female. Six of them were young students/researchers in neuroscience, psicology, and computer science aged between 24 and 30. Two of them (subject five and eight in

Table 5. Real-time classification of motor imagery tasks with OpenBCI Mark IV.

Subjects	Accuracy
subject 1	0.78
subject 2	0.88
subject 3	0.83
subject 4	0.85
subject 5	0.62
subject 6	0.73
subject 7	0.74
subject 8	0.6
average	0.75375

) were, respectively, a male neuroscience professor, aged 53, and a female computer science professor, aged 48.

6.3.2. Apparatus OpenBCI Mark IV

Procedure. The training data were previously filtered and processed, as described in Section 5. Repeating steps 1 to 5, as described in Section 5.3.6, with a training/test set ration of 80–20%, with 2000 divisions of the dataset. As the classifier’s parameters, we used uniform weight and a number of filters ranging from 8 to 10. Performance evaluation was performed through the K-Fold cross-validation, and, given the reduced number of available training samples, the leave-one-out approach was used. In particular, we used the Stratified Shuffle Split cross-validator, which is a merge of Stratified KFold and Shuffle Split, returning stratified randomized folds (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html#:~:text=Provides%20train%2Ftest) (accessed on 23 December 2022). The folds are made by preserving the percentage of samples for each class. In particular, the Stratified Shuffle Split cross-validator divides the training set n times, each time, by choosing a different test set but always having the same number of left- and right-hand tasks. By repeating training and test sets with different data, it is possible to obtain the better-performing classifier in terms of accuracy. For each subject, the process is then repeated 2000 times, with a training size of 80% and a test size of 20%, respectively. The classifier obtaining the best result in term of accuracy is kept. The real-time accuracy is then calculated using the best classifier found.

Results. We obtained the real-time results shown in Table 5.

If we exclude the two older subjects (subject five and subject eight), the average accuracy increases to 0.8.

6.4. Discussion

After several acquisitions, both with Emotiv and OpenBCI headsets, we noticed that, in line with the current literature [17], the performances have great variations among the participants, probably due to individual differences or variability in the individual person’s capacity to concentrate and engage in the imagery task. Several studies, indeed, show that different individual psychological, i.e., attention, concentration, motivation, and visuo-motor coordination [18,81] and personality factors [82], influence BCI users’ performance. In line with the relevant literature, also in our study, these factors influenced the quality of the collected data. Our future studies should develop specific and personalized BCI training protocols adapted to the profile of each user [82,83,84] in order to minimize these differences and increase the quality of the data. In particular, Jeunet et al. [82] point out how the personality psychological users’ profiles influence MI-BCI control ability. However, the authors suggest that a possible solution is to design novel MI-BCI training protocols adapted to the profile of each user. Collectively, these results indicate that mental state is closely related to BCI performance, encouraging future development of psychologically adaptive BCIs [83].

We here presented methodological and experimental results reached so far. They represent a proof of concept testifying that the devised procedure, along with the developed algorithm, constitutes a plausible BCI and can yield satisfactory performance even in a real-time configuration. Our results seem to be in line with the ones reported in the literature in similar contexts and approaches, although with the chance for improvement. The correctness of our approach is witnessed by the fact that about half of the 89 studies reported in a 2022 review paper on EEG in motor imagery BCI [73] used the CSP in combination with classification based on an SVM [117], suggesting that classic combination of CSP and SVM is effective. Their statistically-based analysis highlights that the most performing feature selection approach achieved at least 85% and 83% accuracy in classifying EEG signals associated with MI, respectively, in the binary and multi-class cases.

Many other researchers have examined the applicability of machine learning methods for feature selection in MI-BCI; however, the results obtained do not support the existence of an algorithm that is clearly more efficient than the others, with high variance in accuracy results. For instance, using a spatial filter with common spatial patterns (CSP) with OpenBCI 8 channels at a 250 Hz sample rate, Haji [85] proposed a comparison of different machine learning methods for MI classification, obtaining the best accuracy results with Linear Discriminant Analysis (LDA) (~80%), followed by logistic regression (~78%), Random Forest (RF ~73%), and SVM (~72%).

Behri et al. [86] used an EEG database from five volunteers to try to classify the imagery movements of the right hand and right foot. For this, Decision Trees (DT), multilayer perceptron (MLP), SVM, k-nearest neighbors (kNN), Naïve Bayes (NB), and RF algorithms were used after noise reduction, feature extraction, and dimension reduction. In terms of the classification accuracy achieved, the 53% result of NB proved to be the worst. The DT (64%), MLP (67%), RF (78%), and SVM (89%) methods performed significantly better, but the best result, almost 95% accuracy in the average of the five volunteers, was provided by the kNN algorithm. By applying a small window size (1-s window size) and using a purely convolutional neural network (CNN) on the MI-BCI PhysioNet dataset (https://physionet.org/, accessed on 23 December 2022. The MI-BCI database was recorded using the international BCI2000 instrumentation system. It contains 64 channels of scalp EEG recordings sampled at 160 Hz. The total database includes 109 subjects who performed 14 motor imagery trials. Electrodes were positioned as per the 10–10 international system, a standard of the American clinical neurophysiology society and the international federation of clinical neurophysiology), [72] achieved 97.7% recognition accuracy. Assi et al. [87] proposed a right-hand MI_BCI, wherein, using an LDA vs. SVM, they were able to improve classification accuracy results from 66% to 88.10% after removing ocular artifacts from the PhysioNet dataset.

In an approach more similar to the one reported in this paper, Costantini et al. [88] proposed an MI-BCI using 61 electrodes at a 256 Hz sample rate, based on SVM Classification of EEG signals, which obtained very poor results (52.2% of average accuracy) in discriminating between the movement of the right hand and left hand, while they obtained better results in case of discriminating between the thought of a carol and a mathematical operation (63.4%), or mathematical operation or carol and hand movements (68% and 73.8%, respectively). Stock and Balbinot [89] used Emotiv Epoc+ with CSP and an NB classifier. They tested the approach in a MI experiment with two users and analyzed the collected raw data offline. The results obtained reached a maximum classification rate of 85%.

Regarding the comparison between Emotiv and OpenBCI, Aldridge et al. [71] used the Ultracortex Mark IV for the P300 Speller classification task with nine experimental participants, with the accuracy being measured along with the time required to set up the Mark IV, the participants’ comfort, and the participants’ perceived ease of setup. Their results were then compared to previous evaluations of other EEG headsets. The OpenBCI Ultracortex Mark IV’s classification accuracy outperformed the one of Emotiv Epoch (82.9% vs. 61.7%), which was better evaluated for comfort and ease of setup.

Real-time performance classification studies, also known as online classification studies, are rarely available. The Lotte et al. [74] review of studies evaluating BCI signal-classification algorithms found the most used offline analyses, and over the year, the situation does not seem to have changed so much, even in MI-BCI studies. Among the few, Irimia et al. [90], in a real-time evaluation of an MI-BCI with five stroke patients wearing a 64-channel EEG cap, obtained a grand average accuracy of 87.4% and a mean maximum accuracy of 96.9%, using a CSP filter and LDA for classification. Mondini et al. [91] developed a BCI system to control the flexion–extension of a DOF-modelled arm using an MI strategy. They implemented an adaptive strategy, including a simple scheme, based on a common spatial pattern (CSP) method and support vector machine (SVM) classification. The system was tested online on 10 participants, of whom 7 reached the criterion level of 70% with both peak and average accuracy in 3 days. Lehtonen et al. [92] investigated whether inexperienced subjects could control a BCI accurately by means of visually-cued left versus right index finger movements. In the second trial, seven of the ten subjects were able to control the BCI well. Their mean single trial classification rate was 80%, and the bit rate was 10 bits/min. Hazrati and Erfanian [93] presented an online single-trial EEG-based BCI for controlling hand-holding, sequence of hand grasping, and opening in an interactive virtual reality environment, using an adaptive probabilistic neural network to overcome the subject training challenge. The experimental evaluation on ten naïve subjects demonstrated that an average classification accuracy of 75.4% was obtained during the first experiment session and 81.4% during the second session. The average rates during the third and eighth sessions were 79.0% and 84.0%, respectively, using a previously calculated classifier during the first sessions, without online training and without the need to calibrate.

Notably, the algorithms have proved to work well with two different headsets, characterized by a different number of electrodes and different configurations. Although the present study was not conceived to test and compare the two systems, we can report the main differences and our subjective impressions.

“Technologically” different electrodes The Emotiv Epoc+ headset uses sponge-type electrodes soaked in a saline solution that favors the skin-electrode electrical contact, while the OpenBCI Mark IV headset uses plastic electrodes with tips pressing on the scalp. On the one hand, the headset has elastic branches that impart a slight force on the scalp; on the other hand, the rigid OpenBCI headset allows to tighten up the electrodes so that they press on the skin with an adjustable force but are generally stronger than the Emotiv system.

Electrodes in different positions of the scalp. The Emotiv Epoc+ headset has 14 prefixed electrode positions, while the OpenBCI Mark IV allows deciding where to insert the electrodes (ranging from 8 to 16) and choosing between different options.

Data Communication. While OpenBCI provides tools and routines allowing to implement the data acquisition in a quite straightforward way under different platforms and languages (e.g., Matlab, Python, etc.), the Emotive system provides an easy collection of the data only at the end of the recording session. Online data acquisition and processing with Emotive appears to be more complicated and requires an additional software license.

Comfort. The Emotive headset appears to be more comfortable, which could, in principle, allow for a longer-lasting experimental session. The dry electrodes of OpenBCI are, conversely, a bit painful but have the advantage of providing contacts that are more stable in time (no risk of drying up). Both systems are quite fast to set up.

Obviously, the presented BCIs require numerous other tests with new subjects to verify the stability of the performance and, eventually, adjust the system parameters. The results obtained may become more promising with the integration of additional data also acquired from other devices. Significant tests with ALS patients have not yet been carried out due to the lockdown and social distancing imposed by the COVID-19 pandemic.

7. BciAi4Sla Proof of Concept Prototype

In the following, we illustrate a proof of concept prototype for the BciAi4Sla user experience created using the Justinmind prototyping tool (https://www.justinmind.com/) (accessed on 23 December 2022). The prototype is intended to be used on a tablet/PC where the application will be running.

The designed layout (see Figure 3 and the following ones) has a simplified and similar structure for all the pages to maintain internal consistency. For example, in the header, we always find the project logo linked to the home page, the choices are always graphically represented by uniform boxes, identifiable by an icon and a descriptive field, etc., a sound is always returned after a user’s action as audio-feedback, etc. The basic idea is to make the application usable both for the patient and for the family members or caregivers who interact with it. For this reason, a previous version of the prototype has been evaluated by two usability experts to ensure easiness of use and of learnability.

We envision that, in order to make a preferential choice [94] in the final system, the user will have to intensively concentrate on the object of his/her choice when a light signal occurs on the element under consideration (see Figure 4 and Figure 5, for example), and then imagining the preferred hand movement to express the choice. The lighting has a duration of a certain number of predefined seconds based on the patient’s state and his/her ability to respond (the duration can be modified based on the user’s need in the Settings section) in order to allow him/her to carry out the mental task.

The home page (Figure 3) presents two buttons, the first is called “BciAi4Sla”, which can be used by the patient to access all the functions allowing communication/relaxation, while the second one called “Settings,” which allows the platform customization by the caregivers.

By accessing the BciAi4Sla section, a default page is displayed (Figure 4), but this is completely customizable. The default page presents the following options:

• Communication;
• Activities;
• Health;
• Music;
• Reading,

which will be detailed in the following sections.

7.1. Communication

In this section, the user may choose whether to communicate using a keyboard or a phrasebook, consisting of both predefined expressions and the classic answers that can be provided, for example, “Yes”, “No”, “Thank you”, etc. (see Figure 6).

The keyboard (Figure 7) has the goal to facilitate and speed up the composition of the words that express the patient’s thought according to the paradigm of the binary search: it consists of an alphabet and of numbers sections grouped and divided into blocks to facilitate selection; the software, according to the selected letters, proposes a guided composition in the typing of alphanumeric strings of the possible statistically most used words to speed up digitization. There is also a button allowing the deletion of one or more letters and a button to insert spaces. At the end of the composition of the words, it is possible, through the “Read” button, to listen to a voice reproducing the written definition. Using the “New sentence” option, the writing field is cleared to allow a new formulation.

The phrasebook (Figure 6) is made up of definitions that are predefined through the settings menu and are completely customizable according to the patient’s needs. It is recommended not to enter many expressions that would occupy the entire interface, making the patient’s activity difficult. To speed up the interaction, the choice of keyboard is present in the section to allow a direct connection with typing in case that the desired word is not present. As for the other default choices, the phrasebook is customizable by the caregiver in the Settings section.

7.2. Activities

Within this category, physical or manual activities are proposed to the patient. The choices will be fully set and customized in the Settings section. Some predefined examples include turning the TV on or off, activating an alarm sound to attract the attention of nearby people, etc.

7.3. Health

In this section, we have inserted some expressions allowing the patient’s state of health and needs to be expressed, such as I’m thirsty, I feel good/bad, I’m cold, I’m hot, etc. Additionally, in this case, the sentences are customizable.

7.4. Music and Reading

To make the application more innovative, the communication tool has been combined with functions allowing for playful and entertaining use. In the music section, the patient can choose between listening to the radio, where the main radio stations are listed, or playing songs from a playlist, which has been preloaded in the appropriate Settings section.

In the reading section, the user has the possibility to choose the category of the desired book genre. Once the choice has been made, a list of e-books that can be heard through the application is proposed.

7.5. Settings

In the Settings section, the caregiver can completely customize the application. He/she can:

• Adjust the brightness and zoom;
• Add or remove the various categories or sub-categories on the home page to act more quickly on the fundamental actions for the patient;
• Set the order of the sections;
• Add or remove pre-set phrases and possible answers in the various categories, activities, phrasebook, health, etc.;
• Load or remove songs from the playlist;
• Load or remove e-books;
• Set different background and foreground colors;
• Increase/decrease font, etc.

8. Conclusions and Future Work

Currently, the first release of the BciAi4Sla software platform has been developed, allowing the interaction through BCI as input devices between the user and the BCI application that can enable him/her to communicate and give commands, as described in the proof of concept prototype in Section 7. The BciAi4Sla’s goal is to enable LIS and CLIS patients to communicate, adapting the application and the interaction to the characteristics and needs of the single subject. Toward the end of Phase 4, the platform will be tested with patients and modified based on the feedback that will emerge.

BCI devices do not replace the assistance of caregivers, but they contribute significantly to the lives of patients. They will have a greater sense of autonomy in carrying out small daily actions, making them feel more independent.

Even today, BCIs are being studied in medical research studies to improve and make great progress in understanding and reading brain signals in order to be used more and more by different types of users, improving their lifestyles. Studies are proceeding with great constancy, and technological developments make it possible to overcome those existing limits due to the lack of sufficient knowledge and tools available. Nonetheless, there are many jobs to focus on in the future. In fact, BCIs need, for example, precautions to make them more effective and easier to use and to improve responsiveness to feedback, which is very important in communication paradigms.

There are many topics on which research could focus, but given the results obtained so far, a largely positive development of the neural interface can be expected. Regarding our current and future work, we are integrating into the BciAi4Sla platform an infrared eye tracker aimed at detecting the pupillary movement that is voluntarily controllable (through the accommodative response, for details, see [95]) and, apparently, preserved in ALS. The dual functionality (BCI and eye-tracker) will intend to compensate for the frequent occurrence of patients with impaired pupil function or with difficulty in learning and performing the tasks required by BCI based only on EEG signals.