The BciAi4SLA Project: Towards a User-Centered BCI

Round 1

Reviewer 1 Report

Authors presented a prototype for doing BCI experiments, oriented for ALS people. The paper seems to not provide very novel results or advancement with respect to the state of the art, by doing just experiments with few subjects, and by showing classification accuracy comparable (or even lower) with respect to existing literature. Also, all the previous projects regarding BCI for ALS people were not mentioned at all. I suggest an overall improvement of the paper befor publishing it. Here below some comments:

Abstract: passive BCI for example does not depend on voluntarily intentions. Please revise the definition.

Line 149: The sentence is not clear at all, in fact the acquisition of the cerebral signal was always considered in all the research as the main pillar to generate a good BCI.

Line 206: Authors did not mentioned passive BCI research (e.g. https://pubmed.ncbi.nlm.nih.gov/21436512/). I understand that the paper is most focused on active BCI, but at least this field should be mentioned, since it is part of BCI research. In addition, the authors did not mentioned anymore the previous BCI projects for ASL people (e.g. TOBI, SM4All...) They should insert a section talking on these projects, and state in which their actual project is different. 

Line 236: "BCI headsets via EEG" does not make any sense. 

Line 243: Real time, or online interaction? I think authors intend the latter.

Line 255: what about P3 based BCI? They are not mentioned at all.

Line 376: Actually Emotiv is not considered by the whole scientific community a high quality device. (https://pubmed.ncbi.nlm.nih.gov/28497557/). Before authors said that EEG acquisition device is a very important thing (line 149).

Line 421: CSP realizes a spacial filtering, that can enhance the difference between right and left hand detection. Anyhow, it is not clear which is the features that has been used by the SVM (e.g. frequency features? Or anything else?).

Line 462: Authors did not reported ethical approvals for the experiments. 

Results section: Authors involved just few healthy subjects for each paradigm, by reaching a fair level of accuracy. Authors should increase the number of partecipants, and once done, at least compare with existing other similar studies.

Author Response

Authors presented a prototype for doing BCI experiments, oriented for ALS people. The paper seems to not provide very novel results or advancement with respect to the state of the art, by doing just experiments with few subjects, and by showing classification accuracy comparable (or even lower) with respect to existing literature. 

We would like to thank the reviewer for the useful comments and suggestions, which helped us to improve the the paper

Concerning this first point, we would like to highlight that the key point of our paper is indeed the use of low cost BCI headsets, which let us obtain results absolutely comparable to the ones of other research works,  using headsets with a higher number of channels (. We would like to remark that the number of subjects in our experiment is similar to the one of other research work, see works reported in  lines 778-80.

Another key point of our paper are the on-line evaluation with users, to whom the system reacts in real time, while most of BCI papers are offline experiments with dataset 

 Moroverover the project has been carried out during the COVID outbreak

Also, all the previous projects regarding BCI for ALS people were not mentioned at all. I suggest an overall improvement of the paper befor publishing it. Here below some comments:

Abstract: passive BCI for example does not depend on voluntarily intentions. Please revise the definition.

Definition revised 

Line 149: The sentence is not clear at all, in fact the acquisition of the cerebral signal was always considered in all the research as the main pillar to generate a good BCI.

We removed this ambiguous sentence from the the paper

Line 206: Authors did not mentioned passive BCI research (e.g. https://pubmed.ncbi.nlm.nih.gov/21436512/). I understand that the paper is most focused on active BCI, but at least this field should be mentioned, since it is part of BCI research. 

Two references have been added  added 

In addition, the authors did not mentioned anymore the previous BCI projects for ASL people (e.g. TOBI, SM4All...) They should insert a section talking on these projects, and state in which their actual project is different. 

We added references to the mentioned projects, and one more

Line 236: "BCI headsets via EEG" does not make any sense. 

Corrected 

Line 243: Real time, or online interaction? I think authors intend the latter.

Yes, If the reviewer intends  ‘on-line’ as  the real-time interaction, with the system reacting in real-time to the brain commends 

Line 255: what about P3 based BCI? They are not mentioned at all.

It was one of the only comparison between the 2 headsets, this is why has been reported, also because it mentions the comfort of Emotiv

Line 376: Actually Emotiv is not considered by the whole scientific community a high quality device. (https://pubmed.ncbi.nlm.nih.gov/28497557/). 

The goal of the overall project is to test low cost device, as already happened in TRL research project as https://www.braincontrol.eu/en/

Also in the paper mentioned by the reviewer the conclusions about Emotiv are not so discouraging, as well as reported by https://pubmed.ncbi.nlm.nih.gov/33614271/.  Indeed, as we reported in section 6, we preferred to concentrate on the OpenBCI headset since literature reported best accuracy results compared to the ones of Emotiv Epoch+ [69], and delays in the transmission of markers has been reported while using the last one, especially in real-time classifications [66]

Before authors said that EEG acquisition device is a very important thing (line 149).

This ambiguous sentence has been removed

Line 421: CSP realizes a spacial filtering, that can enhance the difference between right and left hand detection. Anyhow, it is not clear which is the features that has been used by the SVM (e.g. frequency features? Or anything else?).

We created the whitening matrix W by feeding as input to the CSP algorithm a matrix of representing the first class trials (left hand imagined movement) and the second class (right hand imagined movement).
From this W matrix, we took the first and last 4 columns representing the eigenvalues of the spatial filter, simply called also filters. With these filters we transformed each trial.
At last, we extracted the features related to the trials (which will be used by the SVM), by taking for each of the 8 rows of the matrix, their logarithm of the variance and obtaining 8 feature numbers. 

Line 462: Authors did not reported ethical approvals for the experiments. 

It has been reported 

Results section: Authors involved just few healthy subjects for each paradigm, by reaching a fair level of accuracy. Authors should increase the number of partecipants, and once done, at least compare with existing other similar studies.

We are aware of it. That part of the  project has been carried out since Autumn 2019 to September 2021. Now are integrating in the system pupil accomodative response, before starting new trials. 

During BciAi4Sla, we have almost always been in lockdown period and severe restrictions. For this reason we never involved patients, and also all the experimental subjects were colleagues and staff of the university because it was not possible for us to reach anyone else. With the COVID-19 outbreak many things changed and conducting evaluations with users became challenging, as recognized by the HCI research community

Author Response File: Author Response.pdf

Reviewer 2 Report

Timely topic. Clear presentation of research aims, objectives and methods amd results.

Providing a table to illustrate participant demographic information and clarify who involved with an ASL would enhance quality of the paper and results.

Author Response

Providing a table to illustrate participant demographic information and clarify who involved with an ASL would enhance quality of the paper and results

We would like to thank the reviewer for the useful comments and suggestions, which helped us to improve the the paper

The involved subjects were all neuro-typical.  During the project, we have almost always been in lockdown period and severe restrictions. For this reason we never involved patients, and also all the experimental subjects were colleagues and staff of the university because it was not possible for us to reach anyone else. We have information about age, sex and instruction of the subjects, which we reported in the sub-section Participants of every experiment.  If required we could add a Table, but we believe that it is not so informative

Round 2

Reviewer 1 Report

Authors replied to all the comments.