Predicting Perceptual Decision-Making Errors Using EEG and Machine Learning

Round 1

Reviewer 1 Report

Predicting Perceptual Decision-Making Errors is investigated by Using EEG and Machine Learning. Distinguishing between correct and 2 erroneous responses in the perceptual decision-making task using 32 channels EEG and training an ANN. The proposed methods and obtained results are pretty good. However, there are some problems need to modify.

1.     The paper is well organized, however, the expression need to pay more attention, for example “1D” or “1-D”is should be uniform in table 1 and table 2.

2.     The English is good, but which also need to polish more. And there are some typos and grammar mistakes?

3.     The style of the references is not good, which need to check carefully.

Author Response

Predicting Perceptual Decision-Making Errors is investigated by Using EEG and Machine Learning. Distinguishing between correct and 2 erroneous responses in the perceptual decision-making task using 32 channels EEG and training an ANN. The proposed methods and obtained results are pretty good. However, there are some problems need to modify.

1. The paper is well organized, however, the expression need to pay more attention, for example “1D” or “1-D”is should be uniform in table 1 and table 2.
2. The English is good, but which also need to polish more. And there are some typos and grammar mistakes?
3. The style of the references is not good, which need to check carefully.

We thank the referee for careful reading of our manuscript. In the new version, we addressed all issues: provided uniform writing for 1D and 2D through the text; fixed grammar and typos, and changed the style of references.

Reviewer 2 Report

(1)     I suggest listing the main contributions point by point in Section 1.

(2)     The structure of this paper should be given in the last paragraph of Section 1.

(3)     In Section2, the overall framework of the model proposed in this paper should be given. For example, adding an overall learning framework figure or a pseudo code.

(4)     In Section 4, I suggest listing the main conclusions point by point.

(5)     For medical data, sensitivity and specificity are two widely concerning indicators. In this paper, how to balance sensitivity and specificity?

(6)     Further, how can wrong prediction results of perceptual errors impact real life? Discussions should be added.

Author Response

1. I suggest listing the main contributions point by point in Section 1.

We listed the main contributions as follows:

• We demonstrated that an artificial neural network can predict errors in the perceptual-decision-making task using EEG signals recorded before the behavioral response with an accuracy above 80%.
• To create the input data for ANN, we averaged neural activity over time and sensors. In both cases, the classification accuracy remained above 80% manifesting that both dimensions contain valuable discriminating features.
• Using pre-stimulus and post-stimulus EEG segments as input data resulted in the classification accuracy above 80%. The pre-stimulus EEG reflects the human state while the post-stimulus EEG reflects the sensory processing mechanisms. Thus, we concluded that these processes affect the final decision.

2. The structure of this paper should be given in the last paragraph of Section 1.

Following the referee’s suggestion, we gave the structure in the last paragraph of Section 1.

This manuscript has the following structure. The methods section contains a description of the experiment including the recruitment, EEG recording, and the type of perceptual decision-making task, as well as the data analysis pipeline including preprocessing, feature selection, machine-learning model, its hyperparameters, and cross-validation procedure. The results section presents the outcomes of the ANN's application in terms of accuracy, F1 score, sensitivity, and specificity. This section also reports the effect of ANN's hyperparameters and the class imbalance on the classification scores. Finally, the Discussion section summarizes the results, pays attention to the limitation of our paradigm, and its potential use in the brain-computer interfaces to predict and prevent human errors in critical situations.

3. In Section2, the overall framework of the model proposed in this paper should be given. For example, adding an overall learning framework figure or a pseudo code.

The overall learning framework includes three main steps: preprocessing (1), feature selection (2), and cross-validation (3). In the preprocessing block, we filtered raw EEG, removed ICAs with artefacts, segmented signals into the trials and removed bad segments. In the feature selection block, we reduced dimensions of the input data by averaging EEG signals over time and sensors. In the cross-validation block, we divided trials into the test (10%) and train (90%) sets and trained the model. This procedure was repeated five times followed by randomizing the trials. All scores were averaged over the five rounds of cross-validation.

We added the new figure with an overall data analysis framework (see Fig. 2 in the revised ms)

4.     In Section 4, I suggest listing the main conclusions point by point.

• We demonstrated that an artificial neural network can predict errors in the perceptual-decision-making task using EEG signals recorded before the behavioral response with an accuracy above 80%.
• To form the input data for ANN, we averaged neural activity over time and the sensors. In both cases, the classification accuracy remained above 80% manifesting that both dimensions contain valuable discriminative features.
• Using pre-stimulus and post-stimulus EEG segments as input data resulted in the classification accuracy above 80%. The prestimulus EEG reflects the human state while the post-stimulus EEG reflects the sensory processing mechanisms. Thus, we concluded that these processes affect the final decision.
• While this study reports the possibility to predict errors, further research is required to build an optimal machine learning model to achieve the best classification metrics.
• Our approach performs a single-trial classification; therefore, it can be implemented in BCI. Further studies will test this possibility using the BCI paradigm.

5.     For medical data, sensitivity and specificity are two widely concerning indicators. In this paper, how to balance sensitivity and specificity? 

Following the referee’s comment, we reported data for the sensitivity and specificity

6.     Further, how can wrong prediction results of perceptual errors impact real life? Discussions should be added.

Important to note that our model has a nonzero false positive rate (mean FPR is 10%). False positive errors are the most dangerous errors for a BCI operator because they can provoke improper action of BCI, thus risking the possibility of normal functional neural activity disturbance. For these cases to minimize the negative effects of the BCI action, a careful choice of the subject stimulation strategy is essential.

This passage has been added to the Discussion of the manuscript

Reviewer 3 Report

1 Section 

Rephrase - Addressing this issue, we consider a perceptual decision-making task involving processing bistable visual stimuli,  deciding on its interpretation,  and reporting the decision by pressing joystick keys.

It is unclear why the authors define the research questions as hypotheses? They are referencing the papers and so cannot be defined as hypotheses, See e.g. “First, the neural activity registered before the stimulus onset carries biomarkers of the human state impacting the decision on an ongoing stimulus [11].” 

The authors are expected to clearly describe new contribution to the subject area.

2 Section

The authors are expected to explain why they use a set of recordings made from 30 participants, whilst there are many similar benchmarks typically used in the research community for experiments and testing of hypotheses.   

The Eq 1 - 4  are known in the literature and should be removed. 

3 Section

The diagrams are not informative for support of the research goals and objectives. 

4 Section

The authors need to provide cross-validation results in order to estimate the confidence intervals. 

Tables 4 and 5 do not seem informative as well and do not support the research hypotheses.

Overall comments - the authors are expected to clearly describe working hypotheses, explain their motivations, describe their research methodology and how these hypotheses could be tested within the standard statistical framework. 

Author Response

1 Section 

Rephrase - Addressing this issue, we consider a perceptual decision-making task involving processing bistable visual stimuli, deciding on its interpretation, and reporting the decision by pressing joystick keys.

We rewrote this part as follows:

In this manuscript, we made the first step towards the perceptual errors’ prediction in the brain-computer interfaces - we proposed a machine learning model that predicts errors from the short EEG segments on a single-trial basis. To collect behavioral data and brain activity signals, we used a perceptual decision-making task, an experimental paradigm that requires participants to perceive visual stimuli on the screen and respond to them using a joystick.

It is unclear why the authors define the research questions as hypotheses? They are referencing the papers and so cannot be defined as hypotheses, See e.g. “First, the neural activity registered before the stimulus onset carries biomarkers of the human state impacting the decision on an ongoing stimulus [11].” 

We rewrote this part as follows:

The literature suggests that the neural activity registered before the stimulus onset carries biomarkers of the observer’s state, affecting the decision on an ongoing stimulus [11]. Therefore, we expect that ANN will learn to distinguish between the errors and correct responses using the pre-stimulus EEG segments as the input.

Recent studies on perceptual decision-making report that this process involves two components: sensory processing and decision-making. The sensory processing dominates during the early time window (about 300 ms, 30% of the whole decision time) [12,13]. There is a view that the brain matches sensory information with an internal template, even in the early sensory-processing stages. Therefore, we expect that neural activity registered during 300 ms post-stimulus onset also influences the final decision on the stimuli and may also serve as ANN input.

The authors are expected to clearly describe new contribution to the subject area.

• We demonstrated that an artificial neural network can predict errors in the perceptual-decision-making task using EEG signals recorded before the behavioral response with an accuracy above 80%.
• To form the input data for ANN, we averaged neural activity over time and the sensors. In both cases, the classification accuracy remained above 80% manifesting that both dimensions contain valuable discriminative features.
• Using pre-stimulus and post-stimulus EEG segments as input data resulted in the classification accuracy above 80%. The prestimulus EEG reflects the human state while the post-stimulus EEG reflects the sensory processing mechanisms. Thus, we concluded that these processes affect the final decision.

2 Section

The authors are expected to explain why they use a set of recordings made from 30 participants, whilst there are many similar benchmarks typically used in the research community for experiments and testing of hypotheses.   

Individual neuroimaging studies typically involve 12–20 participants [Robert M. Hardwick, Svenja Caspers, Simon B. Eickhoff, Stephan P. Swinnen. Neural correlates of action: Comparing meta-analyses of imagery, observation, and execution. Neuroscience and Biobehavioral Reviews 94 (2018) 31–44]. At the same time, 30 is the smallest sample size that allows using central limit theorem [Chang, H. J., K. Huang, and C. Wu. "Determination of sample size in using central limit theorem for weibull distribution." International Journal of Information and Management Sciences, Vol. 17, No. 3. 2006, pp. 153-174]

The Eq 1 - 4 are known in the literature and should be removed. 

We removed them from the manuscript

3 Section

The diagrams are not informative for support of the research goals and objectives. 

We removed them from the manuscript

4 Section

The authors need to provide cross-validation results in order to estimate the confidence intervals. 

Following the referees’ comment, we reported these data

Tables 4 and 5 do not seem informative as well and do not support the research hypotheses.

We removed them from the manuscript

Overall comments - the authors are expected to clearly describe working hypotheses, explain their motivations, describe their research methodology and how these hypotheses could be tested within the standard statistical framework.

We thank the reviewer for valuable comments. We addressed all issues in the revised version of the manuscript. In the new version, we clearly stated motivation and hypothesis, gave the structure of data analysis framework and research methodology. All corrections are marked in red.

Reviewer 4 Report

The authors present an interesting topic - Brain Computer Interfaces, but the presented algorithms are used and established in the specialized literature for the analysis of brain waves with Deep Learning.

First of all, the article does not show what is desired. We are discussing about Brain Computer Interface but it is not mentioned what are the conditions for performing the experiment. What is the paradigm used for these tests. It also does not result in what is desired to be obtained in the end. The proposed mathematical algorithms are established in the specialized literature for solving problems related to the Brain Computer Interface, therefore the authors must clarify why their method is more efficient than the similar methods reported in the specialized literature.

I ask the authors to respond to these clarifications so that the article has a greater impact.

Author Response

The authors present an interesting topic - Brain Computer Interfaces, but the presented algorithms are used and established in the specialized literature for the analysis of brain waves with Deep Learning.

First of all, the article does not show what is desired. We are discussing about Brain Computer Interface but it is not mentioned what are the conditions for performing the experiment. What is the paradigm used for these tests? It also does not result in what is desired to be obtained in the end. The proposed mathematical algorithms are established in the specialized literature for solving problems related to the Brain Computer Interface, therefore the authors must clarify why their method is more efficient than the similar methods reported in the specialized literature.

In this manuscript, we made the first step towards the perceptual errors’ prediction in the brain-computer interfaces - we proposed a machine learning model that predicts errors from the short EEG segments on a single-trial basis. While this study reports the possibility to predict perceptual errors, further research is required to build an optimal machine learning model to achieve the best classification metrics.

To collect behavioral data and brain activity signals, we used a perceptual decision-making task, an experimental paradigm that requires participants to perceive visual stimuli on the screen and respond to them using a joystick. In this paradigm, the participants were subjected to perceive 400 stimuli in a row with a brief interval. It simulated a real-life situation that implies decision-making under stress and pressure. To increase the probability of errors, we used ambiguous stimuli and decreases the stimulus exhibition time. As the result, the subjects make errors in 13% of their responses.

For implementing our model to BCI technology it should allow real-time operation. Although this was not demonstrated in this study, our method can be easily applied for real-time perceptual error prediction. This is facilitated by two key features of our model: data preprocessing is implemented automatically and after training the artificial neural network works immediately without delay.

I ask the authors to respond to these clarifications so that the article has a greater impact.

We thank the reviewer for valuable recommendations. We believe that in the present form, our manuscript is more interesting for the readers

Reviewer 5 Report

In this paper, the authors present an interesting development for predicting perceptual decision-making using EEG signals and machine learning. In general, the article displays a consistent development; however, some aspects to review and improve are the following:

1. It is suggested to make a figure in the introduction section that relates the topics identified.

2. Authors should consider adding an introductory paragraph in Section 2 ‘‘Materials and Methods’’ indicating the topics covered in this section.

3. It is recommended to place the figures after being cited. For example, see Figure 1.

4. To have a continuity in reading, it is suggested to add an introductory paragraph in Section 2.5 ‘‘Data analysis’’.

5. It is recommended to better structure Section 3 ‘‘Results’’ since this section starts with two figures.

6. It is recommended to extend the description and analysis of Tables 4 and 5.

7. It is suggested to considered add an experimental design using a statistical analysis via parametric or non-parametric tests (ANOVA, Kruskal Wallis, Bonferroni…).

8. The discussion section is adequate; however, it is recommended to place the conclusions section separately.

9. A general revision of the article is suggested to improve the writing. Some typos are identified for example see lines 321, and 327.

Author Response

In this paper, the authors present an interesting development for predicting perceptual decision-making using EEG signals and machine learning. In general, the article displays a consistent development; however, some aspects to review and improve are the following:

1. It is suggested to make a figure in the introduction section that relates the topics identified.

In line with the second referee, we added Figure 2 (see revised manuscript) illustrating the proposed approach for data analysis.

2. Authors should consider adding an introductory paragraph in Section 2 ‘‘Materials and Methods’’ indicating the topics covered in this section.

In this section, we described experimental paradigm and data analysis framework. The description of experimental paradigm includes recruitment process, visual stimuli (Necker cubes), design of experiment, and behavioral estimates resulted in separating data in two classes. The description of data analysis framework includes preprocessing, feature selection, machine learning model, and cross-validation.

3. It is recommended to place the figures after being cited. For example, see Figure 1.

We placed Figure 1 after its first citation.

4. To have a continuity in reading, it is suggested to add an introductory paragraph in Section 2.5 ‘‘Data analysis’’.

In line with the second referee, we added an introductory paragraph:

The overall data analysis framework includes three main steps: preprocessing (1), feature selection (2), and cross-validation (3). In the preprocessing block, we filtered raw EEG, removed ICAs with artefacts, segmented signals into the trials and removed bad segments. In the convolution block, we performed dimension reduction by averaging EEG signals over time and/or over the sensors. In the cross-validation block, we combined data of all subjects and divided trials into the test (10\%) and train ($90\%$) sets. This procedure was repeated five times followed by randomising the trials. All scores were averaged over the five rounds of cross-validation.

5. It is recommended to better structure Section 3 ‘‘Results’’ since this section starts with two figures.

In line with the third referee, we removed figures from the results, hence improving its structure.

6. It is recommended to extend the description and analysis of Tables 4 and 5.

In line with the third referee, we removed these tables from the manuscript and extended description of these results in the text.

7. It is suggested to considered add an experimental design using a statistical analysis via parametric or non-parametric tests (ANOVA, Kruskal Wallis, Bonferroni…).

Following the referee’s suggestions, we tested how the ER depends on the type of stimulus using repeated measures ANOVA. Stimulus ambiguity (High vs Low) and stimulus orientation (Left vs Right) were taken as within-subject factors. As a result, we found a significant main effect of ambiguity: F(1,25)=65.13, p<0.001, The main effect of orientation was insignificant F(1,25)=0.88, p=0.375. Finally, we reported an insignificant interaction effect of Ambiguity and Orientation: F(1,25)=0.02, p=0.888. The post-hoc t-test revealed, the ER for High ambiguity (M=18.2%, SE=2.02) exceeded ER for Low ambiguity (M=3.3%, SE=0.53): p<0.001, Bonferroni correction.

8. The discussion section is adequate; however, it is recommended to place the conclusions section separately.

Following this suggestion, we added conclusion as a separate section.

9. A general revision of the article is suggested to improve the writing. Some typos are identified for example see lines 321, and 327.

We fixed typos through the text

Round 2

Reviewer 3 Report

The authors have addressed the main comments and in my opinion improved the manuscript. Thus I am positive for acceptance.

Reviewer 4 Report

Thanks to the authors for clarifying the suggestions.