EEG Correlates of Distractions and Hesitations in Human–Robot Interaction: A LabLinking Pilot Study

Round 1

Reviewer 1 Report

The paper presents an interesting idea but should be modified in some of its parts:

- A section on the state of the art is missing (although some references have been included in the introduction);

- An image representing the workflow of the proposed framework (duly commented) should be inserted to make the proposal clearer;

- Implementation details about the code used are missing;

- More accurate performance measures such as Recall, Precision, F-measure should be used to measure performance;

- Object recognition techniques based on structured data would be interesting to evaluate in the specific context. An interesting recent paper in this regard should be cited:

Manzo, M., & Pellino, S. (2021). FastGCN+ ARSRGemb: a novel framework for object recognition. Journal of Electronic Imaging, 30(3), 033011-033011.

Author Response

Dear Reviewer 1,

We addressed your comments as follows:

• a section state of the art is added
• an additional image depicts the workflow
• the code is installable via the CITK  
• additional measures are provided and
• the future work section is extended.

Thank you very much for the constructive feedback.

Kind regards,

Birte Richter

Reviewer 2 Report

The authors investigated human EEG correlates of distractions and hesitations from a robot during human-robot interactions (HRI). They showed that (i) EEG responses between the ambient (road noise) and distraction (robot’s voice) conditions can be differentiated using spectral power features from the theta and alpha bands (4-12Hz), and (ii) a more distinct difference can be found between the distraction and hesitation conditions using features from the theta, alpha, and low-beta band (4-20 Hz). Besides, the authors also claimed that this is a proof-of-concept study, intended to show the potential of their LabLinking method when conducting collaborative HRI studies in remote laboratories.

The motivation and relevance of the research are clearly described. The used methodologies are clear, and the results and discussion are well-focused. I addressed some issues that need the authors’ attention:

Major comments:

1. The authors spent a lot of effort in explaining the setup and the advantages of the proposed LabLinking method. But are there other attempts to solve the remote experiment problem in the literature? What are the advantages of LabLinking compared to others? Where is the literature review on that? 

2. How does LabLinking contribute to the scientific society? For example, can other labs have access to it? 

3. Critically, how to evaluate whether the LabLinking method is good or not? It seems better to use some metrics (e.g., signal latency) to quantify the remote communication efficiency.

4. When investigating the EEG responses to a specific event (here, it can be robot distraction or hesitation), the time-frequency analysis of EEG signals is a straightforward as well as meaningful way, but why do the authors only consider the classification performance? For example, it is interesting to show event-related spectral perturbations (ERSP) and brain topography changes during HRI. In this way, how EEG correlates with distractions and hesitations of a robot could become more clear.

Minor comments:

1. In the Abstract, it is better to change the focus of the second sentence, as it didn’t mention why hesitation is worth investigating during HRI.

2. Line 110, the abbreviation of ‘Brain Computer Interfaces’ should be in Line 103, when the authors first mention it.

3. In Table 2, to be consistent with Lines 311-312, it should be ‘DistNoHes’ instead of ‘NoHes’.

4. How many EEG channels were used in the experiment? Give what is mentioned in Line 364, I assume, it is 64, but in any case clarify it.

5. Line 381, the dimensions of the features are wrongly calculated.

6. Line 407, what do the three parameters in the parenthesis mean when referring to Figure 4? The same question for Line 425.

Author Response

Dear Reviewer 2,

We addressed your comments as follows:

• all minor comments are addressed in the text
• 1: a literature review is added 
• 2: the code is online available; therefore other labs could use it
• 3: additional information about the signal latency is added
• 4: addressed in the section 3.5

Thank you very much for the constructive feedback.

Kind regards,

Birte Richter

Reviewer 3 Report

The manuscript reports about a LabLinking-based study aiming to evaluate the EEG correlates of Distractions and Hesitations in Human-Robot Interaction. The manuscript is well written, and the results are very interesting. In my opinion, only few minor concerns need to be addressed before publication:

1)     Please provide some information regarding the EEG instrumentation (e.g., the model, the number of channels).

2)     Were the classes of the classification task balanced? Or some trials had to be discarded and it affected the balance of the classes?

3)     It could be interesting to also report the confusion matrix and the ROC curve associated to the classification, in order to assess the sensibility and specificity of the procedure.

4)     It could be interesting to reports the three level classification results as well (i.e., NODIST vs. DISTNOHES vs NOHES). 

Author Response

Dear Reviewer 3,

We addressed your comments as follows:

• 1: additional information regarding EEG is provided
• 2: additional information about the balanced is provided
• 3: additional measurements are added
• 4: The two classification problems were chosen because they were considered to be the most essential and meaningful ones. If a three-class classification problem were attempted, it would involve discriminating between the "NoDist" and "DistHes" classes, which would aim to identify neural differences in the baseline trials and the trials where both manipulations occur simultaneously (during distraction and after hesitation intervention). Therefore, this setup does not allow for a meaningful interpretation of potential differences in EEG data because we cannot differentiate whether classification depends on one or the other manipulation. Nonetheless, this specific case was tested and yielded discrimination accuracies of 63% +/- 10%, which were better than those of the first classification task, but worse than those of the second task.
  

Thank you very much for the constructive feedback.

Kind regards,

Birte Richter

Round 2

Reviewer 1 Report

No further changes are required apart from the missing reference:

Manzo, M., & Pellino, S. (2021). FastGCN+ ARSRGemb: a novel framework for object recognition. Journal of Electronic Imaging, 30(3), 033011-033011.