From When to When: Evaluating Naturalness of Reaction Time via Viewing Turn around Behaviors

Round 1

Reviewer 1 Report

In the reviewed paper, the authors investigated visual reaction times in touch interaction. In general, the paper has no novelty and the proposal is very poor. My main issues:
1) The novelty is missing.
2) The abstract does not say how these studies impact the current state of the art.
3) The introduction is outdated. The authors should analyze mainly the papers from the last 3 years mostly.
4) The model is not defined in a proper way. There is no mathematical formulation, no implementation idea like pseudocode.
5) Some visualization would be nice to add. 
6) The paper should have a separated experiment section.
7) Some comparison with state-of-art must be made.
8) There is no analysis of other robot models. It should be made on more test groups. 
9) The statistical analysis should be made on a larger group of participants.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Overall, I find this work means a valuable contribution to the field. I have only a minor concern:

As a conclusion to your work, you state that it's more relevant to do a good calibration of the parameters of the robots' behavior design than to base them on observation of human behaviors. Could you explain this conclusion a bit more? Maybe just a few comments could clarify this concept a bit more.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

This paper addresses an interesting question in the domain of human-robot interaction, related to the perceived naturalness of a particular interactions (turn around behavior of a robot when touched by a human).

If the topic is relevant and the size of the sample as well as the experimental plan seems to be well designed, I think that some important points should be significantly improved to make this paper publishable.

----------------

Introduction

-----------------

1. The authors studies the question of interaction naturalness, and provides some elements from the literature studying this criterium. Nevertheless, no clear definition was given to define this concept.
2. Moreover, the authors later use a 10 point scale to assess the naturalness of robot interaction. It sounds strange that there is no pre-existing questionnaire or evaluation approach in the literature?
3. As mentioned very late in the paper, different factors, other than time and robot type, can explain the perceived naturalness of interaction (like the context of the interaction, given the level of surprise or task involvement in the task). These factor should be listed in the literature review.

----------------

Material & Method

----------------

I have several questions about the method. I hope that can help improve the paper, either to better detail the approach, or to better analyze the results.

4. How do you manage the order of the video to counterbalance the sequence effect?

5. How do you explain that more than one third of your sample consider as natural some instantaneous reaction?

6. Are you sure you can use ANOVA for ordinal dependent variables?

7. Why do not use 2-factor or 3-factor ANOVA (or non parametric ANOVA) to study the effect of the different reaction time level and model type on perceived naturalness?

8. Are you sure that there is a significant effect of all your factor (type of model, time before and time after?)

9. I do not understand why make an ANOVA between best combination of timings? I really have important concern about your methodology to compare only the best combination. For many models (human or robot), it seems to be that different combinations are not too far from the "best combination", and could have been of interest...

10. The section 3.4 is not clear to me. Authors mention the question of size and appearance, then they discuss about the difference between autonomous VS teleoperated robots... Moreover, I am not sure to catch what authors did in this additional study. I am not sure to catch what the authors studied in this part. Do they compare the data collected in the previous survey about human models with this new survey?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

My comments were addressed.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

I have read this new version, and there are noticeable improvement brought by these authors to the paper.

Nevertheless, I am still not convinced in how the authors analysed their data, especially when they do some ANOVA by comparing only the best combinations of times. To me, that analysis is not really consistent (since many combinations are very close the best combinations and are finally discarded), and the significant effect reported in the paper can be biased by this methodological choice.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The paper is an attempt to addresses the effects of visual reaction times of a turn around behavior toward  touch stimulus in the context of perceived naturalness. The paper investigated appropriate reaction times for three robots (Sota, Nao, and Pepper) and humans (male and female).

Given that the experiments have been performed on very minimal conditions, thus the results are very limited, and the concluded guidelines cannot be generalized.

The authors claim " The appearance and size effects toward reaction time design are limited" need to be justified with experiments.

The paper assumes many restrictions that oversimplify the robot's reaction time of a turn around behavior to touch stimuli. The authors assume that a turn around behavior is employed by rotating head only and ignoring the waist . Also, the strength of touch, direction and location of touch are oversimplified to one scenario.  Moreover, application space and user group of this research study, which may affect the outcome of this study, are not clarified. Thus, the results of this research study can not be extrapolated to other robot design and behavior as well as other user group.

Reviewer 2 Report

This paper investigated the robot’s natural reaction time when someone touches the robot’s shoulder. I agree with authors that the natural reaction time is one of the important aspects of HRI. However, the experimental model and results are too simple even though this study conducted the reasonable sized survey. The authors need to define more challenging and comprehensive research problem. The current version of paper is not enough to publish in Applied Sciences.

Reviewer 3 Report

I find this work seems to be interesting to the community. I have only one minor concern about your presented paper:

It's a good idea that you include a brief paragraph to comment on the paper structure, maybe at the end of the introduction section. This will ease the reader to know what is going to appear in the text afterward.

Reviewer 4 Report

The article “From When to When: Evaluating Naturalness of Reaction Time via Observing of Turn Around Behaviors” addresses the effects of visual reaction times of a turn around behavior toward touch stimulus in the context of perceived naturalness. It is an interesting overview, however, there are some expositional and conceptual flaws, as detailed in the comments below. 

The details such as how many sensors does SOTA/Nao/Pepper robots have which are responsible for the turnaround behavior are missing. These details can be added in section 2.2 along with the degree of freedom of each robot. The details of how the robots are able to perform the turnaround behavior are missing. For example, the authors mentioned that the touch on the left shoulder on the robots triggers the turn around behavior.  The authors have provided photographs for each model but I still believe the research would be stronger if there was evidence of how these models exhibited that turnaround behavior. Was it the wizard of oz or how the robots were programmed? If the robots are already built-in with such behavior then such experiments do not contribute to the research.

The experimental data suggest that multiple videos were recorded and data were analyzed as before- reaction time and after-reaction time for each model. The experimental data should include the details of how robots are trained for affective touch in human-robot interaction and therefore related references should be provided.

The references have repetition. I believe the authors wanted to cite these references multiple times in the article and so ended up having repetition. However, this also leads to limited reference which does not provide sufficient background and relevant information on how significant their work is.

References #1 and #6: repeated source

References #2 and #10: repeated source

References #3 and #9: repeated source

References #4 and #5: repeated source

The original contribution and the relevance of each part are not obvious. The summary and conclusions do not explain what is the novelty of the presented considerations in terms of the algorithmic/conceptual approach.