Press Start to Play: Classifying Multi-Robot Operators and Predicting Their Strategies through a Videogame

Round  1

Reviewer 1 Report

In the abstract the author suggests a change of paradigm from system centric to user centric. As the paper develops it appears the objective is more towards operator preference and prediction. May I suggest that the authors reflect on this and possible redevelop their abstract and introduction.

In addition, the ability to predict operator preference should be discussed in the context of enhancing operator performance. Does the ability reduce cognitive workload and increase situation awareness?

Author Response

First of all, we would like to thank you for your comments, which have contributed to improve our paper. Our answers are below:

In the abstract the author suggests a change of paradigm from system centric to user centric. As the paper develops it appears the objective is more towards operator preference and prediction. May I suggest that the authors reflect on this and possible redevelop their abstract and introduction.

We have tried to reflect the following idea in the abstract: our work aims at knowing the preferences of operators and predicting their behaviors so that multi-robot interfaces can adapt the information and commands displayed to operators, thus facilitating their tasks.

In addition, the ability to predict operator preference should be discussed in the context of enhancing operator performance. Does the ability reduce cognitive workload and increase situation awareness?

We have written a paragraph discussing the proposals of adaptive interfaces in the literature: motivation, implementation and results.

Reviewer 2 Report

There are minor problems with the English language, more specifically there are numerous dangling modifiers (that, they, ones ...) which makes the text difficult to understand.
The paper does not present clear aim and research questions, which renders the contribution of this paper confusing (robotics, games, human-computer interaction, or machine learning)?
The abstract is missing the presentation of results and conclusions from the study.
Regarding the Introduction, multi-robot operators were not explained in the text, their definition or their methods.
Moreover, The Introduction explains the background to this study, but it does not clearly outline the problem or the aim of this paper.
Line 36: The authors state that 'so they will feel more comfortable.', however, it is not clear what do they mean by 'comfortable' parameter?
Line 41: The next to paragraphs present results and methodology, which does not belong in the Introduction, but rather in their respective sections. Especially since not all of the background information necessary for understanding has been presented yet.
The background section is missing, detailing multi-robot operators and clustering theory. Also, previous related work section is missing.
Line 53: this paragraph details related work and belongs to its respective section. Moreover, it needs to state why is this previous work important, and how did it influence the current study.
Line 85: Why is the split of training and test sets (90% and 10% respectively), compared to 80% - 20% standard?
Line 91: The experiment design paragraph is not-understandable. What are profiles, variables, and what is the motivation to normalize just some of them?
Line 95 - 118: This whole section needs restructuring and explanation to the reader what is this section detailing.
Line 119 - 143: This data analysis section is missing standard statistical tests for mean difference and regression on profile data.
Line 145 - 182: This section belongs to the background info
Line 183 - 200: This section belongs to the methodology of data analysis
The paper is missing the discussion on the specific results acquired and how do they relate to the aim and the research questions of the study. I believe this problem comes from the fact that this paper does not clearly define neither of those.
Regarding the conclusion section, it barely summarizes the results presented, without presenting a critical discussion on them. It even presents more findings, not present in the Results section. Furthermore, the future work and broader perspective on what these results mean, how would someone use them and for what purpose.
Threats to validity and generalizability are present in the study but are not discussed, especially if we consider that the experiment did not have any control condition. This might render the results of this study to a one-sided interpretation.

Author Response

First of all, we would like to thank you for your comments, which have contributed to improve our paper. Our answers are below:

There are minor problems with the English language, more specifically there are numerous dangling modifiers (that, they, ones ...) which makes the text difficult to understand.

We have extensively corrected the language of the whole manuscript, simplifying complex sentences and paying special atention to dangling references.

The paper does not present clear aim and research questions, which renders the contribution of this paper confusing (robotics, games, human-computer interaction, or machine learning)?

We have made an effort to make clear the contribution and study adddressed. The actions that have been taken were: Rewrite the Abstract and the Introduction. We have made clear that the goal of this work was to propose an approach to improve teleoperated multi-robot missions by classifying the operators behaviours and then using them to predict new behaviours. Finally, the research questions were formulated.

The abstract is missing the presentation of results and conclusions from the study.

We have edited the abstract to improve the explanation of the work, as well as to include the results of the study (percentaje of improvement in prediction success by using previous data and operator clusters).

Regarding the Introduction, multi-robot operators were not explained in the text, their definition or their methods.

The role of multi-robot operators, some of the task they need to execute and specifically why this is a complex task has been clarified in the introduction.

Moreover, The Introduction explains the background to this study, but it does not clearly outline the problem or the aim of this paper.

As mentioned in a previous comment, the Introduction was restructured in order to enhance the contribution of this work. Furthermore, the research questions were formulated.

Line 36: The authors state that 'so they will feel more comfortable.', however, it is not clear what do they mean by 'comfortable' parameter?

Currently, operators are trained to use interfaces, but in the near future this can be turned inside out: the interfaces will adapt to operators, so as to facilitate their work, possibly by adapting the information and available commands shown to operators.

Line 41: The next to paragraphs present results and methodology, which does not belong in the Introduction, but rather in their respective sections. Especially since not all of the background information necessary for understanding has been presented yet.

We have moved the first paragraph to the "Analysis/Classification" section and the second one to the "Analysis/Prediction" section. In this way, the introduction section presents the problem and the approach of the paper and the background one describes the related work and the techniques used in the paper. Then, the readers have the information necessary to understand the experiments and analysis sections.

The background section is missing, detailing multi-robot operators and clustering theory. Also, previous related work section is missing.

We have created a background section with the related work: multiple robots-single operator scenarios, studies of operators in multi-robot missions, clustering and prediction techniques...

Line 53: this paragraph details related work and belongs to its respective section. Moreover, it needs to state why is this previous work important, and how did it influence the current study.

We have moved this paragraph to the background section, explaining that a serious game allows to collect massive and relevant data of robot operators, and the cited works have been used as a reference for the development of the videogame.

Line 85: Why is the split of training and test sets (90% and 10% respectively), compared to 80% - 20% standard?

We have repeated the data analysis using a training set of 80% and a validation set of 20%. As it can be seen in tables 2, 3 and 4, the data are slightly different, but the results of the study are similar to the obtained with a training set of 90% and a validation set of 10%.

Line 91: The experiment design paragraph is not-understandable. What are profiles, variables, and what is the motivation to normalize just some of them?

We have added the definition of profiles (sets of features that define the styles of players) and variables (each one of these features) in the first paragraph of the "definition of profiles" section. Some variables have raw values between 0 and 1, whereas other ones must be normalized. We have added this explanation too.

Line 95 - 118: This whole section needs restructuring and explanation to the reader what is this section detailing.

We have improved the explanation of the variables, including the meaning, computation and range of values.

Line 119 - 143: This data analysis section is missing standard statistical tests for mean difference and regression on profile data.

We have perfomed two statistical tests (t-test and anova) to demonstrate the difference between the means is significant. The results have been added to the paper.

Line 145 - 182: This section belongs to the background info

We have created a section of background, where we have moved this information of clustering techniques.

Line 183 - 200: This section belongs to the methodology of data analysis

We have created a section of background, where we have moved this information of quality metrics.

The paper is missing the discussion on the specific results acquired and how do they relate to the aim and the research questions of the study. I believe this problem comes from the fact that this paper does not clearly define neither of those.

We have added a discussion section to answer the research questions posed at the beginning of the paper taking into account the results of the study.

Regarding the conclusion section, it barely summarizes the results presented, without presenting a critical discussion on them. It even presents more findings, not present in the Results section. Furthermore, the future work and broader perspective on what these results mean, how would someone use them and for what purpose.

The conclusions have been revised and rewritten to better clarify this aspects, All results are now introduced in previous sections and we have introduced some of the future work and remaining challenges.

Threats to validity and generalizability are present in the study but are not discussed, especially if we consider that the experiment did not have any control condition. This might render the results of this study to a one-sided interpretation.

We have mentioned the limitations of the study in the respective sections of the paper, as well as we have discussed them in the conclusions of the paper. 

Round  2

Reviewer 2 Report

The quality and the presentation of the manuscript has been greatly improved from the original submission.

Since the authors have addressed all of my extensive comments in this revision, I believe this manuscript is ready for publication in the present form.