Socio-Economic Decision Making and Emotion Elicitation with a Serious Game in the Wild

Round 1

Reviewer 1 Report

The paper describes a serious game to elicit emotional response of players when dealing with socio-economic decision making. The focus is specifically on the elicitation of emotions "in the wild", that is, in an uncontrolled setting. This allows to not interrupt flow for assessment during the gameplay and also allows for larger experiments and data collection, since in the next development stages, it will allow testing without the researchers' presence.

That said, the main novelty is in the introduction of an emotion self-assessment widget after each decision. The contribution of this paper is in the demonstration that their SG, comprehending decision-making issues from the literature, provokes the same emotional pattern as the non digital games, and that the collected emotional responses through the self-assessment widget are sound and usable for further experiments.

Beside being a limited, first experimental test, the paper reaches its objectives. The methodology is sound and provides details and analyse the results extensively.

There are a few unclear points and overall I find the paper too verbose. Aiming for extreme clarity, the result is too many repetitions and redundancy.  The style becomes more "heavy" from section 3.2 on. 

I strongly suggest to revise the text removing too many redundancies and make it "lighter" by avoiding passive forms as much as possible.

Examples:

- emotion detection and analysis is useful in mitigating mental disorders: introduction,  lines 26-30, 48-49, 78-81 (three times in the same section); reaction time for PDG is higher: appears in line 515 and is repeated at line 517 (two lines after!)

- the subject "elicit emotional responses and decision-making patterns in the wild" is repeated many many times, and what "in the wild" means as well. Once is enough, two is ok, more is too many ;-)

- Too many references to other sections and to the literature outside the "related works" section: limit references to other sections when you are anticipating something that is defined later on, without repeating what already said. Examples:  line 438 "The emotion data, as discussed in section 3 had six dimensions....). Lines 444-453 on emotional collection in controlled settings and the pros of RPGs (already said elsewhere).

These are just a few examples. Please, revise the text to avoid repetitions! It will be more pleasant to read. Also, please do not underestimate semicolon; they are useful to introduce longer pauses (e.g., line 223: "... in the participants of those studies; hence, we...")

Other minor issues:

- I suggest to select either the emotion or the personal feeling of the player in an homogeneous manner: "Happy" and "Disgust" -> "Happyness" and "Disgust" OR "happy" and "disgusted"

- Table 2: I suggest to invert the last two columns, having value type before value range.  "Quantity" could be replaced by "feature". Decision type (description): why the focus on depressed individuals? type categorical, ok; as for the values I'd put some "", meaning that the string with the name of the scenario is the value. Resource: health is not mentioned here, and should have a numeric (continuous?) value. I don't see the logic behing defining both "Emotion Vector" and "Incidental emotional response" if you discriminate them by "class" and "instance". This tables describes classes at abstract level, the instance has the same "type" and "range" of the class anyway. Instead, the two have different types "continuous" (which is wrong for a vector) vs Emotion Vector. I suggest instead to define "Emotion valence" as a single, continuous value for the six considered emotions, and then Incidental emotional response as an "aggregated set of emotional valences determining the emotional state of the player" Type=set of continuous values; range: positive real numbers. Prior decision is updated with the game progression, but I think the order of the scenarios is fixed, so it is not a critical information. Conversely, for the analysis, you will need to know the RESPONSE to that scenario, so I would expect "prior decision" to be a couple of <scenario, answer>, if not a triple <scenario, answer, emotional response>. Player role, I think it is fixed, e.g., in the Prisoners' dilemma the palyer is always a prisoner. So why do you need this information? Indeed, tha fact that the roles are fixed limits a lot the game, but this is probably due to the aim of analysing known emotional responses that have been analysed in the literature.

- line 617: the game is not really "novel" since it implements known scenarios.

- I am not sure the term "naturalistic setting" can be used as "in the wild"

- As far as I know, Presence is related to the credibility, realism of the environment and sensorial feedback. So, the fact that user rate Presence as low might be related to the graphics (which is not realistic and quite "puppet-like" rather than to cognitive load (it does not seem so demanding after all)

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Interesting reseach work.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper presents an interesting and promising study on a serious game for decision-making under 4 socio-economic contexts while measuring 6 core emotions +  engagement questionnaire. The idea is original and has the potential of contribute to the state of the art in the field.

But, some major concerns exist that seriously question the study relevance and novelty:

1) Some important features of the games are missing. The general or base game mechanic should be explicitly described, that is, the way in which the player is expected to "move" from one task/decision-game to another. Algo gamification elements are missing. Are there any points, ranking, or badges present? Figure 2 in the top-left corner shows 3 progress bars and some threshold rules for progression are mentioned. However, no formal descriptions of such game elements are present. Also are missing types of feedback provided to the player. 

2) on the experimentation design a major concern is the authors used a sample size of 8 participants, based on usability tests standards. However, the results and analysis provided afterward based on those participants do not correspond to any usability instruments or techniques to reveal potential human-computer interaction problems with the game. Instead, they measure self-reports on 6 core emotions (Happy, Surprise, Sad, Angry, Fear, and Disgust)+ Game Engagement Questionnaire (GEQ) with 4 dimensions: Immersion,  Presence, Flow, and Absorption. Those are psycho-emotional constructs, that need internal validation and consistency measures together with a robust quantitative analysis that a sample of 8 participants cannot provide.  My suggestion is that the authors should decide if they really develop a usability study with those 8 participants, revealing usability problems on the software game or decide to provide a full-fledged quantitative study for the effects measured with more subjects that allow to the application of traditional parametric statistics. 

minor concerns:

3) paper editing is missing. The manuscript has many typos: 

4) an important paper argument or point regarding the non-controlled setting of their study, something that they called "in the wild". a brief explanation exists on this point at the related work section giving some examples of more "wild" setting on some features, but not describing which are all the dimension or characteristics in which controlled vs will setting may diverge.  

5) related work also needs further elaboration as it is too descriptive. The section should provide a clear view to the reader at least on a) what other studies have been carried out in this problem (emotions and engagement in serious games for decision-making), b) which are the similarities/differences with the current study and c) which were their main findings. b and c are missing in the review so it is not possible to clearly assess the novelty level of the present paper.  Furthermore, The discussion section does not contrast own findings against literature, thus, reducing depth to the analysis itself.

 

 

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

1) Regarding point 1 of past round, Some important features of the games are missing: the new version has improved. But, the game requirements section seems too extensive and artificial. The game description should focus on the replication principle; Will a reader be able to replicate the games and experiment with the information provided in the paper?

2) On past review point 2: related work, still has a lack of depth. There is no related work on papers acknowledging the relevance of non-controlled settings, studying why emotional states should be different when changing from controlled to uncontrolled conditions, etc. What changes among both that make it relevant to check possible changes in the results?

Also, authors reference [7,9,21,23,26–29,29,30]. But their main characteristics on study objectives, game used, variables studied, experimental design decisions, and main findings is not provided. 

3) On past review point 4: In the new manuscript, the description of what is meant by "in the wild" in lines 566 to 572, should be provided before, following lines 66-73 where the "controlled environment" concept is described. Also should be provided references of other papers defining the "in the wild" concept, the need of studies involving these settings and drawbacks of controlled settings.

4) The novelty stated in section 3.4 of Combining multiple socio-economic game paradigms into player-NPC interactions in one cohesive storyline, should be provided in the related work section. Also authors should give support to the argument of why is it worth/useful to combine those games and their descisión making into a single SG. What is the added value, in terms of the study purposes, when comparing to the single games? 

5)  when describing GEQ in the methodology section, some item/question examples should be provided and/or full questionnaires should be added as appendix. 

 

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf