The EmojiGrid as a Tool to Assess Experienced and Perceived Emotions
Round 1
Reviewer 1 Report
l. 49
how can authors say already here (without reference) that the users are enabled ‘to report their feelings efficiently and intuitively’?
l. 59
What are examples of ‘complex (mixed) emotions that can sometimes be difficult to verbalize’
l. 78
This sentence looks lost (or may not have a complete sentence grammar?)
‘Emoji with facial expressions that are similar typically associated with similar feelings [28,40], independent of the language of the observer [41]. ‘
l.98
Is the neutralness of the middle emoji’s tested? They look ‘surprised’ to me, especially the top one (a bit resembling Munch’s ‘The Scream’ painting).
l. 172
‘Participants implicitly reported’: what’s meant by this ‘implicit’ reporting?
General comment
The y-axis is meant to model ‘intensity’, and ‘arousal’, but are these two concepts the same? I have the impression that both are used in this paper, but it’s not always clear if the y-axis is an intensity scoring of the emotion on the x-axis, or that the y-axis is to score a different concept, viz. ‘arousal’, which is independent of the x-axis.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The EmojiGrid as a tool to assess experienced and perceived emotions
This manuscript is generally well written. The rationale for the present study is set up well, and there is a need for effective measures of emotion. However, there are several concerns and reservations with the submitted manuscript that must be addressed before it can be accepted for publication.
Good abstract, with a clear rationale for the study. However, given the simplicity of the study, participating numbers are low.
Introduction
The notion that ‘it does not seem to be clear what verbal emotion-assessment tools actually measure’ (page 1 line 45 – page 2 line 46-47) should appear earlier in the introduction before then going on to identify reported challenges with verbal questionnaires. Whilst limitations with existing verbal questionnaires are noted, there is a need to explore in greater depth factors that have been shown to influence the ability to rate own (induced) emotions, and perceived emotion of persons shown in different affective situations. This presents an opportunity to identify how the emojigrid might help overcome these.
Full stop missing after [29] on page 2 line 56.
How can an emoji measure mixed emotions? Can an emoji capture the intensity of emotional experience with sufficient sensitivity, particularly in view of use of the circumplex model of affect in setting the context for this study.
Methods
If the experiment was performed as an anonymous online survey, how did participants received compensation of one Euro for completing the study?
The EmojiGrid presents 17 emoji, with the central emoji presenting a neutral point. Given the aim of the submitted manuscript was to evaluate the Emojigrid as a tool to assess experienced and perceived emotions, it is important to present evidence that the facial emojis are interpreted as expected in terms of valence and arousal. Thus supporting the contention that the EmojoGrid is an affective self-report tool based on facial emojis? (p2 line 73-74).
The study used a subset (presumably the 90 noted on page 5 line 176) of 170 images covering the entire dimensional affective space from the Necki Affective Picture system. These images have associated normative ratings for valence, arousal and approach-avoidance. However, research indicates an individual’s appraisal of an affective situation influences perceived emotions (Burgess et al., 2001), and as the affective situation may be appraised differently this may influence the perceived emotions of others. As such, despite normative data, is gathering appraisal of the affective situation something that should be addressed?
The instructions stated ‘Click a point on the grid that best matches your [how the person(s) in the picture) feeling towards the picture’ – was there a tendency to orientate towards one of the 17 images rather than use the full grid space? Was there no attempt to explain the valence and arousal characteristics of the grid?
Study 1 results
How were points selected on the Emojigrid converted into a score for comparison with the 9-point SAM scale? Please explain the quadratic fits and how these are calculated. These important details are omitted. If the images were intended to cover the entire dimensional affective space, then why do valence and arousal ratings not reflect this?
The introduction is critical of the self-assessment mannikin (SAM), noting that ‘people often misinterpret the emotions it depicts’ (page 2 line 67). As such, can it be argued that using this measure in providing corresponding normative ratings present a limitation? Indeed, given this limitation, is the finding that the EmojiGrid shows excellent agreement with the SAM a good thing?
Experiment II
More information is required on the PiSCES database, an illustrative image would be beneficial. Were all 203 images used in the present study?
Why use a different population for study 2? Why not the same participants? The point regarding appraisal of a situation holds in rating emotional valence and arousal.
There appears to be an error, on page 6 line 226 it is suggested that PiSCES has 203 images, thereafter it is suggested there are 230.
Methodological details such as where the study was completed are absent, for example, was this study also undertaken remotely?
Concerns regarding the results of study 1 hold for study 2.
Discussion and conclusion
The limitations noted above should be accommodated in the discussion of results. If individuals experience a range of emotions in response to a situation, how can this be captured using the EmojiGrid, does this present a limitation in oversimplifying emotion experiences? The discussion fails to acknowledge limitations with the study design, the remote nature of its administration for example. Using SAM, a measure critiqued in the present study as a comparator.
Author Response
Please see the attachment.
we would like to thank the reviewer for all the useful comments and suggestions, and for spending valuable time on the review of our manuscript
Author Response File: Author Response.pdf
Reviewer 3 Report
The authors present an interesting follow up study on the validity of a new method for the assessment of the valence and arousal of emotions. In general I like the paper and I truly belief in the added value of the EmojiGrid, but I don’t like the way in which the EmojiGrid is presented. In my opinion, the EmojiGrid is too positively presented, certainly in the introduction (the tone of voice is better/more nuanced in the last section). A more nuanced discussion of the EmojiGrid is advisable. For example, your claim that emoji are is not well and correctly supported. The references you use (e.g. ref 14 & 15) are studies on the verbal scales and as far as I know, emoji’s don’t have a universal, ubiquitous meaning neither. Ref 41 is also only about European languages and most likely only covers predominantly Western people. So, the claim that emoji’s don’t need to be translated is at least not supported, neither by previous research, nor by your research, and in my opinion even false.
See e.g.
Takahashi, K., Oishi, T., & Shimada, M. (2017). Is☺ smiling? Cross-cultural study on recognition of emoticon’s emotion. Journal of Cross-Cultural Psychology, 48(10), 1578-1586.
Miller, H. J., Thebault-Spieker, J., Chang, S., Johnson, I., Terveen, L., & Hecht, B. (2016, March). “Blissfully Happy” or “Ready toFight”: Varying Interpretations of Emoji. In Tenth International AAAI Conference on Web and Social Media.
Tigwell, G. W., & Flatla, D. R. (2016, September). Oh that's what you meant!: reducing emoji misunderstanding. In Proceedings of the 18th international conference on human-computer interaction with mobile devices and services adjunct (pp. 859-866). ACM.
The popularity of emoticons also differs between cultures. See e.g.
Kavanagh, B. (2010). A cross-cultural analysis of Japanese and English non-verbal online communication: The use of emoticons in weblogs. Intercultural Communication Studies, 19(3), 65-80.
Park, J., Baek, Y. M., & Cha, M. (2012). Cross-cultural comparison of nonverbal cues in emoticons on twitter: Evidence from big data analysis. Journal of Communication, 64(2), 333-354.
Other claims on the advantages of the EmojiGrid are not well supported neither (e.g. better suited for children, illiterate people, …)
You criticize SAM because the validity of its dominance dimension (represented by its size) may be low. So far so good, but the EmojiGrid does not assess dominance at all. I don’t find this 100% fair. The weaknesses and shortcomings of the EmojiGrid should be discussed as well. In line with that, you criticize the SAM scale in that it’s not so good in assessing arousal. However, in the data you report, the EmojiGrid seems not to do a better job on that perspective (if anything, the EmojiGrid seems even to perform worse on the arousal dimension).
In addition, I also find that the discussion at the end of the manuscripts should be much better elaborated (at least quadrupled in size). You only provide another piece of evidence that the EmojiGrid may be a valid tool for the assessment of the valence and arousal of emotions. Yet, much more work needs to be done before we can be sure the EmojiGrid is a good or better way to assess emotions. Unanswered questions include its incremental validity, discriminant validity, ecological validity, … . Have a look to e.g. Borsboom & Mellenbergh (2004) for an excellent discussion on validity (and also to the articles that critics Borsboom & Mellenbergh’s work.)
I wish the authors good luck in revising this manuscript. I look forward to receiving the revised paper!
Author Response
we would like to thank the reviewer for all the useful comments and suggestions, and for spending valuable time on the review of our manuscript
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
The additional information and clarification offered in response to reviewer feedback have strengthened this manuscript. The limitations section and recommendations for further research is a welcomed addition. However, the limitations of the SAM should be more precisely acknowledged when describing the use of this measure for normative ratings. ‘The best we have at present’ is a poor defence, and this should underpin a call for further research to enhance the psychometric integrity of measures used. This would be a final recommended change prior to publication.
I wish the author(s) well in continuing this line of investigation.
|
The introduction is critical of the self-assessment mannikin (SAM), noting that ‘people often misinterpret the emotions it depicts’ (page 2 line 67). As such, can it be argued that using this measure in providing corresponding normative ratings present a limitation? Indeed, given this limitation, is the finding that the EmojiGrid shows excellent agreement with the SAM a good thing? |
Although the SAM has its shortcomings, it is currently considered as a generally accepted and validated tool. The normative ratings provided by the SAM are currently the only available standard and as such the best we have. |
Author Response
Dear reviewer,
thank you for your helpful comments and suggestiosn.
we now more precisely acknowledge the SAM's shortcomings in the limitations section of the discussion.
kind regards,
Lex Toet
Reviewer 3 Report
The issues that I raised in my previous review were adequately addressed. I don't have any further question.
I'm looking forward to using the EmojiGrid :-)
Author Response
Dear reviewer,
thank you for your review, helpful comments, and positive feedback!
Kind regards,
Lex Toet
