The EmojiGrid as a Tool to Assess Experienced and Perceived Emotions

: In a recent study on food-evoked emotions, we observed that people often misunderstood the currently available a ﬀ ective self-report tools. We, therefore, developed a new intuitive and language-independent self-report instrument called the EmojiGrid: a rectangular response grid labeled with facial icons (emoji) that express di ﬀ erent degrees of valence and arousal. We found that participants intuitively and reliably reported their a ﬀ ective appraisal of food by clicking on the EmojiGrid, even without verbal instructions. In this study, we investigated whether the EmojiGrid can also serve as a tool to assess one’s own (experienced) emotions and perceived emotions of others. In the ﬁrst experiment, participants (N = 90) used the EmojiGrid to report their own emotions, evoked by a ﬀ ective images from a database with corresponding normative ratings (obtained with a 9-point self-assessment mannikin scale). In the second experiment, participants (N = 61) used the EmojiGrid to report the perceived emotional state of persons shown in di ﬀ erent a ﬀ ective situations, in pictures from a database with corresponding normative ratings (obtained with a 7-point Likert scale). For both experiments, the a ﬀ ective (valence and arousal) ratings obtained with the EmojiGrid show excellent agreement with the data provided in the literature (intraclass correlations of at least 0.90). Also, the relation between valence and arousal shows the classic U-shape at the group level. Thus, the EmojiGrid appears to be a useful graphical self-report instrument for the assessment of evoked and perceived emotions.

Using verbal questionnaires, people can report their affective state by rating or selecting the words that most closely reflect their current feelings.Since it does not seem to be clear what verbal emotion-assessment tools actually measure, the focus should be more on measures of core affect, such as valence and arousal [12].However, questionnaires have several shortcomings: (1) emotions are sometimes hard to express in words, and the words describing the emotions are typically ambiguous [13], (2) both the number and connotation of emotional words vary between languages and cultures and [14][15][16], and (3) individuals vary widely in their vocabulary and general language skills [12].Consequently, the description of emotions may be differently interpreted by people from different cultures and languages [17], and differences in emotion intensity, context, and other semantics among cultures may be lost in translation [18].Also, verbal tools require mental effort (interpretation) and are time-consuming to carry out (a disadvantage that increases when such tools have to be filled out multiple times throughout an experiment), making their application rather demanding for the user.
Graphical affective self-report tools are an attractive alternative to verbal instruments, since they enable users to report their feelings more intuitively through figural elements that represent their current affective state (for extensive discussions on the benefits of these tools see [19,20]).Instead of asking users to phrase their emotions, these tools use the human capability to intuitively and reliably relate graphical elements to human emotions [21][22][23][24].This holds true especially for iconic representations of facial expressions [25][26][27]: people can accurately identify discrete emotions from facial expressions [28] across different cultures [29].Visually expressed emotions are hypothesized to more closely resemble intuitively experienced emotions [30].Evidence for this hypothesis stems from electroencephalogram (EEG) experiments showing that emotion processing is faster for facial expressions than for emotional words ( [31][32][33]).Facial emoji (iconic faces showing different emotional expressions) have, therefore, recently become popular as self-report instruments [34].Emoji-based ratings scales have, for instance, been used to evaluate online training simulations [35], and the user experience of electronic questionnaires [36].Since emoji-based self-report tools do not need verbal labels, they do not require translation [15,16].While verbal labels trigger analytical and rational responses, emoji afford a more intuitive and affective response.Additional advantages of emoji-based tools are that they may also be used for children [37][38][39] or people who are illiterate [40,41].
The circumplex model of affect [42] suggests that emotions can be represented in a two-dimensional circular space by their valence (pleasantness; the degree of positive or negative affective response to a stimulus) and arousal (the degree of activation or deactivation) components.The self-assessment mannikin (SAM: [6]) is a widely used graphical tool for rating both valence and arousal.It allows users to report the valence, arousal, and dominance components of their affective state by indicating, from a set of human-like figures, the ones that most closely reflect their own feelings.Although the SAM is widely used, it has some practical drawbacks.First, people often misinterpret the emotions it depicts.Children especially tend to misunderstand the SAM [43,44].While the SAM's valence dimension is quite intuitive (a facial expression going from a frown to a smile), its dominance dimension (represented by its size) is harder to interpret, and the arousal dimension (which looks like an 'explosion' in the figure's stomach) is often misunderstood [11,45,46].Second, the SAM requires a successive assessment of valence and arousal.
We, therefore, introduced the EmojiGrid, which is an affective self-report tool based on facial emojis ( [19]; see Figure 1) In electronic messages and on web pages, facial emoji are often used to supplement or replace written text [47].In computer-mediated communication, people use facial emoji to more clearly and explicitly express their intentions [48].While people may find it hard to verbalize their emotions, they appear to communicate their affective experiences more spontaneously and intuitively using facial emoji [49].Although facial emoji are poly-interpretable [50,51] it has been found that similar facial expressions are typically associated with similar feelings [35,52], independent of the language of the observer [53].Facial emoji can represent a wide range of emotions, with different degrees of valence (e.g., angry face vs. smiling face) and arousal (e.g., sleepy face vs. excited face).The EmojiGrid enables users to report affective states with any degree of valence and arousal, in contrast to previous emoji-based affect rating scales that only varied along the valence dimension [35,36,54].
In previous studies, we found that participants intuitively used the EmojiGrid to report their food-related emotions without any further verbal instructions [19,55].Also, the EmojiGrid yielded a quadratic (U-shaped) relation between the mean (across individuals) valence and arousal ratings for food images [19,55], similar to the one that has repeatedly been reported in the literature for affective stimuli in other sensory modalities, such as movies, facial expressions, paintings, images, music, sounds, words, and odors [56,57].Hence, we concluded that the EmojiGrid might also be a more general instrument to assess human affective responses.In this study, we evaluated the EmojiGrid as a self-report tool to assess experienced (own, induced) emotions and perceived emotions (of others [58]).We measured experienced and perceived valence and arousal for images from two validated affective image databases using the EmojiGrid, and compared the results with the normative ratings that were obtained with conventional validated affective rating tools, and that were provided with these databases.
Psych 2019, 1, FOR PEER REVIEW 3 that were obtained with conventional validated affective rating tools, and that were provided with these databases.

Participants
From the Prolific database (https://prolific.ac)we recruited English speaking participants, aged between 18 and 35 years and without any known color vision deficiencies.
The TNO Ethics Committee reviewed and approved the experimental protocol (Ethical Approval Ref: 2017-012), which was in agreement with the Helsinki Declaration of 1975, as revised in 2013 [59].Participation in this study was voluntary.After completing the study, all participants received a compensation of one Euro in their Prolific account.

Demographics
Participants reported their age, gender, and nationality.

Valence and Arousal: The EmojiGrid
Valence and arousal were measured with the EmojiGrid (see Figure 1; this tool was introduced by Toet et al. in xxx).The EmojiGrid is a square grid (similar to the Affect Grid, [60]) that is labeled with emoji showing different facial expressions.Each side of the grid is labeled with five emoji, and there is one (neutral) emoji located in its center.Therefore, the grid contains 17 emoji in total.The central emoji serves as a neutral point (i.e., has a neutral expression).The facial expressions of the emoji along a horizontal (valence) axis vary from disliking (unpleasant), via neutral, to liking

Participants
From the Prolific database (https://prolific.ac)we recruited English speaking participants, aged between 18 and 35 years and without any known color vision deficiencies.
The TNO Ethics Committee reviewed and approved the experimental protocol (Ethical Approval Ref: 2017-012), which was in agreement with the Helsinki Declaration of 1975, as revised in 2013 [59].Participation in this study was voluntary.After completing the study, all participants received a compensation of one Euro in their Prolific account.

Demographics
Participants reported their age, gender, and nationality.

Valence and Arousal: The EmojiGrid
Valence and arousal were measured with the EmojiGrid (see Figure 1; this tool was introduced by Toet et al. in [19]).The EmojiGrid is a square grid (similar to the Affect Grid, [60]) that is labeled with emoji showing different facial expressions.Each side of the grid is labeled with five emoji, and there is one (neutral) emoji located in its center.Therefore, the grid contains 17 emoji in total.The central emoji serves as a neutral point (i.e., has a neutral expression).The facial expressions of the emoji along a horizontal (valence) axis vary from disliking (unpleasant), via neutral, to liking (pleasant), and the arousal component of their expression increases gradually along the vertical axis.The expressions of the emoji are characterized by their eyebrows, eyes, and mouth, and are inspired by the facial action coding system [61].The opening of the mouth and the shape of the eyes represent the degree of arousal, while the concavity of the mouth, the orientation, and curvature of the eyebrows, and the vertical position of these features in the facial area, correspond to the valence dimension.Users can report their affective state by placing a checkmark at the appropriate location on the grid.Previous validation studies confirmed that the facial expressions of the emoji and their arrangement over the valence-arousal space agreed with the users' intuition [19].

Procedure
Both experiments in this study were performed as (anonymous) online surveys.Each survey started by thanking the participants for their interest in the experiment and then continued with the presentation of some general information about the experiment.The participants were instructed to perform the experiment on a (laptop) computer and not on a device with a smaller screen (such as a smartphone).They were also asked to make their web browser full-screen and to avoid any external distractions.The participants were then informed that they would be presented with different images over the course of the experiment, and they were asked to either rate their own feelings elicited by each image (experienced emotions: Experiment I), or to rate the emotions that were being felt by the people shown in each image (perceived emotions of others: Experiment II).It was emphasized that there were no correct or incorrect answers.After the participants signed a printed informed consent form, they reported some demographic variables (nationality, age, gender).
Next, the participants were introduced to the EmojiGrid response tool and were told how they could use this tool to report their (experienced or perceived) affective rating for each image that they would see.To measure evoked emotions (Experiment I) the instructions merely stated: "Click on a point in the grid that best matches your feelings towards the picture."To measure perceived emotions (Experiment II) the instructions were "Click on a point of the grid that best indicates how the person(s) in the picture feel(s)."No further explanation was given since we wanted the participants to use the EmojiGrid tool intuitively.Then they performed two practice trials to familiarize themselves with the EmojiGrid and its use.The actual experiment started directly following these practice trials.The images were presented in random order throughout the experiment.After seeing each image, the participants responded by clicking on the EmojiGrid (see Figure 1).Immediately after responding, the next image appeared.Participants performed the experiment at their own pace.On average, both experiments typically lasted about 10 min.

Data Analysis
The response data (i.e., the horizontal or valence and vertical or arousal coordinates of the check marks on the EmojiGrid) were quantified as integers between 0 and 550, and then scaled between 1 and 9 for comparison with previous results obtained with a 9-point Likert scale (Experiment I), or between 1 and 7 for comparison with a 7-point Likert scale (Experiment II).
IBM SPSS Statistics 25 (www.ibm.com) for Windows was used to perform all statistical analyses.Intraclass correlation coefficient (ICC) estimates and their 95% confident intervals were based on a mean-rating (k = 3), consistency, 2-way mixed-effects model [62,63].ICC values less than 0.50 were indicative of poor reliability, values between 0.50 and 0.75 indicated moderate reliability, values between 0.75 and 0.90 indicated good reliability, while values greater than 0.90 indicated excellent reliability [62].For all other analyses, a probability level of p < 0.05 was considered to be statistically significant.
For each of the images, we computed the mean valence and arousal responses over all participants.We used Matlab 2019a (www.mathworks.com) to investigate the relation between the (mean) valence and arousal ratings and to plot the data.The Curve Fitting Toolbox (version 3.5.7) in Matlab was used to compute a least-squares fit of a quadratic function to the data points.All results from this study are freely available as supplementary material from the Open Science Framework (OSF) repository at https://osf.io/4v7zq.

Experiment I: Experienced Emotions
This experiment was performed to investigate whether the EmojiGrid can serve as a self-report instrument for the assessment of image-evoked emotions.Participants reported their experienced valence and arousal for a selection of images from a validated image database by marking corresponding locations on the EmojiGrid (see Figure 2 for a screenshot of the screen layout during the rating phase of this experiment).The results were compared with the corresponding normative ratings provided for the images in this database.

Experiment I: Experienced Emotions
This experiment was performed to investigate whether the EmojiGrid can serve as a self-report instrument for the assessment of image-evoked emotions.Participants reported their experienced valence and arousal for a selection of images from a validated image database by marking corresponding locations on the EmojiGrid (see Figure 2 for a screenshot of the screen layout during the rating phase of this experiment).The results were compared with the corresponding normative ratings provided for the images in this database.

Stimuli
The stimuli consisted of a subset of 90 images from the Nencki Affective Picture System (NAPS [64]; see http://lobi.nencki.gov.pl).The NAPS is a standardized set of 1356 realistic, emotionallycharged high-quality (minimal resolution of 1600 by 1200 pixels) photographs divided into five general categories (people, faces, animals, objects, and landscapes) with associated normative ratings for valence, arousal and approach-avoidance [64,65].
Riegel, et al. [66] recently performed a study in which a sample of 39 students (aged between 18 and 35, mean age = 23.5, SD = 4.7) of various European nationalities rated valence and arousal for a subset of 170 NAPS images using a 9-point SAM (self-assessment mannikin [6]) scale.For this study, we selected a subset of these 170 images with mean valence and arousal ratings (as reported in [66]) maximally covering the dimensional affective space.The selection contained 13 images of animals, 25 faces, 11 landscapes, 24 objects, and 17 images of people.Figure 2 shows an example image in combination with the EmojiGrid.

Stimuli
The stimuli consisted of a subset of 90 images from the Nencki Affective Picture System (NAPS [64]; see http://lobi.nencki.gov.pl).The NAPS is a standardized set of 1356 realistic, emotionally-charged high-quality (minimal resolution of 1600 by 1200 pixels) photographs divided into five general categories (people, faces, animals, objects, and landscapes) with associated normative ratings for valence, arousal and approach-avoidance [64,65].
Riegel et al. [66] recently performed a study in which a sample of 39 students (aged between 18 and 35, mean age = 23.5, SD = 4.7) of various European nationalities rated valence and arousal for a subset of 170 NAPS images using a 9-point SAM (self-assessment mannikin [6]) scale.For this study, we selected a subset of these 170 images with mean valence and arousal ratings (as reported in [66]) maximally covering the dimensional affective space.The selection contained 13 images of animals, 25 faces, 11 landscapes, 24 objects, and 17 images of people.Figure 2 shows an example image in combination with the EmojiGrid.

Participants
A total of 90 persons from six different countries (40 from the United Kingdom, 16 from Italy, 13 from Spain, 8 from Germany, 8 from the Netherlands, and 5 from France), comprising 45 females (mean age = 26.5 years, SD = 5.1) and 45 males (mean age = 26.6 years, SD = 4.4), participated in this experiment.

Results
Figure 3 shows the relation between the mean valence and arousal ratings for the 90 Nencki images tested, as measured with the EmojiGrid in this study and with a 9-point SAM scale by Riegel et al. [66].The curves represent least-squares quadratic fits to the data points.The adjusted R-squared values (representing the agreement between the data and the quadratic fits) are respectively 0.54 and 0.65, indicating good fits.This figure shows that the relation between valence and arousal ratings provided by both self-assessment methods is closely described by a quadratic (U-shaped) relation at the nomothetic (group) level.

Results
Figure 3 shows the relation between the mean valence and arousal ratings for the 90 Nencki images tested, as measured with the EmojiGrid in this study and with a 9-point SAM scale by Riegel et al. [66].The curves represent least-squares quadratic fits to the data points.The adjusted R-squared values (representing the agreement between the data and the quadratic fits) are respectively 0.54 and 0.65, indicating good fits.This figure shows that the relation between valence and arousal ratings provided by both self-assessment methods is closely described by a quadratic (U-shaped) relation at the nomothetic (group) level.obtained with the self-assessment mannikin (SAM) (blue dots: [66]) and with the EmojiGrid (red dots: this study).The curves represent quadratic fits to the corresponding data points.
To quantify the agreement between the ratings obtained with the Emojigrid (present study) and with the SAM [66], we computed intraclass correlation coefficient (ICC) estimates and their 95% confidence intervals for the mean valence and arousal ratings between both studies.The ICC for valence was 0.950 (with a 95% confidence interval ranging between 0.924 and 0.967) and the ICC for arousal was 0.916 (with a 95% confidence interval ranging between 0.873 and 0.945), indicating excellent reliability (even though both studies were performed through the Internet and could not control for many factors, as in a lab experiment).

Experiment II: Perceived Emotions
This experiment was performed to investigate whether the EmojiGrid can serve as a self-report instrument for the assessment of perceived emotions of others.Participants reported their perceived valence and arousal for persons shown in a wide range of different daily life situations, as depicted in the drawings from the Pictures with Social Context and Emotional Scenes (PiSCES) database [67] Figure 4 shows a screenshot of the screen layout during the rating phase of this experiment.The results were compared with the corresponding normative ratings provided for the images in this database.Relation between the mean valence and arousal ratings for images from the Nencki database, obtained with the self-assessment mannikin (SAM) (blue dots: [66]) and with the EmojiGrid (red dots: this study).The curves represent quadratic fits to the corresponding data points.
To quantify the agreement between the ratings obtained with the Emojigrid (present study) and with the SAM [66], we computed intraclass correlation coefficient (ICC) estimates and their 95% confidence intervals for the mean valence and arousal ratings between both studies.The ICC for valence was 0.950 (with a 95% confidence interval ranging between 0.924 and 0.967) and the ICC for arousal was 0.916 (with a 95% confidence interval ranging between 0.873 and 0.945), indicating excellent reliability (even though both studies were performed through the Internet and could not control for many factors, as in a lab experiment).

Experiment II: Perceived Emotions
This experiment was performed to investigate whether the EmojiGrid can serve as a self-report instrument for the assessment of perceived emotions of others.Participants reported their perceived valence and arousal for persons shown in a wide range of different daily life situations, as depicted in the drawings from the Pictures with Social Context and Emotional Scenes (PiSCES) database [67] Figure 4 shows a screenshot of the screen layout during the rating phase of this experiment.The results were compared with the corresponding normative ratings provided for the images in this database.
activity (e.g., eating, reading, playing, talking, etc.) in a familiar situational context.Half of the pictures show a single person, and the other half contain two or more persons, to represent the range of situations and activities that people typically encounter in real-life.The normative ratings on perceived emotional valence, arousal and social engagement that are provided for each image in the PiSCES database, were collected by Teh, Yap and Liow [67] for 62 young adults (30 males, 32 females, mean age 22 years) using 7-point Likert scales.

Results
Figure 5 shows the relation between the mean valence and arousal ratings for all 203 PiSCES images, as measured with the EmojiGrid in this study and with a 7-point Likert scale in the study by Teh, Yap and Liow [67].The curves represent least-squares quadratic fits to the data points.The adjusted R-squared values are respectively 0.61 and 0.63, indicating good fits.This figure shows that the relation between the mean valence and arousal ratings provided by both self-assessment methods is closely described by a quadratic (U-shaped) relation at the nomothetic (group) level.To quantify the agreement between the ratings obtained with the EmojiGrid (present study) and with the 7-point Likert scales [67], we computed intraclass correlation coefficient (ICC) estimates and their 95% confidence intervals for the mean valence and arousal ratings between both studies.The ICC for valence was 0.987 (with a 95% confidence interval ranging between 0.982 and 0.990) and the ICC for arousal was 0.902 (with a 95% confidence interval ranging between 0.871 and 0.926), indicating excellent reliability.

Stimuli
The stimuli used in this experiment were all 203 images from the PiSCES database [67].PiSCES consists of 203 black-and-white line drawings showing people in various daily life situations.The database has specifically been designed for studies on the interpretation of emotion in others (perceived emotion).The pictures vary systematically on emotional valence (positive, negative, and neutral) and social engagement.All pictures show one or more person(s) performing an everyday activity (e.g., eating, reading, playing, talking, etc.) in a familiar situational context.Half of the pictures show a single person, and the other half contain two or more persons, to represent the range of situations and activities that people typically encounter in real-life.The normative ratings on perceived emotional valence, arousal and social engagement that are provided for each image in the PiSCES database, were collected by Teh, Yap and Liow [67] for 62 young adults (30 males, 32 females, mean age 22 years) using 7-point Likert scales.

Results
Figure 5 shows the relation between the mean valence and arousal ratings for all 203 PiSCES images, as measured with the EmojiGrid in this study and with a 7-point Likert scale in the study by Teh, Yap and Liow [67].The curves represent least-squares quadratic fits to the data points.The adjusted R-squared values are respectively 0.61 and 0.63, indicating good fits.This figure shows that the relation between the mean valence and arousal ratings provided by both self-assessment methods is closely described by a quadratic (U-shaped) relation at the nomothetic (group) level.[67]) and with the EmojiGrid (red dots: this study).The curves represent quadratic fits to the corresponding data points.

Discussion
Using the EmojiGrid, participants subjectively reported their own (experienced) emotions and the perceived emotions of others, for images from two validated affective databases.We compared the results with the corresponding normative ratings provided with these databases.Both for experienced (own) and for perceived (others) emotions, the subjective valence and arousal ratings obtained with the EmojiGrid show excellent agreement with the data provided in the literature and obtained with alternative methods (a 9-point SAM scale and a 7-point Likert scale): all intraclass correlation coefficients exceeded 0.90.In addition, the relation between the mean valence and arousal ratings obtained with the EmojiGrid show the classic U-shape at the nomothetic level, both for experienced and for perceived emotions.Hence, it appears that the EmojiGrid can serve as a valid alternative to these existing affective self-report tools.In contrast to other methods, the EmojiGrid requires no verbal labels (is intuitive and language independent) and affords efficient responding (only a single click).

Limitations
This study has several limitations.We only measured valence and arousal through subjective self-report.Since no objective truth was available, we compared our present results with the normative ratings provided with the image databases that were also obtained with subjective rating methods that have their own limitations.For instance, the normative rating provided with the NAPS image database were collected using the SAM, which has important limitations, amongst others, because its dominance dimension is difficult to interpret, the arousal dimension is often misunderstood [11,45,46] and the valence and arousal ratings are assessed sequentially.Future To quantify the agreement between the ratings obtained with the EmojiGrid (present study) and with the 7-point Likert scales [67], we computed intraclass correlation coefficient (ICC) estimates and their 95% confidence intervals for the mean valence and arousal ratings between both studies.The ICC for valence was 0.987 (with a 95% confidence interval ranging between 0.982 and 0.990) and the ICC for arousal was 0.902 (with a 95% confidence interval ranging between 0.871 and 0.926), indicating excellent reliability.

Discussion
Using the EmojiGrid, participants subjectively reported their own (experienced) emotions and the perceived emotions of others, for images from two validated affective databases.We compared the results with the corresponding normative ratings provided with these databases.Both for experienced (own) and for perceived (others) emotions, the subjective valence and arousal ratings obtained with the EmojiGrid show excellent agreement with the data provided in the literature and obtained with alternative methods (a 9-point SAM scale and a 7-point Likert scale): all intraclass correlation coefficients exceeded 0.90.In addition, the relation between the mean valence and arousal ratings obtained with the EmojiGrid show the classic U-shape at the nomothetic level, both for experienced and for perceived emotions.Hence, it appears that the EmojiGrid can serve as a valid alternative to these existing affective self-report tools.In contrast to other methods, the EmojiGrid requires no verbal labels (is intuitive and language independent) and affords efficient responding (only a single click).

Limitations
This study has several limitations.We only measured valence and arousal through subjective self-report.Since no objective truth was available, we compared our present results with the normative ratings provided with the image databases that were also obtained with subjective rating methods that have their own limitations.For instance, the normative rating provided with the NAPS image database were collected using the SAM, which has important limitations, amongst others, because its dominance dimension is difficult to interpret, the arousal dimension is often misunderstood [11,45,46] and the valence and arousal ratings are assessed sequentially.Future studies using the EmojiGrid to measure the affective appraisal of perceived and experience emotions should, therefore, include physiological (objective) measures to obtain more objective reference data.
The experiments reported in this study were both performed online, and therefore guaranteed no control over the experimental conditions.However, it has been shown that online surveys typically yield similar results to those of lab studies [75][76][77], while limiting several disadvantages that are typically associated with central location studies.
In contrast to the SAM, the EmojiGrid currently does not measure dominance.Future studies should investigate whether this can be resolved by interactively scaling the size of the emojis (e.g., by using the mouse wheel).
We did not include participants younger than 18 years in this study.However, it is likely that our findings will also apply to young people since it has been found that both the use of emoji [78] and their interpretation [79] are independent of age.Their intuitive visual display of emotion also makes emoji particularly suitable both for use with children who may not have the vocabulary to convey all their emotions [37][38][39] and with individuals with variable education levels [12].
We did not investigate cultural differences in this study.It has, for instance, been observed that Japanese focus on the eyes, while Americans focus on the mouth when interpreting facial emotions [80].However, given that emotions in facial expressions, gestures and body postures are to a large extent similarly perceived across different cultures [28,29], cross-cultural differences in the interpretation of emoji may also be smaller than the influences of culture and language on verbal affective self-report tasks [52,81].A previous cross-cultural study [55] showed that the EmojiGrid was able to pick up established cultural response biases (e.g., the Western extreme response style vs. the Eastern middle response style), suggesting that the cross-cultural interpretation was largely similar and outweighed additional variations due to interpretation differences.However, the capability to perceive emotions from emoji may also depend on their frequency of use and familiarity [82].Since we recruited our participants online and, therefore, were probably regular Internet users, we assume that they were at least to some degree familiar with emojis.
Finally, in this study we found that the EmojiGrid demonstrates good convergent validity with two established affective self-report methods (the 7-point labeled Likert scale and the 9-point SAM scale) for the assessment of own and perceived emotions.However, further research using more diverse (preferably multisensory) affective stimuli is needed to assess its full (incremental, discriminant, and ecological) validity.

Conclusions
Overall, our present results show that self-reported valence and arousal ratings obtained with the EmojiGrid resemble those obtained with other validated affective self-report tools, underlining the general validity of this tool.Since the scales used in the different methods may vary somewhat locally (since corresponding anchor points on the scales need not be related to the same emotional state), and since their neutral points need not coincide, there may be some variation in the agreement between the

Figure 1 .
Figure 1.The EmojiGrid.The facial expressions of the emoji along the horizontal (valence) axis gradually change from unpleasant, via neutral, to pleasant, while the arousal component of the facial expressions gradually increases in the vertical (arousal) direction.

Figure 1 .
Figure 1.The EmojiGrid.The facial expressions of the emoji along the horizontal (valence) axis gradually change from unpleasant, via neutral, to pleasant, while the arousal component of the facial expressions gradually increases in the vertical (arousal) direction.

Figure 2 .
Figure 2. Screen layout during the rating phase in Experiment I, showing the image to be rated (left) and the EmojiGrid response tool (right).The red star indicates the location in the grid where the participant clicked.

A
total of 90 persons from six different countries (40 from the United Kingdom, 16 from Italy, 13 from Spain, 8 from Germany, 8 from the Netherlands, and 5 from France), comprising 45 females (mean age = 26.5 years, SD = 5.1) and 45 males (mean age = 26.6 years, SD = 4.4), participated in this experiment.

Figure 2 .
Figure 2. Screen layout during the rating phase in Experiment I, showing the image to be rated (left) and the EmojiGrid response tool (right).The red star indicates the location in the grid where the participant clicked.

Figure 3 .
Figure 3. Relation between the mean valence and arousal ratings for images from the Nencki database,

Figure 3 .
Figure 3. Relation between the mean valence and arousal ratings for images from the Nencki database, obtained with the self-assessment mannikin (SAM) (blue dots:[66]) and with the EmojiGrid (red dots: this study).The curves represent quadratic fits to the corresponding data points.

Figure 4 .
Figure 4. Screen layout during the rating phase in Experiment II, showing the image to be rated (left) and the EmojiGrid response tool (right).The red star indicates the location in the grid where the participant clicked.

Figure 4 .
Figure 4. Screen layout during the rating phase in Experiment II, showing the image to be rated (left) and the EmojiGrid response tool (right).The red star indicates the location in the grid where the participant clicked.

Figure 5 .
Figure 5. Relation between the mean valence and arousal ratings for images from the Pictures with Social Context and Emotional Scenes (PiSCES) database, obtained with the a 7-point Likert rating scale (blue dots:[67]) and with the EmojiGrid (red dots: this study).The curves represent quadratic

Figure 5 .
Figure 5. Relation between the mean valence and arousal ratings for images from the Pictures with Social Context and Emotional Scenes (PiSCES) database, obtained with the a 7-point Likert rating scale (blue dots: [67]) and with the EmojiGrid (red dots: this study).The curves represent quadratic fits to the corresponding data points.