Emojis with a Stable Interpretation Among Individuals in Japan

Abstract

Emojis are widely used to measure users’ emotional states; however, their interpretations can vary over time. While some emojis exhibit consistent meanings, others may be perceived differently at other times. To utilize emojis as indicators in consumer studies, it is essential to ensure that their interpretations remain stable over time. However, the long-term stability of emoji interpretations remains uncertain. Therefore, this study aims to identify emojis with stable and unstable interpretations. We collected 256 responses in an online survey twice, one week apart, in which participants rated the valence and arousal levels of 74 facial emojis on a nine-point scale. Wilcoxon rank-sum tests showed unstable interpretations for seven of the seventy-four emojis. Further, a hierarchical cluster analysis categorized 67 stable emojis into the following four clusters based on valence and arousal dimensions: strong positive sentiment, moderately positive sentiment, neutral sentiment, and negative sentiment. Consequently, we recommend the use of the 67 emojis with stable interpretations as reliable measures of emotional states in consumer studies.

1. Introduction

Emojis, a powerful non-verbal communication tool, are widely utilized worldwide to convey emotional states in daily communication. Recently, they have also been incorporated into research fields such as consumer studies to assess users’ emotions. For instance, human facial emojis have been employed to evaluate food products and gauge participants’ preferences (e.g., da Quinta et al., 2023; Jaeger et al., 2019; Jaeger et al., 2018). Compared to traditional text-based methods, emojis offer advantages such as ease of response and accessibility for individuals across different linguistic backgrounds. Therefore, a deeper understanding of how each emoji corresponds to specific human emotions can enhance their applicability in research.

In order for emojis to be used as appropriate indicators to assess users’ emotional responses, emojis need to be consistent with people’s emotions even at different times. Multiple studies in recent years have consistently shown that most emojis match well with people’s emotions (da Quinta et al., 2023; Jaeger et al., 2018, 2019; Kaneko et al., 2019; Kutsuzawa et al., 2022a, 2022b; Sick et al., 2020). For example, Kutsuzawa et al. (2022a) examined the interpretation of emotional states expressed by 74 emojis from the perspective of core affect among 1082 adults. They found that these emojis were classified into six groups based on core affect and correspond to various human emotions. Additionally, Jaeger et al. (2019) examined the emotional meanings of 33 emojis using open-ended questions among 1085 participants and found that these emojis were perceived as expressing a wide range of human emotions. However, all of these studies have evaluated the relationship between emojis and emotion only once and have never evaluated their stability over time. Schmid and Schmid Mast (2010) reported that the interpretation of human facial expressions is influenced by the interpreter’s mood at the time of assessment. This suggests that the interpretation of emojis may change from time to time. Therefore, the present study aimed to assess the stability of emoji interpretations over time.

The purpose of this study was, therefore, to understand the emojis with stable and unstable interpretations over time. A previous study indicated that the recognition of mood-incongruent expressions tends to be hampered, but neutral expressions were not affected by interpreters’ moods. Based on this, we hypothesize that emojis representing strong positive or strong negative emotions are more likely to have unstable interpretations over time, as they may be more influenced by the respondent’s mood at the time of assessment. To test this hypothesis, participants were asked to interpret the same emojis twice, with a one-week interval, and the consistency of their responses was analyzed. Emojis that exhibited a statistically significant difference between the two responses were considered to have an unstable interpretation, while those without a significant difference were deemed stable. In this study, young adults were selected as participants for the following three reasons: (1) Prior research has exhibited that emoji interpretations tend to differ between young and middle-aged adults (Kutsuzawa et al., 2022b); (2) Young adults are the largest users of emojis compared to other age groups (Evans, 2015); (3) As the first study to examine the stability of emoji interpretation over time, it is essential to establish fundamental findings prior to expanding to other demographics.

2. Materials and Methods

2.1. Participants

An online survey was conducted in Tokyo, Japan, involving 256 valid participants aged 20–39 years (responses were collected to ensure no gender bias). A review by Oshio (2016) on the stability of psychological indicator interpretation noted that most previous research has been conducted with fewer than 300 subjects. Therefore, the sample size for this study was set at approximately 250, and the data was obtained from an online research company. The participants were recruited from an online panel maintained by a marketing research firm (https://candc.co.jp/) and received an honorarium of JPY 200 (approximately USD 1.3). All participants were fluent in Japanese, the language of the survey. Ethical approval was obtained prior to data collection, and eligible participants were informed about data confidentiality. Informed consent was obtained online prior to their participation in the study. This study was conducted in accordance with the World Medical Association’s Declaration of Helsinki, and all study protocols were reviewed and determined that an ethical review was not required for this study by the local institutional review board (Committee on Ergonomic Experiments of the National Institute of Advanced Industrial Science and Technology).

2.2. Emojis Used in This Study

This study utilized human facial emojis, similar to those employed in prior studies (Jaeger et al., 2019; Kutsuzawa et al., 2022a, 2022b), as facial emojis are the most commonly utilized category (Swiftkey, 2015). To build upon prior studies exploring emojis as an emotion measurement index, we selected the same emojis utilized in prior studies (Kutsuzawa et al., 2022a, 2022b). Specifically, out of the 89 facial emojis available on Twemoji, 74 were selected. We excluded 15 (e.g., 🤒 and 🤕) owing to difficulties in defining their emotional meanings during the preliminary survey (see the Supplementary Materials for more information). Since emoji designs vary slightly across platforms (e.g., Twitter, Instagram, Facebook, and WhatsApp), this study utilized Twitter’s emoji designs, as their graphics license is clearly defined (licensed under CC-BY 4.0: https://creativecommons.org/licenses/by/4.0/ (accessed on 8 January 2025)). The emojis were saved as image files and displayed on a screen at an appropriate size (2.16 × 2.16 cm) to ensure clarity. To prevent bias in participant evaluations, no additional information (e.g., emoji labels) was provided.

2.3. Questionnaire

The survey was administered to the same participants on two occasions, with a one-week interval between sessions. The first session collected data on participants’ sociodemographic characteristics and assessed their valence and arousal ratings for each emoji, following the methodology of prior studies (Jaeger et al., 2019; Kutsuzawa et al., 2022a, 2022b). For emoji interpretation, participants responded to the prompt “Please tell us your intended emotional state when using the following emojis in daily life (messages, social networking, etc.)”. Then they rated each emoji on a nine-point Likert scale for valence (“Do you think the emotions indicated by the emojis are pleasant or unpleasant?” where 1 = displeasure and 9 = pleasure) and arousal (“How much emotional intensity do you think the emojis express?” where 1 = weak and 9 = strong). To minimize the participant workload, each individual rated only 30 out of the 74 emojis. To ensure balanced data collection across all emojis, 16 predefined presentation patterns were created and participants were randomly assigned to one of these patterns. After one week, participants rated the same set of emojis again for valence and arousal. Each emoji was evaluated by a minimum of 96 participants.

To verify response validity, two dummy questions were included after every 10 questions (i.e., Questions 11 and 21) following Miura and Kobayashi (2015). These questions were designed to be easily answerable if the instructions were properly understood (e.g., “What is the subject of this questionnaire that you are being asked to answer?”). Participants who provided incorrect responses to these questions were excluded from the study.

2.4. Data Analysis

To examine the stability of emoji interpretations over time, Wilcoxon rank-sum tests were conducted to compare valence and arousal scores across sessions for each emoji. This approach was chosen because several studies recommended utilizing non-parametric tests for Likert scale data, as the intervals between response options (e.g., between 1 and 2, or between 2 and 3) are not strictly equal (e.g., Kuzon et al., 1996; Jamieson, 2004). Results were considered statistically significant if the p-values were less than 0.05 and the effect size (r) exceeded 0.30.

Furthermore, to classify emojis with stable interpretations into distinct clusters, hierarchical cluster analysis was conducted on the valence and arousal ratings of each emoji. The analysis employed Euclidean distance and Ward’s aggregation criterion, with Z-scores computed for each rating prior to implementation. The optimum number of clusters was determined utilizing a dendrogram and the Calinski and Harabasz index 11, following the methodology of a prior study. After clustering, valence and arousal scores were compared across clusters using Kruskal–Wallis tests. When a significant main effect was detected, post hoc analysis was conducted utilizing the Bonferroni correction. Kruskal–Wallis tests results were considered statistically significant if the p-values were less than 0.05.

All the statistical analyses except for the Calinski–Harabasz index calculation were performed using IBM SPSS Statistics version 26 (IBM Corp., Armonk, NY, USA). The Calinski–Harabasz index was calculated using R software (R Development Core Team).

3. Results

3.1. Analyzed Data and Emojis with Stable Interpretation

Data from 256 participants (128 men and 128 women; M age = 29.63 years; SD = 5.61) were analyzed. As a result of statistical analyses, no significant differences were found in either valence or arousal for 67 emojis, and significant differences were found in either or both valence or/and arousal for seven emojis, as shown in

Table 1. List of facial emojis whose interpretations were stable.

Name	Emoji	N	Valence		Arousal
Grinning face with smiling eyes		112	7.411	(1.333)	6.866	(1.580)
		112	7.170	(1.488)	6.527	(1.524)
Beaming face with smiling eyes		112	7.580	(1.299)	7.348	(1.609)
		112	7.250	(1.405)	6.920	(1.549)
Rolling on the floor laughing		112	6.946	(1.825)	6.902	(1.816)
		112	6.768	(1.750)	6.857	(1.854)
Smiling face with smiling eyes		112	7.768	(1.375)	7.241	(1.453)
		112	7.536	(1.375)	6.777	(1.569)
Smiling face		112	7.804	(1.300)	6.991	(1.568)
		112	7.589	(1.480)	6.848	(1.428)
Smiling face with heart-eyes		112	7.991	(1.143)	7.491	(1.266)
		112	7.795	(1.363)	7.652	(1.431)
Face blowing a kiss		112	7.420	(1.380)	6.973	(1.411)
		112	7.134	(1.332)	6.723	(1.210)
Kissing face with closed eyes		112	7.313	(1.452)	7.054	(1.681)
		112	7.170	(1.381)	6.848	(1.416)
Squinting face with tongue		112	7.134	(1.580)	6.804	(1.654)
		112	6.821	(1.390)	6.473	(1.536)
Partying face		96	7.719	(1.574)	7.260	(1.510)
		96	7.615	(1.417)	7.354	(1.376)
Star-struck		96	7.406	(1.560)	7.313	(1.618)
		96	7.229	(1.511)	7.354	(1.451)
Grinning face with big eyes		112	6.991	(1.498)	6.313	(1.724)
		112	6.527	(1.536)	5.839	(1.824)
Smiling face with halo		96	6.031	(1.987)	5.719	(1.703)
		96	5.771	(1.798)	5.594	(1.567)
Face savoring food		112	7.161	(1.366)	6.509	(1.594)
		112	6.705	(1.326)	5.991	(1.562)
Relieved face		96	6.625	(1.324)	5.156	(1.737)
		96	6.365	(1.258)	5.125	(1.624)
Kissing face with smiling eyes		112	7.054	(1.438)	6.438	(1.570)
		112	6.795	(1.246)	6.205	(1.295)
Winking face with tongue		96	6.010	(1.625)	5.948	(1.719)
		96	5.635	(1.680)	5.698	(1.642)
Face with tongue		96	5.813	(1.394)	5.479	(1.667)
		96	5.677	(1.310)	5.479	(1.465)
Smiling face with sunglasses		96	6.000	(1.248)	5.458	(1.589)
		96	5.896	(1.326)	5.219	(1.474)
Cowboy hat face		96	6.521	(1.465)	5.240	(1.621)
		96	6.125	(1.267)	5.385	(1.618)
Clown face		96	6.042	(1.622)	5.479	(1.692)
		96	5.740	(1.578)	5.229	(1.645)
Thinking face		96	5.063	(1.094)	5.208	(1.535)
		96	4.865	(0.936)	5.167	(1.262)
Face with hand over mouth		96	6.604	(1.599)	6.073	(1.630)
		96	6.229	(1.380)	5.906	(1.392)
Flushed face		96	6.073	(1.481)	5.813	(1.802)
		96	6.177	(1.536)	5.802	(1.775)
Face with open mouth		96	5.531	(1.281)	5.604	(1.670)
		96	5.219	(1.018)	5.240	(1.492)
Grinning face with sweat		112	4.232	(1.414)	4.607	(1.673)
		112	4.196	(1.368)	4.616	(1.555)
Face with monocle		96	5.385	(1.060)	5.052	(1.612)
		96	5.125	(0.811)	4.646	(1.542)
Face without mouth		96	5.104	(1.209)	4.406	(1.677)
		96	4.896	(0.968)	4.521	(1.576)
Neutral face		96	4.563	(1.168)	4.448	(1.569)
		96	4.646	(1.248)	4.615	(1.618)
Expressionless face		96	3.885	(1.352)	5.458	(1.660)
		96	3.979	(1.314)	5.073	(1.564)
Unamused face		112	3.580	(1.675)	5.161	(1.789)
		112	3.696	(1.610)	5.411	(1.563)
Face with rolling eyes		96	4.500	(1.392)	5.135	(1.411)
		96	4.365	(1.162)	5.406	(1.319)
Face with raised eyebrow		96	4.354	(1.353)	5.271	(1.632)
		96	4.302	(1.315)	5.500	(1.465)
Disappointed face		96	4.354	(1.338)	4.865	(1.607)
		96	4.208	(1.289)	4.698	(1.557)
Pensive face		112	3.804	(1.432)	5.018	(1.692)
		112	3.518	(1.349)	5.134	(1.768)
Confused face		112	4.241	(1.254)	4.473	(1.588)
		112	4.009	(1.197)	4.696	(1.643)
Slightly frowning face		112	4.402	(1.291)	4.509	(1.666)
		112	4.098	(1.154)	4.589	(1.569)
Grimacing face		112	4.375	(1.316)	5.063	(1.731)
		112	4.116	(1.257)	5.161	(1.563)
Pleading face		112	4.607	(1.410)	4.938	(1.538)
		112	4.402	(1.436)	5.089	(1.763)
Fearful face		112	3.411	(1.557)	5.500	(1.786)
		112	3.268	(1.369)	5.438	(1.775)
Hushed face		96	5.281	(1.287)	4.938	(1.722)
		96	4.885	(1.247)	4.583	(1.614)
Frowning face with open mouth		112	4.393	(1.157)	4.464	(1.785)
		112	4.366	(1.082)	4.188	(1.658)
Anguished face		96	4.167	(1.262)	4.396	(1.695)
		96	4.271	(1.156)	4.771	(1.676)
Sleepy face		96	4.396	(1.294)	4.896	(1.645)
		96	4.427	(1.397)	5.094	(1.577)
Downcast face with sweat		96	4.073	(1.039)	4.948	(1.317)
		96	3.927	(1.259)	5.063	(1.634)
Woozy face		96	4.521	(1.522)	4.948	(1.855)
		96	4.646	(1.306)	5.313	(1.332)
Astonished face		112	5.036	(1.200)	4.884	(1.581)
		112	4.875	(1.124)	4.670	(1.562)
Kissing face		112	5.223	(1.243)	4.491	(1.554)
		112	5.143	(1.056)	4.732	(1.655)
Worried face		96	3.302	(1.215)	5.740	(1.675)
		96	3.469	(1.314)	5.844	(1.489)
Angry face		96	2.979	(1.654)	6.656	(1.602)
		96	3.219	(1.975)	6.479	(1.800)
Pouting face		96	2.156	(1.682)	7.667	(1.799)
		96	2.688	(2.377)	7.781	(1.718)
Face with symbols on mouth		96	2.479	(2.137)	7.594	(1.993)
		96	2.229	(1.976)	7.740	(2.007)
Persevering face		112	3.330	(1.613)	5.536	(1.922)
		112	3.321	(1.601)	5.866	(1.998)
Confounded face		112	3.348	(1.680)	5.893	(1.867)
		112	3.295	(1.733)	6.089	(1.799)
Tired face		112	3.080	(1.612)	6.402	(1.828)
		112	3.054	(1.638)	6.179	(1.987)
Weary face		112	3.071	(1.529)	6.054	(1.845)
		112	3.000	(1.414)	6.188	(1.843)
Face with steam from nose		112	3.027	(1.711)	6.714	(1.956)
		112	3.250	(1.853)	6.804	(1.749)
Face screaming in fear		112	3.455	(1.859)	7.205	(1.709)
		112	3.214	(1.747)	6.607	(2.011)
Anxious face with sweat		112	2.893	(1.503)	6.250	(1.848)
		112	2.830	(1.660)	6.455	(1.898)
Crying face		96	3.604	(1.341)	5.750	(1.704)
		96	3.427	(1.390)	5.917	(1.652)
Sad but relieved face		96	3.146	(1.231)	5.708	(1.759)
		96	3.292	(1.486)	6.156	(1.572)
Loudly crying face		96	3.188	(1.755)	7.021	(1.995)
		96	3.125	(1.617)	6.719	(1.998)
Dizzy face		96	3.698	(1.577)	6.240	(1.924)
		96	3.938	(1.398)	6.125	(1.598)
Exploding head		96	4.417	(1.567)	6.135	(1.763)
		96	4.469	(1.723)	6.490	(1.661)
Nauseated face		96	2.313	(1.489)	6.771	(1.849)
		96	2.521	(1.747)	6.688	(2.109)
Hot face		96	2.844	(1.496)	6.417	(1.972)
		96	2.625	(1.743)	6.979	(1.875)
Cold face		96	2.521	(1.589)	6.760	(2.261)
		96	2.688	(1.826)	7.063	(2.041)

Note. The names of the emojis are those registered in Let’s emoji. The figures in parentheses are standard deviations. Numbers in the upper row indicate the average of the first survey, and those in the lower row indicate the average of the second one.

and

Table 2. List of facial emojis whose interpretations were unstable.

Name	Emoji	N	Valence		p	r	Arousal		p	r
Smiling face with hearts		112	7.991	(1.143)	**	0.32	7.491	(1.266)	*	0.21
		112	7.500	(1.477)			7.205	(1.288)
Grinning squinting face		112	7.848	(1.337)	**	0.36	7.732	(1.414)	**	0.41
		112	7.375	(1.531)			7.107	(1.562)
Winking face		112	7.330	(1.226)	**	0.42	6.348	(1.626)	**	0.45
		112	6.723	(1.357)			5.500	(1.605)
Hugging face		96	7.104	(1.440)	**	0.35	6.594	(1.567)	n.s.	0.03
		96	6.635	(1.616)			6.542	(1.406)
Grinning face		112	7.045	(1.442)	**	0.25	6.366	(1.571)	**	0.34
		112	6.652	(1.406)			5.741	(1.670)
Slightly smiling face		112	6.429	(1.475)	**	0.40	5.116	(1.790)	**	0.26
		112	5.786	(1.297)			4.670	(1.857)
Frowning face		112	4.152	(1.330)	**	0.40	4.661	(1.574)	n.s.	0.00
		112	3.652	(1.198)			4.661	(1.597)

Note: The names of the emojis are those registered in Let’s emoji. The figures in parentheses are standard deviations. Numbers in the upper row indicate the average of the first survey, and those in the lower row indicate the average of the second one; * indicates a significant difference at the 5% level; ** indicates a significant difference at the 1% level; n.s. indicates not significant.

.

3.2. Classifications of Stable Emojis

Based on the results of a cluster analysis, a four-cluster solution was retained as exhibited in Figure 1. The horizontal axis represents the valence level, while the vertical axis represents the arousal level. The error bars in the figure indicate one standard deviation for each variable (i.e., valence and arousal). Additionally, each colored circle represents the range of ± 1 standard deviation from the mean valence and arousal for each cluster. The dendrogram produced by cluster analysis is shown in Figure S2. Among the four clusters, a significant main effect was found in the valence (χ²(3) = 58.249, p < 0.001) and arousal levels (χ²(3) = 54.232, p < 0.001), and post hoc analysis revealed the independence of all clusters on the two axes (ps < 0.05).

4. Discussion

This research aimed to identify emojis with stable and unstable interpretations over time in order to provide insights for consumer surveys using emojis as an indicator. We initially hypothesized that emojis that represent strong positive or strong negative emotions tend to be more unstable in interpretation over time. As a result of two online surveys, one week apart, we found significant statistical differences in either valence or arousal, or both, between surveys for seven emojis listed in Table 2. Of the seven emojis with unstable interpretations, six of them (🥰, 😆, 😉, 🤗, 😀, and 🙂) represent strong positive emotions, as all of them have an average valence of six or higher in the two surveys. The one remaining unstable emoji (☹️) represents a strong negative emotion, as indicated by an average valence of less than four in the two surveys. These results partially support our initial hypothesis and allow us to conclude that these emojis are not suitable to use as indicators for consumer surveys because they can be interpreted in different ways over time.

One reason for the discrepancy between our initial hypothesis and the results of our study may be due to the cultural characteristics of Japanese participants. It has been reported that Japanese individuals are more attuned to negative emotions in ambiguous facial expressions and demonstrate higher accuracy in recognizing negative facial expressions compared to individuals from other countries, due to their cultural characteristics (Cao & Taka, 2020). Therefore, when using emojis as indicators in consumer surveys targeting Japanese individuals, it would be important to use emojis that represent clear positive emotions.

The 67 emojis with stable interpretations distributed in a U-shape on the two axes (valence and arousal), are similar to those used in the previous studies (Kutsuzawa et al., 2022a, 2022b) (Figure 1). A hierarchical cluster analysis classified 67 emojis into the following four clusters. In cluster 1, the following 11 emojis are classified: 😄, 😁, 🤣, 😊, , 😍, 😘, 😚, 😝, 🥳 and 🤩. Because both average valence and arousal were greater than 7, this cluster can be interpreted as “strong positive sentiment”. In cluster 2, the following 14 emojis are classified: 😃, 😇, 😋, 😌, 😙, 😜, 😛, 😎, 🤠, 🤡, 🤔, 🤭, 😳 and 😮. Average valence was 6.12 and average arousal was 5.66. Therefore, this cluster can be interpreted as “moderately positive sentiment”. In cluster 3, the following 21 emojis are classified: 😅, 🧐, 😶, 😐, 😑, 😒, 🙄, 🤨, 😞, 😔, 😕, 🙁, 😬, 🥺, 😨, 😧, 😪, 😓, 🥴, 😲 and 😗. Average valence was 4.37 and average arousal was 4.89. Therefore, this cluster can be interpreted as “neutral sentiment”. Finally, in cluster 4, the following 19 emojis are classified: 😟, 😠, 😡, 🤬, 😣, 😖, 😫, 😩, 😤, 😱, 😰, 😢, 😥, 😭, 😵, 🤯, 🤢, 🥵 and 🥶. From the low average valence (3.12) and high average arousal (6.49), this cluster can be interpreted as “negative sentiment”.

This study has several limitations that must be acknowledged when interpreting the results. First, some readers may point out the relatively small sample size of this study. As described above, we asked 96 to 112 participants to evaluate each emoji. A small sample size may affect the results of a statistical analysis. Therefore, we conducted a post hoc power analysis based on the sample size and effect size of the results. As a result, we found that the power of the test was 81.1% for 96 participants and 86.7% for 112 participants. Accordingly, we can interpret that the sample size of this experiment is sufficient to detect significant differences. Second, all the participants of this study were Japanese individuals aged 20 to 39 years. The interpretation of emojis may be influenced by various factors, such as age (Kutsuzawa et al., 2022b), culture (Jaeger et al., 2019; Kaneko et al., 2019; Kralj Novak et al., 2015), and experience with daily IoT device usage. Therefore, readers should be aware of these factors when applying the findings of this study to consumer research. Third, this study used an online survey to identify the stability of emoji interpretation, although the same emoji could have different meanings in different social situations and seasons (da Quinta et al., 2023; Robertson et al., 2021). More importantly, this study did not examine the ecological validity of emoji interpretation and responsiveness. To effectively utilize emojis as indicators of users’ emotional states in psychology and consumer research, it is essential to identify emojis that are suitable for measuring emotions in everyday life and capable of detecting significant changes. Therefore, future studies should address these aspects to enhance the applicability of emojis as reliable indicators in consumer studies.

5. Conclusions

This study attempted to clarify the stability of emoji interpretation. The results demonstrated that facial emojis are a mixture of emojis with stable and unstable interpretations. We classified 67 of the 74 emojis with stable interpretations into four clusters on valence and arousal axes. These included (1) negative sentiment, (2) neutral sentiment, (3) moderately positive sentiment, and (4) strong positive sentiment clusters. We concluded that emojis correspond stably to detailed human emotions. The results of this study support the potential of utilizing emojis to measure emotions and suggest their applicability beyond food research to various contexts, including everyday situations where emotions need to be assessed easily.