Next Article in Journal
When Robots Get Bored and Invent Team Sports: A More Suitable Test than the Turing Test?
Next Article in Special Issue
TwitPersonality: Computing Personality Traits from Tweets Using Word Embeddings and Supervised Learning
Previous Article in Journal
Vector Spatial Big Data Storage and Optimized Query Based on the Multi-Level Hilbert Grid Index in HBase
Previous Article in Special Issue
Towards Aiding Decision-Making in Social Networks by Using Sentiment and Stress Combined Analysis
Article Menu

Export Article

Open AccessArticle
Information 2018, 9(5), 117;

A Comparison of Emotion Annotation Approaches for Text

Insight Centre for Data Analytics, National University of Ireland Galway, Galway H91 AEX4, Ireland
Aylien Ltd., Dublin D02 RH68, Irenland
Author to whom correspondence should be addressed.
Received: 27 March 2018 / Revised: 25 April 2018 / Accepted: 9 May 2018 / Published: 11 May 2018
(This article belongs to the Special Issue Love & Hate in the Time of Social Media and Social Networks)
Full-Text   |   PDF [443 KB, uploaded 11 May 2018]   |  


While the recognition of positive/negative sentiment in text is an established task with many standard data sets and well developed methodologies, the recognition of a more nuanced affect has received less attention: there are few publicly available annotated resources and there are a number of competing emotion representation schemes with as yet no clear approach to choose between them. To address this lack, we present a series of emotion annotation studies on tweets, providing methods for comparisons between annotation methods (relative vs. absolute) and between different representation schemes. We find improved annotator agreement with a relative annotation scheme (comparisons) on a dimensional emotion model over a categorical annotation scheme on Ekman’s six basic emotions; however, when we compare inter-annotator agreement for comparisons with agreement for a rating scale annotation scheme (both with the same dimensional emotion model), we find improved inter-annotator agreement with rating scales, challenging a common belief that relative judgements are more reliable. To support these studies and as a contribution in itself, we further present a publicly available collection of 2019 tweets annotated with scores on each of four emotion dimensions: valence, arousal, dominance and surprise, following the emotion representation model identified by Fontaine et al. in 2007. View Full-Text
Keywords: emotion; annotation; annotator-agreement; social-media; affective-computing emotion; annotation; annotator-agreement; social-media; affective-computing

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).

Share & Cite This Article

MDPI and ACS Style

Wood, I.D.; McCrae, J.P.; Andryushechkin, V.; Buitelaar, P. A Comparison of Emotion Annotation Approaches for Text. Information 2018, 9, 117.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Information EISSN 2078-2489 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top