The Impact of Face Masks Used for COVID-19 Prevention on Emotion Recognition in Facial Expressions: An Experimental Study ^†

Abstract

This study aimed to evaluate the impact of face masks used for COVID-19 prevention on emotion recognition in facial expressions. Seventy-two (72) adult participants (48 females, 24 males) attempted to correctly identify different emotions displayed by a female and a male actor’s facial expressions. Simulated emotions included neutrality, happiness, surprise, disgust, sadness, fear, and anger at two levels of intensity, with or without wearing a surgical mask. Accuracy rates of facial expression recognition and response times were collected. The GLM analysis for the accuracy revealed a main effect of emotions (F (5.350) = 57.47, p < 0.001) and face masks (without > with) (F (1.70) = 338.95, p < 0.001), as well as a three-way interaction between emotions, masks, and actors (F (5.350) = 9.69, p < 0.001). Disgust was the least recognized emotion, followed by sadness, while happiness, anger, and surprise were the easiest to identify. The analysis of the response times suggested that, when partially covered by a mask, facial expressions can be more ambiguous and difficult to read, and a larger amount of time was required to provide a response. In line with the results on accuracy, sadness was generally the most difficult emotion to identify. Male and female participants had similar response times. Globally, these results show that wearing masks can significantly reduce the ability to detect emotions in facial expressions. However, when emotions are expressed at higher intensity levels, this effect may be mitigated.

1. Introduction

In response to the COVID-19 pandemic, several hygienic measures have been adopted worldwide, including the use of protective face masks and respirators in public spaces [1]. Faces play a central role in communicating emotion-related nonverbal cues, and the ability to correctly recognize emotions by analyzing facial expressions is associated with improved wellbeing and better life satisfaction [2]. With surgical face masks, only visual cues from the upper part of the face can be collected, and expressions may be unclear or misinterpreted [3,4].

Therefore, the primary objective of this study is to understand if using surgical masks can significantly impair social interactions by reducing the accuracy of recognizing emotions shown by facial expressions. A secondary objective is to assess the impact of the emotion intensity, type, and individual gender on the accuracy of recognition.

2. Methods

2.1. Participants and Study Design

Two-hundred-sixty-nine (269) adult volunteers, 152 females and 117 males, aged between 19 and 71 years old (median: 38, IQR: 27–48), were recruited for this study in April 2020. The subjects took part in the survey on a voluntary basis and were free to withdraw from the study at any time. None of them left the test before its completion. A dedicated online database platform (called “Progetto Udine Parma”: https://www.progettoudineparma.it/) was used for this research project. Once registered, participants were recommended to find a quiet place without surrounding distractions, for completing the test.

Visual stimuli consisted of several pictures with an actress’s or an actor’s face expressing different basic emotions, as per Ekman’s classification [5] (happiness, surprise, disgust, sadness, fear, anger), and a neutral expression (Figure 1). The actors simulated two degrees of emotion intensity, classified as high and low, through different facial expressions, with their face either fully exposed or partially covered by a surgical mask. Pictures were displayed one after another in a random sequence to account for any priming bias and to compensate for the potential boredom effects due to task performance. For each item, study participants were asked to classify the facial expression they were looking at by choosing the correct emotion type. Individual responses and response times were recorded and analyzed. Further details about the study protocol are reported in the Supplementary Materials (Section A).

2.2. Preliminary Analysis

Following the data collection, only the responses provided by participants capable of correctly identifying emotions simulated by the unmasked actors at least six out of seven times (86% of correct responses) were considered reliable. This conservative approach reduced the size of the initial sample from 269 to 72 valid cases (N of females = 48, mean age = 34.7, SE (Standard Error) = 1.49; N of males = 24, mean age = 39.13, SE = 2.61). This sample size was still considered sufficient, since the preliminary power analysis recommended a minimum of 42 participants in total (G*Power: effect size of f = 0.15, probability error = 0.05, power estimate = 0.9, 2 groups, and 12 measures).

A frequency analysis delineated the success rate of recognizing each target emotion at the baseline, with the actors’ face uncovered. The majority of emotions were correctly identified above chance (binomial test, test proportion = 0.50, p < 0.001), as reported in the Supplementary Materials (Section A). Only fear was identified at the level of chance in both levels of intensity. This led to the exclusion of fear from the subsequent comparisons in the with-mask condition. For all of the other emotions, those associated with the highest ratings between high and low intensity were selected for further analysis (hit range (%) for the actress: [86.1; 97.2]; and for the actor: [83.3; 100]). Then, each emotion was compared with itself in the two conditions of interest (with and without a face mask on).

2.3. Data Analysis

Two sets of analyses were carried out: a quantitative assessment of the differences in the participants’ correct responses and response times when the two conditions of interest (no-mask versus with-mask) were compared with each other, also as a function of the actor’s gender and the participant’s gender; a qualitative evaluation of the most common error types made by participants when trying to recognize emotions expressed by the two masked actors.

Statistical analyses were carried out with the IBM Statistical Software Platform SPSS, Armonk, NY, USA (v. 19.0). To evaluate the possible differences in emotion identification (no-mask versus with-mask conditions), a 6 × 2 × 2 × 2 repeated measures general linear model (GLM) analysis was carried out with six levels of emotion (disgust, happiness, anger, surprise, sadness, neutrality), two conditions of interest (with or without a mask), and the two actors’ gender (female and male) as the within-subject variables. The participant’s gender was the between-subject factor. The dependent variables were the accuracy rates (hit proportion of the correct responses over the total) and mean response time (sec.). The Greenhouse–Geisser correction was used for the violations of Mauchly’s test of sphericity (p > 0.05). All post-hoc comparisons were Bonferroni corrected.

3. Results

3.1. Accuracy Rates of the Emotion Recognition

The GLM analysis for the accuracy rates revealed a main effect of emotions: F (5.350) = 57.47, p < 0.001, partial-η2 = 0.45, δ = 1; a main effect of masks (without > with): F (1.70) = 338.95, p < 0.001, partial-η2 = 0.83, δ = 1; a significant interaction between emotions and masks: F (5.350) = 61.29, p < 0.001, partial-η2 = 0.47, δ = 1; a significant interaction between emotions and actors: F (5.350) = 8.57, p < 0.001, partial-η2 = 0.11, δ = 1; and a three-way interaction between emotions, masks, and actors: F (5.350) = 9.69, p < 0.001, partial-η2 = 0.12, δ = 1. No significant differences were found as a function of the participant’s gender (p > 0.05). The post-hoc analysis showed that, regardless of wearing or not wearing a mask, disgust was the least recognized emotion, followed by sadness, whose recognition rate was significantly lower, if compared to happiness, anger, and surprise (p < 0.01). Happiness, anger, and surprise recognition rates did not demonstrate any significant differences, when compared with each other (p > 0.05). The overall accuracy rates of emotion recognition are plotted in Figure 2 (see the Supplementary Materials—Section B for further details).

3.2. Response Times

The GLM analysis for the response time revealed a main effect of emotions: F (5.350) = 5.20, p < 0.001, partial-η2 = 0.07, δ = 0.90; a main effect of masks (with > without): F (1.70) = 29.42, p < 0.001, partial-η2 = 0.30, δ = 1; a main effect of actors: F (1.70) = 6.18, p < 0.05, partial-η2 = 0.08, δ = 0.69; a significant interaction between emotions and actors: F (5.350) = 2,97, p < 0.05, partial-η2 = 0.04, δ = 0.68; and a significant interaction between actors and gender: F (1.70) =5.06, p < 0.05, partial-η2 = 0.07, δ = 0.60. The post-hoc analysis showed that the hardest emotion to recognize was sadness, in particular with respect to happiness, surprise, and neutral expression (p < 0.05). Mean response times to each emotion are plotted in a figure, reported in the Supplementary Materials (Section B).

3.3. Response Frequency Distribution

Table 1. Correct response rates of the study participants attempting to identify emotions simulated by the two actors with their face masks on.

Emotion	Disgust		Happiness		Anger		Surprise		Sadness		Neutral
Actor’s gender	F	M	F	M	F	M	F	M	F	M	F	M
Correct hits	19%	10%	60%	99%	100%	62%	81%	88%	61%	63%	89%	93%

Legends: F = female; M = male.

shows the correct response rates (%) of the study participants attempting to identify emotions simulated by the two actors with their face masks on. Most emotions were recognized quite accurately, even when the two actors wore a mask. The most misinterpreted emotion was disgust, followed by sadness. For disgust, ambivalence tended to fall on happiness, sadness, and fear, whereas for sadness, it mostly fell on disgust and fear (see the Supplementary Materials—Section B for further details).

4. Discussion

No significant differences in the emotion recognition accuracy were found when pictures of the two actors were compared with each other in the no-mask condition, and this was important to ensure the test validation at the baseline. As expected, the results showed that face masks can significantly reduce the accuracy of successfully recognizing emotions in facial expressions. Regardless of wearing a mask or not, disgust was the least recognized emotion, followed by sadness. Moreover, happiness, anger, and surprise were the most easily recognized. Interestingly, the three-way interaction showed no differences in happiness recognition between the no-mask and with-mask conditions for the actor, and no differences in anger recognition for the actress: both emotions were recognized quite correctly by the study participants even when the two actors wore a mask.

The analysis of the response times suggested that, when partially covered by masks, facial expressions can be more ambiguous and difficult to read, and a larger amount of time was required to provide a response. In line with the results on the recognition accuracy, sadness was generally the most difficult emotion to identify. When considering the male actor only, the participants struggled more to recognize anger, with respect to happiness and surprise (these two emotions were better recognized when expressed by the actress). This result is consistent with the data on accuracy, thus suggesting that anger was better recognized in the actress, while happiness was more easily identified in the actor. Male participants provided a quicker response to the actor’s emotions, compared to the actress’s ones.

Globally, these findings are not surprising, and the first and more obvious explanation is that, partially covering the face (in particular its lower part) can reduce the number of facial cues available to our brain for deciphering the underlying emotion. However, another possible explanation, or concurrent factor, may be that face masks can inhibit the experience sharing system (the neural basis for empathic accuracy), after partially concealing the most relevant visual cues of the facial expressions, normally mirrored by the observer to ensure proper emotion recognition through enteroception [6,7]. In other words, reducing the number of facial expression-related cues available for the mirror system might entail a harder task for the brain insula to reproduce an appropriate enteroception of the other’s emotional state, thus hindering the recognition of facial expressions for the “feeling” it evokes in the observer [8,9,10]. Another explanation may involve the socio-cultural factors at a more cognitive level, due to specific interpretations of the visual cues related to protective equipment. For example, surgical masks might be associated with danger, as opposed to scarves or other “culturally neutral” ways to conceal the lower part of the face, and this association could trigger individual anxiety, thus negatively affecting the whole process of properly reading emotions in facial expressions [11,12,13]. Furthermore, possible mechanistic explanations, face masks generally represent, and may be perceived as, barriers to effective communication, and people wearing them may be less motivated to put effort into fully expressing their emotions. In the long run, this may even lead to a reduced or flat affect display [3]. Moreover, in everyday life, general attention required for face reading tends to be inferior to the attention levels exhibited in experimental settings, thus potentially resulting in the even greater influence of face masks in real-life conditions [12]. Further explanations for differences in the accuracy of emotion recognition are reported in the Supplementary Materials (Section C).

Limitations

Only two actors simulated the basic emotions with different facial expressions, and pictures were not shown a second time to the study participants. Nevertheless, a multiple-response study design was avoided to shorten the test duration and to maximize the participants’ compliance.

5. Conclusions

In conclusion, surgical masks can reduce the accuracy of recognizing basic emotions in facial expressions. All the same, higher intensity levels of emotional expression may partially mitigate this effect. The decline in recognition accuracy across different emotions may follow a particular pattern, with a potential influence of gender-specific characteristics. However, these findings should not be interpreted as a contraindication to wearing face masks when required for health reasons. In fact, it is highly plausible that our society will gradually adapt to this new scenario with new strategies, aimed to improve interpersonal communication, including transparent face masks, stronger facial expressions, more attention to other body signals (posture, speech intonation, gestures), or simply describing emotions with feeling words. Additionally, our study showed that wearing face masks may have a less pronounced effect on recognizing some emotions, such as surprise, happiness, and anger. Finally, direct implications of this study are important for any setting where accurate empathic communication is necessary, for example in healthcare, politics, and education, especially for promoting and inspiring adaptive communication strategies.