Abstract
Emotion recognition remains a challenging task despite substantial progress in machine learning and affective computing. This study examines challenges in emotion recognition through a comparative analysis of two widely used modalities: facial images and speech signals. The analysis was conducted using FER-2013 for facial emotion recognition and the TESS and RAVDESS datasets for speech emotion recognition. A MobileNetV2-based approach was applied to visual data, while speech analysis employed MFCC-based representations and both classical and deep learning models. The study combines quantitative performance evaluation with qualitative analysis of classification behavior, focusing on emotion-specific recognition difficulties and recurring error patterns across modalities. Model performance was assessed using accuracy, precision, recall, F1-score, and confusion matrices. Across the analysed datasets, overall classification accuracy ranged from approximately 73% to 96%, while class-level F1-scores ranged from 0.48 to 0.89 depending on the emotion and modality. Happiness and surprise consistently achieved the highest recognition performance, whereas neutral emotion, fear, and disgust exhibited the lowest class-level F1-scores and generated the highest numbers of misclassifications. The experimental results confirmed that happiness and surprise achieved the highest classification performance across modalities, while neutral emotion, fear, and disgust showed reduced recognition accuracy due to weak expressive cues and overlapping feature representations. These difficulties are associated with weak or ambiguous expressive signals, overlap between emotional categories, and variability in emotional expression. The comparative findings suggest that recognition challenges arise from both modality-specific limitations and the inherent properties of emotional expression. The results highlight the importance of multimodal approaches and more flexible representations for improving emotion recognition systems.
1. Introduction
Emotion recognition is currently one of the most important directions in the development of intelligent systems, especially those designed to enable more natural human–technology interaction. In classical communication models, computer systems respond primarily to the content of commands, input signals, or predefined control parameters. In contrast, the affective approach assumes that effective interaction also requires consideration of the user’s emotional state, since emotions influence decision making processes, information interpretation, response speed, and levels of engagement [1,2,3]. From this perspective, emotion recognition is not merely a classification task but a component of a broader process of modeling human behavior in digital environments.
The importance of this issue increases as systems used in everyday, educational, medical, and industrial environments develop. In educational applications, recognising frustration, fatigue, or engagement may support adaptive adjustments to instructional materials. In the context of mental health, analysis of emotional expression can serve as an auxiliary tool for monitoring a patient’s condition, supporting therapy, and observing behavioural changes over time [3,4]. In recommender systems and marketing, emotions are treated as a key component of user responses to content, interfaces, and visual and acoustic stimuli. Meanwhile, in human–computer interaction, the development of affective technologies is associated with efforts to create interfaces that are more responsive, context-aware, and aligned with natural communication patterns [5,6].
The theoretical foundations of emotion recognition originate from the psychology of emotions, where researchers have long attempted to determine whether emotions are universal or strongly dependent on cultural and situational contexts. Classical approaches point to the existence of basic emotions with relatively consistent expression patterns that can be identified independently of culture [1]. At the same time, more complex models have been developed that conceptualize emotions as relational systems differing in intensity, similarity, and polarity [2]. From the perspective of learning systems, this has important implications: if emotions are not entirely discrete categories but rather form a continuum of states with partially overlapping features, then their automatic recognition becomes significantly more challenging than a simple multiclass classification problem would suggest.
Emotions can be recognized from various modalities, among which facial expressions and speech are the most frequently analysed [5,7,8,9]. Recent research increasingly emphasizes multimodal approaches that integrate visual and acoustic information in order to improve robustness and interpretability of emotion recognition systems [10]. Although recent advances in deep learning have substantially improved performance in both visual and acoustic analysis [11,12], recognition accuracy remains uneven across emotional classes. The literature consistently indicates that some emotions are recognised with high accuracy, while others present significantly greater challenges for models [6,9,13]. These differences may result from subtle or overlapping expressive patterns, class imbalance, and modality-specific limitations.
This problem remains relatively less explored than comparisons of models and architectures themselves. A large proportion of publications focus on achieving increasingly higher values of accuracy, precision, or F1-score, while less attention is paid to systematic interpretation of which emotions are most frequently confused, which error patterns recur across modalities, and what causes these difficulties [5,6,9]. Yet, analysing emotions that are difficult to recognise may offer particular cognitive value. It enables not only a better understanding of the limitations of current systems but also identification of which properties of data, models, and emotional categories themselves constrain the effectiveness of automatic classification.
Unlike multimodal emotion recognition studies that integrate multiple information sources within a single predictive framework, the present work performs a comparative cross-modal analysis of facial and speech emotion recognition systems considered independently. The objective is not to develop or experimentally evaluate a multimodal fusion architecture, but to identify common and modality-specific recognition challenges through quantitative performance analysis and recurring confusion patterns. The resulting observations are intended to provide methodological insights and design guidelines for future multimodal emotion recognition systems rather than to assess feature-level or decision-level fusion approaches.
The aim of this article is to perform a quantitative and qualitative comparative analysis of emotion recognition challenges across two major modalities: facial images and speech signals. Unlike studies focused only on maximizing overall classification accuracy, this work investigates emotion-specific limitations by analysing class-level performance, confusion matrices, and recurring misclassification patterns.
The main contributions of this study are summarized as follows:
- A comparative evaluation of facial emotion recognition (FER) and speech emotion recognition (SER) systems using commonly used benchmark datasets.
- An emotion-level analysis of recognition difficulty based on accuracy metrics, F1-score values, and confusion matrices.
- Identification of recurring cross-modal error patterns, including fear–surprise and neutral-related misclassifications.
- Analysis of possible causes of recognition errors resulting from data characteristics, modality-specific limitations, and properties of emotional expression.
Research Hypothesis and Research Questions
The present study is based on the following research hypothesis:
Hypothesis 1 (H1).
Despite differences in data representation and recognition models, facial and speech emotion recognition systems exhibit consistent emotion-dependent recognition behaviour that can be quantitatively characterized using class-level F1-scores and confusion patterns.
To verify this hypothesis, the study addresses the following research questions:
- RQ1: Which emotional categories consistently exhibit high recognition difficulty across facial and speech emotion recognition?
- RQ2: Which misclassification patterns are shared between FER and SER, and which remain modality-specific?
- RQ3: How can the identified cross-modal error patterns support the design of future multimodal emotion recognition systems?
The present study does not propose or evaluate a multimodal fusion architecture. Instead, it investigates facial and speech emotion recognition independently to identify common and modality-specific recognition challenges. These findings are intended to provide design guidelines for future multimodal emotion recognition systems.
2. Background
2.1. Models of Emotion
The theoretical foundations of automatic emotion recognition originate from psychological models of emotion, which provide interpretative frameworks for classifying affective states. One of the most frequently cited approaches is the basic emotions model, according to which there exists a limited set of universal emotional states characterized by relatively stable patterns of expression [1]. In the classical perspective, these include emotions such as joy, sadness, anger, fear, surprise, and disgust, which are considered relatively independent and identifiable based on observable behavioral signals.
From the perspective of automated systems, the basic emotions model has important methodological significance because it enables emotions to be treated as classes in a multiclass classification task. At the same time, emotional expressions are not always unambiguous and may exhibit considerable similarity to one another [3]. This means that approaches based solely on discrete emotional categories may fail to capture the full complexity of the phenomenon, which constitutes one of the sources of difficulty in their automatic recognition.
An alternative to discrete models is provided by dimensional approaches, in which emotions are described as points in a multidimensional space. One of the most widely used models of this type is the emotion wheel concept, where emotions are arranged in a space defined by the dimensions of arousal and valence [2]. This approach makes it possible to capture relationships among emotions and their graded nature, indicating that many emotional states do not have sharp boundaries but instead transition smoothly into one another. Contemporary emotion recognition research often combines both perspectives, using discrete labels in classification while interpreting results in light of dimensional relationships between emotions. These observations are also consistent with recent findings suggesting that emotional perception and categorization are influenced by cognitive and contextual factors, which may contribute to variability in both human and automatic emotion recognition [14].
2.2. Emotion Recognition Modalities
Automatic emotion recognition is most commonly performed using facial expressions and speech signals. In facial emotion recognition, emotional states are inferred from visual information contained in facial appearance and expression patterns [5,13]. In speech emotion recognition, classification is based on acoustic and prosodic properties of the signal, including pitch, energy, speech rate, timbre, and spectral structure [7,8,9].
The development of deep learning substantially improved the ability of models to learn discriminative representations in both modalities [11,12]. In visual analysis, convolutional architectures enabled more effective modeling of spatial structures in facial data, while in speech analysis modern models improved the representation of both local acoustic patterns and temporal variation [6]. Despite these advances, the two modalities remain fundamentally different in the type of information they encode. Facial analysis emphasizes visible expression patterns, whereas speech analysis reflects emotional variation through acoustic changes unfolding over time.
From the perspective of this study, the importance of comparing these modalities lies in the fact that they may differ not only in overall performance but also in the kinds of emotions they represent most effectively. As a result, similarities and differences observed across facial- and speech-based recognition may provide insight into whether recognition difficulty arises from the nature of a given emotion or from the limitations of a specific modality.
Table 1 illustrates the progressive transition from handcrafted feature engineering toward representation learning and multimodal integration. While classical approaches relied heavily on manually designed descriptors and domain expertise, contemporary systems increasingly employ deep neural networks capable of learning hierarchical feature representations directly from data. The most recent developments emphasize transformer-based architectures and multimodal fusion strategies, reflecting the growing recognition that emotional expression is inherently complex and cannot always be captured reliably through a single modality [10,15].
Table 1.
Evolution of emotion recognition approaches across modalities.
Although transformer-based architectures currently represent the state of the art in many emotion recognition benchmarks, the present study intentionally employs representative CNN-based and MFCC-based pipelines. This choice enables a clearer interpretation of modality-specific recognition behavior while maintaining compatibility with widely adopted benchmark datasets. Consequently, the discussion of transformer-based systems serves to position the present analysis within the broader evolution of affective computing rather than to compare different deep learning architectures directly.
2.3. Datasets Used in Emotion Recognition
The development of emotion recognition systems has been strongly influenced by the availability of publicly accessible benchmark datasets. These datasets provide standardized evaluation protocols and enable direct comparison between different machine learning and deep learning approaches. In both facial emotion recognition (FER) and speech emotion recognition (SER), several benchmark datasets have become widely adopted and continue to serve as reference resources for algorithm development and evaluation.
In facial emotion recognition, one of the most frequently used datasets is FER-2013, which contains grayscale facial images collected from the Internet and categorized into seven basic emotional classes [16]. Despite its relatively low image resolution, FER-2013 remains an important benchmark due to its large scale and challenging real-world conditions. Other widely used datasets include CK+ (Extended Cohn–Kanade Dataset), which contains posed facial expressions recorded under controlled laboratory conditions [17], RAF-DB, which provides crowd-annotated facial images collected from the web [18], and AffectNet, one of the largest facial expression datasets containing images captured in unconstrained environments [13].
Speech emotion recognition research relies on several benchmark corpora that differ in recording conditions, speaker diversity, and emotional intensity. RAVDESS is one of the most widely used datasets and contains professionally recorded emotional speech and song samples produced by multiple speakers [19]. The Toronto Emotional Speech Set (TESS) provides recordings of emotional utterances spoken by female actors and is commonly used for supervised emotion classification tasks [20]. Additional benchmark resources include the Berlin Emotional Speech Database (EmoDB), which contains acted emotional speech in German [21], and the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database, which includes multimodal recordings of dyadic interactions and has become one of the most important datasets for advanced speech emotion recognition research [22].
The diversity of available datasets contributes significantly to the development of robust emotion recognition systems. However, differences in annotation protocols, cultural backgrounds, recording conditions, and emotional intensity often make direct comparison between studies difficult. These factors also contribute to variations in reported recognition performance and influence the generalizability of developed models.
Table 2 summarizes the most frequently used benchmark datasets in FER and SER research. The variety of available datasets reflects the diversity of recording conditions, annotation strategies, and emotional representations employed in contemporary emotion recognition studies.
Table 2.
Representative datasets used in emotion recognition research.
Recent research has increasingly focused on large-scale and multimodal datasets that better reflect real-world emotional communication. In facial emotion recognition, datasets such as AffectNet and RAF-DB have become important benchmarks due to their diversity, large number of samples, and in-the-wild acquisition conditions. These datasets contain substantial variability in illumination, pose, occlusion, and demographic characteristics, making them suitable for evaluating the robustness and generalization capabilities of modern deep learning models [13,18].
In speech and multimodal emotion recognition, contemporary research increasingly relies on databases such as IEMOCAP, which combines audio, visual, and textual information collected during natural interactions. Recent studies also emphasize the importance of multimodal datasets that integrate speech, facial expressions, physiological signals, and contextual information, enabling the development of cross-modal and foundation-model-based approaches to emotion recognition [23,24,25].
2.4. Evolution of Emotion Recognition Approaches
Building upon the historical overview presented in Section 2.2, this subsection focuses specifically on recent advances in transformer-based, foundation, and multimodal emotion recognition architectures that define the current state of the art.
The field of emotion recognition has undergone substantial development over the past two decades. Early approaches relied primarily on handcrafted feature extraction techniques combined with conventional machine learning algorithms. More recently, advances in deep learning, transformer architectures, multimodal fusion, and foundation models have significantly expanded the capabilities of emotion recognition systems. This evolution reflects a broader transition in artificial intelligence from manually engineered representations toward data-driven learning frameworks capable of capturing complex emotional patterns across different modalities.
2.4.1. Classical Machine Learning Era
The earliest emotion recognition systems were predominantly based on handcrafted features and traditional machine learning methods. In facial emotion recognition (FER), researchers commonly employed visual descriptors such as Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG), Gabor filters, and facial landmark representations to characterize facial expressions [5,13]. These features were subsequently classified using algorithms such as Support Vector Machines (SVMs), Decision Trees, Random Forests, and k-Nearest Neighbors.
Similarly, speech emotion recognition (SER) systems relied on manually extracted acoustic features. Common descriptors included Mel-Frequency Cepstral Coefficients (MFCCs), pitch, energy, spectral statistics, jitter, shimmer, and prosodic characteristics [7,8,9]. These features were typically processed using SVMs, Gaussian Mixture Models, and Hidden Markov Models. Although such approaches achieved satisfactory results in controlled laboratory settings, their performance was often limited by feature engineering quality and poor generalization to unseen data.
2.4.2. Deep Learning Revolution
The introduction of deep learning significantly transformed emotion recognition research. Instead of relying on handcrafted descriptors, deep neural networks enabled automatic extraction of hierarchical feature representations directly from raw data. In FER, Convolutional Neural Networks (CNNs) became the dominant architecture due to their ability to learn spatial structures and discriminative visual patterns from facial images [13,26,27,28].
Speech emotion recognition experienced a similar transition. Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and hybrid CNN-LSTM architectures demonstrated the ability to model both local acoustic characteristics and long-range temporal dependencies in speech signals [6,29]. Numerous comparative studies reported substantial performance improvements over traditional machine learning methods, particularly when large-scale datasets were available for training [30,31].
Recent surveys indicate that deep learning approaches currently represent the dominant paradigm in both FER and SER due to their superior ability to learn robust emotional representations from heterogeneous data sources [10,32].
2.4.3. Transformer-Based Approaches
The emergence of transformer architectures represents the next major stage in the evolution of emotion recognition systems. Originally developed for natural language processing, transformers have increasingly been adapted to visual and acoustic emotion analysis tasks due to their ability to capture long-range dependencies and contextual relationships.
In facial emotion recognition, Vision Transformers (ViTs) enable global modeling of image regions and have demonstrated competitive performance compared with CNN-based architectures [33]. Their attention mechanisms allow models to focus selectively on emotionally relevant facial regions while maintaining awareness of broader contextual information.
Speech emotion recognition has also benefited from transformer-based architectures. Speech transformers and self-supervised learning models have demonstrated improved capability for modeling contextual and temporal information in long utterances [34]. Recent reviews suggest that transformer-based systems are becoming increasingly prevalent in state-of-the-art FER and SER applications due to their scalability and representation learning capabilities [25,35].
2.4.4. Multimodal Learning
A major research trend in recent years involves multimodal emotion recognition, where information originating from facial expressions, speech signals, text, physiological measurements, and contextual cues is integrated within a unified framework. The motivation for multimodal learning stems from the observation that emotions are inherently complex phenomena that are rarely expressed through a single communication channel.
Numerous studies have demonstrated that multimodal systems consistently outperform unimodal approaches, particularly in realistic environments characterized by noise, occlusions, speaker variability, and ambiguous emotional expressions [25,31,32]. The complementary nature of different modalities allows one information source to compensate for limitations present in another modality. For example, speech signals may remain informative when facial expressions are partially occluded, while visual information can support recognition under noisy acoustic conditions.
Recent reviews emphasize that multimodal fusion techniques constitute one of the most promising research directions in affective computing and human-centered artificial intelligence [10,23,35,36]. Furthermore, several benchmark initiatives and international challenges have focused specifically on multimodal emotion recognition, highlighting its growing importance within the research community [37].
2.4.5. Foundation Models and Future Trends
The latest stage in the evolution of emotion recognition involves foundation models and multimodal large language models (MLLMs). These systems leverage large-scale pretraining and self-supervised learning strategies to learn generalized representations from heterogeneous multimodal data.
Recent studies suggest that foundation models can overcome several limitations associated with conventional emotion recognition systems, particularly in situations involving limited labeled data, domain shifts, and complex contextual dependencies [38,39]. Their ability to perform cross-modal reasoning enables richer interpretation of emotional states by jointly analyzing visual, acoustic, textual, and contextual information.
Another emerging research direction concerns open-vocabulary emotion recognition, where systems are no longer restricted to predefined emotional categories but instead learn more flexible representations of affective states [37]. Such approaches are closely related to broader developments in generative artificial intelligence and multimodal foundation models.
Overall, the evolution of emotion recognition systems demonstrates a gradual transition from handcrafted feature engineering toward deep representation learning, transformer architectures, multimodal fusion, and foundation-model-based approaches. This progression reflects broader trends in artificial intelligence and highlights the increasing importance of contextual understanding, cross-modal integration, and robust representation learning in the analysis of human emotions.
2.5. Challenges in Emotion Recognition
Despite significant technological progress, automatic emotion recognition is still associated with many challenges. One of the most important problems is the similarity between emotions, which often leads to classification errors. Some emotions have similar expression patterns, making them difficult to distinguish unambiguously both for humans and automated systems [3]. In discrete classification approaches, this results in frequent confusion between emotions with similar characteristics.
Another important factor is the quality of the data used to train models. Emotion recognition systems are particularly sensitive to disturbances in input data, such as noise, varying lighting conditions, low image resolution, or distortions in acoustic signals [6,9]. Limitations of datasets, including class imbalance and limited sample diversity, also affect model performance and their ability to generalize.
Situational context also plays a significant role, as facial or acoustic expressions may carry different meanings depending on the situation in which they occur [4]. In addition, cultural and individual differences in emotional expression increase data variability and complicate the design of robust models [1,3]. Taken together, these factors indicate that effective emotion recognition depends not only on computational performance but also on the ability to account for the complexity of emotional expression itself. Recent reviews further highlight that variability in emotional expression, data quality limitations, and contextual dependencies remain key challenges for current AI-based emotion recognition systems [15]. The dataset-related limitations summarized in Table 2 are directly connected with several recognition challenges discussed in the following section, particularly class imbalance, signal variability, and ambiguity of emotional labels.
3. Materials and Methods
3.1. FER Dataset and Model
Facial emotion recognition was analysed using the FER-2013 dataset. This dataset contains grayscale facial images with a resolution of 48 × 48 pixels, assigned to seven emotional classes: anger, disgust, fear, happiness, sadness, surprise, and neutral. The dataset includes predefined training, validation, and test splits, which allows consistent evaluation of classification results. Although larger and more recent facial emotion datasets such as AffectNet are currently available, FER-2013 was selected because it remains one of the most frequently used benchmark datasets for evaluating FER algorithms. Moreover, its challenging characteristics, including low image resolution, class imbalance, variations in facial appearance, and annotation noise, make it particularly suitable for analysing recognition limitations and emotion confusion patterns.
The analysis employed a MobileNetV2-based approach. This architecture belongs to lightweight models designed for efficient processing under limited computational resources and is based on depthwise separable convolutions and inverted residual blocks. These design choices reduce the number of parameters while preserving strong representational capacity, making the architecture suitable for image-based emotion classification tasks.
Before training, standard preprocessing procedures were applied. These included pixel value normalization, resizing images to match the model input size, and conversion into a format compatible with the network. Data augmentation techniques were also used to increase training variability, including small rotations, translations, horizontal flips, and brightness adjustments.
No explicit class weighting or oversampling strategy was applied during network training. Instead, standard data augmentation techniques were used to increase the diversity of the training samples while preserving the original class distribution. Consequently, class imbalance remained an inherent characteristic of the FER-2013 dataset and was considered during the interpretation of class-level recognition performance, particularly for underrepresented emotions such as disgust. Although MobileNetV2 was originally designed for RGB images with an input resolution of pixels, transfer learning has been widely adopted for FER-2013 despite its original image resolution of only pixels. In the present study, all images were resized prior to training using standard interpolation techniques. While this preprocessing step does not introduce additional discriminative facial information and may amplify interpolation artifacts caused by the limited original resolution, it represents a commonly adopted practice for benchmark evaluation using pretrained convolutional neural networks. Consequently, the reported results should be interpreted primarily as an analysis of emotion-specific recognition behaviour under standardized benchmark conditions rather than the maximum achievable performance of the MobileNetV2 architecture.
The FER pipeline consisted of face detection, preprocessing, feature extraction using the pretrained MobileNetV2 backbone, and emotion classification using a modified classification head. The original ImageNet classification layer was replaced by a Global Average Pooling layer followed by a fully connected layer with 256 neurons and ReLU activation, a Dropout layer with a dropout rate of 0.5 to reduce overfitting, and a final fully connected Softmax output layer consisting of seven neurons corresponding to the seven FER-2013 emotion classes.
The detailed configuration of the MobileNetV2-based FER model is presented in Table 3. The model was trained using transfer learning with a modified classification head adapted to seven emotional categories. The final dense layer used the softmax activation function for multiclass emotion classification.
Table 3.
Training configuration of the MobileNetV2-based FER model.
Although MobileNetV2 was originally pre-trained on the ImageNet dataset containing RGB images with a resolution of 224 × 224 pixels, transfer learning has been widely adopted for facial emotion recognition tasks involving smaller grayscale images, including FER-2013. In the present study, the network was fine-tuned on FER-2013 following grayscale preprocessing and resizing, enabling adaptation to the target domain. Nevertheless, the discrepancy between the source and target domains remains a potential limitation and may reduce sensitivity to subtle facial action units, particularly for visually similar emotions such as fear and surprise.
3.2. SER Datasets and Models
Speech emotion recognition was analysed using two widely used benchmark datasets: the Toronto Emotional Speech Set (TESS) and the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). Both datasets contain recordings labeled with basic emotional categories and differ in speaker variability, recording conditions, and expressive intensity, which allows evaluation across heterogeneous acoustic conditions.
TESS includes recordings of short utterances spoken by female speakers and annotated with emotional labels such as anger, disgust, fear, happiness, sadness, surprise, and neutral. RAVDESS contains recordings of multiple speakers expressing similar emotional categories in both speech and song formats, providing greater variability in vocal expression and acoustic structure.
A central stage of the SER pipeline was acoustic feature extraction. Mel-frequency cepstral coefficients (MFCCs) were used as the primary representation of the speech signal due to their effectiveness in capturing spectral characteristics relevant to emotional expression. These features were complemented by prosodic descriptors, including pitch, signal energy, and speech rate. Feature extraction was performed using short temporal windows to capture both local acoustic structure and temporal variability. The detailed MFCC extraction parameters are summarized in Table 4. The configuration was selected according to commonly used SER preprocessing strategies and was kept constant for all analysed speech datasets.
Table 4.
MFCC feature extraction parameters used for SER analysis.
The adopted configuration of 13 MFCC coefficients corresponds to the standard representation widely used in speech emotion recognition research. This number of coefficients provides a compact description of the spectral envelope while reducing feature redundancy and computational complexity. In addition, first- and second-order temporal derivatives (delta and delta-delta coefficients) were included, enabling the representation to capture both static spectral characteristics and dynamic temporal changes in speech. Since the objective of the present study was a comparative analysis of representative FER and SER pipelines rather than feature engineering or feature optimization, a commonly adopted benchmark MFCC configuration was intentionally retained throughout all experiments.
Two categories of models were considered in the analysis. Classical machine learning approaches included feature-space-based classifiers trained on extracted acoustic descriptors. In parallel, deep learning models were applied to spectrogram representations of the signal, enabling automatic learning of higher-level acoustic features and temporal dependencies.
The SER pipeline included signal segmentation, feature extraction, normalization of acoustic features, and final emotion classification based on the resulting representations.
For both analysed modalities, the datasets were divided into independent training, validation, and testing subsets. Training data were used for model optimization, validation data were applied to monitor convergence and reduce overfitting, while final evaluation was performed only on unseen test samples. Classification performance was analysed using accuracy, precision, recall, F1-score, and confusion matrices.
As shown in Figure 1, both modalities follow structurally similar processing pipelines, consisting of preprocessing, feature extraction, and classification stages, despite relying on different input representations and signal characteristics.
Figure 1.
Overview of the methodological pipelines used for facial and speech emotion recognition. The diagram illustrates the main processing stages in both modalities, including preprocessing, feature extraction, model-based classification, and final emotion prediction.
3.3. Evaluation and Analytical Approach
The evaluation strategy combined quantitative performance assessment with qualitative interpretation of classification errors. The models were evaluated using standard classification metrics, including accuracy, precision, recall, and F1-score. In addition, confusion matrices were analysed to identify recurring misclassification patterns between emotional categories.
To compare recognition difficulty across modalities, an emotion-level difficulty score was introduced based on class-specific F1-score values. The following criteria were adopted:
- Low difficulty—F1-score above 0.80, indicating stable and reliable recognition.
- Medium difficulty—F1-score between 0.60 and 0.80, indicating moderate classification uncertainty.
- High difficulty—F1-score below 0.60, indicating frequent misclassification and significant overlap with other emotions.
This approach enabled a direct comparison of emotional categories independently of the overall model accuracy and allowed identification of emotions that consistently remain challenging across different recognition modalities.
Because the objective of this study is to compare emotion-level recognition behaviour rather than temporal modeling strategies, each modality was analysed using representative benchmark pipelines commonly adopted in the literature. Consequently, the comparison focuses on recurring emotion-specific confusion patterns rather than differences arising from static or temporal signal representations.
4. Comparative Analysis of Emotion Recognition
A comparative analysis of emotion recognition performance was conducted based on observed trends in representative FER and SER studies for two modalities: facial images and speech signals. To describe model behaviour, emotions were divided into two groups: relatively easy to recognise emotions and emotions posing the greatest classification challenges. This division was based on observed prediction distributions and recurring error patterns during classification.
4.1. Easy Emotions
In both analyzed modalities, happiness was among the emotions that were relatively easy to recognize. In facial image analysis, this was associated with clear and consistent visual features such as raised mouth corners and characteristic changes around the eyes. In speech analysis, happiness was reflected in distinct acoustic patterns, including higher signal energy, increased pitch, and greater intonational variability.
Relatively high recognizability was also observed for surprise. In facial images, this emotion was characterized by distinctive changes in the eye and mouth regions, while in speech it appeared through abrupt shifts in prosodic parameters.
4.2. Difficult Emotions
The greatest classification difficulties were observed for neutral emotion, fear, and disgust. Neutral emotion was frequently misclassified across both modalities. In facial image analysis, neutrality was often confused with low-intensity emotions such as sadness or fear. A similar pattern appeared in speech analysis, where neutral utterances showed substantial acoustic variability.
From an acoustic perspective, the observed confusion between neutral and sadness can be explained by their similar low-arousal characteristics. Both emotional states are typically associated with relatively flat spectral envelopes, limited pitch variation, reduced energy dynamics, and weak prosodic modulation. As a result, their MFCC representations partially overlap, reducing the discriminative capability of the classifier. Consequently, the observed confusion reflects intrinsic similarities in the acoustic realization of these emotions rather than solely limitations of the recognition model.
Fear was also difficult to recognize consistently. In facial image analysis, its expression frequently resembled surprise, particularly in the eye region. In speech analysis, fear showed considerable variability in acoustic features, contributing to classification inconsistencies.
Disgust demonstrated relatively low classification stability, especially in speech analysis. In facial image analysis, it was occasionally confused with anger due to partially similar facial muscle activation patterns.
These observations are summarized in Table 5, which presents a comparative overview of recognition difficulty across emotions and highlights recurring confusion patterns observed in both modalities.
Table 5.
Emotion-level recognition performance and difficulty assessment.
To reduce subjectivity in the interpretation of recognition difficulty, difficulty levels were assigned based on the obtained F1-score values. Emotions with F1-score values above 0.80 were classified as low difficulty, values between 0.60 and 0.80 as medium difficulty, and values below 0.60 as high difficulty.
As shown in Table 5, the results indicate that happiness and surprise achieved the most stable recognition performance across modalities. The lowest F1-score values were observed mainly for fear and disgust, confirming that emotions characterized by ambiguous expressive patterns remain the most challenging. Neutral and sadness categories also showed reduced performance, especially in facial emotion recognition, due to their low-intensity visual characteristics.
To further investigate the sources of classification errors, confusion matrices were analysed to identify the most frequent emotion-level misclassification patterns. Table 6 summarizes the dominant confusion pairs together with the number and percentage of incorrect predictions observed for each modality.
Table 6.
Dominant emotion misclassification patterns observed in FER and SER experiments.
The quantitative analysis of misclassification patterns confirms that classification errors are not randomly distributed across emotional categories. In FER, the largest number of errors was associated with visually similar expressions, particularly fear–surprise and neutral–sadness pairs. In SER, most errors occurred between emotions with similar acoustic properties, especially low-arousal emotional states.
Table 7 and Table 8 summarize the characteristic expressive features associated with individual emotional categories across modalities. The comparison indicates that recognition performance is strongly influenced by the clarity and stability of expressive signals, with visually and acoustically distinctive emotions showing more consistent classification behavior.
Table 7.
Characteristic facial features associated with emotional categories.
Table 8.
Characteristic acoustic features associated with emotional categories.
4.3. Error Patterns
Error analysis revealed recurring patterns of misclassification in both modalities. One of the most frequent observations was confusion between emotions with similar expressive features. In facial image analysis, this primarily involved pairs such as fear–surprise and sadness–neutral. In speech analysis, similar confusion occurred between neutral and low-intensity emotions.
The number of errors was also associated with the subtlety of emotional expression. Emotions with clear and distinctive expressive patterns showed fewer misclassifications, whereas more ambiguous expressions resulted in greater variability in predictions. In facial analysis, errors were often linked to small differences in facial configuration, while in speech they were associated with variability in acoustic parameters.
These recurring misclassification relationships are illustrated in Figure 2, which highlights the most common confusion patterns observed across both modalities. As shown in Figure 2, confusion most frequently occurs between emotions with overlapping expressive characteristics. In facial image analysis, this primarily involves fear and surprise as well as neutrality-related confusions, whereas in speech-based systems the dominant errors are associated with neutral and low-intensity emotional states.
Figure 2.
Recurring misclassification patterns observed across facial- and speech-based emotion recognition systems. Arrows indicate the most frequent confusion relationships between emotional categories in each modality.
To provide direct quantitative evidence for the error patterns observed in speech emotion recognition, confusion matrices were constructed for the LSTM models evaluated on the RAVDESS and TESS datasets. The matrices visualize class-level prediction distributions and enable comparison of error concentration across two datasets characterized by substantially different levels of acoustic separability.
The confusion matrices presented in Figure 3 reveal substantial differences in class separability between the two speech datasets. The RAVDESS matrix exhibits a more dispersed error structure, with the lowest class-level recognition observed for disgust, fear, and neutral utterances. These classes were assigned to several alternative categories, indicating considerable overlap in their acoustic representations.
Figure 3.
Confusion matrices for the LSTM-based speech emotion recognition experiments: (a) RAVDESS and (b) TESS. Numbers denote absolute prediction counts and percentages are normalized with respect to each true emotion class.
The qualitative observations discussed throughout this section are supported by the quantitative results obtained using the implemented FER and SER recognition pipelines. The confusion matrices together with class-level performance metrics constitute the empirical basis for the subsequent cross-modal interpretation.
In contrast, the TESS matrix is strongly diagonal, and most classes achieved correct classification rates above 90%. The few remaining errors were concentrated primarily around fear, happiness, and pleasant surprise. This difference is consistent with the higher F1-score obtained for TESS and indicates that controlled recording conditions and strongly acted vocal expressions facilitate clearer separation of emotional categories.
The comparison confirms that error patterns in speech emotion recognition are strongly influenced by dataset characteristics. Although the same model architecture was used, RAVDESS produced broader confusion across classes, whereas TESS yielded substantially more stable and concentrated predictions.
For FER, the analysis of class-level performance and representative misclassification examples indicated that fear–surprise and neutral–sadness relationships remain the most relevant visual confusion patterns. However, a complete FER confusion matrix was not available in the source experiment and was therefore not included in the present comparison.
To complement the confusion analysis, representative examples of correctly classified and challenging recognition cases are presented in Figure 4. These examples provide qualitative evidence illustrating how similarities in facial appearance and acoustic structure contribute to recurring recognition errors.
Figure 4.
Representative examples of emotion recognition cases. (a) Correctly classified facial expression of happiness characterized by a distinctive smile and eye contraction. (b) Fear facial expression exhibiting visual characteristics similar to surprise, resulting in misclassification. (c) Spectrogram of a correctly recognized happy utterance showing strong harmonic structure and pronounced prosodic variation. (d) Spectrogram of a neutral utterance exhibiting acoustic characteristics partially overlapping with sadness.
Figure 4 demonstrates that correctly recognized emotions usually contain highly discriminative visual or acoustic characteristics. In contrast, the misclassified examples illustrate situations in which expressive cues are weak, ambiguous, or partially overlap with neighboring emotional categories. In facial emotion recognition, the similarity between fear and surprise is primarily associated with comparable eye opening and eyebrow configuration. In speech emotion recognition, neutral utterances frequently exhibit flat prosodic contours and reduced spectral variability, making them acoustically similar to sadness.
5. Cross-Modal Comparison
Similarities
The observations described in the previous section suggest several consistent patterns shared across modalities. One of the most prominent findings concerns the difficulty of recognizing emotions with weak or ambiguous expressive signals. Across both facial- and speech-based systems, low-intensity and intermediate emotional states consistently produced greater variability in classification outcomes.
For each factor, the qualitative level was assigned according to its observed impact on recognition performance across the analysed datasets. A factor was classified as High when it consistently produced large reductions in class-level F1-score and frequent recurring misclassifications in both datasets. Medium denotes factors that affected recognition only under selected conditions or for specific emotional categories, whereas Low indicates a relatively minor influence on the final classification results. Another shared tendency relates to the graded nature of emotional expression. Many emotions appear to form continua rather than discrete categories, which contributes to frequent misclassification between neighboring emotional states. This was particularly evident in cases where expressive signals overlapped across categories, indicating that classification difficulty may stem from the structure of emotional expression itself rather than modality-specific limitations alone.
The main sources of classification difficulty observed across modalities are summarized in Table 9.
Table 9.
Sources of difficulty in emotion recognition across modalities.
As indicated in Table 9, subtle and low-intensity emotional expressions represent the most consistent source of difficulty in both modalities, whereas modality-specific factors such as environmental conditions and signal variability show more differentiated effects.
The comparison highlights structural differences between modalities, particularly in terms of temporal dynamics and sensitivity to environmental factors, which helps explain observed differences in classification behavior.
While both modalities share several recognition challenges, they differ in the type of information they encode and in the dominant sources of classification instability. Facial emotion recognition is more strongly influenced by visual conditions such as illumination, pose, and facial visibility, whereas speech emotion recognition is more sensitive to acoustic variability, recording conditions, and speaker-dependent characteristics. These differences indicate that recognition difficulty is shaped not only by the properties of emotions themselves but also by the constraints of the signal through which emotion is expressed.
Table 10 summarizes the main challenges identified across both modalities together with potential strategies that may reduce their impact. Several difficulties, including neutral emotion recognition, subtle emotional expression, and class imbalance, were found to affect both FER and SER systems. These challenges appear to originate primarily from the nature of emotional expression rather than from modality-specific limitations.
Table 10.
Major challenges in emotion recognition and potential mitigation strategies.
In contrast, some challenges are strongly modality-dependent. Facial emotion recognition is particularly sensitive to lighting conditions and pose variations, whereas speech emotion recognition is more vulnerable to environmental noise and speaker variability. The comparison suggests that many of these limitations may be mitigated through multimodal approaches, where complementary information from different modalities can compensate for weaknesses present in individual signals.
The results further indicate that future emotion recognition systems should move beyond isolated classification pipelines and incorporate temporal modeling, contextual information, and multimodal fusion mechanisms. Such approaches may improve robustness in real-world environments, where emotional expressions are often ambiguous, subtle, and affected by external factors.
6. Discussion
The findings of this study highlight that many challenges in automatic emotion recognition stem not only from model limitations but also from the nature of emotional expression itself. Emotions do not function as strictly separated categories but rather form a continuum of states with partially overlapping properties. As a result, emotional expressions frequently contain features associated with multiple categories simultaneously, which complicates categorical classification.
It should be emphasized that the comparison performed in this work concerns recognition behaviour at the emotion level rather than direct comparison of recognition accuracy between heterogeneous datasets. This distinction enables meaningful cross-modal interpretation despite differences in data representation and acquisition protocols.
Contextual factors further complicate interpretation. Emotional signals may carry different meanings depending on situational conditions, interpersonal relationships, and communicative intent. Automated systems typically analyze signals in isolation, which limits their ability to interpret complex or ambiguous expressions.
The challenges identified throughout the analysis originate from multiple interacting factors rather than from model limitations alone. These factors include characteristics of the training data, the intrinsic properties of emotional expression, and constraints associated with individual recognition modalities. Figure 5 summarizes the major sources of difficulty that influence the performance of contemporary emotion recognition systems.
Figure 5.
Major sources of difficulty in automatic emotion recognition. The diagram summarizes challenges related to dataset characteristics, the intrinsic properties of emotional expression, and modality-specific limitations affecting facial and speech emotion recognition systems.
As illustrated in Figure 5, the sources of difficulty in emotion recognition can be grouped into data-related factors, intrinsic properties of emotional expression, and modality-specific limitations. Dataset quality, emotional ambiguity, and signal-specific constraints jointly affect recognition performance and contribute to classification errors across modalities.
The taxonomy highlights that recognition challenges rarely originate from a single source. Instead, they emerge from the interaction of multiple factors, further emphasizing the importance of robust multimodal approaches capable of reducing uncertainty in complex recognition scenarios.
The comparison of modalities also indicates that facial- and speech-based systems capture different aspects of emotional expression. Facial analysis reflects structural and spatial variation in visible expression patterns, whereas speech analysis captures temporal and prosodic variation. These differences support the view that multimodal approaches may provide a more comprehensive representation of emotional states.
6.1. Implications for Multimodal Systems
The observed differences between facial- and speech-based emotion recognition have important implications for the development of multimodal affective computing systems. Although both modalities aim to infer the same underlying emotional state, they rely on fundamentally different sources of information and are therefore affected by different types of uncertainty. As a result, classification errors that occur in one modality are not necessarily reproduced in another, creating opportunities for complementary decision-making.
One of the most important observations is that facial and speech modalities often fail under different conditions. Facial emotion recognition systems are highly dependent on image quality and environmental factors such as illumination, face orientation, occlusions, and camera resolution. Under unfavorable visual conditions, expressive facial cues may become difficult to detect, leading to increased classification uncertainty. In contrast, speech-based systems remain capable of capturing emotional information even when facial data are partially unavailable or visually degraded.
The complementary nature of the two modalities becomes particularly important for emotions that were identified as difficult to recognize in the present analysis. Neutral emotion, fear, and disgust demonstrated substantial ambiguity in both modalities, although the sources of confusion were often different. For example, fear was frequently confused with surprise in facial analysis because of similarities in eye-region activation, whereas speech-based confusion was more closely related to variability in acoustic patterns. Combining both sources of information may therefore reduce uncertainty by incorporating additional evidence unavailable to a single modality.
A similar effect can be observed for neutral emotional states. In facial analysis, neutral expressions often overlap visually with low-intensity emotions, whereas in speech analysis neutrality may be confused with emotionally weak vocal expressions. Integrating facial and acoustic information can improve discrimination between truly neutral states and subtle emotional manifestations that might remain ambiguous when only one modality is considered.
The findings therefore support the growing interest in multimodal fusion approaches. By combining complementary information from facial expressions and speech signals, multimodal systems can improve robustness, reduce modality-specific errors, and provide a more comprehensive representation of emotional behavior. Such systems are particularly promising in real-world environments, where signal quality, environmental conditions, and expressive variability frequently challenge unimodal emotion recognition models.
From a practical perspective, the results suggest that future emotion recognition systems should not rely exclusively on a single source of information. Instead, adaptive multimodal architectures capable of dynamically weighting available modalities may offer a more reliable solution for handling ambiguous emotions and challenging operating conditions. This direction appears particularly relevant for applications in healthcare, education, human–computer interaction, and socially assistive technologies, where robustness and interpretability are essential requirements.
Table 11 illustrates representative situations in which multimodal fusion may improve recognition robustness by compensating for weaknesses associated with individual modalities.
Table 11.
Examples of modality complementarity in emotion recognition.
6.2. Limitations
Several limitations of the present study should be acknowledged. First, the analysis was primarily qualitative in nature and focused on interpreting recognition difficulty, recurring error patterns, and modality-specific challenges rather than performing a detailed quantitative comparison of classification performance. Consequently, the conclusions emphasize behavioral trends observed across modalities rather than statistically validated performance differences.
Second, the study did not involve direct experimental evaluation of multiple models under identical training and testing conditions. Instead, the analysis relied on representative facial and speech emotion recognition pipelines and on observations derived from widely used benchmark datasets. While this approach enables a broader discussion of recognition challenges, it limits the ability to attribute observed effects to specific model architectures.
Another limitation concerns the scope of modalities considered in the comparison. The analysis focused exclusively on facial image-based and speech-based emotion recognition, as these represent two of the most established modalities in affective computing. However, emotional information may also be conveyed through physiological signals, body gestures, eye movements, textual content, and contextual information, which were not included in the present study.
Finally, the study relied on publicly available benchmark datasets. Although such datasets facilitate reproducibility and comparison across studies, they may not fully reflect the complexity of real-world emotional expression. Many benchmark datasets contain acted emotions, controlled recording conditions, or limited demographic diversity, potentially affecting the generalizability of conclusions to practical applications.
A formal statistical comparison of modality-specific error distributions, for example using chi-square or permutation-based tests, was not performed because the present study primarily focuses on qualitative cross-modal interpretation based on reported class-level results. Such statistical validation constitutes an important direction for future work involving complete prediction-level outputs.
Despite these limitations, the study provides a structured perspective on the sources of recognition difficulty and highlights common challenges shared across facial and speech emotion recognition systems.
Moreover, the identified error patterns may be partially influenced by the selected model architectures. MobileNetV2- and MFCC-based classifiers represent widely used FER and SER approaches; however, transformer-based models may produce different attention mechanisms and error distributions. Therefore, the reported patterns should be interpreted as representative trends rather than universal properties of all emotion recognition systems.
6.3. Future Research Directions
The findings of this study suggest several promising directions for future research. One of the most important areas is the development of multimodal emotion recognition systems capable of integrating complementary information from facial expressions, speech signals, physiological measurements, and contextual cues. Such approaches may reduce ambiguity associated with individual modalities and improve robustness under challenging operating conditions.
Another promising direction involves the growing use of transformer-based architectures and attention mechanisms. Recent advances in vision transformers, speech transformers, and multimodal transformer frameworks have demonstrated strong potential for learning richer emotional representations and modeling long-range dependencies within emotional signals.
Future research may also benefit from greater emphasis on contextual emotion recognition. Emotional expressions rarely occur in isolation and are strongly influenced by environmental, social, and situational factors. Incorporating contextual information may therefore improve interpretation of ambiguous emotional states and reduce classification uncertainty.
In addition, the emergence of foundation models and large-scale pretrained architectures opens new opportunities for affective computing. Models trained on massive multimodal datasets may provide more generalizable emotional representations and reduce dependence on task-specific feature engineering.
Finally, cross-cultural emotion recognition remains an important open challenge. Although many existing systems assume relatively universal emotional expressions, cultural norms and social conventions may substantially influence how emotions are expressed and perceived. Developing models capable of adapting to diverse cultural contexts may therefore improve both fairness and generalization in future emotion recognition systems.
Taken together, these directions suggest a gradual transition from isolated unimodal classification systems toward more adaptive, context-aware, and multimodal approaches capable of capturing the complexity of emotional behavior in real-world environments.
7. Conclusions
This study examined challenges in emotion recognition through a comparative analysis of facial image- and speech-based systems. The results indicate that classification performance varies substantially across emotional categories, with happiness and surprise consistently showing higher recognizability, while neutral emotion, fear, and disgust remained the most difficult to classify.
The comparison further showed that recognition challenges arise both from the characteristics of emotional expression and from modality-specific constraints. Facial-based systems were more affected by visual conditions, whereas speech-based systems were more sensitive to acoustic variability and recording conditions.
The quantitative analysis demonstrated that recognition performance differed by approximately 30-40 percentage points between the best- and worst-recognized emotional categories. Rather than focusing solely on improving overall classification accuracy, future emotion recognition systems should emphasize robust modelling of ambiguous, low-intensity, and overlapping emotional expressions.
These findings suggest that improving emotion recognition systems requires greater emphasis on multimodal integration, more robust representations of ambiguous emotional states, and approaches capable of handling variability in real-world data.
The conclusions presented in this study should therefore be interpreted as comparative observations derived from representative benchmark implementations rather than as evidence obtained from a unified multimodal fusion framework. Nevertheless, the identified common error patterns provide valuable guidance for future multimodal emotion recognition research.
Funding
This research received no external funding.
Data Availability Statement
The datasets analysed in this study are publicly available from their respective repositories. No new datasets were generated. The implementation used in this study is available from the corresponding author upon reasonable request.
Acknowledgments
During the preparation of this manuscript, the author used OpenAI ChatGPT (GPT-5) for generating conceptual figures and assisting with language refinement. Grammarly Education v1.173.2.0 and the language-support tools available within the Overleaf platform were used for grammar checking, stylistic improvements, and translation support. All generated content was carefully reviewed, verified, and edited by the author. The author takes full responsibility for the content, interpretations, and conclusions presented in this publication.
Conflicts of Interest
The author declares no conflicts of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| AI | Artificial Intelligence |
| CNN | Convolutional Neural Network |
| FER | Facial Emotion Recognition |
| FER-2013 | Facial Expression Recognition 2013 Dataset |
| GMM | Gaussian Mixture Model |
| HMM | Hidden Markov Model |
| HOG | Histogram of Oriented Gradients |
| LBP | Local Binary Pattern |
| LSTM | Long Short-Term Memory |
| MFCC | Mel-Frequency Cepstral Coefficients |
| ML | Machine Learning |
| RAF-DB | Real-world Affective Faces Database |
| RAVDESS | Ryerson Audio-Visual Database of Emotional Speech and Song |
| RNN | Recurrent Neural Network |
| SER | Speech Emotion Recognition |
| SVM | Support Vector Machine |
| TESS | Toronto Emotional Speech Set |
| ViT | Vision Transformer |
References
- Ekman, P. An Argument for Basic Emotions. Cogn. Emot. 1992, 6, 169–200. [Google Scholar] [CrossRef] [Scilit]
- Plutchik, R. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am. Sci. 2001, 89, 344–350. [Google Scholar] [CrossRef]
- Scherer, K.R. What Are Emotions? And How Can They Be Measured? Soc. Sci. Inf. 2005, 44, 695–729. [Google Scholar] [CrossRef] [Scilit]
- Zeng, Z.; Pantic, M.; Roisman, G.I.; Huang, T.S. A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Trans. Pattern Anal. Mach. Intell. 2009, 31, 39–58. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Pantic, M.; Rothkrantz, L.J.M. Automatic Analysis of Facial Expressions: The State of the Art. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1424–1445. [Google Scholar] [CrossRef] [Scilit]
- Wani, T.M.; Gunawan, T.S.; Qadri, S.A.A.; Kartiwi, M.; Ambikairajah, E. A Comprehensive Review of Speech Emotion Recognition Systems. IEEE Access 2021, 9, 47795–47814. [Google Scholar] [CrossRef] [Scilit]
- Cowie, R.; Cornelius, R.R. Describing the emotional states that are expressed in speech. Speech Commun. 2003, 40, 5–32. [Google Scholar] [CrossRef] [Scilit]
- Ververidis, D.; Kotropoulos, C. Emotional Speech Recognition: Resources, Features, and Methods. Speech Commun. 2006, 48, 1162–1181. [Google Scholar] [CrossRef] [Scilit]
- Ayadi, M.E.; Kamel, M.S.; Karray, F. Survey on Speech Emotion Recognition: Features, Classification Schemes, and Databases. Pattern Recognit. 2011, 44, 572–587. [Google Scholar] [CrossRef] [Scilit]
- Bougriche, Z.; Gasz, R.; Tomaszewski, M. Current capabilities in machine emotion recognition. In Proceedings of the Machine Intelligence and Digital Interaction Conference; Springer: Cham, Switzerland, 2024; pp. 106–116. [Google Scholar]
- Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep Learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Mollahosseini, A.; Hasani, B.; Mahoor, M.H. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Trans. Affect. Comput. 2019, 10, 18–31. [Google Scholar] [CrossRef] [Scilit]
- Gasz, R.; Bougriche, Z.; Osuchowski, J.; Tomaszewski, M. A review of current capabilities and future directions in machine-based emotion recognition. Cogn. Affect. Behav. Neurosci. 2026, 1–27. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Tomaszewski, M.; Bougriche, Z.; Osuchowski, J.; Gasz, R. Systematic Review of Image-Based Emotion Recognition Datasets: A Comprehensive Analysis. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2025, 15, e70047. [Google Scholar] [CrossRef] [Scilit]
- Goodfellow, I.J.; Erhan, D.; Carrier, P.L.; Courville, A.; Mirza, M.; Hamner, B.; Cukierski, W.; Tang, Y.; Thaler, D.; Lee, D.H.; et al. Challenges in representation learning: A report on three machine learning contests. In Proceedings of the 20th International Conference on Neural Information Processing, ICONIP 2013, Daegu, Republic of Korea, 3–7 November 2013; pp. 117–124. [Google Scholar]
- Lucey, P.; Cohn, J.F.; Kanade, T.; Saragih, J.; Ambadar, Z.; Matthews, I. The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13–18 June 2010; pp. 94–101. [Google Scholar]
- Li, S.; Deng, W.; Du, J. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2852–2861. [Google Scholar]
- Livingstone, S.R.; Russo, F.A. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE 2018, 13, e0196391. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Dupuis, K.; Pichora-Fuller, M.K. Toronto Emotional Speech Set (TESS); University of Toronto, Psychology Department: Toronto, ON, Canada, 2010. [Google Scholar]
- Burkhardt, F.; Paeschke, A.; Rolfes, M.; Sendlmeier, W.F.; Weiss, B. A database of German emotional speech. Proc. Interspeech 2005, 1517–1520. [Google Scholar] [CrossRef] [Scilit]
- Busso, C.; Bulut, M.; Lee, C.C.; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, J.N.; Lee, S.; Narayanan, S.S. IEMOCAP: Interactive emotional dyadic motion capture database. Lang. Resour. Eval. 2008, 42, 335–359. [Google Scholar] [CrossRef] [Scilit]
- Al-Azani, S.; El-Alfy, E.S.M. A review and critical analysis of multimodal datasets for emotional AI. Artif. Intell. Rev. 2025, 58, 334. [Google Scholar] [CrossRef] [Scilit]
- Moon, A.; Kim, H.; Ye-Chan, P.; Lee, J. A Survey on Multimodal Emotion Recognition: Methods, Datasets, and Future Directions. Comput. Mater. Contin. 2026, 87, 1. [Google Scholar] [CrossRef] [Scilit]
- Wu, Y.; Mi, Q.; Gao, T. A comprehensive review of multimodal emotion recognition: Techniques, challenges, and future directions. Biomimetics 2025, 10, 418. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet classification with deep convolutional neural networks. Commun. ACM 2017, 60, 84–90. [Google Scholar] [CrossRef] [Scilit]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
- Wang, Y.; Song, W.; Tao, W.; Liotta, A.; Yang, D.; Li, X.; Gao, S.; Sun, Y.; Ge, W.; Zhang, W.; et al. A systematic review on affective computing: Emotion models, databases, and recent advances. Inf. Fusion 2022, 83, 19–52. [Google Scholar] [CrossRef] [Scilit]
- Ahmed, N.; Al Aghbari, Z.; Girija, S. A systematic survey on multimodal emotion recognition using learning algorithms. Intell. Syst. Appl. 2023, 17, 200171. [Google Scholar] [CrossRef] [Scilit]
- Ramaswamy, M.P.A.; Palaniswamy, S. Multimodal emotion recognition: A comprehensive review, trends, and challenges. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2024, 14, e1563. [Google Scholar] [CrossRef] [Scilit]
- Kalateh, S.; Estrada-Jimenez, L.A.; Nikghadam-Hojjati, S.; Barata, J. A systematic review on multimodal emotion recognition: Building blocks, current state, applications, and challenges. IEEE Access 2024, 12, 103976–104019. [Google Scholar] [CrossRef] [Scilit]
- Ma, H.; Wang, J.; Lin, H.; Zhang, B.; Zhang, Y.; Xu, B. A transformer-based model with self-distillation for multimodal emotion recognition in conversations. IEEE Trans. Multimed. 2023, 26, 776–788. [Google Scholar]
- Mobbs, R.; Makris, D.; Argyriou, V. Emotion recognition and generation: A comprehensive review of face, speech, and text modalities. arXiv 2025, arXiv:2502.06803. [Google Scholar]
- Chang, K.C.; Chen, S.Q. A survey of multimodal emotion recognition: Fusion techniques, datasets, challenges and future directions. Int. J. Biom. 2025, 17, 485–510. [Google Scholar] [CrossRef] [Scilit]
- Yazici, A.; Kucukyilmaz, T.; Dokeroglu, T.; Sharipbay, A.; Lee, M.H.; Tyler, B. State-of-the-art multimodal emotion recognition: A comprehensive survey and taxonomy. Intell. Syst. Appl. 2026, 30, 200642. [Google Scholar] [CrossRef] [Scilit]
- Lian, Z.; Sun, H.; Sun, L.; Wen, Z.; Zhang, S.; Chen, S.; Gu, H.; Zhao, J.; Ma, Z.; Chen, X.; et al. Mer 2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition. In Proceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing; Association for Computing Machinery: New York, NY, USA, 2024; pp. 41–48. [Google Scholar]
- Shou, Y.; Meng, T.; Ai, W.; Li, K. Multimodal large language models meet multimodal emotion recognition and reasoning: A survey. arXiv 2025, arXiv:2509.24322. [Google Scholar]
- Khan, M.; Tran, P.N.; Pham, N.T.; El Saddik, A.; Othmani, A. MemoCMT: Multimodal emotion recognition using cross-modal transformer-based feature fusion. Sci. Rep. 2025, 15, 5473. [Google Scholar] [CrossRef] [Scilit] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.




