A Method for Automatic Emotion Detection Through Machine Learning

Abstract

Facial expression recognition (FER) is a fundamental component of Affective Computing and is gaining increasing relevance in mental health applications. This study presents an approach for facial expression recognition using feature extraction and machine learning techniques. Starting from a publicly available dataset, a manual cleaning and relabeling process led to the creation of a refined dataset of 35,625 facial images grouped into four emotional macroclasses. Features were extracted using the SqueezeNet and Inception v3 embedders and classified using various algorithms. The experimental results show that Inception v3 consistently outperforms SqueezeNet and that feature normalization improves classification stability and robustness. The results highlight the importance of data quality and preprocessing in applied FER systems.

1. Introduction

The wide adoption of Artificial Intelligence (AI) is now well established in several fields, including mental health. The integration of such tools presents both several opportunities and inherent challenges, making a thorough understanding of how they work imperative. This is crucial to ensure their ethical and correct use, preserving the central role of mental health professionals and, in particular, ensuring the protection of patients’ privacy.

In a clinical setting, facial emotion recognition plays a crucial role. This field of research, part of Affective Computing, focuses on the analysis of facial expressions (using images and video) to interpret human emotional states. Affective Computing was defined by Picard as “computing that relates to, arises from, or deliberately influences emotions” [1] to create a computer system that perceives, recognizes, and understands human emotions, resulting in intelligent and natural responses useful for better human–computer interaction [2].

To understand human emotions, the analysis of facial expressions is fundamental, a process that relies on the detection of the positions of facial landmarks (e.g., the tip of the nose, eyebrows) and, in videos, their variations to identify the contractions of a muscle group [3].

However, emotion recognition through facial expressions is often influenced by factors such as occlusions, pose variations, lighting, and low image quality. To address these issues, it is important to consider facial muscle movements, which represent the main mechanism underlying emotional expressions. Some muscle movements are in fact specific and mutually exclusive between different emotions, while certain facial regions, although in the minority, provide discriminating information even in the presence of occlusions [4]. In this context, models that exploit the relationships between muscle movements can improve the performance of facial expression recognition (FER). An example is the Muscle Movement-Aware Transformer (MMATrans), which uses discriminative feature generation (DFG) and muscle relationship mining (MRM). The DFG module produces robust representations, reducing the impact of any labelling errors, while the MRM module leverages semantic relationships between facial muscle movements to improve FER accuracy. By removing redundant information, the model can focus on minority but critical facial regions, while reducing computational complexity and achieving superior results compared with several approaches in the literature [4].

Head position is often studied using infrared images, which are often degraded or ambiguous. To address this, Liu et al. (2021) [5] propose a nonuniform Gaussian label distribution learning network (NGDNet), coupled with a multi-label Gaussian-like distribution that reduces the effects of low-quality IR images. This method proves to be robust and effective, achieving an accuracy of 77.39% on the dataset created by the authors (IRHPE dataset) and 99.08% and 87.41% on two other public datasets, proving to be a state-of-the-art model [5].

Another key feature is facial symmetry, which allows us to predict the characteristics of the other side of the face even when it is occluded. Liu et al. (2024) [6] propose a method composed of Context-Aware Local Feature Learning (CALL) and Adaptive Feature Relationship Induction (AFRI). CALL detects local features, allowing the model to understand symmetrical and harmonious facial features. AFRI applies a Transformer to capture global patterns of faces. The model appears robust and effective in facial expression recognition, achieving accuracies of 89% and 95% on two distinct datasets [6].

FER analysis, conducted by Artificial Intelligence algorithms, can be particularly useful for mental health professionals, especially during online sessions. Indeed, the real-time detection of a patient’s emotions can support the monitoring of his or her emotional state, decision making processes for diagnosis, and, more generally, the work of the psychologist. Moreover, Affective Computing has been extended to the study of several psychiatric pathologies, such as Alzheimer’s, Parkinson’s, bipolar disorder, and PTSD, offering an objective identification and assessment of emotions, complementary to traditional diagnostic tools (characterized by greater subjectivity) and promising for the screening and evaluation of therapeutic efficacy [2].

An algorithm’s primary task in this context is to attribute meaning to non-verbal signals using pre-labelled and supervised datasets. A central aspect of our work is the pivotal role assigned to the psychologist, who analyzed an open-source dataset and then re-labelled it based on his own expertise. This process allowed the creation of a new dataset, to overcome the uncritical automatism of many Al systems, promoting a model in which technology acts as an objective amplifier of the practitioner’s clinical sensitivity.

Based on the foregoing discussion, this study introduces a method aimed at analyzing emotions by means of facial expressions using Deep Learning. The implementation of this approach was conducted through the Orange software, facilitating a critical comparison between two distinct datasets: the Kaggle dataset, widely accessible as an open-source resource, and a new dataset reworked by our research group. The latter was created in response to the detection of misclassifications in the original dataset, which included errors in emotion labelling and the presence of irrelevant images (e.g., blank or textual images instead of faces). Specifically, our study focuses on the identification and classification of the seven basic emotions: anger, fear, happiness, sadness, surprise, disgust, and neutrality.

This paper is structured as follows: Section 2 offers a review of the most recent scientific literature concerning emotion recognition. Section 3 describes in detail the methodology employed for the labelling of facial expressions and the algorithmic analysis conducted on the Orange platform. Section 4 presents an evaluation of the results in terms of precision and recall for the dataset analyzed by our team. Finally, Section 5 outlines future directions of research in the field of emotion recognition.

2. Related Work

Facial expression recognition (FER) systems include several approaches and technologies that are part of Affective Computing (AC), a field of Artificial Intelligence (AI) [7]. Among these technologies we have conventional approaches, based on hand-created features, Deep Learning (DL)-based approaches, based on features learned from data [8], and more recent attention- and Transformer-based architectures.

Our work will involve the analysis of images, from a large dataset (black and white), relating to the primary emotions, first described by Ekman (1969), namely, surprise, sadness, happiness, fear, disgust, and anger [9], to which we have also added the “neutral” class. Over the years, the technologies used for the recognition and emotional analysis of faces have gradually improved: between the 1960s and the 1990s the focus was on the development of algorithms based on facial features such as the mouth, nose and, eyes [10], with Eigenfaces techniques and algorithms such as Elastic Bunch Graph Matching (EBGM) and the Active Appearance Model (AAM) [11], while since the 2000s, machine learning models (e.g., Hidden Markov Models), Deep Learning models, and convolutional neural networks (CNNs) have taken hold [12,13,14].

The FER system involves data acquisition, preprocessing, face detection, feature extraction, and classification algorithms [10,13,14]. Among the algorithms used in face detection we have the Viola–Jones algorithm [15], Haar’s cascade classifier [16], and Adaboost Contour Points [17]. Meanwhile, to extract features from an identified face, commonly used algorithms are Local Binary Patterns (LBPs), Histogram of Oriented Gradients (HOG), and Active Shape Models (ASMs) [8]. In contrast, the extraction of facial features to classify emotional states is achieved by means of traditional ML techniques, such as Support Vector Machine (SVM) [18], Random Forest (RF) [19], and Dynamic Bayesian Network (DBN) [20].

As suggested by a review by Adyapady & Annapp (2023) [21], several ML techniques for emotion classification have been proposed in the literature, such as the work of Ghimire et al. [22] who, building upon the effectiveness of Local Binary Pattern (LBP) as a local texture descriptor for facial image representation [23], integrated it with the geometric descriptor Normalized Central Moments (NCMs). This combined feature set (NCM + LBP) was then fed into a Support Vector Machine (SVM) for classification, demonstrating that feature descriptors from specific local regions are better than global representations. Instead, Zhong et al. [24], in addition to using LBP and its uniform variant to extract features, also employed two-stage Multi-Task Sparse Learning (MTSL) to analyze both general and specific information related to different facial expressions. Although based on the LBP technique, Liong et al. [25] took a slightly different approach to encoding facial expressions, as they synthesize LBP features from three orthogonal planes (LBP-TOP) with the Bi-Weighted Oriented Optical Flow (Bi-WOOF) feature extractor. Another strategy reported in the review [21] is the extraction of features from the face region of interest (ROI) using Gabor filters and, subsequently, selecting the most relevant features using Principal Component Analysis (PCA).

Other work investigated by the review under consideration [21] continues in this vein, aimed at detecting even minimal changes in ROI. Two works are interesting in this sense: Guo et al. [26] proposed Extended Local Binary Patterns on Three Orthogonal Planes (ELBPTOP) by taking into account the radial and angular differences in a local patch, thus integrating the differences between a pixel and its neighbours (LBPTOP). Likewise, Rashmi and Annappa [27] developed a technique that combines geometric and texture features, employing the Delaunay triangulation (DT) and Voronoi Diagram (VD) approach. This combination is then sent to the DL classifier to categorize the micro-expressions.

Although ML techniques are very valid and popular, several Deep Learning (DL) architectures have been proposed in recent years to improve the performance of facial expression recognition (FER). As suggested by the analyzed review [21], Deep Neural Networks (DNNs) are mainly used, which learn discriminative patterns when fed with extracted features (e.g., via Extracted Scale-Invariant Feature Transform), and convolutional neural networks (CNNs) in combination with image processing techniques to extract specific features of facial expressions.

Other uses of CNNs are based on spatiotemporal characteristics. For example, Kim et al. [28] trained a short-term memory (LSTM), generating a discriminative spatiotemporal representation of facial expressions. Zhang et al. [29] used the bidirectional part-based hierarchical Recurrent Neural Network (PHRNN) and multi-signal CNN (MSCNN) to extract dynamic and morphological variations in facial expressions. Finally, Sun et al. [30] extracted both the spatial characteristics from a grey-level image and the optical flux characteristics from the X and Y components of a neutral and an emotional face; subsequently they combined these different features using the multi-channel deep space–time fusion (MDSTFN) neural network to analyze facial expressions.

Despite the progress made by DL, recognizing facial expressions in unconstrained contexts involves many challenges, such as database variations in ecological contexts, cross-cultural problems, computational complexity, overfitting, and poor data samples [21]. To address these challenges, different methods have been tried, such as combining a CNN with Bag-of-Visual-Words (BOVW) handcrafted features architectures, in order to predict an emotional label [31]. Feature Decomposition and Reconstruction Learning (FDRL) is suitable for modelling expression similarities, characterizing specific variations and reconstructing characteristics [32]. In addition, equally relevant are the Identity-Free Conditional Generative Adversarial Network (IF-GAN) [33] and emotion-rich feature learning network (CEFLNet) [34], respectively, useful for reducing intersubjective variations due to attributes that give a specific identity to a specific face [33] and for identifying dynamic facial expressions [34].

Concerning the difficulties related to ethnic differences, the works of Xie et al. [35] and Li and Deng [36] stand out by proposing, each, a convolutional multi-path attentional deep neural network (DAM-CNN), and the Deep Locality-Preserving CNN (DLP-CNN) together with Loss Locality Preserving (LP). Finally, in order to overcome the overfitting problem, Sun and Xia [37] proposed the AlexNet [38] and GoogleNet [39] architectures to improve the accuracy of the CNN architecture by augmenting the data through an “artificial face”. Shao and Qian [40] also devoted themselves to solving the same limitation, proposing three new CNN architectures that ultimately overcame the high computational complexity and the scarcity of training samples.

Recent studies point to other methodologies employed in FER, such as a modified AlexNet architecture integrating Gabor and LBP, also introducing depthwise-separable convolution layers (Safarov et al., 2025 [41]). In their study, the authors conclude that the proposed methodology outperforms previously used methodologies, including RS-exception, Custom Lightweight CNN-based Model (CLCM), and Lightweight Facial Network with Spatial Bias, in terms of accuracy, reaching 98.10% for the FER2013 dataset and 93.34% for the RAF-DB dataset [41].

Other authors emphasize the importance of the extracted facial features employed in FER; specifically, Shiwei Li et al. (2025) [42] propose a fine-grained human facial key feature extraction and fusion method compared with the face recognition models usually employed, which instead focus on the extraction of global features or representations of local features of faces. The authors, for emotion recognition, employ a two-branched convolutional neural network (CNN) to merge global and local features, allowing the model to capture all the characteristics involved in facial emotion changes during training. Their study demonstrates that the approach they propose improves and exceeds the accuracy of emotion recognition compared with existing models [42].

Still concerning facial features, Jaehyun So et al. (2025) [43] designed a Network for utilizing Keypoint Features (NKF). These features are extracted from significant facial regions within the combined feature maps of a dorsal network and a facial alignment network. This approach allows for the more effective and comprehensive use of detailed information from specific facial regions, thus overcoming the limitations of methods that rely solely on reference point coordinates for image alignment or the holistic interpretation of feature maps. The NKF proposed by the authors outperforms several state-of-the-art methods [43].

To address frequent problems in FER, specifically the issue of assigning an emotional label to a certain facial expression (label ambiguity), together with the numerical disparity of images present in a certain emotional class compared with another (class imbalance), JunGyu Lee et al. (2025) [44] proposed a new framework, the Navigating Label Ambiguity (NLA), which addresses label ambiguity to mitigate both noise and class imbalance in FER. The authors demonstrate that NLA, compared with other pre-trained methods for FER, has a higher overall and average accuracy on different datasets used for FER [44].

In recent years, head position estimation (HPE) has acquired a central role in areas such as image processing [45], human–machine interaction [46], and facial landmark analysis [47]. Convolutional neural networks (CNNs) have so far been the primary tool for HPE, achieving competitive results [48], but CNNs show limitations in scenarios characterized by significant occlusions, unfavourable illumination, or extreme orientations.

Recently, several studies have explored the use of Transformers as an alternative to CNNs, due to their ability to model long-range, high-level relationships between image patches. Liu et al. (2023) [49] propose the TokenHPE model, which combines visual tokens and face orientation tokens to learn the relationships between different face parts and face orientation features in baseline regions. TokenHPE achieves a state-of-the-art performance, successfully addressing challenges arising from poor lighting, occlusions, and extreme orientations, highlighting the importance of relationships between facial features and facial orientation, which have been often overlooked in previous approaches [49].

Narayan et al. (2025) [50] propose FaceXFormer, an end-to-end unified Transformer model capable of simultaneously tackling multiple facial analysis tasks, including facial landmark detection, head pose estimation, and expression recognition. The model integrates a lightweight decoder, equipped with a bidirectional cross-attention mechanism, which improves the learning of robust and generalizable facial representations. The experimental results show an excellent state-of-the-art performance in several tasks, maintaining real-time processing capabilities at 33.21 FPS [50]. Finally, Wu et al. (2025) [51] propose an approach for estimating 3D human poses from 2D sequences in monocular videos, based on a Transformer–LSTM Encoder and a Body Joint Transformer Encoder. The model is designed to capture both the spatiotemporal characteristics and structural information of joints, integrating the characteristics of individual body parts and considering kinematic coordination between joints. This combination allows for the more accurate estimation of 3D poses and better generalization ability. The experimental results demonstrate that the method outperforms previous approaches even in scenarios with limited supervision, highlighting the flexibility of the model and its applicability to different body movements and configurations [51].

In conclusion, several scientific articles can be found in the literature regarding the development of methodologies used in face and emotion recognition. But precisely because it is a rapidly growing field, it deserves further research and development of algorithms that improve the recognition of facial emotions.

3. Materials and Methods

The emotion recognition method adopted in this study is based on a deep learning architecture (Figure 1) and consists of two main phases: the first is training (shown in Figure 2), aimed at creating a predictive model for emotion recognition, and the second is testing (shown in Figure 3), aimed at evaluating the effectiveness of the model.

Figure 1. The Deep Learning model architecture step.

Figure 2. Training step.

Figure 3. Testing step.

The training phase begins with the collection of the dataset (i.e., Facial Expressions in Figure 2), considering facial expressions that have one or more movements or positions of the muscles under the skin of the face. Facial expressions are a form of voluntary or involuntary non-verbal communication, the underlying neural mechanisms of which differ in each case. Voluntary facial expressions are often socially conditioned and follow a cortical pathway in the brain. In contrast, involuntary facial expressions are innate and follow a subcortical pathway [3].

The images considered are labelled with one of the following emotions: anger, fear, happiness, sadness, surprise, disgust, and neutrality (i.e., the emotion labels in Figure 1). Once we have obtained the facial expression images, we make them uniform through a preprocessing step (in Figure 1), in which we crop the images aligned on the face, resize them all to the same target size of 48 × 48 pixels, with a resolution of 96 dpi and bit depth of 8, and finally convert the images into a single grayscale channel.

All preprocessing, training and testing phases were performed using Orange Data Mining version 3.38.1 (Bioinformatics Laboratory, University of Ljubljana, Ljubljana, Slovenia). Regarding models for emotion detection, in this paper we experiment with the following embedders:

• SqueezeNet: a deep model for image recognition with a thin structure, fewer structural parameters, and fewer calculations, in fact it only has 1 × 1 and 3 × 3 convolution kernels, with the aim of simplifying the network, while maintaining a classification accuracy similar to a public network [52].
• Inception v3: a deep Convolutional Neural Network’s framework that is particularly effective for image classification and recognition problems such as those of the ImageNet dataset. It has the innate ability to obtain fewer errors. It also has the advantage of minimizing concatenation operations to highly associated nodes so that they remain scattered [53].

For each embedder we consider the following algorithms (Table 1):

Table 1. Hyperparameters set for each algorithm.

Algorithms	Hyperparameters
Logistic Regression	Regularization type ridge L2, strength C = 1
Neural Network	Neurons in hidden layers 100, activation ReLu, solver Adam, regularization alfa = 0.0001, maximal number of iterations 200, replicable training
Gradient Boosting (Experiment 1)	Number of trees 100, learning rate 0.300; replicable training, regularization Lambda 1, limit depth of individual trees 6, fraction of training instances 1.00, fraction of features for each tree 1.00, fraction of features for each level 1.00, fraction of features for each split 1.00
Gradient Boosting (Experiment 2)	Number of trees 150, learning rate 0.100; replicable training, regularization Lambda 1, limit depth of individual trees 4, fraction of training instances 0.80, fraction of features for each tree 0.80, fraction of features for each level 0.80, fraction of features for each split 0.80
Random Forest (Experiment 2)	Number of trees 100, number of attributes considered at each split 20, limit depth of individual trees 15, do not spilt smaller than 10

• Logistic Regression: a classification algorithm with LASSO (L1) or ridge (L2) regularization. It learns a Logistic Regression model from the data and it only works for classification tasks [54].
• Neural Network: uses sklearn’s Multi-layer Perceptron algorithm that can learn non-linear models as well as linear [55].
• Gradient Boosting: a distributed gradient enhancement optimized library designed to be efficient and flexible. It provides parallel tree boosting, implementing machine learning algorithms under the Gradient Boosting framework [56].
• Random Forest: an ensemble learning method used for classification, regression, and other tasks [57].

At this point, we have the dataset, or input, to feed into the convolutional neural network we have designed (Deep Learning Network in Figure 1), which consists of 14 layers that can be grouped into the following six levels:

• Conv2D: A 2-dimensional (2D) convolution layer with the purpose of generating a convolution kernel to produce an output tensor;
• MaxPooling2D: Performs down sampling of the input along its spatial dimensions to obtain the maximum value on an input window for each input channel;
• Batch Normalization: Essential for stabilizing the training of neural networks, its goal is to standardize the output, keeping the mean close to 0 and the standard deviation close to 1. During the training phase, it normalizes the data using the mean and standard deviation of the input batch. Instead, during inference, it uses a moving average of the statistics (mean and standard deviation) collected during training;
• Flatten: Converts multidimensional matrices into a two-dimensional matrix, typically in the transition from the convolutional layer to the fully connected layer, without influencing the batch size;
• Dropout: Prevents overfitting by randomly setting inputs to 0 during the training phase at a rate frequency at each step from this layer, while inputs not set to 0 are scaled by 1/(1 − rate) so that the sum of all inputs does not change;
• Dense: Each neuron takes input from every neuron in the prior layer, executing a matrix–vector multiplication. The values in this matrix, which serve as the layer’s parameters, can be learned and refined using backpropagation.

When the model has been built, we assess its effectiveness during the testing phase (Figure 3). This second phase uses images of facial expressions not used during the training phase (i.e., Face image in Figure 3). As in the previous phase, in this case we proceed with a preprocessing step to adapt the images to the format used for the images used during the training phase. Once the preprocessed facial image is entered into the constructed model, it will generate an emotional label for the analyzed face image (e.g., Emotion Detection in Figure 3).

4. Results

Our work began with an analysis of the ‘Emotion Detection’ dataset, obtained from the Kaggle repository, which is available online free of charge for research purposes [58]. Specifically, we carried out a quality check to verify whether the classification of the seven basic emotions (anger, fear, happiness, sadness, surprise, disgust, and neutrality) was correct. This check revealed several misclassifications, such as the presence of unclear images, completely black images, written words instead of faces, or even the presence of images belonging to a certain emotional class under another label. Considering this, we created a new dataset according to a new labelling of emotions, eliminating ‘useless’ images. For example, Figure 4 shows examples of ‘useless’ images, such as black images, memes, symbols, unclear images, drawings, and words; Figure 5, Figure 6 and Figure 7 show some misclassifications in the emotion labelling.

Figure 4. Images that pollute the ‘Emotion Detection’ dataset from the Kaggle repository.

Figure 5. Sad images classified in angry.

Figure 6. Some misclassifications in neutral.

Figure 7. Some misclassifications in sadness.

Furthermore, to optimize the performance of embedders, we have grouped the seven classes of emotions into four macroclasses of emotional labels following the criterion of the positivity and negativity of their emotional value. The four macroclasses are as follows:

• Disgust + Fear
• Happiness + Surprise
• Angry + Sadness
• Neutral

Table 1 shows the number of facial expression images for each emotional label we considered in the experiment, i.e., from the dataset created from scratch by our team, a total of 35,625 facial expression images were considered in our study.

Table 2 shows how we set the hyperparameters shown for model training. To perform the experiment, we used a machine with an 8th generation Intel i7 CPU and 16 GB of RAM, running Microsoft Windows 11 Pro. We used the software Orange (version 3.38.1) which is freely available online [59].

Table 2. Our dataset. The # symbol indicates the number of images.

Class (Emotion)	# of Images
Disgust + Fear	4339
Happiness + Surprise	13,417
Angry + Sadness	12,151
Neutral	5718
Total	35,625

4.1. Experiment 1

We trained the models on 80% of the dataset, with the remaining 20% reserved for testing. A manual verification was performed to ensure that no identical images appeared in both the training and testing datasets, maintaining the integrity and independence of the evaluation process. To assess the model’s ability to recognize and distinguish different emotions from facial expression images, we considered the parameters of precision and recall. The results of the experimental analysis, based on the embedder used, are reported in Table 3 and Table 4.

Table 3. Precision, recall, and F1 Score with SqueezeNet.

Embedder: SqueezeNet	Precision	Recall	F1
Logistic Regression	0.589	0.605	0.587
Neutral Network	0.600	0.601	0.600
Gradient Boosting	0.617	0.620	0.618

Table 4. Precision, recall, and F1 Score with Inception v3.

Embedder: Inception v3	Precision	Recall	F1
Logistic Regression	0.605	0.617	0.610
Neural Network	0.614	0.617	0.616
Gradient Boosting	0.622	0.626	0.623

Essentially, Gradient Boosting achieved better results particularly with the Inception v3 embedder (precision 0.622, recall 0.626, F1 score 0.623), outperforming other algorithm–embedder combinations.

4.2. Experiment 2

In this section we show the results related to a subsequent experiment, in which we added a further preprocessing step obtained with a normalization of the interval [0–1], which was applied to all images of the dataset. Maintaining the dataset split into 80% of the images for training and the remaining 20% for testing, we obtained the results shown in Table 5 and Table 6. In this case, we considered all the metrics that Orange’s “Test and Score” widget provides: AUC, AC, F1 score, precision, recall, and MCC.

Table 5. Results with SqueezeNet after normalization preprocessing.

Embedder: SqueezeNet	AUC	CA	F1	Precision	Recall	MCC
Logistic Regression	−3	0.605	0.587	0.589	0.605	0.421
Random Forest	−3	0.573	0.524	0.620	0.573	0.362
Neural Network	−3	0.611	0.610	0.610	0.611	0.445
Gradient Boosting	−3	0.596	0.564	0.599	0.596	0.401

Table 6. Results with Inception v3 after normalization preprocessing.

Embedder: Inception v3	AUC	CA	F1	Precision	Recall	MCC
Logistic Regression	−3	0.611	0.603	0.599	0.611	0.436
Random Forest	−3	0.579	0.524	0.627	0.579	0.371
Neural Network	−4	0.627	0.625	0.624	0.627	0.466
Gradient Boosting	−3	0.611	0.584	0.609	0.611	0.426

Comparing the SqueezeNet and Inception v3 embedders, performance differences are highlighted depending on the classification algorithm used.

For Logistic Regression, the use of Inception v3 resulted in slightly higher values across all reported metrics. Accuracy increased from 0.605 to 0.611, while precision, recall, and F1 score increased from 0.589 to 0.599, from 0.605 to 0.611, and from 0.587 to 0.603, respectively. A corresponding increase in MCC was also observed (from 0.421 to 0.436).

Similar trends were observed for the remaining classifiers. With Random Forest, Inception v3 yielded marginal improvements in precision (0.627 vs. 0.620) and recall (0.579 vs. 0.573), while the F1 score remained unchanged (0.524). For the neural network, higher performance values were obtained when using Inception v3, with precision increasing from 0.610 to 0.624, recall from 0.611 to 0.627, and F1 score from 0.610 to 0.625. Gradient Boosting also showed improved results with Inception v3, with a higher precision (0.609 vs. 0.599), recall (0.611 vs. 0.596), and F1 score (0.584 vs. 0.564).

In addition to the quantitative evaluation, the classification performance was further analyzed through confusion matrices, which provide a detailed view of the distribution of correctly and incorrectly classified samples (Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15).

Figure 8. Confusion matrix for Logistic Regression (SqueezeNet).

Figure 9. Confusion matrix for Random Forest (SqueezeNet).

Figure 10. Confusion matrix for Neural Network (SqueezeNet).

Figure 11. Confusion matrix for Gradient Boosting (SqueezeNet).

Figure 12. Confusion matrix for Logistic Regression (Inception v3).

Figure 13. Confusion matrix for Random Forest (Inception v3).

Figure 14. Confusion matrix for Neural Network (Inception v3).

Figure 15. Confusion matrix for Gradient Boosting (Inception v3).

5. Conclusions and Future Work

In this paper we proposed the adoption of several machine learning models for automatic emotion recognition starting from facial expression images.

The proposed models are designed to classify images of facial expressions into one of seven basic emotions: anger, fear, happiness, sadness, surprise, disgust, and neutrality. A preliminary analysis was conducted on a dataset of 35,625 images. We analyzed the combined effects of data preprocessing, feature extraction through deep embedders, and classical machine learning classifiers. We considered SqueezeNet and Inception v3 embedders and compared model performance both before and after the application of a [0–1] normalization step.

The results obtained in Experiment 1, where no normalization was applied, indicate that classification performance depends on the choice of embedder and classifier. In this setting, Inception v3 consistently outperformed SqueezeNet across all evaluated classifiers, achieving higher precision, recall, and F1 score values. In particular, the combination of Inception v3 with Gradient Boosting yielded the best overall performance; conversely, models based on SqueezeNet generally exhibited a lower performance, highlighting potential limitations in capturing discriminative facial features without additional preprocessing.

The introduction of feature normalization in Experiment 2 combined with the inclusion of additional performance metrics, such as accuracy and MCC, resulted in more balanced classification behaviour. In particular, classifiers that showed limited effectiveness in Experiment 1, such as Random Forest, achieved more meaningful performance levels after normalization.

When comparing the two embedders under normalized conditions, Inception v3 again demonstrated a superior performance. Across all classifiers, the use of Inception v3 resulted in either consistent or marginally improved values for accuracy, F1 score, and MCC compared with SqueezeNet. These improvements were particularly evident for Neural Networks and Gradient Boosting. Logistic Regression also benefited from Inception v3, although the observed gains were moderate.

A further comparison between Experiment 1 and Experiment 2 highlights the positive impact of normalization on overall model behaviour. While Experiment 1 already showed clear differences between embedders, the application of normalization in Experiment 2 reduced performance variability across classifiers and improved result consistency. This suggests that normalization plays a key role in facilitating effective learning from deep feature embeddings, particularly when using heterogeneous classifiers.

In summary, the experimental findings indicate that Inception v3 represents a more effective embedding strategy than SqueezeNet for facial emotion recognition in the considered setting. Moreover, the adoption of feature normalization proves to be a critical preprocessing step, enhancing classifier robustness and enabling more reliable performance comparisons. These results emphasize the importance of jointly considering preprocessing, feature extraction, and classifier selection when designing emotion recognition systems based on deep feature embeddings.

From the limitations point of view, we are aware that mainly there are: class imbalance, poor quality and ambiguity for some images of the dataset and limited detailed control over learning parameters. More specifically:

• Our dataset shows a significant imbalance between the class “Happiness + Surprise” (13,417 images) and the class “Angry + Sadness” (with 12,151 images), compared with the classes “Disgust + Fear” (4339 images) and the single class “Neutral” (5718 images). To begin with, however, the “Disgust” and “Fear” classes, as present on the Kaggle repository, are individually made up of a smaller number of images, with 547 images for “Disgust” and 5121 images for “Fear”, compared with the 8989 images in the “Happiness” folder, 4002 images in the “Surprise” folder, 4953 images of the class “Angry”, and 6077 images of the class “sadness”. Therefore, a potential bias in favour of majority classes cannot be ruled out, which makes it appropriate to conduct future studies on more balanced datasets.
• The quality of some images within the dataset could be another factor influencing the learning capabilities of the model. Moreover, some images are characterized by a certain degree of ambiguity and similarity which makes it difficult to choose which emotional class they belong to, in addition to the fact that each classification, even if to a minimal extent, can be influenced by the subjectivity of the person analyzing the dataset during the classification of emotions.
• The Orange environment primarily provides a graphical interface that abstracts many implementation details. While this limits fine-grained control over learning parameters, it was selected to streamline experimentation and demonstrate model comparisons clearly. While Orange indeed abstracts some implementation details, it was intentionally chosen for its reproducibility and accessibility in comparative model evaluation. Despite limited parameter control, the environment supports standardized preprocessing and model benchmarking, ensuring fair and consistent comparisons across algorithms.

For future developments, remaining faithful to our intent to support the work of mental health professionals, we plan to incorporate explainability techniques such as Grad-CAM, LIME, or SHAP to provide visual and analytical insights into the model’s decision making process. These tools will help highlight the key image regions or features influencing predictions, thereby enhancing the model’s interpretability and clinical applicability. Integrating such activation map illustrations will also support better collaboration between AI systems and clinicians, aligning with this study’s emphasis on human–AI interpretability. We also hope to enrich emotional analysis through a multimodal approach, integrating the recognition of facial expressions with the analysis of vocal signals. Merging data from multiple information channels is a promising field of research, as multimodal recognition can overcome the limitations of a single channel [60]. This approach could provide mental health professionals with more comprehensive and reliable diagnostic and monitoring tools, enriching their assessments with more detailed and complex information.