Analysis of Multiple Emotions from Electroencephalogram Signals Using Machine Learning Models ^†

Abstract

Emotion recognition is a valuable technique to monitor the emotional well-being of human beings. It is found that around 60% of people suffer from different psychological conditions like depression, anxiety, and other mental issues. Mental health studies explore how different emotional expressions are linked to specific psychological conditions. Recognizing these patterns and identifying their emotions is complex in human beings since it varies from each individual. Emotion represents the state of mind in response to a particular situation. These emotions, that are collected using EEG electrodes, need detailed emotional analysis to contribute to clinical analysis and personalized health monitoring. Most of the research works are based on valence and arousal (VA) resulting in two, three, and four emotional classes based on their combinations. The main objective of this paper is to include dominance along with valence and arousal (VAD) resulting in the classification of 16 classes of emotional states and thereby improving the number of emotions to be identified. This paper also considers a 2-class emotion, 4-class emotion, and 16-class emotion classification problem, applies different models, and discusses the evaluation methodology in order to select the best one. Among the six machine learning models, KNN proved to be the best model with the classification accuracy of 95.8% for 2-class, 91.78% for 4-class and 89.26% for 16-class. Performance metrics like Precision, ROC, Recall, F1-Score, and Accuracy are evaluated. Additionally, statistical analysis has been performed using Friedman Chi-square test to validate the results.

1. Introduction

Emotion is a response of a human being when subjected to external stimuli. It affects a person’s psychological and behavioral activities in making decisions and processing information. It is an interesting combination of psychology and technology [1] and can be learnt from different disciplines including marketing, philosophy, neuroscience, psychology, and artificial intelligence. A Brain–Computer Interface (BCI) system is used to provide communication between the machine and the brain [2]. The emergence of BCI [3] has enabled neuroscientists to study the emotions of different individuals and process them using this technology.

Affective Computing is an example of BCI that connects computer science, physiology, and psychology. It is defined as the computational study of emotions and their manifestations within the systems through brain signals [4]. The recognition of emotion through computational means is communicated to healthcare people like doctors, healthcare educators, and medical administrators. The advancement in technology has contributed to various medical applications like rehabilitation, assisting doctors with mental disease diagnoses, like autism, etc., assistance for disabled people like prosthetics, and innovation in medical equipment.

Emotions are presented in two different ways, namely the discrete emotional model and the dimensional model. The discrete model was proposed by Ekman with six emotional states namely happiness, anger, fear, sadness, surprise, and disgust. The dimensional model signifies the affective states in the dimensional space and the dimensions are valence, arousal, dominance, and liking [5]. Here, the emotions are recognized by rating those dimensions. The 2-dimensional models are simpler with valence and arousal whereas the 3-dimensional models are more realistic with valence, arousal, and dominance [6].

Human emotions are analyzed by the quick changes in the electrical activity of the brain. These abrupt changes are measured using an Electroencephalogram (EEG) which is a measuring device placed on the scalp of the head. Various human cognitive and emotional processes of the brain are studied by researchers with the help of EEG signals [7]. EEG signals are in the frequency range of 0.5–100 Hz and the lower frequency range is suitable for cognition [8]. It is commonly preferred by researchers because of easy recording and processing of them into meaningful information. These physiological signals are processed by different machine learning models.

Machine learning is a field of computer science that gives computers the ability to learn without being explicitly programmed [9,10]. It is the subset of artificial intelligence that automates the systems and simplifies the working processes using simple programs. It experiences fast growth and major advancements in various fields. It is indispensable in various fields like healthcare, finance, automotive, and many other promising fields. It is mainly used when the existing solutions require more tuning for a particular problem. It is also for complex problems when there are no good solutions. It has the ability to adapt to new data and find good solutions.

Machine learning systems are classified as supervised, unsupervised, semi-supervised, and reinforcement learning. In supervised earning, the training data have desired solutions called labels before it is given to the algorithm that includes KNN, linear regression, logistic regression, SVM, decision tree, and random forests [10]. In unsupervised learning, the training data are unlabeled and they are not classified. Therefore, the machine uncovers the hidden patterns and creates new labels. The main advantage of this learning is the identification of new unknown patterns. Reinforcement learning is the most advanced learning method since it learns continuously and improves the model by leveraging feedback from past iterations. The term classification in machine learning refers to the task of identifying the data points in the dataset and grouping them into different categories. In a multiclass classification, the data are classified into different classes. It is a statistical problem where the predefined class predicts the output based on features of the dataset [11].

Most emotion recognition-based research focuses on binary classification [12,13,14,15,16]. Few research papers have addressed the 4-class classification [17,18,19]. It is important to classify emotions into different classes so that many numbers of emotions can be estimated [20]. Dominance is either separately classified [21] or not included in the classification of emotions. Nandhini et al. [22] have used the VAD method for emotion recognition for 12 discrete emotions which is a 12-class classification using machine learning algorithms. In this paper, we discuss three different types of classifications namely 2-class, 4-class, and 16-class which are given in

Table 1. Emotional class and its categories.

No. of Class	Categories
2—(V) (A)	Valence(V), Arousal(A)
4—(VA)	High Arousal High Valence (HAHV), High Arousal Low Valence (HALV), Low Arousal High Valence (LAHV), and Low Arousal Low Valence (LALV)
16—(VAD)	Sadness, Shame, Guilt, Envy, Satisfaction, Relief, Hope, Interest, Fear, Disgust, Contempt, Anger, Pride, Elation, Joy, and Surprise

.

The main objectives of this research work are as follows:

• Develop a suitable VAD model to categorize 16 emotions which is high when compared to the existing state-of-the-art techniques.
• Evaluate the performance of the machine learning model for 2-class, 4-class, and 16-class and hence, identify a suitable machine learning model for multiple class classification of emotion.

The remaining sections of the paper are organized in the following order: Section 2 discusses the methodology for preparing the dataset and the explanation of the machine learning models. Section 3 gives the results and discussion of each model and their corresponding performance evaluation. The conclusion is given as the last section in this paper with the summary of the work.

2. Methodology

The interaction between the human brain and the computer follows certain steps so that the computer understands the EEG signal. The DEAP dataset is used for our proposed work [23]. The EEG signal recorded from the human brain has to be collected from the EEG cap which has 48 numbers of electrodes according to the international 10–20 system. Each participant rated the video based on valence, arousal, dominance, and liking on the scale of 1–9. The acquired raw EEG signal was recorded for 63 s after the removal of the baseline signal of 3 s and stored on a computer device. Figure 1 depicts the block diagram of EEG-based emotion recognition for machine learning models. The signal undergoes several processes such as downsampling, filtering, augmentation, feature extraction, and classification of emotions based on the number of classes. At first, the signal was downsampled to 128 Hz to focus on the frequency of interest and to eliminate higher frequency components in the signal. The dimensions of the data are the product of the number of video trials, selected the number of channels and samples which is 40 × 14 × 8064 (63 s × 128 Hz). The windowing technique is used for the data augmentation process; resulting in 19,520 (40 ×488) data samples for a single subject for 40 trials [24]. Power Spectral Density (PSD) was extracted by considering the five frequency bands of EEG and 14 numbers of channels. The machine learning models that are used for classification are SVM, KNN, LDA, random forest, decision tree, and naive Bayes.

The label in the DEAP dataset has rating values of each trail for each subject. The rating is based on four dimensions of valence, arousal, dominance, and liking (VADL) and ranges from 1 to 9. Most studies used binary classification where the labels are classified into positive and negative emotions. Few studies have used a valence and arousal (VA) model resulting in four class classifications. In this paper, the combination of valence, arousal, and dominance (VAD) dimensions is used to categorize into 16 emotions.

3. Result and Discussion

The experimental setup for the investigation is given in

Table 2. Experimental Setup.

Name/Description	Version
CPU	Intel® Core™ i5
RAM	8 GB
OS	Windows 10
Python	Python 3.11.5
TensorFlow	TensorFlow 2.14.0
Scikit-learn	Scikit-learn 1.3.1
Anaconda	2021.05

.

3.1. SVM

Examination of the dataset with the SVM model found that the model had good accuracy results for 2-class classification. It is a powerful machine learning tool that is best suited for binary classification. The percentage of accuracy has dropped for 16-class classification to 37.01%. These accuracy rates can be improved when the SVM model is created for each pair of classes. In this model, the value of the regularization parameter (C) is one and the kernel used for experimentation is linear and RBF. It is important to choose the value of C and a suitable kernel for better classification results.

As shown in

Table 3. Performance evaluation of the SVM model.

Model	SVM-Linear
	Accuracy	Precision	Recall	F1-Score
2- Class	73.81%	72%	64%	65%
4-Class	48.26%	46%	42%	42%
16-Class	37.01%	35%	37%	35%
	SVM-RBF
2- Class	67.35%	34%	50%	40%
4-Class	38.75%	10%	25%	14%
16-Class	24.4%	2%	7%	3%

, the linear kernel has shown a better performance of around 14% higher than the RBF kernel for 16-class classification. This is due to the linearization of data with short time interval considerations for FFT. Generally, EEG-based emotional dataset has the problem of class imbalance and hence, lead to the degradation of the performance [25]. SVM are sensitive to class imbalance and therefore SVM-RBF model has shown poor performance. It needs good tuning to achieve optimal solutions but it requires expertise. Therefore, when the number of classes increased, the model failed to separate the data into different classes.

3.2. LDA

LDA aimed to maximize the separation between the classes while minimizing the variance within each class. However, as the number of classes increased, the overlap between the classes also increased, making it harder to effectively distinguish between them. This happened because the classes are closely related or inherently ambiguous. Also, in multi-class classification, the decision boundaries are highly nonlinear, making it difficult to capture the underlying patterns in the data. Therefore, the model’s accuracy dropped from 73.69% to 33.86% when the number of classes increased. The performance metric of LDA is given in

Table 4. Performance evaluation of the LDA model.

Model	LDA
	Accuracy	Precision	Recall	F1-Score
2-Class	73.69%	70%	66%	67%
4-Class	49.97%	49%	43%	44%
16-Class	33.86%	28%	29%	25%

.

3.3. KNN

In order to select the value of k, different values were investigated to find the best. Since there is no definite method, proper care must be taken for the selection of the k value. A very low k value, such as 1 or 2, may create noise and lead to an outlier effect in the model. Selecting large k values is good and produces stable decision boundaries but creates difficulties in computation. This model is easy to implement and highly robust to noise; it has shown good results due to its ability to perform well with large data. It can process 1-D data and multi-class classification well, resulting in 89.26% accuracy. Among the different k values, it was observed that the model attained the highest accuracy when k = 3 for all three classes.

Table 5. Performance evaluation of the KNN model.

Model	KNN
	Accuracy	Precision	Recall	F1-Score
2-Class	95.81%	95%	95%	95%
4-Class	91.78%	92%	92%	92%
16-Class	89.26%	89%	90%	89%

shows the efficiency of KNN using the performance metrics.

3.4. Decision Tree

The decision tree supports both binary and multi-class classification techniques. The decision tree with entropy is a key factor that helps to make decisions while splitting the data at each node of the tree; it groups homogenous data and maximizes the information gain by reducing the uncertainty. It was observed that the decision tree with entropy has better predictive power for the classification of the tasks due to the ability to quantify the impurity of the dataset. It was estimated that the accuracy of the 2-class was 87.56%, the 4-class was 77.05%, and the 16-class was 70.08%.

Table 6. Performance measure of the Decision Tree.

Model	Decision Tree
	Accuracy		Precision		Recall		F1-Score
	Without Entropy	With Entropy	Without Entropy	With Entropy	Without Entropy	With Entropy	Without Entropy	With Entropy
2-Class	86.71%	87.56%	85%	86%	85%	86%	85%	86%
4-Class	76.39%	77.05%	75%	76%	76%	77%	76%	77%
16-Class	69.68%	70.08%	67%	68%	68%	68%	67%	68%

shows the performance metric of the decision tree with and without entropy.

3.5. Naive Bayes

The naive Bayes model works mostly on the assumption that the features are independent of the given class label. Complexity in capturing the underlying patterns between the variables has caused the model to fail. Confusion has occurred within the model because the features are correlated with each other. As the number of classes increased, the complexity further increased leading to poor classification of the model to about 7%.

Table 7. Performance measure of Naive Bayes.

Model	Naive Bayes
	Accuracy	Precision	Recall	F1-Score
2- Class	58.46%	59%	60%	57%
4-Class	38.29%	29%	28%	24%
16-Class	7.55%	18%	26%	7%

shows the performance evaluation of naive Bayes.

3.6. Random Forest

Random Forest works on multiple numbers of trees instead of relying on a single tree. Therefore, it provides accuracy greater than 84% for all three types of classification and prevents the problem of overfitting. It produces high accuracy even when the dataset is large. The selection in the number of decision trees is based on a trial and error method. For 2-class classification, the number of the decision tree (n) chosen to be high was 24 and the maximum accuracy obtained was 93.2%. For 4-class classification, n was 25 and the maximum accuracy obtained was 87.59%. For 16-class classification, n is chosen to be 24 and the maximum accuracy was 84.7%. The other performance metric of the model was calculated and shown in

Table 8. Performance measure of Random Forest.

Model	Random Forest
	Accuracy	Precision	Recall	F1-Score
2- Class	93.20%	93%	91%	92%
4-Class	87.59%	89%	87%	87%
16-Class	84.70%	87%	82%	84%

.

The receiver operating characteristic (ROC) curve of the 2-class classification is shown in Figure 2a–f for all the machine learning models with predicted value in the x-axis and true value in the y-axis. Figure 3a–f portrays the 4-class classification of the experimented machine learning models. In Figure 4a–f, the ROC curve of 16-class classifications of different machine learning models is shown.

The comparison of different machine learning models based on accuracy is shown in Figure 5. Among the six models, we observed that KNN has better accuracy for multiclass classification. Table 4 shows that better results were obtained for the other performance metrics. This proved good for 16-class which is the highest number of emotions than the existing state-of-the-art classification. Random forest also has proved best with 84.7% for 16-class classification. Naive Bayes showed the worst performance for multiclass classification with 7% of accuracy. Thus, from the complete analysis, it can be deduced that higher classification of emotional states leads to the degradation of the model’s performance. The classification of 16 emotional classes seems to be challenging due to the variation in emotions among each individual and also the label values that are close to one another.

3.7. Statistical Analysis

The Friedman Chi-square statistical analysis test was performed for the machine learning models discussed above in this paper and the corresponding statistic values and p-value is given in

Table 9. Statistical Analysis for 2-class, 4-class, and 16-class.

Statistical Analysis for 2-class, 4-class, and 16-class
2-class
Friedman Test Statistic	19.60431
p-value	0.001482
4-class
Friedman Test Statistic	19.49275
p-value	0.001555
16-class
Friedman Test Statistic	20.0
p-value	0.001249

. For 2-class classification, the estimated p-value is 0.0014 so there is a significant difference between the models. For 4-class classification, the obtained p-value is less than 0.05; therefore, there is a significant difference between the models. The Dunn–Bonferroni test was performed to compare the models in pairs. While performing the Dunn–Bonferroni test, it was evident that there is a significant difference between the KNN and naive Bayes model with p-value of 0.00091, 0.00088, and 0.001123 for 2-class, 4-class, and 16-class, respectively. In addition, random forest is also significantly different from naive Bayes with the p-value of 0.018, 0.0238, and 0.020634 for 2-class, 4-class, and 16-class, respectively. In 16-class classification, the KNN model is also significantly different from LDA with the p-value of 0.01568.

4. Conclusions

The main aim of this analysis is to identify an efficient machine learning model for EEG-based emotion recognition using the VAD model. DEAP dataset has been used for the experimentation of the models. The data were trained using models like SVM, KNN, LDA, naive Bayes, decision tree, and random forest. These models were evaluated using training data and have given varying degrees of accuracy based on the number of classifications and the type of machine learning model used. The performance metrics like accuracy, precision, recall, and F1-score were used to evaluate and compare the model. Among these models, KNN achieved the highest accuracy among all three types of classification, with an accuracy of 87.31% for 16-class, 90.29% for 4-class, and 94.86% for 2-class classification. The naive Bayes model performed least among other models with an accuracy of 7.55% for 16-class, 38.29% for 4-class, and 58.46% for 2-class. Both the random forest and KNN models showed good results for multiclass classification using the VAD model. These results show that these machine learning models could be useful for EEG-based emotion classification with less computational complexity. In the future, these models with proper tuning could be used to provide better results for multiclass classification.