DEEPHER: Human Emotion Recognition Using an EEG-Based DEEP Learning Network Model ^†

Abstract

Emotion identification and categorization have been emerging in the brain machine interface in the current era. Audio, visual, and electroencephalography (EEG) data have all been shown to be useful for automated emotion identification in a number of studies. EEG-based emotion detection is a critical component of psychiatric health assessment for individuals. If EEG sensor data are collected from multiple experimental sessions or participants, the underlying signals are invariably non-stationary. As EEG signals are noisy, non-stationary, and non-linear, creating an intelligent system that can identify emotions with good accuracy is challenging. Many researchers have shown evidence that EEG brain waves may be used to determine feelings. This study introduces a novel automated emotion identification system that employs deep learning principles to recognize emotions through EEG signals from computer games. EEG data were obtained from 28 distinct participants using 14-channel Emotive Epoc+ portable and wearable EEG equipment. Participants played four distinct emotional computer games for five minutes each, with a total of 20 min of EEG data available for each participant. The suggested framework is simple enough to categorize four classes of emotions during game play. The results demonstrate that the suggested model-based emotion detection framework is a viable method for recognizing emotions from EEG data. The network achieves 99.99% accuracyalong with less computational time.

1. Introduction

Emotions are one of humanity’s most distinguishing characteristics; they influence a person’s behavior [1]. Understanding and studying human emotions is a crucial aspect of existence. Automatic emotion categorization using machine learning and artificial intelligence has recently sparked interest, as it may be utilized in a range of human–computer interface (HCI) applications [2,3,4]. To be effective, an artificial emotional intelligence system must have a thorough understanding of human emotional understanding and the connection between affective expression and emotions [5]. Human–machine interfaces and cooperation may be found in a variety of fields, including medicine, therapy, and gaming, to mention a few. Researchers are continually working to improve the flexibility and efficiency of computer–human interaction, with the goal of achieving high levels of user happiness. As a result, HCI systems must be capable of gaining a complete grasp of various human emotions and emotional expressions. Human ideas and emotions can be communicated verbally or nonverbally. Therefore, HCI systems must be able to recognize, interpret, and evaluate nonverbal human expressions.

The rest of the paper is laid out in this manner. Section 2 discusses the associated work. The datasets and experimental methodology, along with model configuration, are described in Section 3. The result of the proposed model is described in Section 4 along with a comparison with the existing models. The conclusion is found in Section 5.

2. Related Work

Specialists are progressively paying attention to emotion identification from EEG data as BCI (brain computer interface) advances, and they often employ a discrete or dimension-based model for emotion categorization. Ekman suggests that anger, contempt, fear, pleasure, sadness, and surprise are six primary emotions in a discrete paradigm [6], whereas Plutchik created an eight-emotion wheel [7]. Because each of these emotions may be exhibited more strongly or less strongly, Russel and Mehrabian suggested both the 2D [8,9] and 3D [10] models portray emotions in electro negativity and enjoyment regions, respectively. Classical machine learning methods have been proposed in the past, but as more publicly released datasets with massive volumes of experimentally acquired brainwaves become available, scholars have begun to use deep learning methods [11]. The authors of [1,12,13,14] used classical ML classifiers SVM, RF, LR, and KNN and achieved accuracy of 93%, 90%, 89.06, and 94.06%, respectively.

Some authors [15,16] also use LSTM architecture and achieve accuracy of 63.6 and 87.25 percent, respectively. Some authors [17,18] reported accuracy rates of 89.66% and 84.21%, respectively, using BiLSTM. The same dataset, GAMEEMO, was used by the author [19] with an accuracy rate of 80%.

Our emotional state might have been disclosed by a brain signal like EEG, and our state of mind could be the only trustworthy means to comprehend it. The most essential challenges for designing a machine learning-based prediction model for emotional state categorization are feature extraction, feature selection, and establishing the correct classifier. To address these concerns, the current study offers a technique for recognizing emotions from EEG data while playing emotional video games.

3. Experimental Methodology

3.1. Selected Data Set and Participants

The GAMEEMO dataset [19], which is freely available, was used in this investigation. The EEG dataset contains signals collected from participants while they were playing emotional video games. Participants were requested to complete the trials for four sessions each in order to compare neurological signals and significant reactions across different individuals and EEG sessions. EEG signals from computer games are included in this collection. They were obtained from 28 distinct people using an Emotive Epoc+ with a 14-channel wearable and portable electroencephalography (EEG) device. Subjects played four different emotional computer games for five minutes each (boring, calm, frightening, and hilarious), with a total of 20 min of EEG data available for each subject. This dataset is intended to serve as an alternative source of data for the emotion recognition process, as well as to compare the performance of wearable EEG devices to standard EEG devices.

3.2. EEG Signal Preprocessing

Because EEG signals respond more quickly to changes in emotional state than other peripheral physiological markers, we will use EEG signals to classify emotions in this work. At a sample rate of 2048 Hz, raw EEG signals were recorded and subsequently down-sampled to 128 Hz in the data gathering experiment. A 0.16–43 Hz band-pass filter is used to remove the EOG artefact from the EEG recordings.

3.3. EEG Signal Classification Method

A memory cell is a new structure introduced by LSTM networks. An input gate (i), a forget gate (f), an output gate (o), and a self-recurrent neuron are the four main aspects of a memory block. Cells can use these gates to store and retrieve info for longer periods of time. The LSTM creates a “constant error carousel” (CEC) for RNNs using the concept of “constant error flow”. This assures that gradients do not decrease. A memory cell serves as the main component, acting as an accumulator (keeping track of the identification relationship) throughout time. Instead of computing the difference between the new and old states as a matrix product, it computes the difference between them. The expressivity is the same, but the gradients are more consistent. The “DEEPHER” deep learning model consists of three LSTM layers of 512,1024, and 2048 neurons, as well as a fully connected embedding layer, one dropout layer, and a dense layer. The LSTM and dropout layers are used to learn attributes from real EEG data. The dropout layer, on the other hand, prevents a large number of units from “co-adapting”, resulting in less over fitting. Finally, the dense layer and the softmax activation function are used to conduct classification. Here, 70% of the EEG data are used to train the model, which is then evaluated on 30% of the EEG data. The Adam optimizer is used in the training phase, with the learning rate of 0.0001.

3.4. Model Complexity

The input length has no influence on the network’s storage needs as LSTM is local in both time and space [20] and, for each time step, the time complexity per weight (for LSTM) is O(1). As a result, the LSTM’s total complexity for every time step is O (w), with w being the number of weights.

4. Result and Discussion

The efficiency of the ‘DEEPHER’ approach on the GAMEEMO database is shown in

Table 1. Performance of the DEEPHER model.

		Precision	Recall	f1-Score	Support
EMOTIONS	0	1.00	1.00	1.00	161,257
	1	1.00	1.00	1.00	160,447
	2	1.00	1.00	1.00	160,493
	3	1.00	1.00	1.00	160,437
	Accuracy			1.00	642,634
	Macro avg	1.00	1.00	1.00	642,634
	Weighted avg	1.00	1.00	1.00	642,634

. Figure 1 shows a confusion matrix of our results influencing the performance factors for each emotion; 0: boring, 1: calm, 2: horror, and 3: funny. It demonstrates the exact precision and proportion of properly detected genuine positives and negatives of sensations. As a result, the information gathered can be utilized as a standard by other researchers. The model is trained using the GAMEEMO database, and it achieves an accuracy of 99.99%. Figure 2 depicts the training-validation accuracy, whereas Figure 3 depicts the training-validation loss. In this study, the greatest identification rate for emotion detection using EEG waves was 99.99 percent.

5. Conclusions and Future Work

In this paper, we describe previously used long short-term memory recurrent neural network (LSTM) architecture for the task of in-depth emotion recognition. This model shows that DEEPHER classification is compatible with cognitive perceptions of emotion, meaning that the model accurately identifies all EEG-related emotions during game play. This allows us to describe the performance of the model and demonstrate what the network is learning. Visualizing what the network learnt provides us with a valuable tool for investigating how emotions were identified and comprehending our network’s behavior. This indicates that the LSTM network is trained to recognize emotions. In this approach, we will be able to correlate our approach to other emotion recognition and classification systems.

During game play, the system can detect four emotion settings for EEG raw data. The technology requires users to wear an EPOC headset and stand in front of a camera to collect raw EEG data. The system can detect emotion in 99.99 percent of EEG data, according to the findings. The suggested model cannot presently detect emotions in real time, which is one of the most significant hurdles in future research.