Abstract
This study proposes 4D-MRSimNet, a framework that employs attention mechanisms to focus on distinct dimensions. The approach applies enhancements to key responses in the spatial and spectral domains and provides a characterization of dynamic evolution in temporal domain, which extracts and integrates complementary emotional features to facilitate final classification. At the feature level, differential entropy (DE) and power spectral density (PSD) are combined within four core frequency bands (, , , and ). These bands are recognized as closely related to emotional processing. This integration constructs a complementary feature representation that preserves both energy distribution and entropy variability. These features are organized into a 4D representation that integrates electrode topology, frequency characteristics, and temporal dependencies inherent in EEG signals. At the network level, a multi-resolution convolutional module embedded with SimAM attention extracts spatial and spectral features at different scales and adaptively emphasizes key information. A bidirectional GRU (BiGRU) integrated with temporal attention further emphasizes critical time segments and strengthens the modeling of temporal dependencies. Experiments show that our method achieves an accuracy of 97.68% for valence and 97.61% for arousal on the DEAP dataset and 99.60% for valence and 99.46% for arousal on the DREAMER dataset. The results demonstrate the effectiveness of complementary feature fusion, multidimensional feature representation, and the complementary dual attention enhancement strategy for EEG emotion recognition.
1. Introduction
As a core component of human cognitive activity, emotion profoundly influences a series of psychological processes including perception, attention, memory, learning, and decision-taking [1]. In affective science’s realm, emotion recognition is a primary focus of current research. It has steadily evolved into a critical technology advancing research and applications in multiple fields encompassing human–computer interaction [2], disease detection [3], and mental workload estimation [4]. Emotion recognition methodologies are primarily categorized into two types. One type relies on non-physiological signals, including facial expressions [5] and speech [6]; the other is based on physiological signals like electroencephalogram (EEG) [7] and electrocardiography (ECG) [8]. EEG not only contains abundant neuro electrical activity features directly associated with emotional states [9,10] but also exhibits advantages in portability, economic efficiency, and temporal resolution [11,12,13]. These advantages provide EEG with high practical applicability, making it a primary data source for emotion recognition.
1.1. EEG Emotion Recognition Methods
Emotion recognition methods are commonly categorized into shallow models and deep models [14]. Their effectiveness and reliability depend primarily on two fundamental processes: feature extraction and emotion classification. Shallow models typically rely on handcrafted feature extraction and employ features that span multiple dimensions. In the time domain, Hjorth parameters [15] effectively reflect signal complexity and dynamic properties, while fractal dimension (FD) [16] describes signal irregularity. In frequency-domain analysis, differential entropy (DE) [17] quantifies the randomness and uncertainty of signals, while power spectral density (PSD) [18] analyzes the energy distribution across frequency components. Wavelet transform (WT) [19] captures both temporal and spectral information, allowing for analysis of signal dynamics across time and frequency. Following feature extraction, emotion classification is typically performed using conventional classifiers, including support vector machines (SVMs) [20] and K-nearest neighbors (KNNs) [17]. Despite their established success, these techniques are limited by their reliance on expert knowledge for feature design and by the inability of handcrafted features to capture high-order nonlinear information associated with emotions. These limitations restrict the effectiveness of shallow models in applying EEG data for emotion recognition.
Due to the limited capacity of shallow models in feature representation and learning, researchers have progressively shifted toward deep models. In recent years, deep learning has become the dominant approach because its ability to perform automatic feature extraction and integrate learning offers clear advantages over shallow methods. For the purpose of detecting in EEG frequency bands connected with emotional states, a deep belief network (DBN) is employed by Zheng and Lu [21]. Yang et al. [22] integrate convolutional neural networks (CNNs) with long short-term memory networks (LSTM) to capture the inherent temporal and spatial dependencies in EEG signals. Regarding the important feature of brain hemispheric asymmetry in EEG emotion research, Huang et al. [23] propose a CNN model constructed based on inter-hemispheric differences, which strengthens the extraction capability of such asymmetric features through targeted network structure design. To model the spatial topological relationships and functional connections of brain regions in EEG signals, graph convolutional networks (GCNs) are introduced into network models [24,25], providing a new approach for effectively using the spatial correlation information between electrodes. Hybrid architectures including 4D-CRNN [26] and DC-ASTGCN [27] have been developed to integrate multi-domain features across frequency, space, and time, demonstrating the effectiveness of synergistic multidimensional feature representation. Compared with shallow models, deep models have exhibited higher efficiency and accuracy in emotion recognition.
1.2. Persistent Challenges in EEG Emotion Recognition
Although deep models achieve better performance than shallow models in emotion recognition, certain limitations still remain. As research on EEG frequency bands and their relation to emotions continues to advance, studies reveal strong associations between the four frequency ranges (: 4–7 Hz, : 8–13 Hz, : 14–30 Hz, and : 31 Hz+) and emotional expression. These findings indicate that each frequency band carries distinct emotional information. Experiments have also confirmed that feature representations fused from the four frequency bands achieve higher performance in emotion classification compared with the independent use of a single frequency band [21]. Li et al. [28] address spatial feature modeling by constructing a 2D sparse graph model. The model incorporates spatial position correlation information of electrode array arrangements to investigate EEG’s spatial topological characteristics. Zhong et al. [29] propose a regularized graph neural network (RGNN) for EEG-based relationship modeling. The RGNN uses biological topological structures across brain regions to capture both global connectivity and local relationships between EEG channels. In emotion recognition, the temporal interdependence of EEG signals holds substantial significance. Chen et al. [30] extract temporal features related to emotion recognition by introducing a gate recurrent unit (GRU) into their model. In the study by Shen et al. [26], differential entropy features extracted from EEG channels are reorganized into a four-dimensional format incorporating frequency, spatial, and temporal domains. A convolutional neural network (CNN) is subsequently employed to perform emotion classification based on these structured features. Although the 4D representation improves the modeling of EEG signals, the distinctions between brain regions and frequency bands are not adequately addressed in their study. To better extract discriminative EEG characteristics and improve the robustness of emotion recognition, attention mechanisms are subsequently employed. This enables models to adaptively emphasize informative features while suppressing irrelevant or noisy components. Neural network models incorporating channel attention [31] and those employing spatiotemporal attention [32] have both been shown to improve the discriminability of key features. The hybrid SMOTE and Trans-CWGAN enhancement strategy proposed by Wang et al. [33] effectively addresses the data imbalance issue in fault diagnosis and provides a referential approach for tackling similar challenges in EEG signal analysis.
These studies have made substantial progress in frequency band synergy, spatial modeling, and temporal capture. However, the challenges still remain in complementary feature fusion across frequency, spatial, and temporal domains; in capturing discriminative local features; and in achieving unified multidimensional modeling of EEG signals.
1.3. Proposed Model: 4D-MRSIMNET
We propose a new framework called a SimAM-Enhanced Multi-Resolution CNN with BiGRU for EEG emotion recognition (4D-MRSimNet).
The 4D-MRSimNet constructs a 4D representation by integrating two features: DE captures the complexity of EEG signals [17,34], while PSD reflects the distribution of signal energy across frequency bands [35]. They provide complementary feature inputs for the emotion recognition model from the dimensions of energy intensity and information complexity, based on a common physical basis [36]. This representation preserves emotional responses associated with the energy attribute and signal irregularity corresponding to the entropy attribute. The representation is constructed by first segmenting the EEG signals into time windows, arranging the electrodes spatially based on a 2D electrode layout map, and then computing the DE and PSD within each frequency band, ultimately forming a four-dimensional spatial–spectral–temporal representation. These features are organized along three key dimensions: electrode topology, frequency band, and time. Compared to using a single type of feature, this integration enables a more comprehensive capture of the affective information contained in EEG signals, providing a more complete and informative feature representation.
Considering both local activation and global responses in emotional arousal, we adopt the multi-resolution convolutions with the SimAM attention mechanism to realize frequency and spatial feature learning. The multi-resolution convolutional is designed to comprehensively capture both local and global features. Large kernel convolutions capture global collaborative responses of emotional arousal with a wide receptive field, characterizing the holistic neural patterns of emotions. Small kernel convolutions focus on local regions, capturing local activation differences in emotional brain areas and restoring fine-grained neural representations of emotions. We introduce the parameter-free SimAM attention mechanism into the multi-resolution CNN. We apply a 3D attention mechanism to the input feature containing spatial–spectral information. This mechanism emphasizes feature points within the three-dimensional structure that are significantly more important than their surroundings. It further enhances the model’s ability to capture spatial and spatial information from EEG signals. This design enhances spatial–spectral features and adds no extra parameters. This keeps the model light and reduces overfitting risk. The generated 3D attention weights adapt to the electrode layout and frequency band structure. This gives two advantages. First, it avoids information loss from separating channel and spatial dimensions. Second, it allows for precise spatial feature extraction without complex designs. Our approach achieves effective spatial–spectral modeling through a simpler and more efficient attention-based design.
To retain the past and future contextual information required for emotion recognition tasks in EEG signals and accurately capture the dynamic characteristics of emotions over the temporal dimension, our study adopts a BiGRU combined with a temporal attention mechanism for temporal modeling. This design focuses on the segments containing key emotional information within the bidirectional context through the temporal attention mechanism. The BiGRU comprehensively models the dynamic emotional processes in EEG signals from both forward and reverse directions, while the temporal attention mechanism assigns higher weights to emotionally salient time steps, strengthening the representation of core features. This mechanism effectively addresses two core issues in EEG emotion recognition: First, traditional unidirectional GRUs can only process signals along a single temporal dimension, leading to the loss of contextual information and making it difficult to fully capture the continuous features of emotional evolution over time. Second, compared with the equal-weight processing of all time steps in EEG signals, the temporal attention mechanism can avoid the dilution of core emotional features by irrelevant temporal information, while suppressing background noise interference, thus resolving the problem that traditional methods lack sensitivity to key dynamic segments of emotions.
In summary, our model starts with a 4D representation. It employs a complementary dual attention design. SimAM attention refines the spatial–spectral features. Temporal attention focuses on key time steps from the BiGRU output. Such a spatial–spectral refinement followed by a temporal focusing technical pathway improves the quality of emotional dynamics representation and enhances the accuracy of emotion classification.
1.4. Main Contributions
The main contributions are as follows:
- By integrating DE and PSD features, we construct four-dimensional emotion discriminative representations that cover the four major EEG frequency bands (, , , and ). This integration allows for a more comprehensive capture of emotional information contained in EEG signals and provides a more complete and informative feature representation compared with the use of a single feature type.
- We propose a framework that employs a complementary dual attention mechanism for multidimensional EEG feature learning. The architecture integrates multi-resolution convolution enhanced by SimAM for spatial–spectral feature extraction, along with a BiGRU model augmented with temporal attention to capture temporal dynamics. This design enables collaborative learning of emotional representations across different dimensions.
- Experiments on the DEAP dataset and the DREAMER dataset demonstrate that the proposed model outperforms the comparative models in both accuracy and stability across emotional categories.
2. Methods
Figure 1 illustrates the overall architecture of the proposed 4D-MRSimNet framework for EEG emotion recognition. The framework is composed of four major modules. The first module generates 4D feature representations by integrating spatial, spectral, and temporal information from EEG signals. The second module employs a multi-resolution convolutional neural network embedded with the SimAM attention mechanism to extract spatial and spectral features at different scales. The third module models temporal dynamics through a BiGRU combined with a temporal attention mechanism to capture contextual dependencies. The final module is a classifier that produces the recognition results.
Figure 1.
Overall framework of 4D-MRSimNet, including signal acquisition, time–frequency feature extraction (PSD and DE), multi-resolution convolutions embedded with SimAM attention, a BiGRU network with temporal attention, and a classifier.
2.1. Four-Dimensional Representation of Spatial–Spectral–Temporal Features
The performance of emotion recognition tasks is closely tied to the way in which features are extracted and characterized. In this study, DE and PSD are adopted as feature computation approaches to ensure that the extracted features retained crucial EEG information under complex and noisy conditions. The raw EEG signals are subsequently transformed into four-dimensional feature representations integrating spectral, spatial, and temporal characteristics. Figure 2 illustrates the construction process of 4D feature representations.
Figure 2.
Framework for 4D EEG feature construction, including raw signal segmentation, frequency band decomposition, PSD and DE feature extraction, and 2D sparse mapping to generate the 4D representation.
To obtain the 4D representation that incorporates spatial–spectral–temporal features, we referred to previous work [22,37]. Firstly, the continuous EEG data is partitioned into successive segments of T seconds, and the related experimental labels are allocated to them. Secondly, each segment is decomposed into four frequency bands (, , , and ) using a fourth-order Butterworth filter, and from every band, DE and PSD features are extracted through a 0.5 s non-overlapping analysis window. According to the study by Zheng et al. [36], these two types of features have been demonstrated to provide stable and reliable performance in the task of emotion recognition.
DE features can distinguish differences in EEG patterns between low-frequency and high-frequency energy. They are also applied to quantitatively evaluate the complexity of EEG signals [17,34], and this process is defined as follows:
where X denotes a random variable, and is the probability density function of X. The calculation of the DE feature for the Gaussian distribution is as follows:
where X follows a Gaussian distribution , e refers to the Euler constant, and denotes the standard deviation of the time series. The PSD feature is used to quantify the intensity of the energy distribution of the signal within a specific frequency band, which is defined as follows:
where x is a random variable that herein corresponds to the signal acquired from a specific EEG channel within a certain frequency band.
Third, we organize the DE features and PSD features of four frequency bands to form 2D maps. These maps are stacked along the channel dimension to derive 3D features. Through these operations, we obtain N segments represented by a 4D structure, which is denoted as where . In the definition of , h and w denote the height and width corresponding to the 2D map; d represents the number of frequency bands; stands for twice the segment length; and N is the total number of samples. The parameter settings in are explained as follows: , with N denoting the total number of samples. h and w are associated with the electrode positions that contain spatial structure information. Based on the 2D map with 62 channels in Figure 3 (where unused channels are marked with the number zero), we set and . The 2D maps of different frequency bands are stacked to form 3D maps that incorporate both diverse frequency bands and spatial structures, where d corresponds to the four frequency bands for both the DE and PSD features. When processing each EEG segment, a 0.5 s window is adopted to compute the features for each frequency band and perform normalization, which provided temporal features for constructing the 4D map. Each window can be denoted as . As a consequence of this setting, the segment length needs to be set to twice the length of the original signal, denoted as .
Figure 3.
Conversion of the distribution of EEG electrode channels into a 2D matrix.
2.2. Frequency and Spatial Feature Learning
To enhance the model’s ability to identify key features, we introduce a lightweight attention module. Similar to the unequal protection strategy based on feature importance [38], this mechanism allows for effective selective enhancement of key features. In our design, the feature extraction network outputs a tensor organized across spatial (electrode topology) and frequency band dimensions. This tensor can be regarded as a sequence of static spatial–spectral representations at each time step. The dynamic modeling of the temporal dimension is entrusted to a dedicated BiGRU module in the subsequent stage. Therefore, we propose three core requirements for the attention mechanism at this phase: (1) It should support holistic optimization of the 3D spatial–spectral tensor to strengthen its inherent spatial–spectral correlations. (2) It must incur minimal increase in computational cost and parameter count to ensure efficiency. (3) It should feature a concise structure that can seamlessly integrate into the current architecture.
We evaluate various classic lightweight attention mechanisms. First, the SE [39] module and its efficient variant ECA-Net [40] are purely channel-wise attention methods. SE applies channel weighting via fully connected layers, while ECA-Net improves efficiency by leveraging 1D convolution. These methods ignore spatial dimension information, a major drawback for our task that relies on spatial–spectral correlations. Second, although the CBAM [41] module attempts to combine channel and spatial attention, its spatial attention is designed based on 2D convolution. Forcing its application to our 3D spatial–spectral structure would result in high computational costs and adaptation challenges. Non-local networks [42] effectively capture long-range dependencies. But their computational complexity is proportional to the square of the input size, leading to a prohibitive computational burden when processing our data. Attentional mechanisms have been similarly validated in other fields such as HVAC fault detection [43], demonstrating their efficacy in processing multi-sensor inputs and complex temporal patterns, further highlighting their potential for challenging signal processing tasks.
Based on the above analysis, we finally selected the SimAM attention mechanism [44]. Grounded in energy function theory, this module derives a unified 3D weight mask for each neuron, eliminating the need for artificial distinction between channel and spatial dimensions. This characteristic allows the generated 3D weights to naturally align with the topological distribution of electrodes and frequency band combinations in the feature maps. This alignment preserves and enhances the intrinsic spatial–spectral correlations of EEG data. SimAM is parameter-free, with weights computed directly through an analytical solution which introduces almost no additional model complexity or training overhead. Its inherent 3D processing capability and extreme lightweight nature make it an ideal choice for this application scenario. Figure 4 illustrates the architecture of the SimAM attention mechanism, and its energy function is formulated as follows:
Figure 4.
Architecture of the SimAM attention mechanism, illustrating weight generation and fusion across channel, spatial, and 3D dimensions.
Each channel contains M energy functions, and the expressions of and are as follows:
and represent the mean and variance of all neurons in this channel except for t. The minimum energy can be obtained by the following formula:
As the formula shows, the lower the energy value of neuron t, the more significant its difference from surrounding neurons and the higher its importance. We can assess the importance through computing , the reciprocal of the energy value. To enable the attention mechanism to efficiently focus on key features, we enhance the weighting of neurons with higher importance. This increases the network’s sensitivity to relevant inputs and assists the model in accurately identifying and utilizing critical information. The specific strengthening process is as follows:
To boost the CNN’s ability to extract feature information, we introduce the SimAM attention mechanism into the spaces between convolutional layers.
In this paper, a terminal pooling multi-resolution CNN architecture is designed for feature extraction. This architecture consists of two parallel branches, each containing three convolutional layers. In this architecture, two separate branches utilize convolutional kernels that differ in dimensions. While smaller kernels focus on capturing intricate local details, larger ones specialize in extracting broader global features. This combination strengthens the model’s ability to retrieve features across a variety of scales. Subsequent to each convolutional layer, we apply the ReLU activation function, and a SimAM attention module is embedded after the first convolutional layer of each branch. This module can dynamically focus on the critical feature regions within sample while specifically suppressing noise and irrelevant background interference. This selective enhancement mechanism enhances the discriminability of initial features while at the same time offering a more targeted and effective basis for feature extraction in subsequent convolutional layers. In contrast to the conventional pooling operations distributed across multiple layers of CNNs, we adopt a terminal pooling design. A unique pooling layer is added only after feature fusion, following all convolutional operations. This design stems from the small size of the 2D maps of sample , which requires maximal preservation of spatial feature details. Terminal pooling reduces parameter redundancy and the risk of overfitting. This approach allows the network to capture and retain critical feature information in sample with greater effectiveness. is first input into the two branches in parallel. Branch 1 (local feature extraction) consists of Conv2D(64, 5, 1), Conv2D(128, 3, 1), and Conv2D(128, 3, 1); Branch 2 (global feature extraction) consists of Conv2D(64, 7, 1), Conv2D(128, 5, 1), and Conv2D(128, 5, 1). The selection of convolutional kernel sizes is based on considerations of the spatial characteristics of EEG signals and multi-scale feature extraction. The input spatial dimensions (8 × 9) correspond to the two-dimensional distribution layout of electrodes. In the local branch, the 5 × 5 convolutional kernel has a receptive field capable of covering multiple adjacent electrodes, enabling the network to capture the basic spatial patterns of neural activity and synchronization between neighboring electrodes at a medium local scale. Subsequently, we introduce a SimAM attention module, which enhances the overall feature representation, allowing the network to adaptively reinforce informative neurons within these local feature maps without introducing additional learnable parameters. Immediately following this, the stacking of smaller 3 × 3 convolutional kernels further focuses on finer local neighborhood relationships while reducing the parameter count. The combination of two 3 × 3 kernels provides more nonlinear transformations, thereby enhancing the representational capacity of the network. In the global branch, the 7 × 7 convolutional kernel possesses a larger receptive field, enabling coverage of a broader electrode area. It aims to capture more macroscopic EEG activity patterns at a global or hemispheric scale. Similarly, a SimAM attention module is applied afterward. Its lightweight nature allows it to effectively focus on critical global features without introducing significant additional computational burden, thereby improving feature discriminability. Building upon this, 5 × 5 convolutional kernels are used to progressively refine and focus the global features, extracting more discriminative regional activity patterns from the broad representations. The stacking of two 5 × 5 kernels further integrates information across medium scales. After fusing the features from the two branches, terminal pooling is performed using MaxPooling2D(2, 2). The feature maps are subsequently converted into a 1D vector through the Flatten operation, and this vector is fed into a fully connected layer with 256 neurons. The final output is reshaped into a 1D structure using Reshape. In this way, the original EEG’s frequency and spatial information may be depicted as . Figure 5 illustrates the complete flow of the aforementioned feature extraction, presenting the architecture of the multi-resolution 2D convolutional module embedded with SimAM attention.
Figure 5.
Architecture of the multi-resolution 2D convolutional module embedded with SimAM attention.
2.3. Temporal Feature Learning
After enhancing the spatial–spectral feature representation of each time step via the SimAM module, the model needs to model the obtained series of dynamic temporal features to ultimately accomplish the emotion classification task. To this end, we adopt a scheme combining BiGRU with a temporal attention mechanism, and this design is mainly based on the following three considerations: (1) Utilizing outputs of all time steps to capture emotional dynamics. Emotions exhibit fluctuating and evolving characteristics over time, and EEG signals can precisely reflect the instantaneous changes in brain activities. Compared with relying solely on the final state of recurrent units, fully leveraging all time-step hidden states of BiGRU enables the model to capture the dynamic evolution trajectory of emotions, thereby acquiring more detailed feature information and effectively improving the classification performance for complex emotions. (2) Selecting GRU to achieve a balance between efficiency and performance. By merging gating units, GRU has a more concise structure and significantly fewer parameters compared to traditional LSTM. In scenarios with small-scale EEG data, this lightweight characteristic not only effectively reduces the risk of model overfitting but also brings a faster training speed and lower resource consumption, achieving an optimized balance between model efficiency and classification performance. (3) Introducing temporal attention to focus on key emotional frames. Although BiGRU can effectively capture temporal dynamic features, these outputs are often treated equally during feature aggregation of all time-step outputs. To address this, we introduce a temporal attention mechanism that can automatically distinguish all hidden states output by BiGRU, assigning higher weights to discriminative moments and realizing hierarchical processing from sequence understanding to key point focusing.
As a classic variant of RNNs, the GRU leverages its streamlined gating mechanism to achieve an optimal balance between temporal modeling capability and computational efficiency [45], which makes it widely used in sequential data processing. The key computations of the GRU are formulated as follows:
denotes the logistic sigmoid function. and represent the reset gate and update gate of the GRU unit. is the candidate unit state. is the hidden state. W and b are the learnable weight and bias parameters. ⊙ denotes element-wise multiplication. The GRU is effective in modeling temporal variations in EEG signals with its gating mechanism, and unlike earlier methods that depend only on the final state, our design makes use of sequential outputs to strengthen classification. This architecture integrates a temporal attention mechanism and BiGRU to achieve temporal modeling and dynamic weighting of feature sequences.
The BiGRU is made up of two unidirectional GRUs: a forward one (GRU+) and a backward one (GRU−). Each GRU employs 128 hidden units, resulting in a concatenated output dimension of 256. This choice balances representational capacity with parameter efficiency: 128 units per direction provide sufficient complexity to model temporal dynamics in EEG sequences while avoiding parameter redundancy that could lead to overfitting on limited physiological data. The total parameter count for the BiGRU module remains moderate, ensuring computational feasibility and fast inference. Compared to the unidirectional GRU, the bidirectional GRU can capture more historical and future temporal information. It models contextual dependencies more accurately. The forward temporal module (GRU+) takes the feature sequence output by the CNN as input, while the backward temporal module (GRU−) takes the reverse-ordered sequence of this feature sequence, , as input. The output tensors of the bidirectional temporal modules, and , are concatenated at each time step to obtain the output of the BiGRU, (where ).
To better capture information from key steps in the sequence and strengthen the emphasis on critical temporal characteristics, a temporal attention module is applied to the feature sequence generated by the BiGRU. The temporal attention mechanism consists of a two-layer fully connected network, including linear transformation layers, nonlinear activation functions, and probability normalization operations. The formula associated with the temporal attention mechanism is outlined below:
The learnable parameter set can map the input to through feature dimensionality reduction. After the normalization process, the obtained represents the importance weights of each time segment, and this module effectively captures the dynamic temporal features of EEG signals via an adaptive weighting mechanism. The high-level representation of the 4D sample can be defined as follows:
integrates the spatial, spectral, and temporal information of the input data. The specific results of the BiGRU incorporating a temporal attention mechanism are shown in Figure 6.
Figure 6.
Architecture of the BiGRU network with temporal attention, showing bidirectional sequence encoding, temporal attention on key segments, and integrated feature representation.
2.4. Classifier
Based on the 256-dimensional representation vector, to forecast the category of a sample, we start with the 256-dimensional feature vector . Sample categories are predicted using a fully connected layer followed by a softmax activation function. The implementation is given as follows:
is the input feature vector that integrates spatial, spectral, and temporal information. is the learnable weight matrix, is the learnable bias vector, and is the predicted probability vector representing the probabilities of the sample belonging to each of the two classes. The final predicted label is determined by the class which holds the greatest probability.
3. Experiment
3.1. Datasets
This study uses two public authoritative benchmark datasets in affective EEG research, DEAP [46] and DREAMER [47], for experimental evaluation. As multimodal affective physiological signal databases, they offer comprehensive resources for multidimensional emotion recognition and facilitate the investigation of emotional states based on physiological signals.
The DEAP dataset records physiological data from 32 healthy participants (half male and half female) while they watch 40 carefully selected one-minute music video clips. These videos are evaluated in advance to ensure that they can effectively induce emotional experiences of varying intensities and types. During data collection, after watching these video, participants rate their perceived arousal, valence, liking/disliking, dominance, and familiarity on a 1–9 scale, where a score of 5 serves as the threshold for binary classification. The EEG signal of each clip includes 3 s of baseline signal and 60 s of trial signal. Before release, the DEAP dataset undergoes preprocessing, including the removal of power line interference and artifacts and the application of a frequency filter in the range of 4.0–45.0 Hz. We perform binary emotion recognition tasks (high/low arousal; positive/negative valence) based on participants’ arousal and valence ratings, with 5 used as the threshold for these classifications.
The DREAMER dataset comprises three types of physiological signals: electroencephalogram (EEG), electrocardiogram (ECG), and galvanic skin response (GSR) signals. EEG signals are collected via 14 electrodes with a sampling rate of 128 Hz, enabling the capture of electrical activity characteristics from multiple brain regions. It records multimodal physiological data of 23 healthy participants while they watch 18 video clips. Similar to the DEAP dataset, after being induced to experience emotions of varying intensities and types, the participants complete multidimensional subjective emotional evaluations based on their own perceptions. The dataset also undergoes a rigorous preprocessing procedure prior to its release. In our study, we perform binary emotion recognition tasks (high/low arousal; positive/negative valence) based on participants’ arousal and valence ratings, with 3 used as the threshold for these classifications (DREAMER dataset uses a 1–5 Likert scale).
3.2. Experimental Setup
Our proposed model adopts a training batch size of 64, a maximum iteration number of 100 training epochs, and a learning rate of 0.001 that is used in conjunction with the Adam optimizer. The model is developed using Keras within Google TensorFlow and is trained on a NVIDIA GeForce RTX 3090 GPU. The performance of all EEG emotion recognition methods is evaluated following a protocol similar to Yang et al. [37]. Each trial is conducted under a five-fold cross-validation framework. This study adopts a subject-dependent experiments approach. We can get 4800 EEG samples on the DEAP dataset and 7452 EEG samples on the DREAMER dataset. Taking the DEAP dataset as an example, a total of 4800 continuous 0.5 s feature segments are obtained for each subject after preprocessing the 40 trials (calculated as 40 trials × 60 s × 2), where “×2” indicates the use of non-overlapping 0.5 s windows to segment each second of the signal. To capture the dynamic changes in emotions, every 6 consecutive segments are concatenated to form a time-series sample, resulting in the construction of 800 samples per subject. A five-fold stratified split is applied independently to each subject’s samples. In each validation round, four folds (640 samples) are used for training, while the remaining one fold (160 samples) is reserved for testing. This process is repeated five times so that each fold serves as an independent test set once. Performance indicators include accuracy (ACC) and standard deviation (STD) for each participant. To obtain the final evaluation of the method, these metrics are aggregated by computing the overall mean ACC and STD across all subjects.
4. Results
4.1. Baseline Models
We compare several representative models spanning diverse architectures, including 4D-CRNN, BiDCNN, RACNN, ACRNN, CapsNET, and DGCNN. The comparative results on the DEAP and DREAMER datasets are summarized in Table 1 and Table 2.
Table 1.
The performance (average ACC ± STD (%)) of the proposed and baseline models on the DEAP dataset.
Table 2.
The performance (average ACC ± STD (%)) of the proposed and baseline models on the DREAMER dataset.
4D-CRNN: Uses a CNN-RNN architecture for 4D EEG feature processing.
BiDCNN: Models hemispheric asymmetry via a multi-input CNN.
RACNN: A regional-asymmetric CNN designed to capture local and inter-hemispheric features.
ACRNN: Integrates attention mechanisms into a CNN-RNN framework.
CapsNET: Employs capsule networks and attention for emotion dimension recognition.
DGCCN: Utilizes dynamic graph convolutional networks to model interactions among EEG channels.
On the DEAP dataset, we achieve significant improvements of 3.30% in valence and 2.89% in arousal compared to the BiDCNN. We also outperform the 4D-CRNN model by significant margins of 3.46% in valence and 3.03% in arousal accuracy. Compared with RACNN, we achieve a 1.03% improvement in valence accuracy while continuing to hold an advantage in arousal. The advantage over ACRNN is the most substantial, with a massive lead of 3.96% in valence and 4.23% in arousal. Compared to the newer CapsNET model, our model also demonstrates advantages in terms of both valence and arousal.
On the DREAMER dataset, our model also demonstrates strong performance. Compared to the ACRNN, our model elevates the valence accuracy from 97.93% to 99.60%, a substantial absolute improvement of 1.67%. For arousal, the accuracy is boosted from 97.98% to 99.46%, marking a 1.48% absolute gain. Compared to the earlier DGCCN model, our medel demonstrates an overwhelming performance advantage, achieving an improvement of 13.37% in valence accuracy and 14.92% in arousal accuracy. When compared to the RACNN model, we still achieve steady improvements of 4.05% in valence and 2.45% in arousal accuracy. It is noteworthy that our model shows significant superiority over the CapsNET model which performs well on DEAP, with absolute gains of 4.64% in valence and 3.92% in arousal. Compared with baseline models, our proposed 4D-MRSimNet model achieves better performance in classification accuracy and standard deviation across both the DEAP and DREAMER datasets, demonstrating its effectiveness and reliability for EEG emotion recognition.
4.2. Comparisons with State-of-the-Art Models
We also compare the 4D-MRSimNet performance with that of several state-of-the-art models (4D-aNN, LResCapsule, DGGN, STFCGAT, SS4-STANN, TDMNN, SAG-CET, DC-ASTGCN, 1D-CNN-BiLSTM, TC-Net, and RGNet-GCN) in a comparative study. These models draw inspiration from the following baseline models for their improvements: for example, 4D-aNN references the designs of 4D-CRNN and ACRNN, LResCapsule evolved from CapsNET, and the other models have also been extended and optimized based on the baseline architectures. The evaluation focuses on the valence and arousal emotional dimensions under subject-dependent experimental settings using the DEAP dataset and DREAMER dataset. For performance measurement, we employ both ACC and the STD as evaluation metrics, which together provide a more reliable assessment of stability and precision. Table 3 presents a comparison with the state-of-the-art model on the DEAP dataset; Table 4 shows the corresponding comparison on the DREAMER dataset.
Table 3.
The performance (average ACC ± STD (%)) of the proposed and state-of-the-art models on the DEAP dataset.
Table 4.
The performance (average ACC ± STD (%)) of the proposed and state-of-the-art models on the DREAMER dataset.
On the DEAP dataset, we achieve significant improvements of 4.32% in valence and 3.19% in arousal compared to the DC-ASTGCN. With respect to the 4D-aNN and DGGN, our method achieved steady gains against DGGN and we improved by 0.70% in valence and 0.42% in arousal compared to 4D-aNN where we achieved gains of 0.78% in valence and 0.22% in arousal. Our model outperforms STFCGAT by 1.98% in valence and 2.57% in arousal and shows advantages over SS4-STANN with improvements of 2.08% in valence and 0.61% in arousal. Compared with LResCapsule, our model also maintains slightly superior performance.
On the DREAMER dataset, our model also demonstrates strong performance. Compared to state-of-the-art models, our method achieves valence accuracy of 99.60% and arousal accuracy of 99.46%. Against recent peer approaches, our model exhibits comprehensive advantages in both metrics: it outperforms 1D-CNN-BiLSTM (98.41%; 98.23%), TC-Net (98.59 ± 1.38%; 98.61 ± 1.34%), TDMNN (99.45 ± 0.91%; 99.51 ± 0.79%), LResCapsule (95.15 ± 3.51%; 95.77 ± 3.82%), and RGNet-GCN (99.17 ± 0.85%; 99.06 ± 1.29%). Compared to LResCapsule with 4.15 M parameters, our model uses only 2.2 M parameters, corresponding to a reduction of approximately 47%, while achieving notable absolute improvements of 4.45% in valence and 3.69% in arousal, which demonstrates superior parameter efficiency. Even compared to TDMNN (4.59 M parameters), whose performance is close to ours, our model maintains comparable accuracy with only about half the parameters, indicating lower model complexity and higher computational efficiency. Our results show good performance both in terms of accuracy and standard deviation, further proving the model’s excellent performance and robust generalization capability.
Compared to existing methods, the proposed 4D-MRSimNet model demonstrates distinct advantages. Prior works like 4D-CRNN, BiDCNN, LResCapsule, 4D-aNN, and DGGN suffer from certain limitations, including insufficient temporal modeling, shallow feature fusion, high computational complexity, and strong reliance on specific feature representations. Our design enhances intrinsic spatial–spectral correlations through the SimAM attention mechanism and achieves precise modeling of emotional dynamics via the synergy between BiGRU and temporal attention. The integration of 4D features (PSD + DE) significantly improve feature representation, while the multi-resolution CNN with SimAM module, terminal pooling, and BiGRU design further optimize computational efficiency. Our model also achieves a balance between high accuracy, low parameter count, and fast inference speed. Experimental results show that the model achieves classification accuracies of 97.68 ± 1.72% for valence and 97.61 ± 2.20% for arousal on the DEAP dataset and 99.60 ± 0.87% for valence and 99.46 ± 0.72% for arousal on the DREAMER dataset. Table 5 and Table 6 summarize the performance of 4D-MRSimNet on the DEAP dataset and DREAMER dataset, listing the ACC and STD for both dimensions (the STD reported in Table 5 and Table 6 are provided solely to describe the dispersion of performance cross-validation folds). Figure 7 and Figure 8 illustrate the overall performance of 4D-MRSimNet in experiments conducted on the DEAP dataset and DREAMER dataset.
Table 5.
Performance of 4D-MRSimNet on the DEAP dataset across the valence and arousal dimensions, evaluated using ACC and STD.
Table 6.
Performance of 4D-MRSimNet on the DREAMER dataset across the valence and arousal dimensions, evaluated using ACC and STD.
Figure 7.
Classification accuracy of the 4D-MRSimNet model on the DEAP dataset across 32 subjects.
Figure 8.
Classification accuracy of the 4D-MRSimNet model on the DREAMER dataset across 23 subjects.
4.3. Ablation Experiments
The 4D-MRSimNet comprises a multi-resolution CNN with an embedded SimAM attention module that fuses DE and PSD features, a BiGRU with a temporal attention module, and a classifier. To systematically evaluate the contribution of each core component, multiple ablation experiments are conducted on the DEAP dataset and DREAMER dataset. Comparisons among different module combinations reveal the influence of complementary feature fusion and network design incorporating the complementary dual attention enhancement strategy on classification accuracy. Figure 9 illustrates the ablation results for valence and arousal on the two datasets. DE + PSD represents our proposed 4D-MRSimNet, while DE and PSD denote the cases using only a single type of feature input. Experiments Exp.1, Exp.2, and Exp.3 correspond to the removal of the SimAM attention module, the removal of the temporal attention module, and the replacement of BiGRU with GRU.
Figure 9.
Ablation study results of the 4D-MRSimNet for the valence and arousal dimensions on the DEAP and DREAMER datasets.
For validation at the complementary feature fusion level, the focus is on the fusion of DE and PSD. Three control experiments are designed. On the DEAP dataset, when only DE features are used as input, the accuracy of the model in the valence and arousal dimensions reaches 96.26 ± 2.86% and 96.78 ± 2.46%. When only PSD features are used, the performance decreases to 88.79 ± 11.43% and 91.79 ± 9.48%. With the adoption of the feature fusion strategy, the accuracy rises markedly to 97.68 ± 1.72% and 97.61 ± 2.20%. Similarly, on the DREAMER dataset, when only DE features are used, the accuracy for valence and arousal is 97.81 ± 2.25% and 97.52 ± 2.62%. When only PSD features are used, the accuracy drops to 83.19 ± 11.00% and 82.34 ± 11.66%. With feature fusion, the accuracy improves significantly to 99.60 ± 0.87% and 99.46 ± 0.72%. Table 7 lists detailed results for direct comparison.
Table 7.
Classification performance comparison of feature fusion (DE + PSD) versus single features (DE; PSD) on the DEAP dataset and DREAMER dataset.
The results indicate that the accuracy in both dimensions of the experiment improves significantly when complementary features are used together compared with the use of a single feature. Comparative visualizations on the DEAP dataset are presented in Figure 10 for valence and in Figure 11 for arousal. DE features capture the instantaneous energy fluctuations of EEG signals, and these fluctuations directly reflect the temporal variation patterns of EEG signals. PSD features are more effective in representing the distribution of frequency components. The complementarity between the two allows the fused feature set to integrate important information from both the time and frequency domains. This integration produces a more comprehensive and information rich representation that overcomes the inherent limitations and performance bottlenecks of using a single feature. The synergistic effect of feature fusion is further evidenced in Figure 12, which presents the t-SNE visualization for the first-fold data of subject 17 under the valence condition on the DEAP dataset. The fusion of DE and PSD features facilitates more compact clustering of instances within the same class and a more distinct separation between different classes. The clustering patterns obtained with DE or PSD alone are less distinct. This visual evidence confirms that the fused feature set enhances feature discriminability more effectively than either feature in isolation.
Figure 10.
Accuracy comparison of DE, PSD, and DE+PSD feature inputs across subjects for the valence dimension on the DEAP dataset.
Figure 11.
Accuracy comparison of DE, PSD, and DE+PSD feature inputs across subjects for the arousal dimension on the DEAP dataset.
Figure 12.
t-SNE visualization of Subject 17’s first fold data. It compares DE, PSD, and DE + PSD features, with fused features showing tighter clusters and clearer boundaries.
At the network design incorporating attention mechanism level, ablation experiments further clarify how the attention mechanism and BiGRU exert a synergistic influence. We take the accuracy of the complete model in the valence and arousal experiment on DEAP (97.68 ± 1.72% and 97.61 ± 2.20%) and DREAMER (99.60 ± 0.87% and 99.46 ± 0.72%) as the benchmark and conduct three sets of comparative experiments: Experiment 1: After removing the SimAM attention mechanism, the model shows blurred feature responses in the recognition of complex EEG patterns, and the accuracy decreases to 97.33 ± 1.86% and 97.25 ± 2.39% on DEAP, 98.52 ± 1.53% and 98.67 ± 1.44% on DREAMER. Experiment 2: When the temporal attention mechanism is removed, the model’s ability to capture emotional variations with strong temporal dependence decreases, and its ability to focus on key temporal nodes also decreases. The accuracy falls to 97.23 ± 2.11% and 97.37 ± 2.40% on DEAP, 98.87 ± 1.29% and 98.40 ± 1.61% on DREAMER. Experiment 3: After removing the bidirectional GRU module and replacing it with a unidirectional GRU, the accuracy of the two dimensions drops to 97.08 ± 2.22% and 97.10 ± 2.71% on DEAP, 98.51 ± 1.65% and 98.74 ± 1.41% on DREAMER. The detailed results are summarized in Table 8 for comparison.
Table 8.
Classification performance comparison of the full 4D-MRSimNet versus its structural ablation variants (Experiments 1–3: removing SimAM, removing temporal attention, and replacing BiGRU with GRU).
The results indicate that the network relies on all core components, and the complete model produces higher accuracy in both valence and arousal than any variant with components removed. Comparative visualizations on the DEAP dataset are provided in Figure 13 for valence and Figure 14 for arousal. The SimAM attention mechanism is crucial for improving the quality of feature responses, and its absence diminishes the recognition of complex EEG patterns, causing accuracy to decline. The temporal attention mechanism strengthens the modeling of time-dependent changes in emotion by assigning greater weight to critical temporal segments, and removing it diminishes the model’s ability to capture variations over time. The BiGRU captures dependencies in both temporal directions, and substituting it with a GRU lowers performance more substantially. These components collaborate hierarchically: SimAM enhances spatial and frequency feature discriminability, BiGRU refines temporal contextual modeling, and temporal attention prioritizes key time segments. This synergy allows 4D-MRSimNet to overcome individual component limitations, validating the structure’s rationality. t-SNE visualizations in Figure 15, based on the first fold experiment data from 10 subjects on the DEAP dataset under the valence condition, show that the complete model generates compact and clearly separated feature clusters. Models with components removed, such as SimAM disabled in (a), temporal attention disabled in (b), or a GRU in (c), display less distinct clusters. These results demonstrate that embedding SimAM into the multi-resolution feature extraction, together with the BiGRU and temporal attention, plays a key role in enhancing feature representations and ensuring effective emotion recognition.
Figure 13.
Accuracy comparison across subjects for the valence dimension on the DEAP dataset, contrasting the full 4D-MRSimNet model with its variants (SimAM ablation, temporal-attention ablation, and GRU substitution).
Figure 14.
Accuracy comparison across subjects for the arousal dimension on the DEAP dataset, contrasting the full 4D-MRSimNet model with its variants (SimAM ablation, temporal-attention ablation, and GRU substitution).
Figure 15.
Row-wise comparisons of feature distributions: (a) SimAM attention (enabled vs. disabled); (b) temporal attention (enabled vs. disabled); (c) bidirectional GRU vs. unidirectional GRU.
5. Conclusions
This study presents 4D-MRSimNet, a multi-resolution CNN and BiGRU architecture with dual attention mechanisms that balances model efficiency and recognition performance for EEG emotion recognition. It integrates DE and PSD features, combining DE’s capture of signal complexity with PSD’s clear band-power information to create a complementary representation. The SimAM attention mechanism is applied to the multi-resolution CNN to enhance critical spatial and spectral patterns. The temporal attention mechanism is applied to the BiGRU to prioritize emotionally salient segments across the sequence. Our model achieves 97.68 ± 1.72%/97.61 ± 2.20% (valence/arousal) on the DEAP dataset and 99.60 ± 0.87%/99.46 ± 0.72% on the DREAMER dataset. The results demonstrate the effectiveness of complementary feature fusion, multidimensional feature representation, and the complementary dual attention enhancement strategy for EEG emotion recognition. Future research will explore finer frequency band analysis methods and adaptive frequency band methods to construct more discriminative spectral features. Furthermore, we will conduct research on multimodal emotion fusion and subject-independent emotion recognition, enhance the cross-dataset generalization ability of the model, and promote the lightweight implementation of the model for wearable device scenarios.
Author Contributions
Methodology, Y.H.; software, Y.H.; validation, Y.H.; formal analysis, Y.H.; investigation, J.D.; writing—original draft preparation, Y.H.; writing—review and editing, J.D.; visualization, Y.H.; supervision, J.D.; project administration, Y.H. and J.D. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Data Availability Statement
The DEAP dataset and DREAMER dataset used in this study are publicly available to support the reported results.
Conflicts of Interest
The authors declare no conflicts of interest.
References
- Dolan, R.J. Emotion, Cognition, and Behavior. Science 2002, 298, 1191–1194. [Google Scholar] [CrossRef] [Scilit]
- Fiorini, L.; Mancioppi, G.; Semeraro, F.; Fujita, H.; Cavallo, F. Unsupervised emotional state classification through physiological parameters for social robotics applications. Knowl.-Based Syst. 2020, 190, 105217. [Google Scholar] [CrossRef] [Scilit]
- Figueiredo, G.R.; Ripka, W.L.; Romaneli, E.F.R.; Ulbricht, L. Attentional bias for emotional faces in depressed and non-depressed individuals: An eye-tracking study. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 5419–5422. [Google Scholar] [CrossRef] [Scilit]
- Cartocci, G.; Maglione, A.G.; Vecchiato, G.; Flumeri, G.D.; Colosimo, A.; Scorpecci, A.; Marsella, P.; Giannantonio, S.; Malerba, P.; Borghini, G.; et al. Mental workload estimations in unilateral deafened children. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; pp. 1654–1657. [Google Scholar]
- Yan, J.; Zheng, W.; Xu, Q.; Lu, G.; Li, H.; Wang, B. Sparse Kernel Reduced-Rank Regression for Bimodal Emotion Recognition From Facial Expression and Speech. IEEE Trans. Multimed. 2016, 18, 1319–1329. [Google Scholar] [CrossRef] [Scilit]
- Zhang, Z.; Wu, B.; Schuller, B. Attention-augmented End-to-end Multi-task Learning for Emotion Prediction from Speech. In Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 6705–6709. [Google Scholar] [CrossRef] [Scilit]
- Chen, X.; Pan, Z.; Wang, P.; Zhang, L.; Yuan, J. EEG oscillations reflect task effects for the change detection in vocal emotion. Cogn. Neurodyn. 2015, 9, 351–358. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Hsu, Y.L.; Wang, J.S.; Chiang, W.C.; Hung, C.H. Automatic ECG-Based Emotion Recognition in Music Listening. IEEE Trans. Affect. Comput. 2020, 11, 85–99. [Google Scholar] [CrossRef] [Scilit]
- Liu, Y.J.; Yu, M.; Zhao, G.; Song, J.; Ge, Y.; Shi, Y. Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals. IEEE Trans. Affect. Comput. 2018, 9, 550–562. [Google Scholar] [CrossRef] [Scilit]
- Pan, J.; Li, Y.; Wang, J. An EEG-Based brain-computer interface for emotion recognition. In Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada, 24–29 July 2016; pp. 2063–2067. [Google Scholar] [CrossRef] [Scilit]
- Al-Nafjan, A.; Hosny, M.; Al-Ohali, Y.; Al-Wabil, A. Review and classification of emotion recognition based on EEG brain-computer interface system research: A systematic review. Appl. Sci. 2017, 7, 1239. [Google Scholar] [CrossRef] [Scilit]
- Torres, E.P.; Torres, E.A.; Hernández-Álvarez, M.; Yoo, S.G. EEG-Based BCI Emotion Recognition: A Survey. Sensors 2020, 20, 5083. [Google Scholar] [CrossRef] [Scilit]
- Huang, H.; Xie, Q.; Pan, J.; He, Y.; Wen, Z.; Yu, R.; Li, Y. An EEG-Based Brain Computer Interface for Emotion Recognition and Its Application in Patients with Disorder of Consciousness. IEEE Trans. Affect. Comput. 2021, 12, 832–842. [Google Scholar] [CrossRef] [Scilit]
- Islam, M.R.; Moni, M.A.; Islam, M.M.; Rashed-Al-Mahfuz, M.; Islam, M.S.; Hasan, M.K.; Hossain, M.S.; Ahmad, M.; Uddin, S.; Azad, A.; et al. Emotion Recognition From EEG Signal Focusing on Deep Learning and Shallow Learning Techniques. IEEE Access 2021, 9, 94601–94624. [Google Scholar] [CrossRef] [Scilit]
- Hjorth, B. EEG analysis based on time domain properties. Electroencephalogr. Clin. Neurophysiol. 1970, 29, 306–310. [Google Scholar] [CrossRef] [Scilit]
- Liu, Y.; Sourina, O. Real-time fractal-based valence level recognition from EEG. In Transactions on Computational Science XVIII: Special Issue on Cyberworlds; Springer: Berlin/Heidelberg, Germany, 2013; pp. 101–120. [Google Scholar]
- Duan, R.N.; Zhu, J.Y.; Lu, B.L. Differential entropy feature for EEG-based emotion classification. In Proceedings of the 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), San Diego, CA, USA, 6–8 November 2013; IEEE: New York, NY, USA, 2013; pp. 81–84. [Google Scholar]
- Lin, Y.P.; Wang, C.H.; Jung, T.P.; Wu, T.L.; Jeng, S.K.; Duann, J.R.; Chen, J.H. EEG-Based Emotion Recognition in Music Listening. IEEE Trans. Biomed. Eng. 2010, 57, 1798–1806. [Google Scholar] [CrossRef] [Scilit]
- Akin, M. Comparison of wavelet transform and FFT methods in the analysis of EEG signals. J. Med. Syst. 2002, 26, 241–247. [Google Scholar] [CrossRef] [Scilit]
- Noble, W.S. What is a support vector machine? Nat. Biotechnol. 2006, 24, 1565–1567. [Google Scholar] [CrossRef] [Scilit]
- Zheng, W.L.; Lu, B.L. Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Trans. Auton. Ment. Dev. 2015, 7, 162–175. [Google Scholar] [CrossRef] [Scilit]
- Yang, Y.; Wu, Q.; Qiu, M.; Wang, Y.; Chen, X. Emotion Recognition from Multi-Channel EEG through Parallel Convolutional Recurrent Neural Network. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil, 8–13 July 2018; pp. 1–7. [Google Scholar] [CrossRef] [Scilit]
- Huang, D.; Chen, S.; Liu, C.; Zheng, L.; Tian, Z.; Jiang, D. Differences first in asymmetric brain: A bi-hemisphere discrepancy convolutional neural network for EEG emotion recognition. Neurocomputing 2021, 448, 140–151. [Google Scholar] [CrossRef] [Scilit]
- Yin, Y.; Zheng, X.; Hu, B.; Zhang, Y.; Cui, X. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl. Soft Comput. 2021, 100, 106954. [Google Scholar] [CrossRef] [Scilit]
- Gu, Y.; Zhong, X.; Qu, C.; Liu, C.; Chen, B. A Domain Generative Graph Network for EEG-Based Emotion Recognition. IEEE J. Biomed. Health Inform. 2023, 27, 2377–2386. [Google Scholar] [CrossRef] [Scilit]
- Shen, F.; Dai, G.; Lin, G.; Zhang, J.; Kong, W.; Zeng, H. EEG-based emotion recognition using 4D convolutional recurrent neural network. Cogn. Neurodynamics 2020, 14, 815–828. [Google Scholar] [CrossRef] [Scilit]
- Yang, X.; Zhu, Z.; Jiang, G.; Wu, D.; He, A.; Wang, J. DC-ASTGCN: EEG emotion recognition based on fusion deep convolutional and adaptive spatio-temporal graph convolutional networks. IEEE J. Biomed. Health Inform. 2024, 29, 2471–2483. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Li, J.; Zhang, Z.; He, H. Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn. Comput. 2018, 10, 368–380. [Google Scholar] [CrossRef] [Scilit]
- Zhong, P.; Wang, D.; Miao, C. EEG-Based Emotion Recognition Using Regularized Graph Neural Networks. IEEE Trans. Affect. Comput. 2022, 13, 1290–1301. [Google Scholar] [CrossRef] [Scilit]
- Chen, J.X.; Jiang, D.M.; Zhang, Y.N. A Hierarchical Bidirectional GRU Model With Attention for EEG-Based Emotion Classification. IEEE Access 2019, 7, 118530–118540. [Google Scholar] [CrossRef] [Scilit]
- Tao, W.; Li, C.; Song, R.; Cheng, J.; Liu, Y.; Wan, F.; Chen, X. EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans. Affect. Comput. 2020, 14, 382–393. [Google Scholar] [CrossRef] [Scilit]
- Feng, L.; Cheng, C.; Zhao, M.; Deng, H.; Zhang, Y. EEG-Based Emotion Recognition Using Spatial-Temporal Graph Convolutional LSTM with Attention Mechanism. IEEE J. Biomed. Health Inform. 2022, 26, 5406–5417. [Google Scholar] [CrossRef] [Scilit]
- Wang, S. A hybrid SMOTE and Trans-CWGAN for data imbalance in real operational AHU AFDD: A case study of an auditorium building. Energy Build. 2025, 348, 116447. [Google Scholar] [CrossRef] [Scilit]
- Shi, L.C.; Jiao, Y.Y.; Lu, B.L. Differential entropy feature for EEG-based vigilance estimation. In Proceedings of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan, 3–7 July 2013; IEEE: NewYork, NY, USA, 2013; pp. 6627–6630. [Google Scholar]
- Hassan, C.A.U.; Ehatisham-ul Haq, M.; Murtaza, F.; Yasin, A.U.; Ullah, S.S. EmoTrans attention based emotion recognition using EEG signals and facial analysis with expert validation. Sci. Rep. 2025, 15, 22004. [Google Scholar] [CrossRef] [Scilit]
- Zheng, W.L.; Zhu, J.Y.; Lu, B.L. Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans. Affect. Comput. 2017, 10, 417–429. [Google Scholar] [CrossRef] [Scilit]
- Yang, Y.; Wu, Q.; Fu, Y.; Chen, X. Continuous convolutional neural network with 3D input for EEG-based emotion recognition. In Proceedings of the International Conference on Neural Information Processing, Siem Reap, Cambodia, 13–16 December 2018; Springer: Berlin/Heidelberg, Germany, 2018; pp. 433–443. [Google Scholar]
- Im, S.K.; Pearmain, A. Unequal error protection with the H. 264 flexible macroblock ordering. In Proceedings of the Visual Communications and Image Processing 2005, Beijing, China, 12–15 July 2005; SPIE: Bellingham, WA, USA, 2005; Volume 5960, pp. 1033–1040. [Google Scholar]
- Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 7132–7141. [Google Scholar]
- Wang, Q.; Wu, B.; Zhu, P.; Li, P.; Zuo, W.; Hu, Q. ECA-Net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 11534–11542. [Google Scholar]
- Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- Wang, X.; Girshick, R.; Gupta, A.; He, K. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 7794–7803. [Google Scholar]
- Wang, S. Evaluating cross-building transferability of attention-based automated fault detection and diagnosis for air handling units: Auditorium and hospital case study. Build. Environ. 2025, 287, 113889. [Google Scholar] [CrossRef] [Scilit]
- Yang, L.; Zhang, R.Y.; Li, L.; Xie, X. Simam: A simple, parameter-free attention module for convolutional neural networks. In Proceedings of the International Conference on Machine Learning, Virtual, 18–24 July 2021; pp. 11863–11874. [Google Scholar]
- Chung, J.; Gulcehre, C.; Cho, K.; Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv 2014, arXiv:1412.3555. [Google Scholar] [CrossRef] [Scilit]
- Koelstra, S.; Muhl, C.; Soleymani, M.; Lee, J.S.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. Deap: A database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 2011, 3, 18–31. [Google Scholar] [CrossRef] [Scilit]
- Katsigiannis, S.; Ramzan, N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 2017, 22, 98–107. [Google Scholar] [CrossRef] [Scilit]
- Cui, H.; Liu, A.; Zhang, X.; Chen, X.; Wang, K.; Chen, X. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl.-Based Syst. 2020, 205, 106243. [Google Scholar] [CrossRef] [Scilit]
- Li, C.; Wang, B.; Zhang, S.; Liu, Y.; Song, R.; Cheng, J.; Chen, X. Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism. Comput. Biol. Med. 2022, 143, 105303. [Google Scholar] [CrossRef] [Scilit]
- Song, T.; Zheng, W.; Song, P.; Cui, Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans. Affect. Comput. 2018, 11, 532–541. [Google Scholar] [CrossRef] [Scilit]
- Xiao, G.; Shi, M.; Ye, M.; Xu, B.; Chen, Z.; Ren, Q. 4D attention-based neural network for EEG emotion recognition. Cogn. Neurodyn. 2022, 16, 805–818. [Google Scholar] [CrossRef] [Scilit]
- Li, Z.; Zhang, G.; Wang, L.; Wei, J.; Dang, J. Emotion recognition using spatial-temporal EEG features through convolutional graph attention network. J. Neural Eng. 2023, 20, 016046. [Google Scholar] [CrossRef] [Scilit]
- Sartipi, S.; Torkamani-Azar, M.; Cetin, M. A hybrid end-to-end spatiotemporal attention neural network with graph-smooth signals for EEG emotion recognition. IEEE Trans. Cogn. Dev. Syst. 2023, 16, 732–743. [Google Scholar] [CrossRef] [Scilit]
- Fan, C.; Wang, J.; Huang, W.; Yang, X.; Pei, G.; Li, T.; Lv, Z. Light-weight residual convolution-based capsule network for EEG emotion recognition. Adv. Eng. Inform. 2024, 61, 102522. [Google Scholar] [CrossRef] [Scilit]
- Du, R.; Zhu, S.; Ni, H.; Mao, T.; Li, J.; Wei, R. Valence-arousal classification of emotion evoked by Chinese ancient-style music using 1D-CNN-BiLSTM model on EEG signals for college students. Multimed. Tools Appl. 2023, 82, 15439–15456. [Google Scholar] [CrossRef] [Scilit]
- Wei, Y.; Liu, Y.; Li, C.; Cheng, J.; Song, R.; Chen, X. TC-Net: A Transformer Capsule Network for EEG-based emotion recognition. Comput. Biol. Med. 2023, 152, 106463. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Ju, X.; Li, M.; Tian, W.; Hu, D. EEG-based emotion recognition using a temporal-difference minimizing neural network. Cogn. Neurodyn. 2024, 18, 405–416. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Fan, Z.; Chen, F.; Xia, X.; Liu, Y. EEG emotion classification based on graph convolutional network. Appl. Sci. 2024, 14, 726. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.














