CATM: A Multi-Feature-Based Cross-Scale Attentional Convolutional EEG Emotion Recognition Model
Abstract
:1. Introduction
- In this study, we extracted four features at different frequencies: differential entropy, power spectral density, nonlinear energy, and fractal dimension. We spatially mapped these features according to the positions of the electrode channels, resulting in four-dimensional spatial features with enhanced discriminative and characterization capabilities. Finally, we performed feature fusion of multiple spatial features to leverage the advantages of each feature type.
- We propose a new spatio-temporal feature attention network to address the limitations of existing EEG emotion recognition methods. The network comprises a cross-scale attention convolution module, a transition module, a Bi-LSTM module, and a classifier. This architecture effectively enhances feature extraction capabilities and fully leverages the spatio-temporal features in EEG signals.
- To enable the proposed network model to fully utilize the information in the four-dimensional structure, we designed a frequency–space attention mechanism. This mechanism comprehensively considers the weights between convolutional channels and the positional relationships between electrode channels. It is embedded within the cross-scale convolutional module, allowing for adaptive weight assignment for both frequency and electrode channel positions in the EEG signal.
- Extensive experiments on the DEAP dataset demonstrate that the model exhibits strong emotion recognition accuracy and robustness in binary, four-class, and few-channel scenarios, validating the effectiveness of the proposed method.
2. Related Work
2.1. Multi-Feature-Based EEG Emotion Recognition
2.2. Machine Learning-Based EEG Emotion Recognition
2.3. Deep Learning-Based EEG Emotion Recognition
2.4. Attentional Mechanisms in EEG Emotion Recognition
2.5. Channel Selection in EEG Emotion Recognition
3. Materials and Methods
3.1. Feature Extraction
3.2. Feature Mapping and Feature Fusion
3.3. Network Model Architecture and Classification
3.3.1. Cross-Scale Attention Module (CSAM)
3.3.2. Frequency–Space Attention Module (FSAM)
3.3.3. Feature Transition Module (FTM)
3.3.4. Temporal Feature Extraction Module (Bi-LSTM)
3.3.5. Deep Classification Module (DCM)
4. Experiments
4.1. Dataset and Dataset Processing
4.2. Experiment Setup and Performance Evaluation Metrics
5. Results and Discussion
5.1. Ablation Experiments
5.1.1. Feature Fusion Ablation Experiments
5.1.2. Feature Ablation Experiments
5.1.3. Module Ablation Experiments
5.2. Experiments with Few Channels
5.3. Comparative Experiments
- (1)
- FSA-3D-CNN [63]: This method constructs a 3D matrix of the EEG containing spatio-temporal information and introduces an attention mechanism to use 3D-CNN for emotion classification tasks.
- (2)
- TSFFN [64]: This method performs de-baselining of the EEG and extracts spatio-temporal features from EEG signals using a parallel transformer and a three-dimensional convolutional neural network (3D-CNN), and finally performs an emotion classification task.
- (3)
- Multi-aCRNN [65]: This method proposes a multi-view feature fusion attentional convolutional recurrent neural network. The interference of label noise is reduced by label smoothing, and GRU and CNN are combined to accomplish the emotion classification task.
- (4)
- RA2-3DCNN [66]: This method introduces segmentation–transformation–merge techniques, residuals, and attention mechanisms into shallow networks to improve the accuracy of the model. It is based on the 2D convolutional neural network and 3D convolutional neural network for emotion recognition.
- (5)
- MDCNAResnet [67]: This method extracts differential entropy features from EEG signals and constructs a three-dimensional feature matrix, uses deformable convolution to extract high-level abstract features, and combines MDCNAResnet with bidirectional gated recurrent units (BiGRUs) to accomplish emotion recognition.
- (6)
- BiTCAN [40]: This method utilizes a bi-hemispheric difference module to extract salience features of brain cognition, fuses salience and spatio-temporal features into an attention module, and inputs them into a classifier for emotion recognition.
- (7)
- RFPN-S2D-CNN [68]: This method uses preprocessed signals, differential entropy (DE), symmetric difference, and the symmetric quotient to construct four EEG signal feature matrices, and proposes a residual feature pyramid network (RFPN) to obtain inter-channel correlation, which is effective in improving the classification accuracy of emotion recognition.
- (8)
- FCAN-XGBoost [55]: This method extracts DE features and PSD features of the EEG and fuses FCAN and XGBoost algorithms for sentiment recognition, which reduces computational cost and improves classification accuracy.
- (9)
- Multi-scale 3D-CRU [69]: This method reconstructs a 3D feature representation of the EEG containing delta (δ) frequencies, combined with a recurrent neural network GRU for emotion classification.
- (10)
- MES-CTNet [70]: This method proposes a new capsule transformer network based on multidomain features, which uses multiple features and multiple attention mechanisms, and achieves high accuracy in emotion classification.
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Torres, E.P.; Torres, E.A.; Hernández-Álvarez, M.; Yoo, S.G. EEG-Based BCI Emotion Recognition: A Survey. Sensors 2020, 20, 5083. [Google Scholar] [CrossRef] [PubMed]
- Lin, W.; Li, C. Review of Studies on Emotion Recognition and Judgment Based on Physiological Signals. Appl. Sci. 2023, 13, 2573. [Google Scholar] [CrossRef]
- Kamble, K.; Sengupta, J. A comprehensive survey on emotion recognition based on electroencephalograph (EEG) signals. Multimed. Tools Appl. 2023, 82, 27269–27304. [Google Scholar] [CrossRef]
- Zhang, J.; Xing, L.; Tan, Z.; Wang, H.; Wang, K. Multi-head attention fusion networks for multi-modal speech emotion recognition. Comput. Ind. Eng. 2022, 168, 108078. [Google Scholar] [CrossRef]
- Rajan, S.; Chenniappan, P.; Devaraj, S.; Madian, N. Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM. IET Image Process. 2020, 14, 1373–1381. [Google Scholar] [CrossRef]
- Ahmed, F.; Bari, A.S.M.H.; Gavrilova, M.L. Emotion Recognition from Body Movement. IEEE Access 2020, 8, 11761–11781. [Google Scholar] [CrossRef]
- Xu, M.; Cheng, J.; Li, C.; Liu, Y.; Chen, X. Spatio-temporal deep forest for emotion recognition based on facial electromyography signals. Comput. Biol. Med. 2023, 156, 106689. [Google Scholar] [CrossRef]
- Zhu, H.; Fu, C.; Shu, F.; Yu, H.; Chen, C.; Chen, W. The Effect of Coupled Electroencephalography Signals in Electrooculography Signals on Sleep Staging Based on Deep Learning Methods. Bioengineering 2023, 10, 573. [Google Scholar] [CrossRef] [PubMed]
- Lee, J.-A.; Kwak, K.-C. Personal Identification Using an Ensemble Approach of 1D-LSTM and 2D-CNN with Electrocardiogram Signals. Appl. Sci. 2022, 12, 2692. [Google Scholar] [CrossRef]
- Han, Z.; Chang, H.; Zhou, X.; Wang, J.; Wang, L.; Shao, Y. E2ENNet: An end-to-end neural network for emotional brain-computer interface. Front. Comput. Neurosci. 2022, 16, 942979. [Google Scholar] [CrossRef]
- Lin, X.; Chen, J.; Ma, W.; Tang, W.; Wang, Y. EEG emotion recognition using improved graph neural network with channel selection. Comput. Methods Programs Biomed. 2023, 231, 107380. [Google Scholar] [CrossRef]
- Chen, J.; Min, C.; Wang, C.; Tang, Z.; Liu, Y.; Hu, X. Electroencephalograph-Based Emotion Recognition Using Brain Connectivity Feature and Domain Adaptive Residual Convolution Model. Front. Neurosci. 2022, 16, 878146. [Google Scholar] [CrossRef] [PubMed]
- Han, L.; Zhang, X.; Yin, J. EEG emotion recognition based on the TimesNet fusion model. Appl. Soft Comput. 2024, 159, 111635. [Google Scholar] [CrossRef]
- Wu, Q.; Yuan, Y.; Cheng, Y.; Ye, T. A novel emotion recognition method based on 1D-DenseNet. J. Intell. Fuzzy Syst. 2023, 44, 5507–5518. [Google Scholar] [CrossRef]
- Singh, K.; Ahirwal, M.K.; Pandey, M. Quaternary classification of emotions based on electroencephalogram signals using hybrid deep learning model. J. Ambient Intell. Humaniz. Comput. 2023, 14, 2429–2441. [Google Scholar] [CrossRef]
- Liu, F.; Yang, P.; Shu, Y.; Liu, N.; Sheng, J.; Luo, J.; Wang, X.; Liu, Y.J. Emotion Recognition from Few-Channel EEG Signals by Integrating Deep Feature Aggregation and Transfer Learning. IEEE Trans. Affect. Comput. 2023, 1–17. [Google Scholar] [CrossRef]
- Jatupaiboon, N.; Pan-ngum, S.; Israsena, P. Emotion classification using minimal EEG channels and frequency bands. In Proceedings of the 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE), Khon Kaen, Thailand, 29–31 May 2013; pp. 21–24. [Google Scholar]
- Mert, A.; Akan, A. Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal. Appl. 2018, 21, 81–89. [Google Scholar] [CrossRef]
- Bazgir, O.; Mohammadi, Z.; Habibi, S. Emotion Recognition with Machine Learning Using EEG Signals. In Proceedings of the 2018 25th National and 3rd International Iranian Conference on Biomedical Engineering (ICBME), Qom, Iran, 29–30 November 2019. [Google Scholar]
- Koelstra, S.; Muhl, C.; Soleymani, M.; Lee, J.S.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. DEAP: A Database for Emotion Analysis; Using Physiological Signals. IEEE Trans. Affect. Comput. 2012, 3, 18–31. [Google Scholar] [CrossRef]
- Zheng, W.L.; Lu, B.L. Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Trans. Auton. Ment. Dev. 2015, 7, 162–175. [Google Scholar] [CrossRef]
- Houssein, E.H.; Hammad, A.; Ali, A.A. Human emotion recognition from EEG-based brain–computer interface using machine learning: A comprehensive review. Neural Comput. Appl. 2022, 34, 12527–12557. [Google Scholar] [CrossRef]
- Rahman, M.M.; Sarkar, A.K.; Hossain, M.A.; Hossain, M.S.; Islam, M.R.; Hossain, M.B.; Quinn, J.M.W.; Moni, M.A. Recognition of human emotions using EEG signals: A review. Comput. Biol. Med. 2021, 136, 104696. [Google Scholar] [CrossRef] [PubMed]
- Jafari, M.; Shoeibi, A.; Khodatars, M.; Bagherzadeh, S.; Shalbaf, A.; García, D.L.; Gorriz, J.M.; Acharya, U.R. Emotion recognition in EEG signals using deep learning methods: A review. Comput. Biol. Med. 2023, 165, 107450. [Google Scholar] [CrossRef] [PubMed]
- Dadebayev, D.; Goh, W.W.; Tan, E.X. EEG-based emotion recognition: Review of commercial EEG devices and machine learning techniques. J. King Saud Univ. Comput. Inf. Sci. 2022, 34, 4385–4401. [Google Scholar] [CrossRef]
- Singh, A.K.; Krishnan, S. Trends in EEG signal feature extraction applications. Front. Artif. Intell. 2023, 5, 1072801. [Google Scholar] [CrossRef] [PubMed]
- Yuvaraj, R.; Thagavel, P.; Thomas, J.; Fogarty, J.; Ali, F. Comprehensive Analysis of Feature Extraction Methods for Emotion Recognition from Multichannel EEG Recordings. Sensors 2023, 23, 915. [Google Scholar] [CrossRef] [PubMed]
- Liu, S.; Wang, X.; Zhao, L.; Zhao, J.; Xin, Q.; Wang, S.H. Subject-Independent Emotion Recognition of EEG Signals Based on Dynamic Empirical Convolutional Neural Network. IEEE/ACM Trans. Comput. Biol. Bioinform. 2021, 18, 1710–1721. [Google Scholar] [CrossRef]
- Çelebi, M.; Öztürk, S.; Kaplan, K. An emotion recognition method based on EWT-3D–CNN–BiLSTM-GRU-AT model. Comput. Biol. Med. 2024, 169, 107954. [Google Scholar] [CrossRef] [PubMed]
- Zhang, J.; Zhang, X.; Chen, G.; Huang, L.; Sun, Y. EEG emotion recognition based on cross-frequency granger causality feature extraction and fusion in the left and right hemispheres. Front. Neurosci. 2022, 16, 974673. [Google Scholar] [CrossRef] [PubMed]
- Cai, J.; Xiao, R.; Cui, W.; Zhang, S.; Liu, G. Application of Electroencephalography-Based Machine Learning in Emotion Recognition: A Review. Front. Syst. Neurosci. 2021, 15, 729707. [Google Scholar] [CrossRef]
- Nawaz, R.; Cheah, K.H.; Nisar, H.; Yap, V.V.J.B.; Engineering, B. Comparison of different feature extraction methods for EEG-based emotion recognition. Biocybern. Biomed. Eng. 2020, 40, 910–926. [Google Scholar] [CrossRef]
- Bhardwaj, A.; Gupta, A.; Jain, P.; Rani, A.; Yadav, J. Classification of human emotions from EEG signals using SVM and LDA Classifiers. In Proceedings of the 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), Noida, India, 19–20 February 2015; pp. 180–185. [Google Scholar]
- Gupta, V.; Chopda, M.D.; Pachori, R.B. Cross-Subject Emotion Recognition Using Flexible Analytic Wavelet Transform From EEG Signals. IEEE Sens. J. 2019, 19, 2266–2274. [Google Scholar] [CrossRef]
- Asghar, M.A.; Khan, M.J.; Fawad; Amin, Y.; Rizwan, M.; Rahman, M.; Badnava, S.; Mirjavadi, S.S. EEG-Based Multi-Modal Emotion Recognition using Bag of Deep Features: An Optimal Feature Selection Approach. Sensors 2019, 19, 5218. [Google Scholar] [CrossRef] [PubMed]
- Li, X.; Song, D.; Zhang, P.; Yu, G.; Hou, Y.; Hu, B. Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network. In Proceedings of the 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Shenzhen, China, 15–18 December 2016; pp. 352–359. [Google Scholar]
- Chakravarthi, B.; Ng, S.-C.; Ezilarasan, M.R.; Leung, M.-F. EEG-based emotion recognition using hybrid CNN and LSTM classification. Front. Comput. Neurosci. 2022, 16, 1019776. [Google Scholar] [CrossRef]
- Yang, Y.; Wu, Q.; Fu, Y.; Chen, X. Continuous Convolutional Neural Network with 3D Input for EEG-Based Emotion Recognition. In Proceedings of the Neural Information Processing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, 13–16 December 2018; pp. 433–443. [Google Scholar]
- Liu, S.; Wang, X.; Zhao, L.; Li, B.; Hu, W.; Yu, J.; Zhang, Y.D. 3DCANN: A Spatio-Temporal Convolution Attention Neural Network for EEG Emotion Recognition. IEEE J. Biomed. Health Inform. 2022, 26, 5321–5331. [Google Scholar] [CrossRef] [PubMed]
- An, Y.; Hu, S.; Liu, S.; Li, B. BiTCAN: A emotion recognition network based on saliency in brain cognition. Math. Biosci. Eng. 2023, 20, 21537–21562. [Google Scholar] [CrossRef]
- Xiao, G.; Shi, M.; Ye, M.; Xu, B.; Chen, Z.; Ren, Q. 4D attention-based neural network for EEG emotion recognition. Cogn. Neurodynamics 2022, 16, 805–818. [Google Scholar] [CrossRef]
- Jia, Z.; Lin, Y.; Cai, X.; Chen, H.; Gou, H.; Wang, J. SST-EmotionNet: Spatial-Spectral-Temporal based Attention 3D Dense Network for EEG Emotion Recognition. In Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA, 12 October 2020; pp. 2909–2917. [Google Scholar]
- Zhang, J.; Chen, M.; Zhao, S.; Hu, S.; Shi, Z.; Cao, Y. ReliefF-Based EEG Sensor Selection Methods for Emotion Recognition. Sensors 2016, 16, 1558. [Google Scholar] [CrossRef]
- Özerdem, M.S.; Polat, H. Emotion recognition based on EEG features in movie clips with channel selection. Brain Inform. 2017, 4, 241–252. [Google Scholar] [CrossRef]
- Topic, A.; Russo, M.; Stella, M.; Saric, M. Emotion Recognition Using a Reduced Set of EEG Channels Based on Holographic Feature Maps. Sensors 2022, 22, 3248. [Google Scholar] [CrossRef]
- Duan, R.N.; Zhu, J.Y.; Lu, B.L. Differential entropy feature for EEG-based emotion classification. In Proceedings of the 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), San Diego, CA, USA, 6–8 November 2013; pp. 81–84. [Google Scholar]
- Chen, J.; Jiang, D.; Zhang, Y.; Zhang, P. Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset. Comput. Commun. 2020, 154, 58–65. [Google Scholar] [CrossRef]
- Toole, J.M.O.; Temko, A.; Stevenson, N. Assessing instantaneous energy in the EEG: A non-negative, frequency-weighted energy operator. In Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA, 26–30 August 2014; pp. 3288–3291. [Google Scholar]
- Zhao, X.; Zhang, R.; Mei, Z.; Chen, C.; Chen, W. Identification of Epileptic Seizures by Characterizing Instantaneous Energy Behavior of EEG. IEEE Access 2019, 7, 70059–70076. [Google Scholar] [CrossRef]
- Vega, C.F.; Noel, J. Parameters analyzed of Higuchi’s fractal dimension for EEG brain signals. In Proceedings of the 2015 Signal Processing Symposium (SPSympo), Debe, Poland, 10–12 June 2015; pp. 1–5. [Google Scholar]
- Jacob, J.E.; Gopakumar, K. Automated Diagnosis of Encephalopathy Using Fractal Dimensions of EEG Sub-Bands. In Proceedings of the 2018 IEEE Recent Advances in Intelligent Computational Systems (RAICS), Thiruvananthapuram, India, 6–8 December 2018; pp. 94–97. [Google Scholar]
- Pavithra, M.; NiranjanaKrupa, B.; Sasidharan, A.; Kutty, B.M.; Lakkannavar, M. Fractal dimension for drowsiness detection in brainwaves. In Proceedings of the 2014 International Conference on Contemporary Computing and Informatics (IC3I), Mysore, India, 27–29 November 2014; pp. 757–761. [Google Scholar]
- Liu, Y.; Sourina, O. EEG-based subject-dependent emotion recognition algorithm using fractal dimension. In Proceedings of the 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), San Diego, CA, USA, 5–8 October 2014; pp. 3166–3171. [Google Scholar]
- Gao, Q.; Wang, C.-h.; Wang, Z.; Song, X.-l.; Dong, E.-z.; Song, Y. EEG based emotion recognition using fusion feature extraction method. Multimed. Tools Appl. 2020, 79, 27057–27074. [Google Scholar] [CrossRef]
- Zong, J.; Xiong, X.; Zhou, J.; Ji, Y.; Zhou, D.; Zhang, Q. FCAN–XGBoost: A Novel Hybrid Model for EEG Emotion Recognition. Sensors 2023, 23, 5680. [Google Scholar] [CrossRef]
- Ma, J.; Tang, H.; Zheng, W.L.; Lu, B.L. Emotion Recognition using Multimodal Residual LSTM Network. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019. [Google Scholar]
- Zhong, P.; Wang, D.; Miao, C. EEG-Based Emotion Recognition Using Regularized Graph Neural Networks. IEEE Trans. Affect. Comput. 2022, 13, 1290–1301. [Google Scholar] [CrossRef]
- Schmidt, L.A.; Trainor, L.J. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cogn. Emot. 2001, 15, 487–500. [Google Scholar] [CrossRef]
- Lin, Y.P.; Wang, C.H.; Jung, T.P.; Wu, T.L.; Jeng, S.K.; Duann, J.R.; Chen, J.H. EEG-Based Emotion Recognition in Music Listening. IEEE Trans. Biomed. Eng. 2010, 57, 1798–1806. [Google Scholar] [CrossRef] [PubMed]
- Costa, T.; Rognoni, E.; Galati, D. EEG phase synchronization during emotional response to positive and negative film stimuli. Neurosci. Lett. 2006, 406, 159–164. [Google Scholar] [CrossRef]
- Mattavelli, G.; Rosanova, M.; Casali, A.G.; Papagno, C.; Lauro, L.J.R. Timing of emotion representation in right and left occipital region: Evidence from combined TMS-EEG. Brain Cogn. 2016, 106, 13–22. [Google Scholar] [CrossRef]
- Javaid, M.M.; Yousaf, M.A.; Sheikh, Q.Z.; Awais, M.M.; Saleem, S.; Khalid, M. Real-Time EEG-based Human Emotion Recognition. In Proceedings of the Neural Information Processing; Springer International Publishing: Cham, Switzerland, 2015; pp. 182–190. [Google Scholar]
- Zhang, J.; Zhang, X.Y.; Chen, G.J.; Yan, C. EEG emotion recognition based on the 3D-CNN and spatial-frequency attention mechanism. J. Xidian Univ. 2022, 49, 191–198. [Google Scholar] [CrossRef]
- Sun, J.; Wang, X.; Zhao, K.; Hao, S.; Wang, T. Multi-Channel EEG Emotion Recognition Based on Parallel Transformer and 3D-Convolutional Neural Network. Mathematics 2022, 10, 3131. [Google Scholar] [CrossRef]
- Xin, R.; Miao, F.; Cong, P.; Zhang, F.; Xin, Y.; Feng, X. Multiview Feature Fusion Attention Convolutional Recurrent Neural Networks for EEG-Based Emotion Recognition. J. Sens. 2023, 2023, 9281230. [Google Scholar] [CrossRef]
- Cui, D.; Xuan, H.; Liu, J.; Gu, G.; Li, X. Emotion Recognition on EEG Signal Using ResNeXt Attention 2D-3D Convolution Neural Networks. Neural Process. Lett. 2023, 55, 5943–5957. [Google Scholar] [CrossRef]
- Du, X.; Meng, Y.; Qiu, S.; Lv, Y.; Liu, Q. EEG Emotion Recognition by Fusion of Multi-Scale Features. Brain Sci. 2023, 13, 1293. [Google Scholar] [CrossRef] [PubMed]
- Hou, F.; Liu, J.; Bai, Z.; Yang, Z.; Liu, J.; Gao, Q.; Song, Y. EEG-Based Emotion Recognition for Hearing Impaired and Normal Individuals with Residual Feature Pyramids Network Based on Time–Frequency–Spatial Features. IEEE Trans. Instrum. Meas. 2023, 72, 2505011. [Google Scholar] [CrossRef]
- Dong, H.; Zhou, J.; Fan, C.; Zheng, W.; Tao, L.; Kwan, H.K. Multi-scale 3D-CRU for EEG emotion recognition. Biomed. Phys. Eng. Express 2024, 10, 045018. [Google Scholar] [CrossRef]
- Du, Y.; Ding, H.; Wu, M.; Chen, F.; Cai, Z. MES-CTNet: A Novel Capsule Transformer Network Base on a Multi-Domain Feature Map for Electroencephalogram-Based Emotion Recognition. Brain Sci. 2024, 14, 344. [Google Scholar] [CrossRef]
Module | Full Name | Function |
---|---|---|
CSAM | Cross-scale attention module | Extracts features of different scales and assigns weights |
FSAM | Frequency–space attention module | Gives higher weight to more important frequency bands and spatial locations |
Bi_LSTM | Bidirectional long short-term memory | Extracts time features |
DCM | Deep classification module | Classifies the features |
Layer | Layer Setting | Output |
---|---|---|
Conv2D (1 × 1) | In_features = 80 | (128 × 128 × 8 × 9) |
Out_feaures = 128 | ||
BatchNorm2d | ||
Activation = ReLU | ||
Conv2D (3 × 3) | In_features = 80 | (128 × 128 × 8 × 9) |
Out_feaures = 128 | ||
BatchNorm2d | ||
Activation = ReLU | ||
Conv2D (5 × 5) | In_features = 80 | (128 × 128 × 8 × 9) |
Out_feaures = 128 | ||
BatchNorm2d | ||
Activation = ReLU | ||
Max pooling(3 × 3) | In_features = 80 | (128 × 80 × 8 × 9) |
Out_feaures = 80 | ||
BatchNorm2d | ||
Activation = ReLU | ||
Concatenate | (128 × 464 × 8 × 9) |
Layer | Layer Setting | Output |
---|---|---|
Linear1 | In_features = 128 | (128 × 64) |
Out_feaures = 64 | ||
Activation = ReLU | ||
Dropout | p = 0.5 | |
Linear2 | In_features = 64 | (128 × 32) |
Out_feaures = 32 | ||
Activation = ReLU | ||
Dropout | p = 0.5 | |
Linear3 | In_features = 32 Out_feaures = num_classes | (128 × num_classes) |
Feature | Accuracy% | Precision% | Recall% | F1-Score% | ||||
---|---|---|---|---|---|---|---|---|
Valence | Arousal | Valence | Arousal | Valence | Arousal | Valence | Arousal | |
98.18 | 98.59 | 98.23 | 98.62 | 98.18 | 98.59 | 98.18 | 98.59 | |
89.84 | 88.79 | 90.08 | 89.16 | 89.84 | 88.79 | 89.78 | 88.40 | |
99.70 | 99.74 | 99.70 | 99.74 | 99.69 | 99.73 | 99.69 | 99.73 |
Feature | Accuracy% | Precision% | Recall% | F1-Score% | ||||
---|---|---|---|---|---|---|---|---|
Valence | Arousal | Valence | Arousal | Valence | Arousal | Valence | Arousal | |
DE | 97.39 | 97.75 | 98.18 | 98.28 | 97.25 | 97.83 | 97.6 | 97.99 |
PSD | 96.44 | 96.58 | 97.80 | 98.34 | 96.68 | 97.11 | 97.06 | 97.50 |
NE | 95.32 | 95.51 | 97.09 | 97.57 | 94.59 | 96.21 | 95.37 | 96.66 |
FD | 90.97 | 97.73 | 95.10 | 96.01 | 90.96 | 92.93 | 91.13 | 92.94 |
DE-PSD | 98.98 | 98.95 | 99.25 | 99.27 | 98.89 | 98.99 | 99.01 | 99.08 |
DE-NE | 98.81 | 98.72 | 99.30 | 98.85 | 98.89 | 98.52 | 99.03 | 98.63 |
DE-FD | 98.64 | 98.88 | 98.99 | 99.15 | 98.50 | 98.79 | 98.67 | 98.92 |
PSD-NE | 97.84 | 98.33 | 98.76 | 98.94 | 97.81 | 98.46 | 98.13 | 98.64 |
PSD-FD | 98.43 | 98.40 | 99.01 | 99.07 | 98.38 | 98.59 | 98.56 | 98.76 |
NE-FD | 98.29 | 98.19 | 98.59 | 98.95 | 97.85 | 97.72 | 98.12 | 97.88 |
DE-PSD-NE | 99.01 | 99.16 | 99.48 | 99.49 | 99.32 | 99.16 | 99.40 | 99.28 |
PSD-NE-FD | 98.81 | 99.13 | 99.22 | 99.43 | 98.42 | 99.16 | 98.51 | 99.25 |
DE-PSD-FD | 99.05 | 99.25 | 99.49 | 99.61 | 99.16 | 99.36 | 99.29 | 99.46 |
DE-NE-FD | 99.14 | 99.19 | 99.42 | 99.03 | 99.24 | 99.04 | 99.33 | 99.13 |
All Feature | 99.70 | 99.74 | 99.70 | 99.74 | 99.69 | 99.73 | 99.69 | 99.73 |
Models | CSAM | FSAM | Bi-LSTM | DCM |
---|---|---|---|---|
Model 1 | × | √ | √ | √ |
Model 2 | √ | × | √ | √ |
Model 3 | √ | √ | × | √ |
Model 4 | √ | √ | √ | × |
Models | Accuracy/% | Precision/% | Recall/% | F1-Score/% | ||||
---|---|---|---|---|---|---|---|---|
Valence | Arousal | Valence | Arousal | Valence | Arousal | Valence | Arousal | |
Model 1 | 91.34 | 95.06 | 97.33 | 98.57 | 88.61 | 94.73 | 87.2 | 94.15 |
Model 2 | 98.95 | 98.97 | 99.36 | 99.27 | 98.95 | 98.98 | 98.95 | 98.97 |
Model 3 | 91.75 | 92.45 | 95.07 | 95.32 | 91.45 | 92.62 | 92.69 | 93.52 |
Model 4 | 98.92 | 98.87 | 98.93 | 98.88 | 98.92 | 98.87 | 98.92 | 98.87 |
CATM | 99.70 | 99.74 | 99.70 | 99.74 | 99.69 | 99.73 | 99.69 | 99.73 |
Kernel Size | Accuracy% | Precision% | Recall% | F1-Score% | ||||
---|---|---|---|---|---|---|---|---|
Valence | Arousal | Valence | Arousal | Valence | Arousal | Valence | Arousal | |
1 | 88.25 | 93.73 | 93.47 | 96.30 | 88.25 | 93.73 | 84.44 | 91.77 |
3 | 95.12 | 97.22 | 97.05 | 98.24 | 95.12 | 97.22 | 93.66 | 96.42 |
5 | 97.63 | 98.52 | 98.50 | 98.75 | 97.63 | 98.52 | 96.98 | 98.32 |
Dimension | Accuracy/% | Precision/% | Recall/% | F1-Score/% |
---|---|---|---|---|
Valence | 97.96 | 98.01 | 97.96 | 97.95 |
Arousal | 98.11 | 98.17 | 98.11 | 98.10 |
V-A | 92.86 | 94.12 | 92.86 | 91.95 |
Dimension | Accuracy/% | Precision/% | Recall/% | F1-Score/% |
---|---|---|---|---|
Valence | 99.59 | 99.61 | 99.59 | 99.59 |
Arousal | 99.53 | 99.54 | 99.53 | 99.53 |
V-A | 94.57 | 95.98 | 94.57 | 93.50 |
Model | Feature | Dataset | Valence | Arousal | V-A | Year |
---|---|---|---|---|---|---|
FSA-3D-CNN | DE | DEAP | 95.87% | 95.23% | 94.53 | 2022 |
TSFFN | Baseline removal | DEAP | 98.27% | 98.53% | - | 2022 |
Multi-aCRNN | DE | DEAP | 96.30% | 96.43% | - | 2023 |
RA2-3DCNN | Baseline removal | DEAP | 97.58% | 97.19% | - | 2022 |
MDCNAResnet | DE | DEAP | 98.63% | 98.89% | - | 2023 |
BiTCAN | Baseline removal | DEAP | 98.46% | 97.65% | - | 2023 |
RFPN–S2D–CNN | DE | DEAP | 96.89% | 96.82% | 93.56% | 2023 |
FCAN–XGBoost | DE, PSD | DEAP | - | - | 95.26% | 2023 |
Multi-scale 3D-CRU | DE | DEAP | 93.12% | 94.31% | - | 2024 |
MES-CTNet | DE, PSD, SE | DEAP | 98.31% | 98.28% | - | 2024 |
Ours | DE, PSD, NE, FD | DEAP | 99.70% | 99.74% | 97.27% | 2024 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yu, H.; Xiong, X.; Zhou, J.; Qian, R.; Sha, K. CATM: A Multi-Feature-Based Cross-Scale Attentional Convolutional EEG Emotion Recognition Model. Sensors 2024, 24, 4837. https://doi.org/10.3390/s24154837
Yu H, Xiong X, Zhou J, Qian R, Sha K. CATM: A Multi-Feature-Based Cross-Scale Attentional Convolutional EEG Emotion Recognition Model. Sensors. 2024; 24(15):4837. https://doi.org/10.3390/s24154837
Chicago/Turabian StyleYu, Hongde, Xin Xiong, Jianhua Zhou, Ren Qian, and Kaiwen Sha. 2024. "CATM: A Multi-Feature-Based Cross-Scale Attentional Convolutional EEG Emotion Recognition Model" Sensors 24, no. 15: 4837. https://doi.org/10.3390/s24154837
APA StyleYu, H., Xiong, X., Zhou, J., Qian, R., & Sha, K. (2024). CATM: A Multi-Feature-Based Cross-Scale Attentional Convolutional EEG Emotion Recognition Model. Sensors, 24(15), 4837. https://doi.org/10.3390/s24154837