Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition
Abstract
:1. Introduction
2. EEG-Based Signal Extraction for Emotion Recognition
3. Optimal EEG Signal Extraction for Emotion Recognition
3.1. EEG Topographic Maps
3.2. Objective Function
3.3. Simplicial Homology Global Optimization
4. Optimization Results and Discussion
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Roshdy, A.; Karar, A.S.; Al-Sabi, A.; Barakeh, Z.A.; El-Sayed, F.; Alkork, S.; Beyrouthy, T.; Nait-ali, A. Towards Human Brain Image Mapping for Emotion Digitization in Robotics. In Proceedings of the 2019 3rd International Conference on Bio-Engineering for Smart Technologies (BioSMART), Paris, France, 24–26 April 2019; pp. 1–5. [Google Scholar] [CrossRef]
- Al Machot, F.; Elmachot, A.; Ali, M.; Al Machot, E.; Kyamakya, K. A Deep-Learning Model for Subject-Independent Human Emotion Recognition Using Electrodermal Activity Sensors. Sensors 2019, 19, 1659. [Google Scholar] [CrossRef] [PubMed]
- Hassouneh, A.; Mutawa, A.; Murugappan, M. Development of a Real-Time Emotion Recognition System Using Facial Expressions and EEG based on machine learning and deep neural network methods. Inform. Med. Unlocked 2020, 20, 100372. [Google Scholar] [CrossRef]
- Bashivan, P.; Rish, I.; Yeasin, M.; Codella, N. Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks. arXiv 2015, arXiv:1511.06448. [Google Scholar]
- Roshdy, A.; Al Kork, S.; Karar, A.; Al Sabi, A.; Al Barakeh, Z.; El-Sayed, F.; Beyrouthy, T.; Nait-ali, A. Machine Empathy: Digitizing Human Emotions. In Proceedings of the 2021 International Symposium on Electrical, Electronics and Information Engineering, New York, NY, USA, 19–21 February 2021; Association for Computing Machinery: New York, NY, USA, 2021; pp. 307–311. [Google Scholar] [CrossRef]
- Bhadangkar, D.; Pujari, J.D.; Yakkundimath, R. Comparison of Tuplet of Techniques for Facial Emotion Detection. In Proceedings of the 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Palladam, India, 7–9 October 2020; pp. 725–730. [Google Scholar] [CrossRef]
- Aguiñaga, A.R.; Hernandez, D.E.; Quezada, A.; Calvillo Téllez, A. Emotion Recognition by Correlating Facial Expressions and EEG Analysis. Appl. Sci. 2021, 11, 6987. [Google Scholar] [CrossRef]
- Xu, W.; Zhou, R.; Liu, Q. Electroencephalogram Emotion Recognition Based on Three-Dimensional Feature Matrix and Multivariate Neural Network. In Proceedings of the 2022 IEEE 25th International Conference on Computational Science and Engineering (CSE), Wuhan, China, 9–11 December 2022; pp. 32–37. [Google Scholar] [CrossRef]
- Schoneveld, L.; Othmani, A.; Abdelkawy, H. Leveraging recent advances in deep learning for audio-Visual emotion recognition. Pattern Recognit. Lett. 2021, 146, 1–7. [Google Scholar] [CrossRef]
- Kosti, R.; Alvarez, J.M.; Recasens, A.; Lapedriza, A. Context Based Emotion Recognition using EMOTIC Dataset. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 2755–2766. [Google Scholar] [CrossRef]
- Alam, F.; Riccardi, G. Predicting Personality Traits using Multimodal Information. In Proceedings of the Association for Computing Machinery, New York, NY, USA, 4–7 November 2014. [Google Scholar] [CrossRef]
- Adiga, S.; Vaishnavi, D.; Saxena, S.; Tripathi, S. Multimodal Emotion Recognition for Human Robot Interaction, Stockholm, Sweden. In Proceedings of the 2020 7th International Conference on Soft Computing & Machine Intelligence (ISCMI), Stockholm, Sweden, 14–15 November 2020; pp. 197–203. [Google Scholar] [CrossRef]
- Debnath, T.; Reza, M.; Rahman, A.; Band, S.S.; Alinejad-Rokny, H. Four-layer Convnet to Facial Emotion Recognition with Minimal Epochs and the Significance of Data Diversity. Sci. Rep. 2021, 12, 6991. [Google Scholar] [CrossRef]
- Nie, J.; Hu, Y.; Wang, Y.; Xia, S.; Jiang, X. SPIDERS: Low-Cost Wireless Glasses for Continuous In-Situ Bio-Signal Acquisition and Emotion Recognition, Sydney, NSW, Australia. In Proceedings of the 2020 IEEE/ACM Fifth International Conference on Internet-of-Things Design and Implementation (IoTDI), Sydney, Australia, 21–24 April 2020; pp. 27–39. [Google Scholar] [CrossRef]
- Franzoni, V.; Biondi, G.; Perri, D.; Gervasi, O. Enhancing Mouth-based Emotion Recognition using Transfer Learning. Sensors 2020, 20, 5222. [Google Scholar] [CrossRef]
- Kreibig, S.D. Autonomic nervous system activity in emotion: A review. Biol. Psychol. 2010, 84, 394–421. [Google Scholar] [CrossRef] [PubMed]
- Orban, M.; Elsamanty, M.; Guo, K.; Zhang, S.; Yang, H. A Review of Brain Activity and EEG-Based Brain—Computer Interfaces for Rehabilitation Application. Bioengineering 2022, 9, 768. [Google Scholar] [CrossRef]
- Lin, C.T.; Wu, P.H.; Chen, Y.P.; Tsai, C.S. Consumer-grade EEG devices in neuroscientific research: A review on cost-effectiveness and user experience. Behav. Res. Methods 2019, 51, 2287–2302. [Google Scholar]
- Tan, Y.; Sun, Z.; Duan, F.; Solé-Casals, J.; Caiafa, C.F. A multimodal emotion recognition method based on facial expressions and electroencephalography. Biomed. Signal Process. Control 2021, 70, 103029. [Google Scholar] [CrossRef]
- Yin, Y.; Zheng, X.; Hu, B.; Zhang, Y.; Cui, X. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl. Soft Comput. 2021, 100, 106954. [Google Scholar] [CrossRef]
- Harmony, T. The functional significance of delta oscillations in cognitive processing. Front. Integr. Neurosci. 2013, 7, 83. [Google Scholar] [CrossRef] [PubMed]
- Aftanas, L.I.; Golocheikine, S.A. Human anterior and frontal midline theta and lower alpha reflect emotionally induced changes in cognitive and affective processing. Int. J. Psychophysiol. 2001, 40, 57–70. [Google Scholar]
- Klimesch, W. EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Res. Rev. 1999, 29, 169–195. [Google Scholar] [CrossRef] [PubMed]
- Miskovic, V.; Schmidt, L.A. Frontal brain electrical asymmetry and cardiac vagal tone predict biased attention to social threat. Biol. Psychol. 2010, 84, 344–348. [Google Scholar] [CrossRef]
- Başar, E.; Başar-Eroğlu, C.; Karakaş, S.; Schürmann, M. Gamma, alpha, delta, and theta oscillations govern cognitive processes. Int. J. Psychophysiol. 2001, 39, 241–248. [Google Scholar] [CrossRef]
- Selvathi, D.; Meera, V.K. Realization of epileptic seizure detection in EEG signal using wavelet transform and SVM classifier. In Proceedings of the 2017 International Conference on Signal Processing and Communication (ICSPC), Coimbatore, India, 28–29 July 2017; pp. 18–22. [Google Scholar] [CrossRef]
- Hwang, S.; Ki, M.; Hong, K.; Byun, H. Subject-Independent EEG-based Emotion Recognition using Adversarial Learning. In Proceedings of the 2020 8th International Winter Conference on Brain-Computer Interface (BCI), Gangwon, Republic of Korea, 26–28 February 2020; pp. 1–4. [Google Scholar] [CrossRef]
- Wei, C.; Chen, L.; Song, Z.; Lou, X.; Li, D. EEG-based emotion recognition using simple recurrent units network and ensemble learning. Biomed. Signal Process. Control 2020, 58, 101756. [Google Scholar] [CrossRef]
- Lee, Y.Y.; Hsieh, S. Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns. PLoS ONE 2014, 9, e95415. [Google Scholar] [CrossRef]
- Emotiv Systems Inc. Emotiv-Brain Computer Interface Technology. Available online: https://www.emotiv.com/ (accessed on 26 June 2023).
- Knyazev, G.G. Motivation, emotion, and their inhibitory control mirrored in brain oscillations. Neurosci. Biobehav. Rev. 2007, 31, 377–395. [Google Scholar] [CrossRef] [PubMed]
- Aftanas, L.; Golocheikine, S. Human anterior and frontal midline theta and lower alpha reflect emotionally positive state and internalized attention: High-resolution EEG investigation of meditation. Neurosci. Lett. 2001, 310, 57–60. [Google Scholar] [CrossRef] [PubMed]
- Keil, A.; Müller, M.M.; Gruber, T.; Wienbruch, C.; Stolarova, M.; Elbert, T. Effects of emotional arousal in the cerebral hemispheres: A study of oscillatory brain activity and event-related potentials. Clin. Neurophysiol. 2001, 112, 2057–2068. [Google Scholar] [CrossRef]
- Mert, A.; Akan, A. Emotion recognition based on time–frequency distribution of EEG signals using multivariate synchrosqueezing transform. Digit. Signal Process. 2018, 81, 106–115. [Google Scholar] [CrossRef]
- Zhang, D.; Yao, L.; Chen, K.; Monaghan, J. A Convolutional Recurrent Attention Model for Subject-Independent EEG Signal Analysis. IEEE Signal Process. Lett. 2019, 26, 715–719. [Google Scholar] [CrossRef]
- Subasi, A.; Tuncer, T.; Dogan, S.; Tanko, D.; Sakoglu, U. EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier. Biomed. Signal Process. Control 2021, 68, 102648. [Google Scholar] [CrossRef]
- Zhang, H. Expression-EEG Based Collaborative Multimodal Emotion Recognition Using Deep AutoEncoder. IEEE Access 2020, 8, 164130–164143. [Google Scholar] [CrossRef]
- Giannakaki, K.; Giannakakis, G.; Farmaki, C.; Sakkalis, V. Emotional State Recognition Using Advanced Machine Learning Techniques on EEG Data. In Proceedings of the 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS), Thessaloniki, Greece, 22–24 June 2017; pp. 337–342. [Google Scholar] [CrossRef]
- Abdel-Hamid, L. An Efficient Machine Learning-Based Emotional Valence Recognition Approach Towards Wearable EEG. Sensors 2023, 23, 1255. [Google Scholar] [CrossRef]
- Topic, A.; Russo, M. Emotion recognition based on EEG feature maps through deep learning network. Eng. Sci. Technol. Int. J. 2021, 24, 1442–1454. [Google Scholar] [CrossRef]
- Rudakov, E.; Laurent, L.; Cousin, V.; Roshdi, A.; Fournier, R.; Nait-ali, A.; Beyrouthy, T.; Kork, S.A. Multi-Task CNN model for emotion recognition from EEG Brain maps. In Proceedings of the 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART), Paris/Créteil, France, 8–10 December 2021. [Google Scholar]
- Stevens, F.; Murphy, D.T.; Smith, S.L. The Self Assessment Manikin And Heart Rate Responses To. In Proceedings of the Interactive Audio Systems Symposium, York, UK, 23 September 2016. [Google Scholar]
- Bradley, M.M.; Lang, P.J. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 1994, 25, 49–59. [Google Scholar] [CrossRef]
- Koelstra, S.; Muhl, C.; Soleymani, M.; Lee, J.S.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. DEAP: A Database for Emotion Analysis Using Physiological Signals. IEEE Trans. Affect. Comput. 2012, 3, 18–31. [Google Scholar] [CrossRef]
- Butpheng, C.; Yeh, K.H.; Xiong, H. Security and Privacy in IoT-Cloud-Based e-Health Systems—A Comprehensive Review. Symmetry 2020, 12, 1191. [Google Scholar] [CrossRef]
- Apicella, A.; Arpaia, P.; Isgrò, F.; Mastrati, G.; Moccaldi, N. A Survey on EEG-Based Solutions for Emotion Recognition with a Low Number of Channels. IEEE Access 2022, 10, 117411–117428. [Google Scholar] [CrossRef]
- Liu, H.; Wang, Z.; Fan, S.; Yang, Z. Optimizing EEG Channel Selection for Emotion Recognition Using DEAP Dataset. IEEE Trans. Affect. Comput. 2020, 11, 27–39. [Google Scholar]
- Patel, N.B.; Patel, A.K. Optimizing EEG Channel Selection for Emotion Recognition Using DEAP Dataset. Int. J. Comput. Sci. Inf. Secur. 2019, 17, 141–148. [Google Scholar]
- Topic, A.; Russo, M.; Stella, M.; Saric, M. Emotion Recognition Using a Reduced Set of EEG Channels Based on Holographic Feature Maps. Sensors 2022, 22, 3248. [Google Scholar] [CrossRef] [PubMed]
- Asghar, M.A.; Khan, M.J.; Fawad; Amin, Y.; Rizwan, M.; Rahman, M.; Badnava, S.; Mirjavadi, S.S. EEG-Based Multi-Modal Emotion Recognition using Bag of Deep Features: An Optimal Feature Selection Approach. Sensors 2019, 19, 5218. [Google Scholar] [CrossRef]
- Wang, Z.M.; Hu, S.Y.; Song, H. Channel Selection Method for EEG Emotion Recognition Using Normalized Mutual Information. IEEE Access 2019, 7, 143303–143311. [Google Scholar] [CrossRef]
- Carper, R.A.; Moses, P.; Tigue, Z.D.; Courchesne, E. Cerebral lobes in autism: Early hyperplasia and abnormal age effects. NeuroImage 2002, 16, 1038–1051. [Google Scholar] [CrossRef]
- EEG-brain activity monitoring and predictive analysis of signals using artificial neural networks. Sensors 2020, 20, 3346. [CrossRef]
- Fingelkurts, A.A.; Fingelkurts, A.A. Morphology and dynamic repertoire of EEG short-term spectral patterns in rest: Explorative study. Neurosci. Res. 2010, 66, 299–312. [Google Scholar] [CrossRef] [PubMed]
- Zhao, W.; Van Someren, E.J.; Li, C.; Chen, X.; Gui, W.; Tian, Y.; Liu, Y.; Lei, X. EEG spectral analysis in insomnia disorder: A systematic review and meta-analysis. Sleep Med. Rev. 2021, 59, 101457. [Google Scholar] [CrossRef] [PubMed]
- Zheng, W.L.; Zhu, J.Y.; Lu, B.L. Identifying Stable Patterns over Time for Emotion Recognition from EEG. IEEE Trans. Affect. Comput. 2016, 10, 417–429. [Google Scholar] [CrossRef]
- Sammler, D.; Grigutsch, M.; Fritz, T.; Koelsch, S. Music and emotion: Electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology 2007, 44, 293–304. [Google Scholar] [CrossRef]
- Roshdy, A.; Alkork, S.; Karar, A.S.; Mhalla, H.; Beyrouthy, T.; Al Barakeh, Z.; Nait-ali, A. Statistical Analysis of Multi-channel EEG Signals for Digitizing Human Emotions. In Proceedings of the 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART), Paris, France, 8–10 December 2021; pp. 1–4. [Google Scholar] [CrossRef]
- Miltiadous, A.; Tzimourta, K.D.; Afrantou, T.; Ioannidis, P.; Grigoriadis, N.; Tsalikakis, D.G.; Angelidis, P.; Tsipouras, M.G.; Glavas, E.; Giannakeas, N.; et al. A Dataset of Scalp EEG Recordings of Alzheimer’s Disease, Frontotemporal Dementia and Healthy Subjects from Routine EEG. Data 2023, 8, 95. [Google Scholar] [CrossRef]
- Oostenveld, R.; Fries, P.; Maris, E.; Schoffelen, J.M. FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data. Int. J. Biomed. Imaging 2011, 2011, 156869. [Google Scholar] [CrossRef]
- Das, R.; Martin, A.; Zurales, T.; Dowling, D.; Khan, A. A Survey on EEG Data Analysis Software. Sci 2023, 5, 23. [Google Scholar] [CrossRef]
- Sun, J.; Rui, C.; Zhou, M.; Hussain, M.; Wang, B.; Xue, J.; Xiang, J. A hybrid deep neural network for classification of schizophrenia using EEG Data. Sci. Rep. 2021, 11, 4706. [Google Scholar] [CrossRef]
- Liu, Z.; Wang, Y.; Shen, X. SHGO: Scalable global optimization via the divide and conquer approach. Appl. Soft Comput. 2021, 103, 107021. [Google Scholar]
- Liu, Z.; Wang, Y.; Shen, X. SHGO-MOGA: A multi-objective optimization algorithm based on scalarized hypervolume contribution. Appl. Soft Comput. 2020, 87, 105984. [Google Scholar]
- Tereshin, N.A.; Padokhin, A.M.; Andreeva, E.S.; Kozlovtseva, E.A. Simplicial Homology Global Optimisation in the Problem of Point-to-Point Ionospheric Ray Tracing. In Proceedings of the 2020 XXXIIIrd General Assembly and Scientific Symposium of the International Union of Radio Science, Rome, Italy, 29 August 2020; pp. 1–4. [Google Scholar] [CrossRef]
- Endres, S.C.; Sandrock, C.; Focke, W.W. A simplicial homology algorithm for Lipschitz optimisation. J. Glob. Optim. 2018, 72, 181–217. [Google Scholar] [CrossRef]
- Lee, S.H.; Lee, J.; Yoon, G.J. Simplicial Homology-Based Global Optimization: Sperner’s Lemma, Criticality, and Applications. IEEE Trans. Cybern. 2020, 51, 4465–4476. [Google Scholar]
Ref. | Channels | Accuracy | Subject | Database | Detection |
---|---|---|---|---|---|
[3] | 14 | 77.60–78.96% | Dependent | Deap | Valence |
Arousal | |||||
Dominance | |||||
[4] | 12 | 81.5–86.87% | Independent | 12 subjects | High |
Low Valence | |||||
Arousal | |||||
[40] | 14 | 87.25% | Dependent | 19 subjects | Happiness and Sadness |
Fear and Anger | |||||
Surprise and Disgust | |||||
[41] | 32 | 96.28–96.62% | Dependent | Deap | High |
Low Valence | |||||
Arousal | |||||
[7] | 62 | 83.33% | Independent | Seed | Neutral |
Sadness and Fear | |||||
Happiness | |||||
[10] | 10 | 58.47–60.90% | Independent | N/A | High |
Low Valence | |||||
Arousal |
Layer (Type) | Output Shape | Param # |
---|---|---|
2D Convolutional | (None, 63, 63, 10) | 130 |
2D Max Pooling | (None, 31, 31, 10) | 0 |
Activation | (None, 31, 31, 10) | 0 |
2D Convolutional | (None, 30, 30, 512) | 20,992 |
2D Max Pooling | (None, 15, 15, 512) | 0 |
Activation | (None, 15, 15, 512) | 0 |
2D Convolutional | (None, 14, 14, 32) | 65,568 |
2D Max Pooling | None, (7, 7, 32) | 0 |
Activation | (None, 7, 7, 32) | 0 |
2D Convolutional | (None, 6, 6, 16) | 2064 |
2D Max Pooling | (None, 3, 3, 16) | 0 |
Activation | (None, 3, 3, 16) | 0 |
Flatten | (None, 144) | 0 |
Dense | (None, 10) | 1450 |
Dense | (None, 2) | 22 |
Total params | 90,226 | |
Trainable params | 90,226 | |
Non-trainable params | 0 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Roshdy, A.; Al Kork, S.; Beyrouthy, T.; Nait-ali, A. Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition. Robotics 2023, 12, 99. https://doi.org/10.3390/robotics12040099
Roshdy A, Al Kork S, Beyrouthy T, Nait-ali A. Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition. Robotics. 2023; 12(4):99. https://doi.org/10.3390/robotics12040099
Chicago/Turabian StyleRoshdy, Ahmed, Samer Al Kork, Taha Beyrouthy, and Amine Nait-ali. 2023. "Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition" Robotics 12, no. 4: 99. https://doi.org/10.3390/robotics12040099