Utilizing Dry Electrode Electroencephalography and AI Robotics for Cognitive Stress Monitoring in Video Gaming
Abstract
:1. Introduction
- The DSI-24 is a state-of-the-art wireless EEG device, highly efficient in capturing brainwave activity in real-time, making it an ideal tool for dynamic and interactive settings such as video gaming.
- The Furhat [19] social robot (available: https://furhatrobotics.com/furhat-robot/, accessed on 29 May 2024) is one of the emerging platforms that can be employed in human–robot interaction with a variety of applications.
- ChatGPT (available: https://openai.com/chatgpt/, accessed on 29 May 2024) has been seen as a flexible tool for text generation, which can be adapted to different context through user prompts. While it is mainly accessed through a web browser, it can be accessed through an API embedded in a program serving a specific application.
2. System-Level Design
2.1. System Components
2.1.1. DSI-24
2.1.2. Furhat
2.2. Experimental Set-Up
2.3. Data Acquisition Procedure
3. Results and Discussion
3.1. Implementation and EEG Measurements
3.2. Machine Learning Classification
3.2.1. Pre-Processing
3.2.2. Baseline Model
3.2.3. Feature Extraction Using Local Binary Pattern
3.2.4. Model Evaluation
4. Limitations and Future Work
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Brilliant, T.D.; Nouchi, R.; Kawashima, R. Does Video Gaming Have Impacts on the Brain: Evidence from a Systematic Review. Brain Sci. 2019, 9, 251. [Google Scholar] [CrossRef]
- Bavelier, D.; Green, C.S.; Pouget, A.; Schrater, P. Brain Plasticity Through the Life Span: Learning to Learn and Action Video Games. Annu. Rev. Neurosci. 2012, 35, 391–416. [Google Scholar] [CrossRef] [PubMed]
- Anderson, C.A.; Shibuya, A.; Ihori, N.; Swing, E.L.; Bushman, B.J.; Sakamoto, A.; Rothstein, H.R.; Saleem, M. Violent Video Game Effects on Aggression, Empathy, and Prosocial Behavior in Eastern and Western Countries: A Meta-Analytic Review. Psychol. Bull. 2010, 136, 151–173. [Google Scholar] [CrossRef]
- Xu, J.; Fan, Y.; Wang, M.; Ji, B.; Li, L.; Jin, M.; Shang, S.; Ni, C.; Cheng, Y.; Dong, L.; et al. Microneedle Array Electrode With Ag-PPS Modification for Superior Bio-Signal Recording on Skin. IEEE Sens. J. 2023, 23, 24196–24204. [Google Scholar] [CrossRef]
- Gao, K.; Wu, N.; Ji, B.; Liu, J. A Film Electrode upon Nanoarchitectonics of Bacterial Cellulose and Conductive Fabric for Forehead Electroencephalogram Measurement. Sensors 2023, 23, 7887. [Google Scholar] [CrossRef]
- Shin, J.; Kwon, J.; Kim, J.; Ryu, H.; Ok, J.; Kwon, S.; Park, H.; Kim, T.i. Wearable EEG electronics for a Brain–AI Closed-Loop System to enhance autonomous machine decision-making. NPJ Flex. Electron. 2022, 6, 32. [Google Scholar] [CrossRef]
- Kappenman, E.S.; Luck, S.J. The Oxford Handbook of Event-Related Potential Components; Oxford University Press: Oxford, UK, 2010. [Google Scholar]
- Palaus, M.; Marrón, E.M.; Viejo-Sobera, R.; Redolar-Ripoll, D. Neurobiological Consequences of Playing Video Games. Front. Hum. Neurosci. 2017, 11, 248. [Google Scholar] [CrossRef]
- Kühn, S.; Gleich, T.; Lorenz, R.C.; Lindenberger, U.; Gallinat, J. Playing Super Mario induces structural brain plasticity: Gray matter changes resulting from training with a commercial video game. Mol. Psychiatry 2014, 19, 265–271. [Google Scholar] [CrossRef]
- Hafeez, T.; Saeed, U.; Arsalan, A.; Anwar, S.M.; Ashraf, M.U.; Alsubhi, K. EEG in game user analysis: A framework for expertise classification during gameplay. PLoS ONE 2021, 16, e0246913. [Google Scholar] [CrossRef]
- Sheikholeslami, C.; Yuan, H.; He, E.J.; Bai, X.; Yang, L.; He, B. A high resolution EEG study of dynamic brain activity during video game play. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, France, 22–26 August 2007; IEEE: Piscataway, NJ, USA, 2007; pp. 2489–2491. [Google Scholar] [CrossRef]
- Youssef, K.; Said, S.; Alkork, S.; Beyrouthy, T. A Survey on Recent Advances in Social Robotics. Robotics 2022, 11, 75. [Google Scholar] [CrossRef]
- Rice, A.; Klęczek, K.; Alimardani, M. The Effectiveness of Social Robots in Stress Management Interventions for University Students. In Social Robotics; Ali, A.A., Cabibihan, J.J., Meskin, N., Rossi, S., Jiang, W., He, H., Ge, S.S., Eds.; Springer: Singapore, 2024; pp. 181–190. [Google Scholar]
- Rasouli, S.; Gupta, G.; Nilsen, E.; Dautenhahn, K. Potential Applications of Social Robots in Robot-Assisted Interventions for Social Anxiety. Int. J. Soc. Robot. 2022, 14, 1–32. [Google Scholar] [CrossRef] [PubMed]
- Wykowska, A. Social Robots to Test Flexibility of Human Social Cognition. Int. J. Soc. Robot. 2020, 12, 1203–1211. [Google Scholar] [CrossRef] [PubMed]
- Sabo, R.; Beňuš, S.; Kevická, V.; Trnka, M.; Rusko, M.; Darjaa, S.; Kejriwal, J. Towards the Use of Social Robot Furhat and Generative AI in Testing Cognitive Abilities. Hum. Aff. 2024, 34, 224–243. [Google Scholar] [CrossRef]
- Galatolo, A.; Leite, I.; Winkle, K. Personality-Adapted Language Generation for Social Robots. In Proceedings of the 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Busan, Republic of Korea, 28–31 August 2023; pp. 1800–1807. [Google Scholar] [CrossRef]
- DSI-24-Wearable Sensing|Dry EEG. Available online: https://wearablesensing.com/dsi-24/ (accessed on 1 May 2023).
- Al Moubayed, S.; Beskow, J.; Skantze, G.; Granström, B. Furhat: A Back-Projected Human-Like Robot Head for Multiparty Human-Machine Interaction. In Cognitive Behavioural Systems, Proceedings of the COST 2102 International, Dresden, Germany, 21–26 February 2011; Esposito, A., Esposito, A.M., Vinciarelli, A., Hoffmann, R., Müller, V.C., Eds.; Springer: Berlin/Heidelberg, Germany, 2012; pp. 114–130. [Google Scholar]
- Li, J.; Masullo, M.; Maffei, L.; Pascale, A.; Chau, C.k.; Lin, M. Improving informational-attentional masking of water sound on traffic noise by spatial variation settings: An in situ study with brain activity measurements. Appl. Acoust. 2024, 218, 109904. [Google Scholar] [CrossRef]
- Kim, S.; Ryu, J.; Lee, Y.; Park, H.; Lee, K. Methods for Selecting Design Alternatives through Integrated Analysis of Energy Performance of Buildings and the Physiological Responses of Occupants. Buildings 2024, 14, 237. [Google Scholar] [CrossRef]
- Park, J.; Wang, S.; Lee, S.; Seo, S.; Lee, N.; Kim, S. Viewer Emotional Response to Webtoon-Based Drama: An EEG Analysis. Int. J. Human–Computer Interact. 2023, 1–15. [Google Scholar] [CrossRef]
- Liu, F.; Yang, P.; Shu, Y.; Liu, N.; Sheng, J.; Luo, J.; Wang, X.; Liu, Y. Emotion Recognition from Few-Channel EEG Signals by Integrating Deep Feature Aggregation and Transfer Learning. IEEE Trans. Affect. Comput. 2023, 1–17. [Google Scholar] [CrossRef]
- Chan, M.M.; Choi, C.X.; Tsoi, T.C.; Shea, C.K.; Yiu, K.W.; Han, Y.M. Effects of multisession cathodal transcranial direct current stimulation with cognitive training on sociocognitive functioning and brain dynamics in ASD: A double-blind, sham-controlled, randomized EEG study. Brain Stimul. 2023, 16, 1604–1616. [Google Scholar] [CrossRef]
- Mizrahi, D.; Laufer, I.; Zuckerman, I. Modulation of Beta Power as a Function of Attachment Style and Feedback Valence. In Proceedings of the International Conference on Brain Informatics, Hoboken, NJ, USA, 1–3 August 2023; Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, Germany, 2023. [Google Scholar] [CrossRef]
- Georgiadis, K.; Kalaganis, F.P.; Oikonomou, V.P.; Nikolopoulos, S.; Laskaris, N.A.; Kompatsiaris, I. Harnessing the Potential of EEG in Neuromarketing with Deep Learning and Riemannian Geometry. In Proceedings of the International Conference on Brain Informatics, Hoboken, NJ, USA, 1–3 August 2023; Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, Germany, 2023. [Google Scholar] [CrossRef]
- Robb, D.A.; Lopes, J.; Ahmad, M.I.; McKenna, P.E.; Liu, X.; Lohan, K.; Hastie, H. Seeing eye to eye: Trustworthy embodiment for task-based conversational agents. Front. Robot. AI 2023, 10, 1234767. [Google Scholar] [CrossRef]
- Casaccia, S.; Revel, G.M.; Scalise, L.; Bevilacqua, R.; Rossi, L.; Paauwe, R.A.; Karkowsky, I.; Ercoli, I.; Artur Serrano, J.; Suijkerbuijk, S.; et al. Social Robot and Sensor Network in Support of Activity of Daily Living for People with Dementia. In Proceedings of the Dementia Lab 2019. Making Design Work: Engaging with Dementia in Context, Eindhoven, The Netherlands, 21–22 October 2019; Brankaert, R., IJsselsteijn, W., Eds.; Springer: Cham, Switzerland, 2019; pp. 128–135. [Google Scholar]
- Jasper, H.H. The ten-twenty electrode system of the International Federation. Electroencephalogr. Clin. Neurophysiol. 1958, 10, 371–375. [Google Scholar]
- Awad, M.; Khanna, R. Support Vector Machines for Classification. In Efficient Learning Machines: Theories, Concepts, and Applications for Engineers and System Designers; Apress: Berkeley, CA, USA, 2015; pp. 39–66. [Google Scholar] [CrossRef]
- Hegedues, C.; Constantino, J.; Dixen, L.; Burelli, P. Investigating Perceived and Mechanical Challenge in Games Through Cognitive Activity. In Proceedings of the IEEE Conference on Game, Boston, MA, USA, 21–24 August 2023; IEEE: Piscataway, NJ, USA, 2023. [Google Scholar] [CrossRef]
- Du, G.; Zhou, W.; Li, C.; Li, D.; Liu, P.X. An Emotion Recognition Method for Game Evaluation Based on Electroencephalogram. IEEE Trans. Affect. Comput. 2023, 14, 591–602. [Google Scholar] [CrossRef]
- Roshdy, A.; Al Kork, S.; Beyrouthy, T.; Nait-ali, A. Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition. Robotics 2023, 12, 99. [Google Scholar] [CrossRef]
- Behera, S.; Mohanty, M.N. Classification of EEG Signal Using SVM. In Advances in Electrical Control and Signal Systems; Pradhan, G., Morris, S., Nayak, N., Eds.; Singapore: Berlin/Heidelberg, Germany, 2020; pp. 859–869. [Google Scholar]
- Shanmugam, S.; Dharmar, S. Implementation of a non-linear SVM classification for seizure EEG signal analysis on FPGA. Eng. Appl. Artif. Intell. 2024, 131, 107826. [Google Scholar] [CrossRef]
- Khan, K.A.; Shanir, P.P.; Khan, Y.U.; Farooq, O. A hybrid Local Binary Pattern and wavelets based approach for EEG classification for diagnosing epilepsy. Expert Syst. Appl. 2020, 140, 112895. [Google Scholar] [CrossRef]
- Kumar, T.S.; Kanhangad, V.; Pachori, R.B. Classification of seizure and seizure-free EEG signals using local binary patterns. Biomed. Signal Process. Control 2015, 15, 33–40. [Google Scholar] [CrossRef]
- Katmah, R.; Al-Shargie, F.; Tariq, U.; Babiloni, F.; Al-Mughairbi, F.; Al-Nashash, H. A Review on Mental Stress Assessment Methods Using EEG Signals. Sensors 2021, 21, 5043. [Google Scholar] [CrossRef]
Classification Metric | Baseline Model | LBPH Model |
---|---|---|
Accuracy | 0.53 | 0.68 |
Recall | 0.63 | 0.71 |
F1-score | 0.71 | 0.73 |
Precision | 0.79 | 0.75 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Alrasheedi, A.A.; Alrabeah, A.Z.; Almuhareb, F.J.; Alras, N.M.Y.; Alduaij, S.N.; Karar, A.S.; Said, S.; Youssef, K.; Kork, S.A. Utilizing Dry Electrode Electroencephalography and AI Robotics for Cognitive Stress Monitoring in Video Gaming. Appl. Syst. Innov. 2024, 7, 68. https://doi.org/10.3390/asi7040068
Alrasheedi AA, Alrabeah AZ, Almuhareb FJ, Alras NMY, Alduaij SN, Karar AS, Said S, Youssef K, Kork SA. Utilizing Dry Electrode Electroencephalography and AI Robotics for Cognitive Stress Monitoring in Video Gaming. Applied System Innovation. 2024; 7(4):68. https://doi.org/10.3390/asi7040068
Chicago/Turabian StyleAlrasheedi, Aseel A., Alyah Z. Alrabeah, Fatemah J. Almuhareb, Noureyah M. Y. Alras, Shaymaa N. Alduaij, Abdullah S. Karar, Sherif Said, Karim Youssef, and Samer Al Kork. 2024. "Utilizing Dry Electrode Electroencephalography and AI Robotics for Cognitive Stress Monitoring in Video Gaming" Applied System Innovation 7, no. 4: 68. https://doi.org/10.3390/asi7040068
APA StyleAlrasheedi, A. A., Alrabeah, A. Z., Almuhareb, F. J., Alras, N. M. Y., Alduaij, S. N., Karar, A. S., Said, S., Youssef, K., & Kork, S. A. (2024). Utilizing Dry Electrode Electroencephalography and AI Robotics for Cognitive Stress Monitoring in Video Gaming. Applied System Innovation, 7(4), 68. https://doi.org/10.3390/asi7040068