Simultaneous Classification of Both Mental Workload and Stress Level Suitable for an Online Passive Brain–Computer Interface
Abstract
:1. Introduction
2. Materials and Methods
2.1. Participants
2.2. Physiological Signal Acquisition
2.3. Experimental Procedure
2.3.1. Structure of Experimental Session
2.3.2. Mental Workload Induction
2.3.3. Affective State Induction
2.4. Validation of Stress and Workload Induction
3. Data Analysis
3.1. EEG-Based Classification of Mental Workload Level and Affective State
3.2. EEG Preprocessing
3.3. EEG Signal Feature Calculation
3.4. Classification
- Subject-specific paradigm: For each target subject, the classifier was trained on the subject’s first two blocks of data and tested on the subject’s final two blocks. A regularized Linear Discriminant Analysis (LDA) algorithm was used for classification [37].
- Cross-subject without TL paradigm: For each target subject, the classifier was trained on the subject’s first two blocks of data combined with all data from the other 17 subjects and tested on the subject’s final two blocks of data. No transfer learning algorithm was applied. A regularized LDA algorithm was used for classification [37].
- Cross-subject with TL paradigm: For each target subject, the classifier was trained on the subject’s first two blocks of data combined with all data from the other 17 subjects and tested on the subject’s final two blocks of data. The InstanceEasyTL transfer learning method was applied to reject the differences between data coming from different subjects; this method was originally proposed in [32] and is described in detail in the section below.
InstanceEasyTL Algorithm
4. Results
5. Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Brunner, C.; Birbaumer, N.; Blankertz, B.; Guger, C.; Kübler, A.; Mattia, D.; Millan, J.D.R.; Miralles, F.; Nijholt, A.; Opisso, E.; et al. BNCI Horizon 2020: Towards a roadmap for the BCI community. Brain-Comput. Interfaces 2015, 2, 1–10. [Google Scholar] [CrossRef] [Green Version]
- Zander, T.O.; Kothe, C. Towards passive brain–computer interfaces: Applying brain–computer interface technology to human–machine systems in general. J. Neural Eng. 2011, 8, 025005. [Google Scholar] [CrossRef]
- Roman-Gonzalez, A. EEG Signal Processing for BCI Applications. In Human-Computer Systems Interaction: Backgrounds and Applications; Springer: Berlin/Heidelberg, Germany, 2012; Volume 98, pp. 571–591. [Google Scholar] [CrossRef] [Green Version]
- Allison, B.Z.; Polich, J. Workload assessment of computer gaming using a single-stimulus event-related potential paradigm. Biol. Psychol. 2008, 77, 277–283. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Baldwin, C.L.; Penaranda, B. Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification. NeuroImage 2011, 59, 48–56. [Google Scholar] [CrossRef] [PubMed]
- Chaouachi, M.; Jraidi, I.; Frasson, C. Adapting to Learners’ Mental States Using a Physiological Computing Approach. In Proceedings of the 28th International Flairs Conference, Hollywood, FL, USA, 18–20 May 2015; pp. 257–262. [Google Scholar]
- Andreessen, L.M.; Gerjets, P.; Meurers, D.; Zander, T.O. Toward neuroadaptive support technologies for improving digital reading: A passive BCI-based assessment of mental workload imposed by text difficulty and presentation speed during reading. User Model. User-Adapt. Interact. 2020, 31, 75–104. [Google Scholar] [CrossRef]
- Hirshfield, L.M.; Gulotta, R.; Hirshfield, S.; Hincks, S.; Russell, M.; Ward, R.; Williams, T.; Jacob, R. This is your brain on interfaces: Enhancing usability testing with functional near-infrared spectroscopy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada, 7–12 May 2011; ACM: New York, NY, USA, 2011; pp. 373–382. [Google Scholar]
- Yi, W.; Qiu, S.; Fan, X.; Zhang, L. Estimation of mental workload induced by different presentation rates in rapid serial visual presentation tasks. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 5552–5555. [Google Scholar] [CrossRef]
- Chin, Z.Y.; Zhang, X.; Wang, C.; Ang, K.K. EEG-based discrimination of different cognitive workload levels from mental arithmetic. In Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA, 18–21 July 2018; pp. 1984–1987. [Google Scholar] [CrossRef]
- Jao, P.-K.; Chavarriaga, R.; Milln, J.D.R. Analysis of EEG Correlates of Perceived Difficulty in Dynamically Changing Flying Tasks. In Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan, 7–10 October 2018; pp. 550–555. [Google Scholar] [CrossRef]
- Duraisingam, A.; Palaniappan, R.; Andrews, S. Cognitive task difficulty analysis using EEG and data mining. In Proceedings of the 2017 Conference on Emerging Devices and Smart Systems (ICEDSS), Mallasamudram, India, 3–4 March 2017; pp. 52–57. [Google Scholar]
- Koyas, E.; Hocaoglu, E.; Patoglu, V.; Cetin, M. Detection of intention level in response to task difficulty from EEG signals. In Proceedings of the 2013 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), Southampton, UK, 22–25 September 2013. [Google Scholar]
- Wang, S.; Gwizdka, J.; Chaovalitwongse, W.A. Using Wireless EEG Signals to Assess Memory Workload in the n-Back Task. IEEE Trans. Hum.-Mach. Syst. 2015, 46, 424–435. [Google Scholar] [CrossRef]
- Smith, M.E.; Gevins, A.S.; Brown, H.; Karnik, A.; Du, R. Monitoring Task Loading with Multivariate EEG Measures during Complex Forms of Human-Computer Interaction. Hum. Factors J. Hum. Factors Ergon. Soc. 2001, 43, 366–380. [Google Scholar] [CrossRef]
- Brookings, J.B.; Wilson, G.F.; Swain, C.R. Psychophysiological responses to changes in workload during simulated air traffic control. Biol. Psychol. 1996, 42, 361–377. [Google Scholar] [CrossRef]
- Borghini, G.; Aricò, P.; Di Flumeri, G.; Cartocci, G.; Colosimo, A.; Bonelli, S.; Golfetti, A.; Imbert, J.P.; Granger, G.; Benhacene, R.; et al. EEG-Based Cognitive Control Behaviour Assessment: An Ecological study with Professional Air Traffic Controllers. Sci. Rep. 2017, 7, 547. [Google Scholar] [CrossRef]
- Aricò, P.; Borghini, G.; Di Flumeri, G.; Colosimo, A.; Bonelli, S.; Golfetti, A.; Pozzi, S.; Imbert, J.-P.; Granger, G.; Benhacene, R.; et al. Adaptive Automation Triggered by EEG-Based Mental Workload Index: A Passive Brain-Computer Interface Application in Realistic Air Traffic Control Environment. Front. Hum. Neurosci. 2016, 10, 539. [Google Scholar] [CrossRef] [Green Version]
- Kakkos, I.; Dimitrakopoulos, G.N.; Gao, L.; Zhang, Y.; Qi, P.; Matsopoulos, G.K.; Thakor, N.; Bezerianos, A.; Sun, Y. Mental Workload Drives Different Reorganizations of Functional Cortical Connectivity Between 2D and 3D Simulated Flight Experiments. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 27, 1704–1713. [Google Scholar] [CrossRef]
- Zhang, P.; Wang, X.; Zhang, W.; Chen, J. Learning Spatial–Spectral–Temporal EEG Features With Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment. IEEE Trans. Neural Syst. Rehabil. Eng. 2018, 27, 31–42. [Google Scholar] [CrossRef]
- Zhang, P.; Wang, X.; Chen, J.; You, W.; Zhang, W. Spectral and Temporal Feature Learning With Two-Stream Neural Networks for Mental Workload Assessment. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 27, 1149–1159. [Google Scholar] [CrossRef] [PubMed]
- Zarjam, P.; Epps, J.; Lovell, N. Beyond Subjective Self-Rating: EEG Signal Classification of Cognitive Workload. IEEE Trans. Auton. Ment. Dev. 2015, 7, 301–310. [Google Scholar] [CrossRef]
- Zammouri, A.; Chraa-Mesbahi, S.; Moussa, A.A.; Zerouali, S.; Sahnoun, M.; Tairi, H.; Mahraz, M.A. Brain waves-based index for workload estimation and mental effort engagement recognition. J. Phys. Conf. Ser. 2017, 904, 12008. [Google Scholar] [CrossRef] [Green Version]
- Mühl, C.; Jeunet, C.; Lotte, F. EEG-based workload estimation across affective contexts. Front. Neurosci. 2014, 8, 114. [Google Scholar]
- Bagheri, M.; Power, S.D. EEG-based detection of mental workload level and stress: The effect of variation in each state on classification of the other. J. Neural Eng. 2020, 17, 056015. [Google Scholar] [CrossRef] [PubMed]
- Bagheri, M.; Power, S.D. Investigating hierarchical and ensemble classification approaches to mitigate the negative effect of varying stress state on EEG-based detection of mental workload level- and vice versa. Brain-Comput. Interfaces 2021, 8, 26–37. [Google Scholar] [CrossRef]
- Bagheri, M.; Power, S.D. EEG-based classification of visual and auditory monitoring tasks. In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada, 11–14 October 2020; pp. 4032–4037. [Google Scholar] [CrossRef]
- Boff, K.R.; Kaufman, L.; Thomas, J.P. Handbook of Perception and Human Performance: Vol. II Cognitive Processes and Performance; John Wiley & Sons: New York, NY, USA, 1986. [Google Scholar]
- Longo, L. Mental workload in medicine: Foundations, applications, open problems, challenges and future perspectives. In Proceedings of the 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), Dublin, Ireland, 20–24 June 2016; pp. 106–111. [Google Scholar]
- Parent, M.; Peysakhovich, V.; Mandrick, K.; Tremblay, S.; Causse, M. The diagnosticity of psychophysiological signatures: Can we disentangle mental workload from acute stress with ECG and fNIRS? Int. J. Psychophysiol. 2019, 146, 139–147. [Google Scholar] [CrossRef] [PubMed]
- Salas, E.; Martin, L.; Flin, R. Decision-Making under Stress: Emerging Themes and Applications, 1st ed.; Routledge: London, UK, 1998. [Google Scholar]
- Zeng, H.; Zhang, J.; Zakaria, W.; Babiloni, F.; Gianluca, B.; Li, X.; Kong, W. InstanceEasyTL: An Improved Transfer-Learning Method for EEG-Based Cross-Subject Fatigue Detection. Sensors 2020, 20, 7251. [Google Scholar] [CrossRef]
- Zijlstra, F.R.H. Efficiency in Work Behaviour: A Design Approach for Modern Tool. Ph.D. Thesis, Delft University of Technology, Delft, The Netherlands, 1993. [Google Scholar]
- Kirschbaum, C.; Pirke, K.M.; Hellhammer, D. The ‘Trier Social Stress Test’—A tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology 1993, 28, 76–81. [Google Scholar] [CrossRef]
- Spielberger, C.D.; Gorsuch, R.L.; Lushene, R.E. Manual for the State-Trait Anxiety Inventory; Consulting Psychologists Press: Palo Alto, CA, USA, 1983. [Google Scholar]
- Klimesch, W. EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Res. Rev. 1999, 29, 169–195. [Google Scholar] [CrossRef]
- Ledoit, O.; Wolf, M. A well-conditioned estimator for large-dimensional covariance matrices. J. Multivar. Anal. 2004, 88, 365–411. [Google Scholar] [CrossRef] [Green Version]
- Krusienski, D.J.; Grosse-Wentrup, M.; Galán, F.; Coyle, D.; Miller, K.J.; Forney, E.; Anderson, C.W. Critical issues in state-of-the-art brain–computer interface signal processing. J. Neural Eng. 2011, 8, 025002. [Google Scholar] [CrossRef]
- Hine, G.E.; Maiorana, E.; Campisi, P. Resting-State EEG: A Study on Its Non-Stationarity for Biometric Applications. In Proceedings of the 2017 International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany, 20–22 September 2017. [Google Scholar]
- Wan, Z.; Yang, R.; Huang, M.; Zeng, N.; Liu, X. A review on transfer learning in EEG signal analysis. Neurocomputing 2020, 421, 1–14. [Google Scholar] [CrossRef]
- Zeng, H.; Yang, C.; Zhang, H.; Wu, Z.; Zhang, J.; Dai, G.; Babiloni, F.; Kong, W. A LightGBM-Based EEG Analysis Method for Driver Mental States Classification. Comput. Intell. Neurosci. 2019, 2019, 3761203. [Google Scholar] [CrossRef]
- Deng, Z.; Xu, P.; Xie, L.; Choi, K.-S.; Wang, S. Transductive Joint-Knowledge-Transfer TSK FS for Recognition of Epileptic EEG Signals. IEEE Trans. Neural Syst. Rehabil. Eng. 2018, 26, 1481–1494. [Google Scholar] [CrossRef]
- Wang, J.; Chen, Y.; Yu, H.; Huang, M.; Yang, Q. Easy Transfer Learning by Exploiting Intra-domain Structures. In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME), Shanghai, China, 8–12 July 2019; pp. 8–12. [Google Scholar]
- Dai, W.; Yang, Q.; Xue, G.-R.; Yu, Y. Boosting for transfer learning. In Proceedings of the 24th International Conference on Machine Learning, Corvallis, OR, USA, 20–24 June 2007; ACM: New York, NY, USA, 2007; pp. 193–200. [Google Scholar]
- Pan, S.J.; Tsang, I.; Kwok, J.T.; Yang, Q. Domain Adaptation via Transfer Component Analysis. IEEE Trans. Neural Netw. 2010, 22, 199–210. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Gong, B.; Shi, Y.; Sha, F.; Grauman, K. Geodesic flow kernel for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition, Providence, RI, USA, 16–21 June 2012. [Google Scholar]
- Ganin, Y.; Ustinova, E.; Ajakan, H.; Germain, P.; Larochelle, H.; Laviolette, F.; Marchand, M.; Lempitsky, V. Domain-Adversarial Training of Neural Networks. J. Mach. Learn. Res. 2016, 17, 1–35. [Google Scholar] [CrossRef] [Green Version]
- Lee, D.-H.; Jeong, J.-H.; Kim, K.; Yu, B.-W.; Lee, S.-W. Continuous EEG Decoding of Pilots’ Mental States Using Multiple Feature Block-Based Convolutional Neural Network. IEEE Access 2020, 8, 121929–121941. [Google Scholar] [CrossRef]
- Chai, X.; Wang, Q.; Zhao, Y.; Li, Y.; Liu, D.; Liu, X.; Bai, O. A Fast, Efficient Domain Adaptation Technique for Cross-Domain Electroencephalography(EEG)-Based Emotion Recognition. Sensors 2017, 17, 1014. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Mental Workload Level Classification Results (Easy vs. Difficult) | ||||||
---|---|---|---|---|---|---|
Subject-Specific | Cross-Subject without TL | Cross-Subject with TL | ||||
Subjects | Accuracy | F1 Score | Accuracy | F1 Score | Accuracy | F1 Score |
1 | 66.7 | 0.66 | 58.2 | 0.58 | 73.5 | 0.73 |
2 | 49.3 | 0.51 | 47.2 | 0.48 | 64.2 | 0.65 |
3 | 64 | 0.63 | 59.7 | 0.59 | 74.8 | 0.74 |
4 | 65.8 | 0.65 | 59 | 0.6 | 75.0 | 0.75 |
5 | 53.0 | 0.54 | 51.3 | 0.52 | 65.9 | 0.66 |
6 | 52.5 | 0.53 | 53.8 | 0.53 | 68.8 | 0.67 |
7 | 56.4 | 0.56 | 50.5 | 0.5 | 66.5 | 0.65 |
8 | 58.1 | 0.57 | 50.9 | 0.5 | 63.4 | 0.63 |
9 | 57.7 | 0.58 | 58.6 | 0.58 | 73.5 | 0.74 |
10 | 66.5 | 0.66 | 59.7 | 0.59 | 77.2 | 0.76 |
11 | 59.2 | 0.6 | 56.8 | 0.55 | 71.6 | 0.72 |
12 | 57.6 | 0.57 | 55.9 | 0.56 | 69.5 | 0.68 |
13 | 66.6 | 0.66 | 65.5 | 0.66 | 79.8 | 0.79 |
14 | 67.7 | 0.67 | 67.7 | 0.67 | 81.4 | 0.81 |
15 | 60.3 | 0.61 | 61.3 | 0.62 | 80.5 | 0.8 |
16 | 60.8 | 0.6 | 56.0 | 0.56 | 72.3 | 0.71 |
17 | 58.8 | 0.58 | 58.5 | 0.58 | 72.7 | 0.72 |
18 | 57.7 | 0.57 | 57.3 | 0.57 | 69.6 | 0.7 |
Mean: | 59.9 ± 5.3 | 0.59 ± 0.04 | 57.1 ± 5.1 | 0.56 ± 0.05 | 72.2 ± 5.3 | 0.71 ± 0.05 |
Affective State Classification Results (Relaxed vs. Stressed) | ||||||
---|---|---|---|---|---|---|
Subject-Specific | Cross-Subject without TL | Cross-Subject with TL | ||||
Subjects | Accuracy | F1 Score | Accuracy | F1 Score | Accuracy | F1 Score |
1 | 68.8 | 0.69 | 65.1 | 0.64 | 78.6 | 0.78 |
2 | 60.0 | 0.59 | 47.7 | 0.47 | 67.4 | 0.66 |
3 | 61.6 | 0.61 | 62.4 | 0.61 | 79.0 | 0.78 |
4 | 64.4 | 0.64 | 64.2 | 0.63 | 80.0 | 0.79 |
5 | 60.8 | 0.61 | 51.4 | 0.52 | 71.5 | 0.73 |
6 | 65.5 | 0.65 | 50.9 | 0.51 | 68.4 | 0.68 |
7 | 60.5 | 0.6 | 54.8 | 0.54 | 67.8 | 0.67 |
8 | 60.6 | 0.61 | 56.5 | 0.56 | 66.9 | 0.67 |
9 | 64.4 | 0.64 | 56.1 | 0.56 | 74.2 | 0.75 |
10 | 64.0 | 0.63 | 64.9 | 0.63 | 78.5 | 0.77 |
11 | 59.0 | 0.59 | 57.6 | 0.56 | 71.3 | 0.71 |
12 | 65.3 | 0.64 | 56.0 | 0.56 | 69.9 | 0.7 |
13 | 72.8 | 0.71 | 65.0 | 0.64 | 81.9 | 0.81 |
14 | 73.6 | 0.74 | 66.1 | 0.67 | 79.0 | 0.78 |
15 | 67.2 | 0.67 | 58.7 | 0.58 | 80.2 | 0.81 |
16 | 62.3 | 0.62 | 59.2 | 0.58 | 77.0 | 0.78 |
17 | 66.8 | 0.66 | 57.2 | 0.57 | 74.3 | 0.74 |
18 | 62.7 | 0.63 | 56.1 | 0.56 | 70.3 | 0.71 |
Mean: | 64.5 ± 4.1 | 0.64 ± 0.04 | 58.3 ± 5.4 | 0.57 ± 0.05 | 74.2 ± 5.1 | 0.74 ± 0.05 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bagheri, M.; Power, S.D. Simultaneous Classification of Both Mental Workload and Stress Level Suitable for an Online Passive Brain–Computer Interface. Sensors 2022, 22, 535. https://doi.org/10.3390/s22020535
Bagheri M, Power SD. Simultaneous Classification of Both Mental Workload and Stress Level Suitable for an Online Passive Brain–Computer Interface. Sensors. 2022; 22(2):535. https://doi.org/10.3390/s22020535
Chicago/Turabian StyleBagheri, Mahsa, and Sarah D. Power. 2022. "Simultaneous Classification of Both Mental Workload and Stress Level Suitable for an Online Passive Brain–Computer Interface" Sensors 22, no. 2: 535. https://doi.org/10.3390/s22020535
APA StyleBagheri, M., & Power, S. D. (2022). Simultaneous Classification of Both Mental Workload and Stress Level Suitable for an Online Passive Brain–Computer Interface. Sensors, 22(2), 535. https://doi.org/10.3390/s22020535