Recognition of Emotion by Brain Connectivity and Eye Movement
Abstract
:1. Introduction
2. Materials and Methods
2.1. Stimuli Selection
2.2. Experiment Design
2.3. Participants
2.4. Experimental Protocol
3. Analysis
4. Results
4.1. Subject Evaluation
4.2. Brain Connectivity Features
4.2.1. Characteristics of Three Distance Connectivity
4.2.2. Power Value Analysis in Three Distance Connectivity
4.3. Clustering Eye Movement Features
5. Conclusions and Discussion
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Conflicts of Interest
References
- Chanel, G.; Kierkels, J.J.M.; Soleymani, M.; Pun, T. Short-term emotion assessment in a recall paradigm. Int. Hum. J. Comput. Stud. 2009, 67, 607–627. [Google Scholar] [CrossRef]
- Friston, K.J. Functional and effective connectivity: A review. Brain Connect. 2011, 1, 13–36. [Google Scholar] [CrossRef]
- Mauss, I.B.; Robinson, M.D. Measures of emotion: A review. Cogn. Emot. 2009, 23, 209–237. [Google Scholar] [CrossRef]
- Kim, J.-H.; Kim, S.-L.; Cha, Y.-S.; Park, S.-I.; Hwang, M.-C. Analysis of CNS functional connectivity by relationship duration in positive emotion sharing relation. In Proceedings of the Korean Society for Emotion and Sensibility Conference, Seoul, Korea, 11 May 2012; pp. 11–12. [Google Scholar]
- Moon, S.-E.; Jang, S.; Lee, J.-S. Convolutional neural network approach for EEG-based emotion recognition using brain connectivity and its spatial information. In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018; pp. 2556–2560. [Google Scholar]
- Voorbij, H.J. Searching scientific information on the Internet: A Dutch academic user survey. Am. J. Soc. Inf. Sci. 1999, 50, 598–615. [Google Scholar] [CrossRef]
- Thompson, K.G.; Biscoe, K.L.; Sato, T.R. Neuronal basis of covert spatial attention in the frontal eye field. Neurosci. J. 2005, 25, 9479–9487. [Google Scholar] [CrossRef]
- Bulling, A.; Ward, J.A.; Gellersen, H.; Tröster, G. Eye movement analysis for activity recognition using electrooculography. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 33, 741–753. [Google Scholar] [CrossRef]
- Võ, M.L.H.; Jacobs, A.M.; Kuchinke, L.; Hofmann, M.; Conrad, M.; Schacht, A.; Hutzler, F. The coupling of emotion and cognition in the eye: Introducing the pupil old/new effect. Psychophysiology 2008, 45, 130–140. [Google Scholar] [CrossRef]
- Tatler, B.W.; Brockmole, J.R.; Carpenter, R.H.S. LATEST: A model of saccadic decisions in space and time. Psychol. Rev. 2017, 124, 267. [Google Scholar] [CrossRef]
- Carpenter, R.H.S. The neural control of looking. Curr. Biol. 2000, 10, R291–R293. [Google Scholar] [CrossRef]
- Glimcher, P.W. The neurobiology of visual-saccadic decision making. Annu. Rev. Neurosci. 2003, 26, 133–179. [Google Scholar] [CrossRef] [Green Version]
- Argyle, M.; Cook, M. Gaze and Mutual Gaze; Cambridge University Press: Cambridge, UK, 1976. [Google Scholar]
- Baron-Cohen, S.; Campbell, R.; Karmiloff-Smith, A.; Grant, J.; Walker, J. Are children with autism blind to the mentalistic significance of the eyes? Br. Dev. J. Psychol. 1995, 13, 379–398. [Google Scholar] [CrossRef]
- Emery, N.J. The eyes have it: The neuroethology, function and evolution of social gaze. Neurosci. Biobehav. Rev. 2000, 24, 581–604. [Google Scholar] [CrossRef]
- Kleinke, C.L. Gaze and eye contact: A research review. Psychol. Bull. 1986, 100, 78. [Google Scholar] [CrossRef] [PubMed]
- Hood, B.M.; Willen, J.D.; Driver, J. Adult’s eyes trigger shifts of visual attention in human infants. Psychol. Sci. 1998, 9, 131–134. [Google Scholar] [CrossRef]
- Pelphrey, K.A.; Sasson, N.J.; Reznick, J.S.; Paul, G.; Goldman, B.D.; Piven, J. Visual scanning of faces in autism. J. Autism Dev. Disord. 2002, 32, 249–261. [Google Scholar] [CrossRef] [PubMed]
- Glaholt, M.G.; Reingold, E.M. Eye movement monitoring as a process tracing methodology in decision making research. Neurosci. J. Psychol. Econ. 2011, 4, 125. [Google Scholar] [CrossRef]
- Peitek, N.; Siegmund, J.; Parnin, C.; Apel, S.; Hofmeister, J.C.; Brechmann, A. Simultaneous measurement of program comprehension with fmri and eye tracking: A case study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Oulu, Finland, 11–12 October 2018; pp. 1–10. [Google Scholar]
- He, Z.; Li, Z.; Yang, F.; Wang, L.; Li, J.; Zhou, C.; Pan, J. Advances in multimodal emotion recognition based on brain–computer interfaces. Brain Sci. 2020, 10, 687. [Google Scholar] [CrossRef]
- Wu, X.; Zheng, W.-L.; Li, Z.; Lu, B.-L. Investigating EEG-based functional connectivity patterns for multimodal emotion recognition. J. Neural Eng. 2022, 19, 16012. [Google Scholar] [CrossRef]
- Lu, Y.; Zheng, W.-L.; Li, B.; Lu, B.-L. Combining eye movements and EEG to enhance emotion recognition. In Proceedings of the 24th International Conference on Artificial Intelligence, Buenos Aires, Argentina, 25–31 July 2015; pp. 1170–1176. [Google Scholar]
- Zheng, W.-L.; Liu, W.; Lu, Y.; Lu, B.-L.; Cichocki, A. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Trans. Cybern. 2018, 49, 1110–1122. [Google Scholar] [CrossRef]
- Soleymani, M.; Pantic, M.; Pun, T. Multimodal emotion recognition in response to videos. IEEE Trans. Affect. Comput. 2011, 3, 211–223. [Google Scholar] [CrossRef] [Green Version]
- Russell, J.A. A circumplex model of affect. Pers. J. Soc. Psychol. 1980, 39, 1161. [Google Scholar] [CrossRef]
- Jamal, W.; Das, S.; Maharatna, K.; Pan, I.; Kuyucu, D. Brain connectivity analysis from EEG signals using stable phase-synchronized states during face perception tasks. Phys. A Stat. Mech. Appl. 2015, 434, 273–295. [Google Scholar] [CrossRef]
- Savine, A.C.; Braver, T.S. Motivated cognitive control: Reward incentives modulate preparatory neural activity during task-switching. Soc Neurosci. 2010, 10294–10305. [Google Scholar] [CrossRef] [PubMed]
- Ashby, F.G.; Isen, A.M. A neuropsychological theory of positive affect and its influence on cognition. Psychol. Rev. 1999, 106, 529. [Google Scholar] [CrossRef]
- Ashby, F.G.; Valentin, V.V. The effects of positive affect and arousal and working memory and executive attention: Neurobiology and computational models. In Emotional Cognition: From Brain to Behaviour; Moore, S.C., Oaksford, M., Eds.; John Benjamins Publishing Company: Amsterdam, The Netherlands, 2002; pp. 245–287. [Google Scholar] [CrossRef]
- Mehdizadehfar, V.; Ghassemi, F.; Fallah, A.; Mohammad-Rezazadeh, I.; Pouretemad, H. Brain connectivity analysis in fathers of children with autism. Cogn. Neurodyn. 2020, 14, 781–793. [Google Scholar] [CrossRef] [PubMed]
- Nejati, V.; Majdi, R.; Salehinejad, M.A.; Nitsche, M.A. The role of dorsolateral and ventromedial prefrontal cortex in the processing of emotional dimensions. Sci. Rep. 2021, 11, 1971. [Google Scholar] [CrossRef] [PubMed]
- Stuss, D.T. New approaches to prefrontal lobe testing. In The Human Frontal Lobes: Functions and Disorders; Miller, B.L., Cummings, J.L., Eds.; The Guilford Press: New York, NY, USA, 2007; pp. 292–305. [Google Scholar]
- Henri-Bhargava, A.; Stuss, D.T.; Freedman, M. Clinical assessment of prefrontal lobe functions. Contin. Lifelong Learn. Neurol. 2018, 24, 704–726. [Google Scholar] [CrossRef]
- Barbey, A.K.; Koenigs, M.; Grafman, J. Dorsolateral prefrontal contributions to human working memory. Cortex 2013, 49, 1195–1205. [Google Scholar] [CrossRef]
- Rahnev, D.; Nee, D.E.; Riddle, J.; Larson, A.S.; D’Esposito, M. Causal evidence for frontal cortex organization for perceptual decision making. Proc. Natl. Acad. Sci. USA 2016, 113, 6059–6064. [Google Scholar] [CrossRef] [Green Version]
- Ghanavati, E.; Salehinejad, M.A.; Nejati, V.; Nitsche, M.A. Differential role of prefrontal, temporal and parietal cortices in verbal and figural fluency: Implications for the supramodal contribution of executive functions. Sci. Rep. 2019, 9, 3700. [Google Scholar] [CrossRef] [PubMed]
- Popov, T.; Steffen, A.; Weisz, N.; Miller, G.A.; Rockstroh, B. Cross-frequency dynamics of neuromagnetic oscillatory activity: Two mechanisms of emotion regulation. Psychophysiology 2012, 49, 1545–1557. [Google Scholar] [CrossRef] [PubMed]
- Joshi, S.; Gold, J.I. Pupil size as a window on neural substrates of cognition. Trends Cogn. Sci. 2020, 24, 466–480. [Google Scholar] [CrossRef]
- Lowenstein, O.; Loewenfeld, I.E. Role of sympathetic and parasympathetic systems in reflex dilatation of the pupil: Pupillographic studies. Arch. Neurol. Psychiatry 1950, 64, 313–340. [Google Scholar] [CrossRef] [PubMed]
- Toates, F.M. Accommodation function of the human eye. Physiol. Rev. 1972, 52, 828–863. [Google Scholar] [CrossRef] [PubMed]
Methods | Strengths | Weaknesses |
---|---|---|
Deep canonical correlation analysis (DCCA) of integrated functional features [22] | Applied machine learning and incorporated and analyzed brain connectivity and eye movement data. | The statistical significance of brain connectivity and eye movement feature variables was not analyzed. |
Designed a six-electrode placement to collect EEG and combined them with eye movements to integrate internal cognitive states and external behaviors [24]. | Demonstrated the effect of modality fusion with a multimodal deep neural network. The mean accuracy was 85.11% for four emotions (happy, sad, fear, and neutral). | The study did not analyze the functional relationship between brainwave connectivity and eye movements. |
User-independent emotion recognition method to identify affective tags for videos using gaze distance, pupillary response, and EEG [25]. | Investigated pupil diameter, gaze distance, eye blinking, and EEG and applied modality fusion strategy at both feature and decision levels. | The experimental session limited the number of videos shown to participants. The study did not investigate brainwave connectivity. |
Recognition of emotion by brain connectivity and eye movement (proposed method). | Explored the characteristics of brainwave connectivity and eye movement eigenvalues and the relationship between the two in a two-dimensional emotional model. | Did not apply machine learning to formulate a model. The analysis was based on one stimulus for each of the four quadrants in the two-dimensional model. |
Emotion Condition 1 | Emotion Condition 2 | Mean Difference | Lower | Upper | Reject |
---|---|---|---|---|---|
Pleasant-aroused | Pleasant-relaxed | −2.2083 | −2.8964 | −1.5202 | True |
Pleasant-aroused | Unpleasant-aroused | 0.9375 | 0.2494 | 1.6256 | True |
Pleasant-aroused | Unpleasant-relaxed | −0.7083 | −1.3964 | −0.0202 | True |
Pleasant-relaxed | Unpleasant-aroused | 3.1458 | 2.4577 | 3.8339 | True |
Pleasant-relaxed | Unpleasant-relaxed | 1.5 | 0.8119 | 2.1881 | True |
Unpleasant-aroused | Unpleasant-relaxed | −1.6458 | −2.3339 | −0.9577 | True |
Emotion Condition 1 | Emotion Condition 2 | Mean Difference | Lower | Upper | Reject |
---|---|---|---|---|---|
Pleasant-aroused | Pleasant-relaxed | −0.125 | −0.6531 | 0.4031 | False |
Pleasant-aroused | Unpleasant-aroused | −3.625 | −4.1531 | −3.0969 | True |
Pleasant-aroused | Unpleasant-relaxed | −3.1042 | −3.6322 | −2.5761 | True |
Pleasant-relaxed | Unpleasant-aroused | −3.5 | −4.0281 | −2.9719 | True |
Pleasant-relaxed | Unpleasant-relaxed | −2.9792 | −3.5072 | −2.4511 | True |
Unpleasant-aroused | Unpleasant-relaxed | −1.6458 | −2.3339 | −0.9577 | True |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhang, J.; Park, S.; Cho, A.; Whang, M. Recognition of Emotion by Brain Connectivity and Eye Movement. Sensors 2022, 22, 6736. https://doi.org/10.3390/s22186736
Zhang J, Park S, Cho A, Whang M. Recognition of Emotion by Brain Connectivity and Eye Movement. Sensors. 2022; 22(18):6736. https://doi.org/10.3390/s22186736
Chicago/Turabian StyleZhang, Jing, Sung Park, Ayoung Cho, and Mincheol Whang. 2022. "Recognition of Emotion by Brain Connectivity and Eye Movement" Sensors 22, no. 18: 6736. https://doi.org/10.3390/s22186736
APA StyleZhang, J., Park, S., Cho, A., & Whang, M. (2022). Recognition of Emotion by Brain Connectivity and Eye Movement. Sensors, 22(18), 6736. https://doi.org/10.3390/s22186736