# Emotion Recognition from Physiological Channels Using Graph Neural Network

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- It combines spatial–temporal convolution and spatial–temporal attention mechanisms;
- It represents the pairwise relationship between nodes to dynamically construct an adjacency matrix;
- It is easy to expand the model by other types of biosignals.

- RQ1—What is the accuracy of the valence and arousal predicted by GraphEmotionNet?
- RQ2—What is the correctness of the emotion state vectors for Ekman’s model with the neutral state predicted by GraphEmotionNet for a single moment of time?
- RQ3—In what way do the modalities used in GraphEmotionNet and the emotion recognized influence the answers of RQ2?

- Transformation of the GraphSleepNet GNN to GraphEmotionNet and then further preparation of the GNN to work in two configurations utilizing the unimodal and multimodal approaches.
- Selection of a dataset that provides continuous annotations, as much as possible, and obtaining emotions treated as ground truth continuously while performing the activity.
- GraphEmotionNet validation for the Circumplex model to prove its applicability for emotion recognition. The aim of this task was not to improve the accuracy in relation to the baseline methods, but to show a similar accuracy to other research. Within this task, two substasks were performed:
- Configuration and training of GraphEmotionNet for the Circumplex model for the unimodal approach (EEG only) and multimodal approach (EEG with at least two other biosignals).
- Analysis of the accuracy of the recognized quadrants in the Circumplex model, where the quadrants are understood in terms of high or low values of valence and arousal. This analysis was performed with respect to baseline methods.

- Analysis of the recognized emotion state vectors representing the six basic emotions and the neutral state. The aim of this task was to check the correctness of the recognized emotion state vectors and provide the main contribution of this paper: results showing which basic emotions can be recognized with the highest correctness utilizing DE features in GraphEmotionNet and in what way the used modalities influence this. Within this task, three subtasks were performed:
- Determination of the method for measuring the correctness of the recognized emotions, which was achieved by choosing the similarity measure, allowing for the comparison of the two emotion state vectors.
- Configuration and training of GraphEmotionNet for the Ekman model with a neutral state for the unimodal approach (EEG only) and multimodal approach (EEG with at least two other biosignals).
- Analysis of the chosen similarity measure of the recognized emotion state vectors, which made it possible to draw some conclusions. This analysis was not performed with respect to baseline methods, as to the best of our knowledge no similar analysis has been published.

## 2. Related Work

#### 2.1. Automated Emotion Recognition from Physiological Channels

#### 2.1.1. Models for Representing Emotions

#### 2.1.2. Observation Channels

#### 2.1.3. Techniques and Approaches for Multimodal Processing

#### 2.1.4. Datasets

#### 2.2. Graph Neural Networks for Automatic Emotion Recognition

## 3. Graph Neural Network

#### 3.1. GraphSleepNet Model

#### 3.2. Adaptation of the GraphSleepNet to Recognize Emotions—GraphEmotionNet

## 4. Experiments Design

#### 4.1. Dataset

- Electroencephalogram (EEG) that contained fourteen channels (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4);
- Electrocardiogram (ECG) that contained two channels;
- Galvanic skin response (GSR) with only one channel.

- The self method was obtained from the participants’ self-assessment made by completing special questionnaires about their emotional state at the beginning of each experiment and about their emotional state, video liking, and familiarity at the end of each experiment.
- The external method was based on ratings of the arousal and valence levels made by three independent annotators every 20 s using the participants’ face recordings.

#### 4.2. Preprocessing

Listing 1. Emotions recognized by Face Reader for the first participant watching the 80th movie. | |

Video Time | Emotion |

00:00:00.000 | Unknown |

00:00:01.083 | Neutral |

00:00:03.750 | Unknown |

00:00:07.833 | Neutral |

00:00:09.500 | Disgusted |

00:00:14.416 | Neutral |

00:00:16.958 | Disgusted |

00:00:17.541 | Surprised |

00:00:20.000 | Neutral |

00:00:29.166 | Surprised |

00:00:31.000 | Neutral |

00:00:35.958 | Disgusted |

00:00:36.791 | Surprised |

00:00:41.166 | Neutral |

00:01:31.625 | Happy |

00:01:32.875 | Neutral |

00:01:42.791 | END |

**Figure 3.**Ekman’s emotions on Circumplex model (own elaboration based on [81]).

#### Analysis of the Accuracy of Recognized Quadrants in the Circumplex Model

#### 4.3. Analysis of Recognized Emotion State Vectors Representing Six Basic Emotions and Neutral State

#### 4.3.1. Determining the Method of Measuring Correctness of Recognized Emotions

- A set of about 100 pairs of emotion state vectors is obtained (one vector is recognized by Face Reader and the other one is generated by GraphEmotionNet). The pairs are chosen in such a way that for each dominant emotion (the one with the highest value representing the probability assigned), the number of pairs is the same.
- Five persons, not knowing the value of the cosine similarity, independently annotate each pair using the values 0, 1, and 2 to denote the pair as non-similar, semi-similar, and similar, respectively.
- The class of the pair is determined based on the average value of the annotated values; if the average value is ≥1.5, the pair is found to be consistent, if the average value is in the range <0.5, 1.5), the sample is found to be semi-consistent, and if the average value is <0.5, the sample is found to be inconsistent.
- The tresholds are determined based on the distribution of values of cosine similarity within each class and the minimum value of cosine similarity within these classes.

#### 4.3.2. Analysis of Correctness of Recognized Emotional State Vectors

## 5. Experiments Results

#### 5.1. Analysis of the Accuracy of the Recognized Quadrants in the Circumplex Model

#### 5.2. Determining the Method of Measuring Correctness of Recognized Emotions

- consistent class-cosine similarity ≥ 0.8;
- semi-consistent class-cosine similarity in the range <0.5, 0.8);
- inconsistent class-cosine similarity < 0.5.

#### 5.3. Analysis of Correctness of Recognized Emotional State Vectors

- The cosine similarity for happiness and sadness is lower compared to the other basic emotions—this may be explained by the fact that the recognized facial expressions typical for sadness and happiness may have no reflection in a person’s emotions (e.g., the person can smile but she/he does not have to feel happy); there may be no correspondence between a person’s facial expressions and the emotions reflected in physiological signals;
- The best recognized emotion is anger, which may be a premise to formulate the conclusion that DE features convey much information about this emotion;
- Adding the ECG and GSR modalities in the process of emotion recognition increases the cosine similarity for almost all basic emotions except for fear. This may be explained by the fact that the emotion of fear is better reflected in EEG biosignals than in ECG or GSR; however, this result must be confirmed with additional research;
- Setting the class weights to increase the sensitivity of Ekman’s emotions increases the cosine similarity for happiness and sadness, but their cosine similarity is still the lowest of all of the basic emotions;
- Setting the class weights to increase the sensitivity influenced the percentage of consistent samples for the one modality (EEG) approach most—it increases this percentage significantly for happiness, anger, surprise, and sadness while in the same time reducing it by almost 30% for fear.

## 6. Discussion

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Ackermann, P.; Kohlschein, C.; Bitsch, J.A.; Wehrle, K.; Jeschke, S. EEG-based automatic emotion recognition: Feature extraction, selection and classification methods. In Proceedings of the 2016 IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom), Munich, Germany, 14–16 September 2016; pp. 1–6. [Google Scholar] [CrossRef]
- Zhong, P.; Wang, D.; Miao, C. EEG-Based Emotion Recognition Using Regularized Graph Neural Networks. IEEE Trans. Affect. Comput.
**2020**. [Google Scholar] [CrossRef] - Jia, Z.; Lin, Y.; Wang, J.; Zhou, R.; Ning, X.; He, Y.; Zhao, Y. GraphSleepNet: Adaptive Spatial-Temporal Graph Convolutional Networks for Sleep Stage Classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20), Yokohama, Japan, 11–17 July 2020; pp. 1324–1330. [Google Scholar] [CrossRef]
- Ekman, P. An argument for basic emotions. Cogn. Emot.
**1992**, 6, 169–200. [Google Scholar] [CrossRef] - Schlosberg, H. Three dimensions of emotion. Psychol. Rev.
**1954**, 61, 81–88. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Chen, D.W.; Miao, R.; Yang, W.Q.; Liang, Y.; Chen, H.H.; Huang, L.; Deng, C.J.; Han, N. A Feature Extraction Method Based on Differential Entropy and Linear Discriminant Analysis for Emotion Recognition. Sensors
**2019**, 19, 1631. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zheng, W.L.; Zhu, J.Y.; Lu, B.L. Identifying Stable Patterns over Time for Emotion Recognition from EEG. IEEE Trans. Affect. Comput.
**2019**, 10, 417–429. [Google Scholar] [CrossRef] [Green Version] - Li, D.; Xie, L.; Chai, B.; Wang, Z. A feature-based on potential and differential entropy information for electroencephalogram emotion recognition. Electron. Lett.
**2022**, 58, 174–177. [Google Scholar] [CrossRef] - Li, Y.; Wong, C.M.; Zheng, Y.; Wan, F.; Mak, P.U.; Pun, S.H.; Vai, M.I. EEG-based Emotion Recognition Under Convolutional Neural Network with Differential Entropy Feature Maps. In Proceedings of the 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), Tianjin, China, 14–16 June 2019; pp. 1–5. [Google Scholar] [CrossRef]
- Li, J.; Zhang, Z.; He, H. Hierarchical Convolutional Neural Networks for EEG-Based Emotion Recognition. Cogn. Comput.
**2018**, 10, 368–380. [Google Scholar] [CrossRef] - Hwang, S.; Hong, K.; Son, G.; Byun, H. Learning CNN features from DE features for EEG-based emotion recognition. Pattern Anal. Appl.
**2020**, 23, 1323–1335. [Google Scholar] [CrossRef] - Jarraya, S.K.; Masmoudi, M.; Hammami, M. Compound Emotion Recognition of Autistic Children During Meltdown Crisis Based on Deep Spatio-Temporal Analysis of Facial Geometric Features. IEEE Access
**2020**, 8, 69311–69326. [Google Scholar] [CrossRef] - Moschona, D.S. An Affective Service based on Multi-Modal Emotion Recognition, using EEG enabled Emotion Tracking and Speech Emotion Recognition. In Proceedings of the 2020 IEEE International Conference on Consumer Electronics—Asia (ICCE-Asia), Seoul, Korea, 1–3 November 2020; pp. 1–3. [Google Scholar] [CrossRef]
- Liu, S.; Zheng, W.; Song, T.; Zong, Y. Sparse Graphic Attention LSTM for EEG Emotion Recognition; Springer: Cham, Switzerland, 2019; pp. 690–697. [Google Scholar] [CrossRef]
- Zhu, J.; Wei, Y.; Feng, Y.; Zhao, X.; Gao, Y. Physiological Signals-Based Emotion Recognition via High-Order Correlation Learning. ACM Trans. Multimedia Comput. Commun. Appl.
**2019**, 15, 1–18. [Google Scholar] [CrossRef] [Green Version] - Chang, E.J.; Rahimi, A.; Benini, L.; Wu, A.Y.A. Hyperdimensional Computing-based Multimodality Emotion Recognition with Physiological Signals. In Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), Hsinchu, Taiwan, 18–20 March 2019; pp. 137–141. [Google Scholar]
- Landowska, A.; Zawadzka, T.; Zawadzki, M. Mining Inconsistent Emotion Recognition Results with the Multidimensional Model. IEEE Access
**2022**, 10, 6737–6759. [Google Scholar] [CrossRef] - Scherer, K.R. What are emotions? Furthermore, how can they be measured? Soc. Sci. Inf.
**2005**, 44, 695–729. [Google Scholar] [CrossRef] - Steunebrink, B.R.; Dastani, M.; Meyer, J.J.C. The OCC Model Revisited. 2009. Available online: https://www.researchgate.net/publication/228952254_The_OCC_model_revisited (accessed on 5 April 2022).
- Yannakakis, G.N.; Cowie, R.; Busso, C. The Ordinal Nature of Emotions: An Emerging Approach. IEEE Trans. Affect. Comput.
**2021**, 12, 16–35. [Google Scholar] [CrossRef] [Green Version] - Scherer, K.R. Towards a Prediction and Data Driven Computational Process Model of Emotion. IEEE Trans. Affect. Comput.
**2021**, 12, 279–292. [Google Scholar] [CrossRef] - Veeranki, Y.R.; Kumar, H.; Ganapathy, N.; Natarajan, B.; Swaminathan, R. A Systematic Review of Sensing and Differentiating Dichotomous Emotional States Using Audio-Visual Stimuli. IEEE Access
**2021**, 9, 124434–124451. [Google Scholar] [CrossRef] - Zawadzka, T.; Wierciński, T.; Meller, G.; Rock, M.; Zwierzycki, R.; Wróbel, M.R. Graph Representation Integrating Signals for Emotion Recognition and Analysis. Sensors
**2021**, 21, 4035. [Google Scholar] [CrossRef] - Islam, M.R.; Moni, M.A.; Islam, M.M.; Rashed-Al-Mahfuz, M.; Islam, M.S.; Hasan, M.K.; Hossain, M.S.; Ahmad, M.; Uddin, S.; Azad, A.; et al. Emotion Recognition From EEG Signal Focusing on Deep Learning and Shallow Learning Techniques. IEEE Access
**2021**, 9, 94601–94624. [Google Scholar] [CrossRef] - Hasnul, M.A.; Aziz, N.A.A.; Alelyani, S.; Mohana, M.; Aziz, A.A. Electrocardiogram-Based Emotion Recognition Systems and Their Applications in Healthcare—A Review. Sensors
**2021**, 21, 5015. [Google Scholar] [CrossRef] - Bin Heyat, M.B.; Akhtar, F.; Khan, A.; Noor, A.; Benjdira, B.; Qamar, Y.; Abbas, S.J.; Lai, D. A Novel Hybrid Machine Learning Classification for the Detection of Bruxism Patients Using Physiological Signals. Appl. Sci.
**2020**, 10, 7410. [Google Scholar] [CrossRef] - Khan, H.; Heyat, M.; Lai, D.; Akhtar, F.; Ansari, M.; If, A.; Alkahtani, F. Progress in Detection of Insomnia Sleep Disorder: A Comprehensive Review. Curr. Drug Targets
**2020**, 21, 672–684. [Google Scholar] [CrossRef] - Suhaimi, N.; Mountstephens, J.; Teo, J. EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities. Comput. Intell. Neurosci.
**2020**, 2020, 8875426. [Google Scholar] [CrossRef] - Santhiya, P.; Chitrakala, S. A Survey on Emotion Recognition from EEG Signals: Approaches, Techniques amp; Challenges. In Proceedings of the 2019 International Conference on Vision Towards Emerging Trends in Communication and Networking (ViTECoN), Vellore, India, 30–31 March 2019; pp. 1–6. [Google Scholar] [CrossRef]
- Joy, E.; Joseph, R.B.; Lakshmi, M.; Joseph, W.; Rajeswari, M. Recent Survey on Emotion Recognition Using Physiological Signals. In Proceedings of the 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, 19–20 March 2021; Volume 1, pp. 1858–1863. [Google Scholar] [CrossRef]
- Nikolova, D.; Petkova, P.; Manolova, A.; Georgieva, P. ECG-based Emotion Recognition: Overview of Methods and Applications. In Proceedings of the ANNA ’18—Advances in Neural Networks and Applications, St. Konstantin and Elena Resort, Bulgaria, 15–17 September 2018; pp. 1–5. [Google Scholar]
- Dawson, M.E.; Schell, A.M.; Filion, D.L. The Electrodermal System. In Handbook of Psychophysiology, 4th ed.; Cacioppo, J.T., Tassinary, L.G., Berntson, G.G., Eds.; Cambridge Handbooks in Psychology; Cambridge University Press: Cambridge, UK, 2016; pp. 217–243. [Google Scholar] [CrossRef]
- Thammasan, N.; Stuldreher, I.V.; Schreuders, E.; Giletta, M.; Brouwer, A.M. A Usability Study of Physiological Measurement in School Using Wearable Sensors. Sensors
**2020**, 20, 5380. [Google Scholar] [CrossRef] [PubMed] - Posada-Quintero, H.F.; Chon, K.H. Innovations in Electrodermal Activity Data Collection and Signal Processing: A Systematic Review. Sensors
**2020**, 20, 479. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Horvers, A.; Tombeng, N.; Bosse, T.; Lazonder, A.W.; Molenaar, I. Detecting Emotions through Electrodermal Activity in Learning Contexts: A Systematic Review. Sensors
**2021**, 21, 7869. [Google Scholar] [CrossRef] [PubMed] - Shukla, J.; Barreda-Ángeles, M.; Oliver, J.; Nandi, G.C.; Puig, D. Feature Extraction and Selection for Emotion Recognition from Electrodermal Activity. IEEE Trans. Affect. Comput.
**2021**, 12, 857–869. [Google Scholar] [CrossRef] - Seo, J.; Laine, T.H.; Sohn, K.A. An Exploration of Machine Learning Methods for Robust Boredom Classification Using EEG and GSR Data. Sensors
**2019**, 19, 4561. [Google Scholar] [CrossRef] [Green Version] - Gunes, H.; Piccardi, M. Affect recognition from face and body: Early fusion vs. late fusion. In Proceedings of the 2005 IEEE International Conference on Systems, Man and Cybernetics, Waikoloa, HI, USA, 12 October 2005; Volume 4, pp. 3437–3443. [Google Scholar] [CrossRef] [Green Version]
- Hupont, I.; Ballano, S.; Baldassarri, S.; Cerezo, E. Scalable multimodal fusion for continuous affect sensing. In Proceedings of the 2011 IEEE Workshop on Affective Computational Intelligence (WACI), Paris, France, 11–15 April 2011; pp. 1–8. [Google Scholar] [CrossRef]
- Shoumy, N.J.; Ang, L.M.; Seng, K.P.; Rahaman, D.; Zia, T. Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals. J. Netw. Comput. Appl.
**2020**, 149, 102447. [Google Scholar] [CrossRef] - Zheng, W. Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis. IEEE Trans. Cogn. Dev. Syst.
**2017**, 9, 281–290. [Google Scholar] [CrossRef] - Li, Y.; Zheng, W.; Cui, Z.; Zong, Y.; Ge, S. EEG Emotion Recognition Based on Graph Regularized Sparse Linear Regression. Neural Process. Lett.
**2019**, 49, 555–571. [Google Scholar] [CrossRef] - Zhou, J.; Cui, G.; Hu, S.; Zhang, Z.; Yang, C.; Liu, Z.; Wang, L.; Li, C.; Sun, M. Graph Neural Networks: A Review of Methods and Applications. arXiv
**2021**, arXiv:1812.08434. [Google Scholar] - Ngoc, Q.T.; Lee, S.; Song, B.C. Facial Landmark-Based Emotion Recognition via Directed Graph Neural Network. Electronics
**2020**, 9, 764. [Google Scholar] [CrossRef] - Ghosal, D.; Majumder, N.; Poria, S.; Chhaya, N.; Gelbukh, A.F. DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation. arXiv
**2019**, arXiv:1908.11540. [Google Scholar] - Shen, W.; Wu, S.; Yang, Y.; Quan, X. Directed Acyclic Graph Network for Conversational Emotion Recognition. arXiv
**2021**, arXiv:2105.12907. [Google Scholar] - Shirian, A.; Guha, T. Compact Graph Architecture for Speech Emotion Recognition. In Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 6–11 June 2021; pp. 6284–6288. [Google Scholar] [CrossRef]
- Tsai, M.F.; Chen, C.H. Spatial Temporal Variation Graph Convolutional Networks (STV-GCN) for Skeleton-Based Emotional Action Recognition. IEEE Access
**2021**, 9, 13870–13877. [Google Scholar] [CrossRef] - Cheng, H.; Tie, Y.; Qi, L.; Jin, C. Context-Aware Based Visual-Audio Feature Fusion for Emotion Recognition. In Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, 18–22 July 2021; pp. 1–8. [Google Scholar] [CrossRef]
- Nie, W.; Ren, M.; Nie, J.; Zhao, S. C-GCN: Correlation Based Graph Convolutional Network for Audio-Video Emotion Recognition. IEEE Trans. Multimed.
**2021**, 23, 3793–3804. [Google Scholar] [CrossRef] - Chakrabarti, D. Graph Mining; Springer: Berlin/Heidelberg, Germany, 2010; pp. 469–471. [Google Scholar] [CrossRef]
- Peng, H.; Li, J.; He, Y.; Liu, Y.; Bao, M.; Wang, L.; Song, Y.; Yang, Q. Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN. In Proceedings of the WWW’18: The Web Conference 2018, Lyon, France, 23–27 April 2018; pp. 1063–1072. [Google Scholar] [CrossRef] [Green Version]
- Marino, K.; Salakhutdinov, R.; Gupta, A. The More You Know: Using Knowledge Graphs for Image Classification. arXiv
**2017**, arXiv:1612.04844. [Google Scholar] - Wu, Z.; Pan, S.; Chen, F.; Long, G.; Zhang, C.; Yu, P.S. A Comprehensive Survey on Graph Neural Networks. IEEE Trans. Neural Netw. Learn. Syst.
**2021**, 32, 4–24. [Google Scholar] [CrossRef] [Green Version] - Chen, H.; Deng, Y.; Cheng, S.; Wang, Y.; Jiang, D.; Sahli, H. Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition. In Proceedings of the MM’19—The 27th ACM International Conference on Multimedia, Nice, France, 21 October 2019; pp. 19–26. [Google Scholar] [CrossRef] [Green Version]
- Bhattacharya, U.; Mittal, T.; Chandra, R.; Randhavane, T.; Bera, A.; Manocha, D. STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–12 February 2020; Volume 34, pp. 1342–1350. [Google Scholar] [CrossRef]
- Shi, J.; Liu, C.; Ishi, C.T.; Ishiguro, H. Skeleton-Based Emotion Recognition Based on Two-Stream Self-Attention Enhanced Spatial-Temporal Graph Convolutional Network. Sensors
**2021**, 21, 205. [Google Scholar] [CrossRef] - Kalaganis, F.P.; Laskaris, N.A.; Chatzilari, E.; Nikolopoulos, S.; Kompatsiaris, I. A Data Augmentation Scheme for Geometric Deep Learning in Personalized Brain—Computer Interfaces. IEEE Access
**2020**, 8, 162218–162229. [Google Scholar] [CrossRef] - Chamanzar, A.; Liu, X.; Jiang, L.Y.; Vogt, K.A.; Moura, J.M.F.; Grover, P. Automated, Scalable and Generalizable Deep Learning for Tracking Cortical Spreading Depression Using EEG. In Proceedings of the 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), Virtual Event, 4–6 May 2021; pp. 416–419. [Google Scholar] [CrossRef]
- Tang, M.; Lu, Y.; Yang, L. Temporal–Spatial Patterns in Dynamic Functional Brain Network for Self-Paced Hand Movement. IEEE Trans. Neural Syst. Rehabil. Eng.
**2019**, 27, 643–651. [Google Scholar] [CrossRef] - Zhu, H.; Han, G.; Shu, L.; Zhao, H. ArvaNet: Deep Recurrent Architecture for PPG-Based Negative Mental-State Monitoring. IEEE Trans. Comput. Soc. Syst.
**2021**, 8, 179–190. [Google Scholar] [CrossRef] - Wang, Z.; Tong, Y.; Heng, X. Phase-Locking Value Based Graph Convolutional Neural Networks for Emotion Recognition. IEEE Access
**2019**, 7, 93711–93722. [Google Scholar] [CrossRef] - Wang, H.; Xu, L.; Bezerianos, A.; Chen, C.; Zhang, Z. Linking Attention-Based Multiscale CNN with Dynamical GCN for Driving Fatigue Detection. IEEE Trans. Instrum. Meas.
**2021**, 70, 1–11. [Google Scholar] [CrossRef] - Liu, D.; Zhang, H.; Zhou, P. Video-based Facial Expression Recognition using Graph Convolutional Networks. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021; pp. 607–614. [Google Scholar] [CrossRef]
- Liu, X.; Liu, W.; Zhang, M.; Chen, J.; Gao, L.; Yan, C.; Mei, T. Social Relation Recognition From Videos via Multi-Scale Spatial-Temporal Reasoning. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 3561–3569. [Google Scholar] [CrossRef]
- You, Z.; Zeng, R.; Lan, X.; Ren, H.; You, Z.; Shi, X.; Zhao, S.; Guo, Y.; Jiang, X.; Hu, X. Alzheimer’s Disease Classification with a Cascade Neural Network. Front. Public Health
**2020**, 8, 665. [Google Scholar] [CrossRef] - Mansouri-Benssassi, E.; Ye, J. Synch-Graph: Multisensory Emotion Recognition Through Neural Synchrony via Graph Convolutional Networks. Proc. Aaai Conf. Artif. Intell.
**2020**, 34, 1351–1358. [Google Scholar] [CrossRef] - Hyvärinen, A. New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit. In Advances in Neural Information Processing Systems; Jordan, M., Kearns, M., Solla, S., Eds.; MIT Press: Cambridge, MA, USA, 1998; Volume 10. [Google Scholar]
- Wang, X.H.; Zhang, T.; Xu, X.M.; Chen, L.; Xing, X.F.; Chen, C.L.P. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks and Broad Learning System. In Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Madrid, Spain, 3–6 December 2018; pp. 1240–1244. [Google Scholar] [CrossRef]
- Correa, J.A.M.; Abadi, M.K.; Sebe, N.; Patras, I. AMIGOS: A dataset for Mood, personality and affect research on Individuals and GrOupS. arXiv
**2017**, arXiv:1702.02510. [Google Scholar] - Zawadzka, T.; Waloszek, W.; Karpus, A.; Zapałowska, S.; Wróbel, M.R. Ontological Model for Contextual Data Defining Time Series for Emotion Recognition and Analysis. IEEE Access
**2021**, 9, 166674–166694. [Google Scholar] [CrossRef] - Hao, Q.; Hu, F. A compressive eletroencephalography (EEG) sensor design. In Proceedings of the SENSORS, 2010 IEEE, Waikoloa, HI, USA, 1–4 November 2010; pp. 318–322. [Google Scholar] [CrossRef]
- Sun, M.; Sclabassi, R. Optimal selection of the sampling rate for efficient EEG data acquisition. In Proceedings of the First Joint BMES/EMBS Conference, Atlanta, GA, USA, 13–16 October 1999; Volume 2, p. 891. [Google Scholar] [CrossRef]
- Zhang, T.; El Ali, A.; Wang, C.; Hanjalic, A.; Cesar, P. RCEA: Real-time, Continuous Emotion Annotation for Collecting Precise Mobile Video Ground Truth Labels. In Proceedings of the CHI’20—The 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020. [Google Scholar] [CrossRef]
- Nagel, F.; Kopiez, R.; Grewe, O.; Altenmüller, E. EMuJoy: Software for continuous measurement of perceived emotions in music. Behav. Res. Methods
**2007**, 39, 283–290. [Google Scholar] [CrossRef] - Soleymani, M.; Asghari-Esfeden, S.; Fu, Y.; Pantic, M. Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection. IEEE Trans. Affect. Comput.
**2016**, 7, 17–28. [Google Scholar] [CrossRef] - Comas, J.; Aspandi, D.; Binefa, X. End-to-end facial and physiological model for Affective Computing and applications. In Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina, 16–20 November 2020. [Google Scholar]
- Yang, H.C.; Lee, C.C. An Attribute-invariant Variational Learning for Emotion Recognition Using Physiology. In Proceedings of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 1184–1188. [Google Scholar] [CrossRef]
- Santamaria-Granados, L.; Munoz-Organero, M.; Ramirez-González, G.; Abdulhay, E.; Arunkumar, N. Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS). IEEE Access
**2019**, 7, 57–67. [Google Scholar] [CrossRef] - Gjoreski, M.; Lustrek, M.; Gams, M.; Mitrevski, B. An inter-domain study for arousal recognition from physiological signals. Informatica
**2018**, 42, 61–68. [Google Scholar] - Plutchik, R. The Emotions; University Press of America: New York, NY, USA, 1991. [Google Scholar]

**Figure 2.**Overall architecture of GraphEmotionNet (own elaboration based on GraphSleepNet [3]).

Article | Recognized Emotion | Predicted Label | Dataset |
---|---|---|---|

[41] | positive, neutral, negative | single label | SEED |

[14] | Ekman model (6 basic emotions) | single label | DEED, MPED |

[42] | positive, neutral, negative | single label | SEED, RCLS |

[15] | valence and arousal | multiple labels | DEAP, ASCERTAIN |

[16] | valence and arousal | multiple labels | AMIGOS |

Name | Type of Signal | Number of Participants | Type of Videos Watched |
---|---|---|---|

AMIGOS | EEG (14 channels), ECG (2 channels), GSR (1 channel) | 40 | 16 movie clips with lengths between 50 and 150 s |

DEAP | EEG (32 channels), ECG (2 channels), GSR, EOG | 32 | 40 one-minute long music videos |

ASCERTAIN | EEG, ECG, GSR, facial features | 58 | 36 movie clips with lengths between 51 and 127 s |

DREAMER | EEG (14 channels), ECG (2 channels) | 23 | 18 movie clips with lengths between 65 and 393 s |

SEED | EEG (62 channels) | 15 | 15 movie clips (duration of every video approx. 4 min) |

**Table 3.**Hyperparameters in the model that predicts emotions to Ekman’s model with additional neutral emotion.

Hyperparameter Description | Value |
---|---|

Layer number of ST-GCN | 1 |

Standard convolution kernels | 10 |

Graph convolution kernels | 10 |

Chebyshev polynomial K | 3 |

Regularization parameter | 0.001 |

Dropout probability | 0.4 |

Batch size | 2048 |

Learning rate | 0.001 |

Optimizer | Adam |

Hyperparameter Description | Value |
---|---|

Layer number of ST-GCN | 1 |

Standard convolution kernels | 10 |

Graph convolution kernels | 10 |

Chebyshev polynomial K | 3 |

Regularization parameter | 0.001 |

Dropout probability | 0.5 |

Batch size | 512 |

Learning rate | 0.001 |

Optimizer | Adam |

**Table 5.**Matrix row representing emotions detected by Face Reader for the first participant watching the 80th video.

Part. | Movie | Neutral | Disgust | Happiness | Surprise | Anger | Fear | Sadness |
---|---|---|---|---|---|---|---|---|

... | ... | ... | ... | ... | ... | ... | ... | ... |

1 | 80 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |

... | ... | ... | ... | ... | ... | ... | ... | ... |

**Table 6.**Matrix row representing annotated emotions for the first participant watching the 80th video.

Part. | Movie | Neutral | Disgust | Happiness | Surprise | Anger | Fear | Sadness |
---|---|---|---|---|---|---|---|---|

... | ... | ... | ... | ... | ... | ... | ... | ... |

1 | 80 | 0 | 0 | 0 | 0 | 1 | 0 | 0 |

... | ... | ... | ... | ... | ... | ... | ... | ... |

Participant | 1 |

Movie | 19 |

Annotated Valence | 1.00 |

Annotated Arousal | 6.46 |

Quadrant | LVHA |

Possible Ekman emotions | Disgust, Anger, Fear |

Annotated Ekman emotions | Disgust, Fear |

Consistent | Yes |

**Table 8.**Experiments specification for analysis of accuracy of recognized quadrant in Circumplex model.

Exp. ID | Emotion Model | Channels | Data | Network Parameters |
---|---|---|---|---|

Exp. 1 | Circumplex model | EEG | Data annotated with valence and arousal | Table 4 |

Exp. 2 | EEG, ECG, GSR |

Exp. ID | Emotion Model | Channels | Data | Network Parameters |
---|---|---|---|---|

Exp. 3 | Ekman model with neutral emotion | EEG | Data annotated with emotions obtained from facial expression analysis | Table 3 |

Exp. 4 | EEG, ECG, GSR | |||

Exp. 5 | Ekman model with neutral emotion | EEG | Data annotated with emotions obtained from facial expression analysis | Table 3 and also class weights that increase sensitivity of Ekman’s emotions |

Exp. 6 | EEG, ECG, GSR |

Method | Valence Acc | Arousal Acc | |
---|---|---|---|

[77] | AE | 65.05% | 87.53% |

[78] | AI-VAE | 68.80% | 67.00% |

[79] | DCNN | 76.00% | 75.00% |

[80] | AdaB | - | 56.00% |

Our method (unimodal) | GNN | 67.20% | 75.88% |

Our method (multimoadl) | GNN | 69.71% | 70.75% |

Average Cosine Similarity | Percentage of Consistent Samples | Percentage of Semi-Consistent Samples | Percentage of Non-Conistent Samples | |||||
---|---|---|---|---|---|---|---|---|

EEG | EEG, ECG, GSR | EEG | EEG, ECG, GSR | EEG | EEG, ECG, GSR | EEG | EEG, ECG, GSR | |

Happiness | 0.4570 | 0.5499 | 02.13% | 16.32% | 45.77% | 44.15% | 52.09% | 39.53% |

Anger | 0.6663 | 0.8246 | 19.49% | 63.34% | 64.31% | 32.57% | 16.21% | 04.09% |

Disgust | 0.6950 | 0.7755 | 21.87% | 50.83% | 68.09% | 42.41% | 10.04% | 06.77% |

Surprise | 0.6625 | 0.7893 | 05.94% | 53.00% | 83.33% | 42.72% | 10.72% | 04.28% |

Sadness | 0.5251 | 0.6369 | 00.82% | 16.49% | 61.58% | 62.70% | 37.60% | 20.81% |

Fear | 0.7527 | 0.7676 | 34.26% | 38.68% | 60.19% | 59.43% | 05.56% | 01.89% |

Neutral | 0.9635 | 0.9707 | 98.07% | 98.59% | 01.92% | 01.37% | 00.01% | 00.04% |

All | 0.9270 | 0.9457 | 89.29% | 93.00% | 08.12% | 05.41% | 02.59% | 01.59% |

Average Cosine Similarity | Percentage of Consistent Samples | Percentage of Semi-Consistent Samples | Percentage of Non-Conistent Samples | |||||
---|---|---|---|---|---|---|---|---|

EEG | EEG, ECG, GSR | EEG | EEG, ECG, GSR | EEG | EEG, ECG, GSR | EEG | EEG, ECG, GSR | |

Happiness | 0.6076 | 0.6084 | 22.71% | 21.66% | 43.54% | 47.34% | 33.75% | 31.00% |

Anger | 0.8069 | 0.8357 | 59.07% | 66.69% | 33.78% | 26.72% | 07.15% | 06.59% |

Disgust | 0.6824 | 0.7255 | 19.68% | 34.70% | 68.96% | 55.15% | 11.36% | 10.16% |

Surprise | 0.7211 | 0.7207 | 32.67% | 33.19% | 57.95% | 60.31% | 09.38% | 06.50% |

Sadness | 0.6147 | 0.6178 | 17.77% | 18.01% | 56.23% | 58.60% | 25.99% | 23.39% |

Fear | 0.6158 | 0.5743 | 06.48% | 08.33% | 75.00% | 63.89% | 18.52% | 27.78% |

Neutral | 0.9088 | 0.9091 | 90.52% | 89.19% | 08.68% | 10.09% | 00.80% | 00.72% |

All | 0.8868 | 0.8880 | 84.53% | 83.72% | 13.00% | 13.99% | 02.47% | 02.29% |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Wierciński, T.; Rock, M.; Zwierzycki, R.; Zawadzka, T.; Zawadzki, M.
Emotion Recognition from Physiological Channels Using Graph Neural Network. *Sensors* **2022**, *22*, 2980.
https://doi.org/10.3390/s22082980

**AMA Style**

Wierciński T, Rock M, Zwierzycki R, Zawadzka T, Zawadzki M.
Emotion Recognition from Physiological Channels Using Graph Neural Network. *Sensors*. 2022; 22(8):2980.
https://doi.org/10.3390/s22082980

**Chicago/Turabian Style**

Wierciński, Tomasz, Mateusz Rock, Robert Zwierzycki, Teresa Zawadzka, and Michał Zawadzki.
2022. "Emotion Recognition from Physiological Channels Using Graph Neural Network" *Sensors* 22, no. 8: 2980.
https://doi.org/10.3390/s22082980