Investigation of Methods to Create Future Multimodal Emotional Data for Robot Interactions in Patients with Schizophrenia: A Case Study
Abstract
:1. Introduction
1.1. Aim
1.2. Literature review of Heart Rate Variability, Facial Emotion Detection, and Speech Emotion Recognition
1.3. Research Hypothesis
2. Materials and Methods
2.1. Design
2.2. Procedure for Data Collection
2.3. Data Analysis Methods
2.3.1. HRV
2.3.2. Subjective Facial Expression Analysis by Expert Healthcare Professionals
2.3.3. Analysis by Facial Expression Recognition Algorithm Using HCC
2.3.4. Sound Emotion Analysis
3. Results
4. Discussion
4.1. Differences between Subjective Analysis by Expert Healthcare Professionals and HRV, HCC, and Empath
4.2. Whether HRV, HCC, and Empath API© Are Useful to Create Future Multimodal Emotional Data about Robot–Patient Interactions
4.3. Limitations and Future Scope
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Schoenhofer, S.O.; van Wynsberghe, A.; Boykin, A. Engaging Robots as Nursing Partners in Caring: Nursing as Caring Meets Care-Centered Value-Sensitive Design. Int. J. Hum. Caring 2019, 23, 157–167. [Google Scholar] [CrossRef]
- Leszczyńska, A. Facial Emotion Perception and Schizophrenia Symptoms. Psychiatr. Pol. 2015, 49, 1159–1168. [Google Scholar] [CrossRef] [PubMed]
- Cohen, L.; Khoramshahi, M.; Salesse, R.N.; Bortolon, C.; Słowiński, P.; Zhai, C.; Tsaneva-Atanasova, K.; Di Bernardo, M.; Capdevielle, D.; Marin, L.; et al. Influence of Facial Feedback During a Cooperative Human–Robot Task in Schizophrenia. Sci. Rep. 2017, 7, 15023. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Niznikiewicz, M.A.; Kubicki, M.; Mulert, C.; Condray, R. Schizophrenia as a Disorder of Communication. Schizophr. Res. Treat. 2013, 2013, 952034. [Google Scholar] [CrossRef] [Green Version]
- Couture, S.M.; Penn, D.L.; Roberts, D.L. The Functional Significance of Social Cognition in Schizophrenia: A Review. Schizophr. Bull. 2006, 32, S44–S63. [Google Scholar] [CrossRef] [Green Version]
- Ito, F.; Matsumoto, K.; Miyakoshi, T.; Ohmuro, N.; Uchida, T.; Matsuoka, H. Emotional Processing During Speech Communication and Positive Symptoms in Schizophrenia. Psychiatry Clin. Neurosci. 2013, 67, 526–531. [Google Scholar] [CrossRef]
- Pinkham, A.E.; Penn, D.L.; Perkins, D.O.; Graham, K.A.; Siegel, M. Emotion Perception and Social Skill over the Course of Psychosis: A Comparison of Individuals “At-Risk” for Psychosis and Individuals with Early and Chronic Schizophrenia Spectrum Illness. Cogn. Neuropsychiatry 2007, 12, 198–212. [Google Scholar] [CrossRef]
- Bowie, C.R.; Harvey, P.D. Communication Abnormalities Predict Functional Outcomes in Chronic Schizophrenia: Differential Associations with Social and Adaptive Functions. Schizophr. Res. 2008, 103, 240–247. [Google Scholar] [CrossRef] [Green Version]
- Rus-Calafell, M.; Gutiérres-Maldo, J.; Ribas-Sabaté, J. A virtual reality-integrated program for improving social skills in patients with schizophrenia: A pilot study. J. Behav. Ther. Exp. Psychiatry 2014, 45, 81–89. [Google Scholar] [CrossRef]
- Raffard, S.; Bortolon, C.; Khoramshahi, M.; Salesse, R.N.; Burca, M.; Marin, L.; Bardy, B.G.; Billard, A.; Macioce, V.; Capdevielle, D. Humanoid robots versus humans: How is emotional valence of facial expressions recognized by individuals with schizophrenia? An exploratory study. Schizophr. Res. 2016, 176, 506–513. [Google Scholar] [CrossRef] [Green Version]
- Aubin, L.; Mostafaoui, G.; Amiel, C.; Serré, H.; Capdevielle, D.; de Menibus, M.H.; Boiché, J.; Schmidt, R.; Raffard, S.; Marin, L. Study of Coordination Between Patients with Schizophrenia and Socially Assistive Robot During Physical Activity. Int. J. Soc. Robot. 2021, 13, 1625–1640. [Google Scholar] [CrossRef]
- SoftBank Robotics. Available online: https://www.softbankrobotics.com/emea/en/nao (accessed on 22 March 2022).
- Staff Shortages Leaving Mental Health Nurses ‘Near Breaking Point’, Survey Finds. Available online: https://www.nursinginpractice.com/latest-news/staff-shortages-leaving-mental-health-nurses-near-breaking-point-survey-finds/ (accessed on 1 February 2020).
- Balzarotti, S.; Biassoni, F.; Colombo, B.; Ciceri, M.R. Cardiac vagal control as a marker of emotion regulation in healthy adults: A review. Biol. psychol. 2017, 130, 54–66. [Google Scholar] [CrossRef] [PubMed]
- Miu, A.C.; Heilman, R.M.; Miclea, M. Reduced Heart rate variability and vagal tone in anxiety: Trait versus state, and the effects of autogenic training. Auton. Neurosci. 2009, 145, 93–103. [Google Scholar] [CrossRef] [PubMed]
- Shaffer, F.; Ginsberg, J.P. An Overview of Heart Rate Variability Metrics and Norms. Front. Public Health 2017, 5, 258. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Jauniaux, J.; Tessier, M.H.; Regueiro, S.; Chouchou, F.; Fortin-Côté, A.; Jackson, P.L. Emotion Regulation of Others’ Positive and Negative Emotions Is Related to Distinct Patterns of Heart Rate Variability and Situational Empathy. PLoS ONE 2020, 15, e0244427. [Google Scholar] [CrossRef]
- Fatouros-Bergman, H.; Spang, J.; Merten, J.; Preisler, G.; Werbart, A. Stability of Facial Affective Expressions in Schizophrenia. Schizophr. Res. Treat. 2012, 2012, 867424. [Google Scholar] [CrossRef] [Green Version]
- Lotzin, A.; Haack-Dees, B.; Resch, F.; Romer, G.; Ramsauer, B. Facial Emotional Expression in Schizophrenia Adolescents During Verbal Interaction with a Parent. Eur. Arch. Psychiatry Clin. Neurosci. 2013, 263, 529–536. [Google Scholar] [CrossRef]
- Zhang, K.Z.; Zhang, Z.; Li, Z.; Qiao, Y. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Process. Lett. 2016, 23, 1499–1503. [Google Scholar] [CrossRef] [Green Version]
- Riyantoko, P.A.; Sugiarto; Hindrayani, K.M. Facial Emotion Detection Using Haar-Cascade Classifier and Convolutional Neural Networks. J. Phys. Conf. S. 2021, 1844, 012004. [Google Scholar] [CrossRef]
- Abbaschian, B.J.; Sierra-Sosa, D.; Elmaghraby, A. Deep Learning Techniques for Speech Emotion Recognition, from Databases to Models. Sensors 2021, 21, 1249. [Google Scholar] [CrossRef]
- Xu, M.; Zhang, F.; Cui, X.; Zhang, W. Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation. In Proceedings of the ICASSP 2021–2021 IEEE International Conference on Acoustics [Speech], and Signal Processing (ICASSP), Toronto, ON, Canada, 6–11 June 2021. [Google Scholar] [CrossRef]
- Drakopoulos, G.; Pikramenos, G.; Spyrou, E.; Perantonis, S. Emotion Recognition from Speech: A Survey. In Proceedings of the 15th International Conference on Web Information Systems and Technologies (WEBIST 2019), Vienna, Austria, 18–20 September 2019; pp. 432–439. [Google Scholar] [CrossRef]
- El Ayadi, M.; Kamel, M.S.; Karray, F. Survey on Speech Emotion Recognition: Features, Classification Schemes, and Databases. Pattern Recognit. 2011, 44, 572–587. [Google Scholar] [CrossRef]
- Shah Fahad, M.S.; Ranjan, A.; Yadav, J.; Deepak, A. A Survey of Speech Emotion Recognition in Natural Environment. Digit. Signal Process. 2021, 110, 102951. [Google Scholar] [CrossRef]
- Reshma, C.V.; Rajasree, R. A Survey on Speech Emotion Recognition. In Proceedings of the IEEE International Conference on Innovations in Communication, Computing and Instrumentation (ICCI), Chennai, India, 23 March 2019; Volume 2019, pp. 193–195. [Google Scholar] [CrossRef]
- Empath. Available online: https://www.webempath.com/products-en (accessed on 1 August 2020).
- Ward, C.B.; Choi, Y.; Skiena, S.; Xavier, E.C. Empath: A Framework for Evaluating Entity-Level Sentiment Analysis. In Proceedings of the 8th International Conference & Expo on Emerging Technologies for a Smarter World, Hauppauge, NY, USA, 2–3 November 2011; Volume 2011, pp. 1–6. [Google Scholar] [CrossRef]
- Fast, E.; Chen, B.; Bernstein, M.S. Empath: Understanding Topic Signals in Large-Scale Text. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, 7–12 May 2016. [Google Scholar]
- Tanioka, T.; Locsin, R.C.; Betriana, F.; Kai, Y.; Osaka, K.; Baua, E.; Schoenhofer, S. Intentional Observational Clinical Research Design: Innovative Design for Complex Clinical Research Using Advanced Technology. Int. J. Environ. Res. Public Health 2021, 18, 11184. [Google Scholar] [CrossRef]
- Tanioka, T.; Yokotani, T.; Tanioka, R.; Betriana, F.; Matsumoto, K.; Locsin, R.; Zhao, Y.; Osaka, K.; Miyagawa, M.; Schoenhofer, S. Development Issues of Healthcare Robots: Compassionate Communication for Older Adults with Dementia. Int. J. Environ. Res. Public Health 2021, 18, 4538. [Google Scholar] [CrossRef] [PubMed]
- Robot. Available online: https://www.softbank.jp/en/robot/ (accessed on 1 January 2020).
- Facial Expression Recognition from Image. Available online: https://pypi.org/project/fer/ (accessed on 1 February 2020).
- Jeganathan, J.; Breakspear, M. An Active Inference Perspective on the Negative Symptoms of Schizophrenia. Lancet Psychiatry 2021, 8, 732–738. [Google Scholar] [CrossRef]
- Goldring, A.; Borne, S.; Hefner, A.; Thanju, A.; Khan, A.; Lindenmayer, J.P. The Psychometric Properties of the Self-Evaluation of Negative Symptoms Scale (SNS) in Treatment-Resistant Schizophrenia (TRS). Schizophr. Res. 2020, 224, 159–166. [Google Scholar] [CrossRef] [PubMed]
- Benjamin, B.R.; Valstad, M.; Elvsåshagen, T.; Jönsson, E.G.; Moberget, T.; Winterton, A.; Haram, M.; Høegh, M.C.; Lagerberg, T.V.; Steen, N.E.; et al. Heart Rate Variability Is Associated with Disease Severity in Psychosis Spectrum Disorders. Prog. Neuropsychopharmacol. Biol. Psychiatry 2021, 111, 110108. [Google Scholar] [CrossRef]
- Gao, Z.; Zhao, W.; Liu, S.; Liu, Z.; Yang, C.; Xu, Y. Facial Emotion Recognition in Schizophrenia. Front. Psychiatry 2021, 12, 633717. [Google Scholar] [CrossRef]
- Clamor, A.; Lincoln, T.M.; Thayer, J.F.; Koenig, J. Resting Vagal Activity in Schizophrenia: Meta-Analysis of Heart Rate Variability as a Potential Endophenotype. Br. J. Psychiatry 2016, 208, 9–16. [Google Scholar] [CrossRef]
- Stogios, N.; Gdanski, A.; Gerretsen, P.; Chintoh, A.F.; Graff-Guerrero, A.; Rajji, T.K.; Remington, G.; Hahn, M.K.; Agarwal, S.M. Autonomic Nervous System Dysfunction in Schizophrenia: Impact on Cognitive and Metabolic Health. NPJ Schizophr. 2021, 7, 22. [Google Scholar] [CrossRef]
- Clamor, A.; Sundag, J.; Lincoln, T.M. Specificity of Resting-State Heart Rate Variability in Psychosis: A Comparison with Clinical High Risk, Anxiety, and Healthy Controls. Schizophr. Res. 2019, 206, 89–95. [Google Scholar] [CrossRef] [PubMed]
- Haigh, S.M.; Walford, T.P.; Brosseau, P. Heart Rate Variability in Schizophrenia and Autism. Front. Psychiatry 2021, 12, 760396. [Google Scholar] [CrossRef] [PubMed]
- Mohammadi, A.; Emamgoli, A.; Shirinkalam, M.; Meftahi, G.H.; Yagoobi, K.; Hatef, B. The Persistent Effect of Acute Psychosocial Stress on Heart Rate Variability. Egypt. Heart J. 2019, 71, 18. [Google Scholar] [CrossRef] [Green Version]
- De Geus, E.J.C.; Gianaros, P.J.; Brindle, R.C.; Jennings, J.R.; Berntson, G.G. Should Heart Rate Variability Be “Corrected” for Heart Rate? Biological, Quantitative, and Interpretive Considerations. Psychophysiology 2019, 56, e13287. [Google Scholar] [CrossRef] [Green Version]
- Wu, Y.; Gu, R.; Yang, Q.; Luo, Y.J. How Do Amusement, Anger and Fear Influence Heart Rate and Heart Rate Variability? Front. Neurosci. 2019, 13, 1131. [Google Scholar] [CrossRef]
- Sato, W.; Hyniewska, S.; Minemoto, K.; Yoshikawa, S. Facial Expressions of Basic Emotions in Japanese Laypeople. Front. Psychol. 2019, 10, 259. [Google Scholar] [CrossRef] [Green Version]
- Gong, B.; Li, Q.; Zhao, Y.; Wu, C. Auditory Emotion Recognition Deficits in Schizophrenia: A Systematic Review and Meta-Analysis. Asian J. Psychiatry 2021, 65, 102820. [Google Scholar] [CrossRef] [PubMed]
- Lötsch, J.; Kringel, D.; Ultsch, A. Explainable Artificial Intelligence (XAI) in Biomedicine: Making AI Decisions Trustworthy for Physicians and Patients. BioMedInformatics 2022, 2, 1–17. [Google Scholar] [CrossRef]
- Explaianble AI. Available online: https://cloud.google.com/explainable-ai (accessed on 1 March 2021).
- Datta, A.; Matlock, M.K.; Le Dang, N.; Moulin, T.; Woeltje, K.F.; Yanik, E.L.; Swamidass, S.J. ‘Black Box’ to ‘Conversational’ Machine Learning: Ondansetron Reduces Risk of Hospital-Acquired Venous Thromboembolism. IEEE J. Biomed. Health Inform. 2021, 25, 2204–2214. [Google Scholar] [CrossRef] [PubMed]
- Datta, A.; Flynn, N.R.; Barnette, D.A.; Woeltje, K.F.; Miller, G.P.; Swamidass, S.J. Machine learning liver-injuring drug interactions with non-steroidal anti-inflammatory drugs (NSAIDs) from a retrospective electronic health record (EHR) cohort. PLoS Comput. Biol. 2021, 17, e1009053. [Google Scholar] [CrossRef]
Time | Time | A | B | C | D |
---|---|---|---|---|---|
Elapsed Time | 9:33:06 | 9:35:48 | 9:36:58 | 9:41:12 | |
Heart rate variability | HR-mean | 102.6 | 106.5 | 105.1 | 109.6 |
HF | 11.357 | 4.803 | 4.093 | 3.001 | |
LF | 28.461 | 16.143 | 13.216 | 31.39 | |
HFnu | 28.525 | 22.932 | 23.645 | 8.726 | |
LFnu | 71.475 | 77.068 | 76.355 | 91.724 | |
Subjective facial expressions | Evaluator A | Happiness and smile | Happiness and smiles | Happiness | Contempt |
Evaluator B | Laugh out loud | Smile | Laugh out loud | Tilted his head a little. (Couldn’t he understand?) | |
Evaluator C | Wry smile | It seemed that he was surprised when asked “what shampoo. Wry smile | Wry smile | He tilted his head at Pepper’s surprising answer (Pepper likes winter because it’s white). | |
Facial expression analysis | HCC | Angry 0.09, disgust 0, fear 0.18, happy 0.58, sad 0.08, surprise 0.01, neutral 0.04 | Angry 0.01, disgust 0, fear 0.01, happy 0.94, sad 0, surprise 0.03, neutral 0.01 | Angry 0.01, disgust 0, fear 0.01, happy 0.86, sad 0, surprise 0.06, neutral 0.06 | Angry 0.07, disgust 0, fear 0.185, happy 0, sad 0.67, surprise 0, neutral 0.086 |
Conversation content | Pepper | You are young | |||
Patient | Hahaha | Ummm | Ha, once more | Ummm, white…, ok Ummm, that’s why white…, ok | |
Sound analysis | Empath API© | Calm: 1.00, anger: 0, joy: 0, sorrow: 0, energy: 0 | Calm: 0.44, anger: 0.18, joy: 0.02, sorrow: 0.32, energy: 0 | Calm: 0.84, anger: 0, joy: 0, sorrow: 0.14, energy: 0 | Calm: 0.35, anger: 0, joy: 0.28, sorrow: 0, energy: 0.28 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Osaka, K.; Matsumoto, K.; Akiyama, T.; Tanioka, R.; Betriana, F.; Zhao, Y.; Kai, Y.; Miyagawa, M.; Tanioka, T.; Locsin, R.C. Investigation of Methods to Create Future Multimodal Emotional Data for Robot Interactions in Patients with Schizophrenia: A Case Study. Healthcare 2022, 10, 848. https://doi.org/10.3390/healthcare10050848
Osaka K, Matsumoto K, Akiyama T, Tanioka R, Betriana F, Zhao Y, Kai Y, Miyagawa M, Tanioka T, Locsin RC. Investigation of Methods to Create Future Multimodal Emotional Data for Robot Interactions in Patients with Schizophrenia: A Case Study. Healthcare. 2022; 10(5):848. https://doi.org/10.3390/healthcare10050848
Chicago/Turabian StyleOsaka, Kyoko, Kazuyuki Matsumoto, Toshiya Akiyama, Ryuichi Tanioka, Feni Betriana, Yueren Zhao, Yoshihiro Kai, Misao Miyagawa, Tetsuya Tanioka, and Rozzano C. Locsin. 2022. "Investigation of Methods to Create Future Multimodal Emotional Data for Robot Interactions in Patients with Schizophrenia: A Case Study" Healthcare 10, no. 5: 848. https://doi.org/10.3390/healthcare10050848
APA StyleOsaka, K., Matsumoto, K., Akiyama, T., Tanioka, R., Betriana, F., Zhao, Y., Kai, Y., Miyagawa, M., Tanioka, T., & Locsin, R. C. (2022). Investigation of Methods to Create Future Multimodal Emotional Data for Robot Interactions in Patients with Schizophrenia: A Case Study. Healthcare, 10(5), 848. https://doi.org/10.3390/healthcare10050848