Personal Resilience Can Be Well Estimated from Heart Rate Variability and Paralinguistic Features during Human–Robot Conversations
Abstract
:1. Introduction
1.1. Background and Motivation
1.2. Theoretical Basis
1.2.1. Physiological Signals
1.2.2. Speech Signals
2. Materials and Methods
2.1. Procedure
2.1.1. Participants
2.1.2. Questionnaires
2.1.3. Signal Recording
2.1.4. Human–Robot Interaction
2.1.5. Memory Retrieval
2.2. Feature Extraction
2.2.1. Physiological Signals
2.2.2. Speech Signals
2.3. Feature Selection
2.4. Model Training and Testing
3. Results
3.1. Personal Resilience
3.2. Correlational Analysis
3.2.1. Big-Five Personality
3.2.2. Physiological Features
3.2.3. Paralinguistic Features
3.2.4. Linguistic Features
3.3. Classification Results
4. Discussion & Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Dembling, B.P.; Chen, D.T.; Vachon, L. Life expectancy and causes of death in a population treated for serious mental illness. Psychiatr. Serv. 1999, 50, 1036–1042. [Google Scholar] [CrossRef]
- Hiroeh, U.; Appleby, L.; Mortensen, P.B.; Dunn, G. Death by homicide, suicide, and other unnatural causes in people with mental illness: A population-based study. Lancet 2001, 358, 2110–2112. [Google Scholar] [CrossRef]
- Hu, T.Q.; Zhang, D.J.; Wang, J.L. A meta-analysis of the trait resilience and mental health. Personal. Individ. Differ. 2015, 76, 18–27. [Google Scholar] [CrossRef]
- Heim, C.; Nemeroff, C.B. The role of childhood trauma in the neurobiology of mood and anxiety disorders: Preclinical and clinical studies. Biol. Psychiatry 2001, 49, 1023–1039. [Google Scholar] [CrossRef] [Green Version]
- Hovens, J.G.; Wiersma, J.E.; Giltay, E.J.; Van Oppen, P.; Spinhoven, P.; Penninx, B.W.; Zitman, F.G. Childhood life events and childhood trauma in adult patients with depressive, anxiety and comorbid disorders vs. controls. Acta Psychiatr. Scand. 2010, 122, 66–74. [Google Scholar] [CrossRef]
- Lauth-Lebens, M.; Lauth, G.W. Risk and resilience factors of post-traumatic stress disorder: A review of current research. Clin. Exp. Psychol. 2016, 2, 1–6. [Google Scholar] [CrossRef] [Green Version]
- Masten, A.S. Ordinary magic: Resilience processes in development. Am. Psychol. 2001, 56, 227. [Google Scholar] [CrossRef]
- Bonanno, G.A. Loss, trauma, and human resilience: Have we underestimated the human capacity to thrive after extremely aversive events? Am. Psychol. 2004, 59, 20. [Google Scholar] [CrossRef] [Green Version]
- Fletcher, D.; Sarkar, M. Psychological resilience. Eur. Psychol. 2013, 18. [Google Scholar] [CrossRef] [Green Version]
- Nishimi, K.; Choi, K.W.; Cerutti, J.; Powers, A.; Bradley, B.; Dunn, E.C. Measures of adult psychological resilience following early-life adversity: How congruent are different measures? Psychol. Med. 2020, 1–10. [Google Scholar] [CrossRef]
- Davydov, D.M.; Stewart, R.; Ritchie, K.; Chaudieu, I. Resilience and mental health. Clin. Psychol. Rev. 2010, 30, 479–495. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Choi, K.W.; Stein, M.B.; Dunn, E.C.; Koenen, K.C.; Smoller, J.W. Genomics and psychological resilience: A research agenda. Mol. Psychiatry 2019, 24, 1770–1778. [Google Scholar] [CrossRef] [PubMed]
- Vinciarelli, A.; Mohammadi, G. A Survey of Personality Computing. IEEE Trans. Affect. Comput. 2014, 5, 273–291. [Google Scholar] [CrossRef] [Green Version]
- Silveira Jacques Junior, J.C.; Gucluturk, Y.; Perez, M.; Guclu, U.; Andujar, C.; Baro, X.; Escalante, H.J.; Guyon, I.; Van Gerven, M.A.J.; Van Lier, R.; et al. First Impressions: A Survey on Vision-based Apparent Personality Trait Analysis. IEEE Trans. Affect. Comput. 2019, 1–20. [Google Scholar] [CrossRef] [Green Version]
- Miranda Correa, J.A.; Abadi, M.K.; Sebe, N.; Patras, I. AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups. IEEE Trans. Affect. Comput. 2021, 12, 1–15. [Google Scholar] [CrossRef] [Green Version]
- Subramanian, R.; Wache, J.; Abadi, M.K.; Vieriu, R.L.; Winkler, S.; Sebe, N. ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors. IEEE Trans. Affect. Comput. 2018, 9, 147–160. [Google Scholar] [CrossRef]
- Scherer, K.R.; Giles, H. Social Markers in Speech; Cambridge University Press: Cambridge, UK, 1979; Volume 6. [Google Scholar]
- Mairesse, F.; Walker, M.A.; Mehl, M.R.; Moore, R.K. Using linguistic cues for the automatic recognition of personality in conversation and text. J. Artif. Intell. Res. 2007, 30, 457–500. [Google Scholar] [CrossRef]
- Pennebaker, J.W.; Mehl, M.R.; Niederhoffer, K.G. Psychological aspects of natural language use: Our words, our selves. Annu. Rev. Psychol. 2003, 54, 547–577. [Google Scholar] [CrossRef] [Green Version]
- Yarkoni, T. Personality in 100,000 Words: A large-scale analysis of personality and word use among bloggers. J. Res. Personal. 2010, 44, 363–373. [Google Scholar] [CrossRef] [Green Version]
- Walker, F.R.; Pfingst, K.; Carnevali, L.; Sgoifo, A.; Nalivaiko, E. In the search for integrative biomarker of resilience to psychological stress. Neurosci. Biobehav. Rev. 2017, 74, 310–320. [Google Scholar] [CrossRef]
- Li, S.J.; Lu, S.; Ni, S.G.; Peng, K.P. Identifying psychological resilience in Chinese migrant youth through multidisciplinary language pattern decoding. Child. Youth Serv. Rev. 2019, 107, 104506. [Google Scholar] [CrossRef]
- Marshall, K.; Abate, A.; Venta, A. Houston Strong: Linguistic markers of resilience after Hurricane Harvey. J. Trauma. Stress Disord. Treat. 2020, 9, e199. [Google Scholar] [CrossRef]
- Zohar, A.H.; Cloninger, C.R.; McCraty, R. Personality and heart rate variability: Exploring pathways from personality to cardiac coherence and health. Open J. Soc. Sci. 2013, 1, 32. [Google Scholar] [CrossRef] [Green Version]
- Agaibi, C.E.; Wilson, J.P. Trauma, PTSD, and resilience: A review of the literature. Trauma Violence Abus. 2005, 6, 195–216. [Google Scholar] [CrossRef]
- Hopper, J.W.; Spinazzola, J.; Simpson, W.B.; van der Kolk, B.A. Preliminary evidence of parasympathetic influence on basal heart rate in posttraumatic stress disorder. J. Psychosom. Res. 2006, 60, 83–90. [Google Scholar] [CrossRef]
- Taylor, C.B. Depression, heart rate related variables and cardiovascular disease. Int. J. Psychophysiol. 2010, 78, 80–88. [Google Scholar] [CrossRef] [PubMed]
- Moreau, C.; Zisook, S. Rationale for a posttraumatic stress spectrum disorder. Psychiat Clin. N. Am. 2002, 25, 775–790. [Google Scholar] [CrossRef]
- Jones, S.E.; LeBaron, C.D. Research on the relationship between verbal and nonverbal communication: Emerging integrations. J. Commun. 2002, 52, 499–521. [Google Scholar] [CrossRef]
- Johar, S. Emotion, Affect and Personality in Speech: The Bias of Language and Paralanguage; Springer: Berlin, Germany, 2015. [Google Scholar]
- Zhang, T.; Kuo, C.C.J. Content-Based Audio Classification and Retrieval for Audiovisual Data Parsing; Kluwer Academic: Boston, MA, USA, 2013; p. 35. [Google Scholar]
- Pennebaker, J.W.; King, L.A. Linguistic styles: Language use as an individual difference. J. Personal. Soc. Psychol. 1999, 77, 1296. [Google Scholar] [CrossRef]
- Chung, C.; Pennebaker, J.W. The psychological functions of function words. Soc. Commun. 2007, 1, 343–359. [Google Scholar]
- Akinnaso, F.N. On the differences between spoken and written language. Lang. Speech 1982, 25, 97–125. [Google Scholar] [CrossRef]
- Smith, B.W.; Dalen, J.; Wiggins, K.; Tooley, E.; Christopher, P.; Bernard, J. The brief resilience scale: Assessing the ability to bounce back. Int J. Behav. Med. 2008, 15, 194–200. [Google Scholar] [CrossRef] [PubMed]
- Connor, K.M.; Davidson, J.R. Development of a new resilience scale: The Connor-Davidson Resilience Scale (CD-RISC). Depress. Anxiety 2003, 18, 76–82. [Google Scholar] [CrossRef]
- Friborg, O.; Barlaug, D.; Martinussen, M.; Rosenvinge, J.H.; Hjemdal, O. Resilience in relation to personality and intelligence. Int. J. Methods Psychiatr. Res. 2005, 14, 29–42. [Google Scholar] [CrossRef]
- Windle, G.; Bennett, K.M.; Noyes, J. A methodological review of resilience measurement scales. Health Qual. Life Outcomes 2011, 9, 8. [Google Scholar] [CrossRef] [Green Version]
- Sutin, A.R.; Robins, R.W. Continuity and correlates of emotions and motives in self-defining memories. J. Personal. 2005, 73, 793–824. [Google Scholar] [CrossRef]
- Makowski, D.; Pham, T.; Lau, Z.J.; Brammer, J.C.; Lespinasse, F.; Schoelzel, C.; Chen, S.A. NeuroKit2: A python toolbox for neurophysiological signal processing. Behav. Res. Methods 2021, 53, 1689–1696. [Google Scholar] [CrossRef]
- Gabrieli, G.; Azhari, A.; Esposito, G. PySiology: A python package for physiological feature extraction. In Neural Approaches to Dynamics of Signal Exchanges; Springer: Berlin, Germany, 2020; pp. 395–402. [Google Scholar]
- Schuller, B.; Steidl, S.; Batliner, A.; Vinciarelli, A.; Scherer, K.; Ringeval, F.; Chetouani, M.; Weninger, F.; Eyben, F.; Marchi, E. The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. In Proceedings of the 14th Annual Conference of the International Speech Communication Association, Lyon, France, 25–29 August 2013. [Google Scholar]
- Eyben, F.; Wöllmer, M.; Schuller, B. Opensmile: The munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy, 25–29 October 2010; pp. 1459–1462. [Google Scholar]
- Eyben, F.; Scherer, K.R.; Schuller, B.W.; Sundberg, J.; André, E.; Busso, C.; Devillers, L.Y.; Epps, J.; Laukka, P.; Narayanan, S.S. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Trans. Affect. Comput. 2015, 7, 190–202. [Google Scholar] [CrossRef] [Green Version]
- Li, P.-H.; Fu, T.-J.; Ma, W.-Y. Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–12 February 2020; pp. 8236–8244. [Google Scholar]
- Kutner, M.H.; Nachtsheim, C.J.; Neter, J.; Li, W. Applied Linear Statistical Models; McGraw-Hill: New York, NY, USA, 2005. [Google Scholar]
- Oshio, A.; Taku, K.; Hirano, M.; Saeed, G. Resilience and Big Five personality traits: A meta-analysis. Personal. Individ. Differ. 2018, 127, 54–60. [Google Scholar] [CrossRef]
- Chida, Y.; Hamer, M. Chronic psychosocial factors and acute physiological responses to laboratory-induced stress in healthy populations: A quantitative review of 30 years of investigations. Psychol. Bull. 2008, 134, 829. [Google Scholar] [CrossRef]
- Lampert, R. ECG signatures of psychological stress. J. Electrocardiol. 2015, 48, 1000–1005. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Norris, C.J.; Larsen, J.T.; Cacioppo, J.T. Neuroticism is associated with larger and more prolonged electrodermal responses to emotionally evocative pictures. Psychophysiology 2007, 44, 823–826. [Google Scholar] [CrossRef]
- Harnmerschmidt, K.; Juergens, U. Acoustical correlates of affective prosody. J. Voice 2007, 21, 531–540. [Google Scholar] [CrossRef] [PubMed]
- Fredrickson, B.L.; Tugade, M.M.; Waugh, C.E.; Larkin, G.R. What good are positive emotions in crises? A prospective study of resilience and emotions following the terrorist attacks on the United States on September 11th, 2001. J. Personal. Soc. Psychol. 2003, 84, 365–376. [Google Scholar] [CrossRef]
- Pennebaker, J.W.; Boyd, R.L.; Jordan, K.; Blackburn, K. The Development and Psychometric Properties of LIWC2015; The University of Texas at Austin: Austin, TX, USA, 2015. [Google Scholar]
- Tausczik, Y.R.; Pennebaker, J.W. The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 2010, 29, 24–54. [Google Scholar] [CrossRef]
- Tugade, M.M.; Fredrickson, B.L. Resilient individuals use positive emotions to bounce back from negative emotional experiences. J. Personal. Soc. Psychol. 2004, 86, 320–333. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Waugh, C.E.; Thompson, R.J.; Gotlib, I.H. Flexible Emotional Responsiveness in Trait Resilience. Emotion 2011, 11, 1059–1067. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Johnson, J.; Wood, A.M.; Gooding, P.; Taylor, P.J.; Tarrier, N. Resilience to suicidality: The buffering hypothesis. Clin. Psychol. Rev. 2011, 31, 563–591. [Google Scholar] [CrossRef]
HRV Features | ECG Features | GSR Features | |
---|---|---|---|
Features | Meaning | Features | Features |
meanNN | The mean of normal-to-normal (NN) intervals | P mean & std R mean & std T mean & std QR mean & std RS mean & std | Rise Time mean & std Amplitude mean & std Apex mean & std Decay Time mean & std Width mean & std |
SDNN | The standard deviation of the NN intervals | ||
RMSSD | The root mean square of the RR intervals | ||
LF | Power in the low-frequency range (0.04–0.15 Hz) | ||
HF | Power in the high-frequency range (0.15–0.40 Hz) | ||
LF/HF | The ratio of LF to HF |
Resilience | PS (M ± SD) | FC (M ± SD) | SR (M ± SD) | SC (M ± SD) | SS (M ± SD) |
---|---|---|---|---|---|
Median/Total | 30/42 | 35.5/49 | 44/56 | 17.5/28 | 20/28 |
High Group | 34.00 ± 3.20 | 39.56 ± 3.12 | 48.44 ± 3.33 | 22.88 ± 2.25 | 23.25 ± 2.86 |
Low Group | 21.40 ± 5.26 | 28.44 ± 4.82 | 36.93 ± 3.91 | 14.06 ± 2.29 | 14.58 ± 2.50 |
Resilience Dimension | Model | Single Modality | Multiple Modalities | |||||
---|---|---|---|---|---|---|---|---|
GSR | ECG | HRV | Acoustic | LIWC | Physiology | Speech | ||
Personal Strength | KNN | 0.61 | 0.70 | 0.75 | 0.69 | 0.60 | 0.65 | 0.65 |
LR | 0.59 | 0.68 | 0.76 | 0.71 | 0.66 | 0.68 | 0.71 | |
SVC | 0.66 | 0.72 | 0.72 | 0.69 | 0.63 | 0.70 | 0.67 | |
RF | 0.66 | 0.72 | 0.72 | 0.67 | 0.63 | 0.70 | 0.64 | |
Family Cohesion | KNN | 0.63 | 0.64 | 0.70 | 0.75 | 0.69 | 0.68 | 0.74 |
LR | 0.42 | 0.68 | 0.69 | 0.71 | 0.68 | 0.57 | 0.69 | |
SVC | 0.60 | 0.63 | 0.69 | 0.72 | 0.70 | 0.55 | 0.68 | |
RF | 0.60 | 0.63 | 0.69 | 0.66 | 0.68 | 0.63 | 0.60 | |
Social Resources | KNN | 0.69 | 0.76 | 0.86 | 0.64 | 0.69 | 0.82 | 0.68 |
LR | 0.60 | 0.71 | 0.82 | 0.65 | 0.64 | 0.78 | 0.53 | |
SVC | 0.73 | 0.73 | 0.84 | 0.67 | 0.65 | 0.79 | 0.63 | |
RF | 0.73 | 0.73 | 0.84 | 0.65 | 0.73 | 0.81 | 0.62 | |
Social Competence | KNN | 0.42 | 0.54 | 0.51 | 0.65 | 0.66 | 0.45 | 0.66 |
LR | 0.49 | 0.46 | 0.48 | 0.58 | 0.67 | 0.49 | 0.66 | |
SVC | 0.46 | 0.50 | 0.49 | 0.60 | 0.70 | 0.47 | 0.63 | |
RF | 0.49 | 0.46 | 0.48 | 0.62 | 0.68 | 0.50 | 0.63 | |
Structured Style | KNN | 0.75 | 0.78 | 0.68 | 0.77 | 0.72 | 0.77 | 0.75 |
LR | 0.76 | 0.80 | 0.81 | 0.79 | 0.73 | 0.74 | 0.76 | |
SVC | 0.72 | 0.74 | 0.69 | 0.81 | 0.72 | 0.77 | 0.79 | |
RF | 0.72 | 0.74 | 0.69 | 0.79 | 0.76 | 0.79 | 0.78 |
Resilience Dimension | Model | Single Modality | Multiple Modalities | |||||
---|---|---|---|---|---|---|---|---|
GSR | ECG | HRV | Acoustic | LIWC | Physiology | Speech | ||
Personal Strength | KNN | 0.57 | 0.63 | 0.77 | 0.62 | 0.64 | 0.48 | 0.64 |
LR | 0.54 | 0.63 | 0.76 | 0.70 | 0.66 | 0.47 | 0.70 | |
SVC | 0.56 | 0.63 | 0.74 | 0.54 | 0.64 | 0.45 | 0.63 | |
RF | 0.56 | 0.63 | 0.74 | 0.64 | 0.66 | 0.44 | 0.67 | |
Family Cohesion | KNN | 0.56 | 0.63 | 0.65 | 0.56 | 0.58 | 0.50 | 0.56 |
LR | 0.43 | 0.68 | 0.68 | 0.68 | 0.55 | 0.43 | 0.66 | |
SVC | 0.52 | 0.63 | 0.67 | 0.59 | 0.54 | 0.38 | 0.56 | |
RF | 0.52 | 0.63 | 0.67 | 0.58 | 0.52 | 0.63 | 0.53 | |
Social Resources | KNN | 0.60 | 0.72 | 0.85 | 0.64 | 0.63 | 0.51 | 0.66 |
LR | 0.53 | 0.66 | 0.80 | 0.67 | 0.65 | 0.47 | 0.68 | |
SVC | 0.58 | 0.73 | 0.84 | 0.62 | 0.68 | 0.45 | 0.50 | |
RF | 0.58 | 0.73 | 0.84 | 0.63 | 0.62 | 0.54 | 0.60 | |
Social Competence | KNN | 0.37 | 0.43 | 0.57 | 0.69 | 0.58 | 0.41 | 0.71 |
LR | 0.48 | 0.46 | 0.53 | 0.69 | 0.57 | 0.30 | 0.71 | |
SVC | 0.37 | 0.43 | 0.55 | 0.69 | 0.58 | 0.28 | 0.65 | |
RF | 0.48 | 0.46 | 0.53 | 0.65 | 0.58 | 0.36 | 0.64 | |
Structured Style | KNN | 0.72 | 0.68 | 0.69 | 0.81 | 0.73 | 0.67 | 0.79 |
LR | 0.72 | 0.79 | 0.82 | 0.86 | 0.74 | 0.58 | 0.86 | |
SVC | 0.74 | 0.69 | 0.69 | 0.81 | 0.72 | 0.69 | 0.75 | |
RF | 0.74 | 0.69 | 0.69 | 0.78 | 0.73 | 0.71 | 0.76 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hsu, S.-M.; Chen, S.-H.; Huang, T.-R. Personal Resilience Can Be Well Estimated from Heart Rate Variability and Paralinguistic Features during Human–Robot Conversations. Sensors 2021, 21, 5844. https://doi.org/10.3390/s21175844
Hsu S-M, Chen S-H, Huang T-R. Personal Resilience Can Be Well Estimated from Heart Rate Variability and Paralinguistic Features during Human–Robot Conversations. Sensors. 2021; 21(17):5844. https://doi.org/10.3390/s21175844
Chicago/Turabian StyleHsu, Shin-Min, Sue-Huei Chen, and Tsung-Ren Huang. 2021. "Personal Resilience Can Be Well Estimated from Heart Rate Variability and Paralinguistic Features during Human–Robot Conversations" Sensors 21, no. 17: 5844. https://doi.org/10.3390/s21175844
APA StyleHsu, S.-M., Chen, S.-H., & Huang, T.-R. (2021). Personal Resilience Can Be Well Estimated from Heart Rate Variability and Paralinguistic Features during Human–Robot Conversations. Sensors, 21(17), 5844. https://doi.org/10.3390/s21175844