Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing
Abstract
:1. Introduction
1.1. Virtual Reality Set-Ups
1.2. Sense of Presence
1.3. Virtual Reality in Human Behaviour Research
1.4. The Validity of Virtual Reality
1.5. Implicit Measures and the Neuroscience Approach
1.6. Affective Computing and Emotion Recognition Systems
2. Materials and Methods
Data Collection
3. Results
3.1. Summary of Previous Research
3.2. Evolution of the Research
3.3. Emotions Analysed
3.4. Implicit Technique, Features used and Participants
3.5. Data Analysis
3.6. VR Set-Ups Used: HMDs and Formats
3.7. Validation of VR
4. Discussion
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Picard, R.W. Affective Computing; MIT Press: Cambridge, MA, USA, 1997. [Google Scholar]
- Cipresso, P.; Chicchi, I.A.; Alcañiz, M.; Riva, G. The Past, Present, and Future of Virtual and Augmented Reality Research: A network and cluster analysis of the literature. Front. Psychol. 2018, 9, 2086. [Google Scholar] [CrossRef] [Green Version]
- Castelvecchi, D. Low-cost headsets boost virtual reality’s lab appeal. Nature 2016, 533, 153–154. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Slater, M.; Usoh, M. Body centred interaction in immersive virtual environments. Artif. Life Virtual Real. 1994, 1, 125–148. [Google Scholar]
- Giglioli, I.A.C.; Pravettoni, G.; Martín, D.L.S.; Parra, E.; Alcañiz, M. A novel integrating virtual reality approach for the assessment of the attachment behavioral system. Front. Psychol. 2017, 8, 1–7. [Google Scholar] [CrossRef] [PubMed]
- Kwartler, M. Visualization in support of public participation. In Visualization in Landscape and Environmental Planning: Technology and Applications; Bishop, I., Lange, E., Eds.; Taylor & Francis: London, UK, 2005; pp. 251–260. [Google Scholar]
- Vince, J. Introduction to Virtual Reality; Media, Springer: Berlin/Heidelberg, Germany, 2004. [Google Scholar]
- Alcañiz, M.; Baños, R.; Botella, C.; Rey, B. The EMMA Project: Emotions as a Determinant of Presence. PsychNology J. 2003, 1, 141–150. [Google Scholar]
- Mengoni, M.; Germani, M.; Peruzzini, M. Benchmarking of virtual reality performance in mechanics education. Int. J. Interact. Des. Manuf. 2011, 5, 103–117. [Google Scholar] [CrossRef]
- Stamps, A.E., III. Use of photographs to simulate environments: A meta-analysis. Percept. Mot. Ski. 1990, 71, 907–913. [Google Scholar] [CrossRef]
- Morinaga, A.; Hara, K.; Inoue, K.; Urahama, K. Classification between natural and graphics images based on generalized Gaussian distributions. Inf. Process. Lett. 2018, 138, 31–34. [Google Scholar] [CrossRef]
- Siriaraya, P.; Ang, C.S. The Social Interaction Experiences of Older People in a 3D Virtual Environment. In Perspectives on Human-Computer Interaction Research with Older People; Sayago, S., Ed.; Springer: Cham, Switzerland, 2019; pp. 101–117. ISBN 978-3-030-06076-3. [Google Scholar]
- Slater, M. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos. Trans. R. Soc. B Biol. Sci. 2009, 364, 3549–3557. [Google Scholar] [CrossRef] [Green Version]
- Kober, S.E.; Kurzmann, J.; Neuper, C. Cortical correlate of spatial presence in 2D and 3D interactive virtual reality: An EEG study. Int. J. Psychophysiol. 2012, 83, 365–374. [Google Scholar] [CrossRef]
- Borrego, A.; Latorre, J.; Llorens, R.; Alcañiz, M.; Noé, E. Feasibility of a walking virtual reality system for rehabilitation: Objective and subjective parameters. J. Neuroeng. Rehabil. 2016, 13, 68. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Clemente, M.; Rodriguez, A.; Rey, B.; Alcañiz, M. Assessment of the influence of navigation control and screen size on the sense of presence in virtual reality using EEG. Expert Syst. Appl. 2014, 41, 1584–1592. [Google Scholar] [CrossRef]
- Borrego, A.; Latorre, J.; Alcañiz, M.; Llorens, R. Comparison of Oculus Rift and HTC Vive: Feasibility for Virtual Reality-Based Exploration, Navigation, Exergaming, and Rehabilitation. Games Health J. 2018, 7. [Google Scholar] [CrossRef] [PubMed]
- Vecchiato, G.; Jelic, A.; Tieri, G.; Maglione, A.G.; De Matteis, F.; Babiloni, F. Neurophysiological correlates of embodiment and motivational factors during the perception of virtual architectural environments. Cogn. Process. 2015, 16, 425–429. [Google Scholar] [CrossRef] [Green Version]
- Jensen, L.; Konradsen, F. A review of the use of virtual reality head-mounted displays in education and training. Educ. Inf. Technol. 2017, 11, 1–15. [Google Scholar] [CrossRef] [Green Version]
- Riecke, B.E.; LaViola, J.J., Jr.; Kruijff, E. 3D user interfaces for virtual reality and games: 3D selection, manipulation, and spatial navigation. In Proceedings of the ACM SIGGRAPH 2018 Courses, Vancouver, BC, Canada, 12–16 August 2018; p. 13. [Google Scholar]
- Templeman, J.N.; Denbrook, P.S.; Sibert, L.E. Virtual locomotion: Walking in place through virtual environments. Presence 1999, 8, 598–617. [Google Scholar] [CrossRef]
- Bozgeyikli, E.; Bozgeyikli, L.; Raij, A.; Katkoori, S.; Alqasemi, R.; Dubey, R. Virtual reality interaction techniques for individuals with autism spectrum disorder: Design considerations and preliminary results. In Proceedings of the International Conference on Human-Computer Interaction, Florence, Italy, 11–15 July 2016; pp. 127–137. [Google Scholar]
- Tregillus, S.; Folmer, E. Vr-step: Walking-in-place using inertial sensing for hands free navigation in mobile vr environments. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems; ACM: New York, NY, USA; pp. 1250–1255.
- Nescher, T.; Huang, Y.-Y.; Kunz, A. Planning redirection techniques for optimal free walking experience using model predictive control. In Proceedings of the 2014 IEEE Symposium on 3D User Interfaces (3DUI), Minneapolis, MN, USA, 29–30 March 2014; pp. 111–118. [Google Scholar]
- Nabiyouni, M.; Saktheeswaran, A.; Bowman, D.A.; Karanth, A. Comparing the performance of natural, semi-natural, and non-natural locomotion techniques in virtual reality. In Proceedings of the 2015 IEEE Symposium on 3D User Interfaces (3DUI), Arles, France, 23–24 March 2015; pp. 3–10. [Google Scholar]
- Bozgeyikli, E.; Raij, A.; Katkoori, S.; Dubey, R. Locomotion in virtual reality for individuals with autism spectrum disorder. In Proceedings of the 2016 Symposium on Spatial User Interaction, Tokyo, Japan, 15–16 October 2016; pp. 33–42. [Google Scholar]
- Boletsis, C. The New Era of Virtual Reality Locomotion: A Systematic Literature Review of Techniques and a Proposed Typology. Multimodal Technol. Interact. 2017, 1, 24. [Google Scholar] [CrossRef] [Green Version]
- Slater, M.; Wilbur, S. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence Teleoperators Virtual Environ. 1997, 6, 603–616. [Google Scholar] [CrossRef]
- Heeter, C. Being There: The Subjective Experience of Presence. Presence Teleoperators Virtual Environ. 1992, 1, 262–271. [Google Scholar] [CrossRef]
- Baños, R.M.; Botella, C.; Alcañiz, M.; Liaño, V.; Guerrero, B.; Rey, B. Immersion and Emotion: Their Impact on the Sense of Presence. CyberPsychol. Behav. 2004, 7, 734–741. [Google Scholar] [CrossRef]
- Slater, M.; Usoh, M.; Steed, A. Depth of Presence in virtual environments. Presence Teleoperators Virtual Environ. 1994, 3, 130–144. [Google Scholar] [CrossRef]
- Usoh, M.; Arthur, K.; Whitton, M.C.; Bastos, R.; Steed, A.; Slater, M.; Brooks, F.P. Walking > walking-in-place > flying, in virtual environments. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques-SIGGRAPH ’99; Waggenspack, W., Ed.; ACM Press/Addison-Wesley Publishing: New York, NY, USA, 1999; pp. 359–364. [Google Scholar]
- Diemer, J.; Alpers, G.W.; Peperkorn, H.M.; Shiban, Y.; Mühlberger, A. The impact of perception and presence on emotional reactions: A review of research in virtual reality. Front. Psychol. 2015, 6, 1–9. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Riches, S.; Elghany, S.; Garety, P.; Rus-Calafell, M.; Valmaggia, L. Factors Affecting Sense of Presence in a Virtual Reality Social Environment: A Qualitative Study. Cyberpsychol. Behav. Soc. Netw. 2019, 22, 288–292. [Google Scholar] [CrossRef] [PubMed]
- Kiryu, T.; So, R.H.Y. Sensation of presence and cybersickness in applications of virtual reality for advanced rehabilitation. J. NeuroEng. Rehabil. 2007, 4, 34. [Google Scholar] [CrossRef] [Green Version]
- Freina, L.; Ott, M. A literature review on immersive virtual reality in education: State of the art and perspectives. In Proceedings of the International Scientific Conference eLearning and Software for Education, Bucharest, Italy, 23–24 April 2015; Volume 1, p. 133. [Google Scholar]
- Alaraj, A.; Lemole, M.G.; Finkle, J.H.; Yudkowsky, R.; Wallace, A.; Luciano, C.; Banerjee, P.P.; Rizzi, S.H.; Charbel, F.T. Virtual reality training in neurosurgery: Review of current status and future applications. Surg. Neurol. Int. 2011, 2, 52. [Google Scholar] [CrossRef] [Green Version]
- Bhagat, K.K.; Liou, W.-K.; Chang, C.-Y. A cost-effective interactive 3D virtual reality system applied to military live firing training. Virtual Real. 2016, 20, 127–140. [Google Scholar] [CrossRef]
- Yavrucuk, I.; Kubali, E.; Tarimci, O. A low cost flight simulator using virtual reality tools. IEEE Aerosp. Electron. Syst. Mag. 2011, 26, 10–14. [Google Scholar] [CrossRef]
- Dols, J.F.; Molina, J.; Camacho, F.J.; Marín-Morales, J.; Pérez-Zuriaga, A.M.; Garcia, A. Design and development of driving simulator scenarios for road validation studies. Transp. Res. Procedia 2016, 18, 289–296. [Google Scholar] [CrossRef]
- de-Juan-Ripoll, C.; Soler-Domínguez, J.L.; Guixeres, J.; Contero, M.; Gutiérrez, N.Á.; Alcañiz, M. Virtual reality as a new approach for risk taking assessment. Front. Psychol. 2018, 9, 1–8. [Google Scholar] [CrossRef] [Green Version]
- Bohil, C.J.; Alicea, B.; Biocca, F.A. Virtual reality in neuroscience research and therapy. Nat. Rev. Neurosci. 2011, 12, 752–762. [Google Scholar] [CrossRef]
- Peperkorn, H.M.; Alpers, G.W.; Mühlberger, A. Triggers of fear: Perceptual cues versus conceptual information in spider phobia. J. Clin. Psychol. 2014, 70, 704–714. [Google Scholar] [CrossRef] [PubMed]
- Park, K.-M.; Ku, J.; Choi, S.-H.; Jang, H.-J.; Park, J.-Y.; Kim, S.I.; Kim, J.-J. A virtual reality application in role-plays of social skills training for schizophrenia: A randomized, controlled trial. Psychiatry Res. 2011, 189, 166–172. [Google Scholar] [CrossRef] [PubMed]
- Didehbani, N.; Allen, T.; Kandalaft, M.; Krawczyk, D.; Chapman, S. Virtual reality social cognition training for children with high functioning autism. Comput. Hum. Behav. 2016, 62, 703–711. [Google Scholar] [CrossRef] [Green Version]
- Lloréns, R.; Noé, E.; Colomer, C.; Alcañiz, M. Effectiveness, usability, and cost-benefit of a virtual reality—Based telerehabilitation program for balance recovery after stroke: A randomized controlled trial. Arch. Phys. Med. Rehabil. 2015, 96, 418–425. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Alcañiz, M.L.; Olmos-Raya, E.; Abad, L. Use of virtual reality for neurodevelopmental disorders. A review of the state of the art and future agenda. Medicina 2019, 79, 77–81. [Google Scholar]
- Portman, M.E.; Natapov, A.; Fisher-Gewirtzman, D. To go where no man has gone before: Virtual reality in architecture, landscape architecture and environmental planning. Comput. Environ. Urban Syst. 2015, 54, 376–384. [Google Scholar] [CrossRef]
- Bigné, E.; Llinares, C.; Torrecilla, C. Elapsed time on first buying triggers brand choices within a category: A virtual reality-based study. J. Bus. Res. 2015. [Google Scholar] [CrossRef]
- Alcañiz, M.; Bigné, E.; Guixeres, J. Virtual Reality in Marketing: A Framework, Review, and Research Agenda. Front. Psychol. 2019, 10, 1–15. [Google Scholar] [CrossRef]
- Picard, R.W. Affective Computing: Challenges. Int. J. Hum. Comput. Stud. 2003, 59, 55–64. [Google Scholar] [CrossRef]
- Riva, G.; Mantovani, F.; Capideville, C.S.; Preziosa, A.; Morganti, F.; Villani, D.; Gaggioli, A.; Botella, C.; Alcañiz, M. Affective Interactions Using Virtual Reality: The Link between Presence and Emotions. CyberPsychol. Behav. 2007, 10, 45–56. [Google Scholar] [CrossRef]
- Guixeres, J.; Saiz, J.; Alcañiz, M.; Cebolla, A.; Escobar, P.; Baños, R.; Botella, C.; Lison, J.F.; Alvarez, J.; Cantero, L.; et al. Effects of virtual reality during exercise in children. J. Univers. Comput. Sci. 2013, 19, 1199–1218. [Google Scholar]
- Felnhofer, A.; Kothgassner, O.D.; Schmidt, M.; Heinzle, A.K.; Beutl, L.; Hlavacs, H.; Kryspin-Exner, I. Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios. Int. J. Hum. Comput. Stud. 2015, 82, 48–56. [Google Scholar] [CrossRef]
- Lorenzo, G.; Lledó, A.; Pomares, J.; Roig, R. Design and application of an immersive virtual reality system to enhance emotional skills for children with autism spectrum disorders. Comput. Educ. 2016, 98, 192–205. [Google Scholar] [CrossRef] [Green Version]
- Rohrmann, B.; Bishop, I.D. Subjective responses to computer simulations of urban environments. J. Environ. Psychol. 2002, 22, 319–331. [Google Scholar] [CrossRef]
- Bishop, I.D.; Rohrmann, B. Subjective responses to simulated and real environments: A comparison. Landsc. Urban Plan. 2003, 65, 261–277. [Google Scholar] [CrossRef]
- de Kort, Y.A.W.; Ijsselsteijn, W.A.; Kooijman, J.; Schuurmans, Y. Virtual laboratories: Comparability of real and virtual environments for environmental psychology. Presence Teleoperators Virtual Environ. 2003, 12, 360–373. [Google Scholar] [CrossRef]
- Yeom, D.; Choi, J.-H.; Zhu, Y. Investigation of the Physiological Differences between Immersive Virtual Environment and Indoor Enviorment in a Building. Indoor Built Environ. 2017, 1–17. [Google Scholar] [CrossRef]
- van der Ham, I.J.; Faber, A.M.; Venselaar, M.; van Kreveld, M.J.; Löffler, M. Ecological validity of virtual environments to assess human navigation ability. Front. Psychol. 2015, 6, 637. [Google Scholar] [CrossRef] [Green Version]
- Heydarian, A.; Carneiro, J.P.; Gerber, D.; Becerik-Gerber, B.; Hayes, T.; Wood, W. Immersive virtual environments versus physical built environments: A benchmarking study for building design and user-built environment explorations. Autom. Constr. 2015, 54, 116–126. [Google Scholar] [CrossRef]
- Chamilothori, K.; Wienold, J.; Andersen, M. Adequacy of Immersive Virtual Reality for the Perception of Daylit Spaces: Comparison of Real and Virtual Environments. LEUKOS J. Illum. Eng. Soc. N. Am. 2018, 1–24. [Google Scholar] [CrossRef] [Green Version]
- Kimura, K.; Reichert, J.F.; Olson, A.; Pouya, O.R.; Wang, X.; Moussavi, Z.; Kelly, D.M. Orientation in Virtual Reality Does Not Fully Measure Up to the Real-World. Sci. Rep. 2017, 7, 6–13. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Higuera-Trujillo, J.L.; López-Tarruella, J.; Llinares, M.C. Psychological and physiological human responses to simulated and real environments: A comparison between Photographs, 360° Panoramas, and Virtual Reality. Appl. Ergon. 2017, 65, 398–409. [Google Scholar] [CrossRef] [PubMed]
- Marín-Morales, J.; Higuera-Trujillo, J.L.; Greco, A.; Guixeres, J.; Llinares, C.; Gentili, C.; Scilingo, E.P.; Alcañiz, M.; Valenza, G. Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum. PLoS ONE 2019, 14, e0223881. [Google Scholar] [CrossRef] [PubMed]
- Brief, A.P. Attitudes in and Around Organizations; Sage: Thousand Oaks, CA, USA, 1998; Volume 9. [Google Scholar]
- Payne, B.K. Prejudice and perception: The role of automatic and controlled processes in misperceiving a weapon. J. Pers. Soc. Psychol. 2001, 81, 181. [Google Scholar] [CrossRef] [PubMed]
- Schmitt, N. Method bias: The importance of theory and measurement. J. Organ. Behav. 1994, 15, 393–398. [Google Scholar] [CrossRef]
- Barsade, S.G.; Ramarajan, L.; Westen, D. Implicit affect in organizations. Res. Organ. Behav. 2009, 29, 135–162. [Google Scholar] [CrossRef]
- Lieberman, M.D. Social cognitive neuroscience: A review of core processes. Annu. Rev. Psychol. 2007, 58, 259–289. [Google Scholar] [CrossRef] [Green Version]
- Camerer, C.; Loewenstein, G.; Prelec, D. Neuroeconomics: How neuroscience can inform economics. J. Econ. Lit. 2005, 43, 9–64. [Google Scholar] [CrossRef] [Green Version]
- Riener, A.; Ferscha, A.; Aly, M. Heart on the road: HRV analysis for monitoring a driver’s affective state. In Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Essen, Germany, 21–22 September 2009; pp. 99–106. [Google Scholar]
- Setz, C.; Arnrich, B.; Schumm, J.; La Marca, R.; Tröster, G.; Ehlert, U. Discriminating stress from cognitive load using a wearable EDA device. IEEE Trans. Inf. Technol. Biomed. 2009, 14, 410–417. [Google Scholar] [CrossRef] [PubMed]
- Berka, C.; Levendowski, D.J.; Lumicao, M.N.; Yau, A.; Davis, G.; Zivkovic, V.T.; Olmstead, R.E.; Tremoulet, P.D.; Craven, P.L. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviat. Space Environ. Med. 2007, 78, B231–B244. [Google Scholar] [PubMed]
- Jack, A.I.; Dawson, A.J.; Begany, K.L.; Leckie, R.L.; Barry, K.P.; Ciccia, A.H.; Snyder, A.Z. fMRI reveals reciprocal inhibition between social and physical cognitive domains. Neuroimage 2013, 66, 385–401. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ernst, L.H.; Plichta, M.M.; Lutz, E.; Zesewitz, A.K.; Tupak, S.V.; Dresler, T.; Ehlis, A.-C.; Fallgatter, A.J. Prefrontal activation patterns of automatic and regulated approach--avoidance reactions--a functional near-infrared spectroscopy (fNIRS) study. Cortex 2013, 49, 131–142. [Google Scholar] [CrossRef]
- Glöckner, A.; Herbold, A.-K. An eye-tracking study on information processing in risky decisions: Evidence for compensatory strategies based on automatic processes. J. Behav. Decis. Mak. 2011, 24, 71–98. [Google Scholar] [CrossRef]
- Bahreini, K.; Nadolski, R.; Westera, W. Towards multimodal emotion recognition in e-learning environments. Interact. Learn. Environ. 2016, 24, 590–605. [Google Scholar] [CrossRef]
- Huang, K.-Y.; Wu, C.-H.; Su, M.-H.; Kuo, Y.-T. Detecting unipolar and bipolar depressive disorders from elicited speech responses using latent affective structure model. IEEE Trans. Affect. Comput. 2018. [Google Scholar] [CrossRef]
- Prokasy, W. Electrodermal Activity in Psychological Research; Elsevier: Amsterdam, The Netherland, 2012. [Google Scholar]
- Kim, H.-G.; Cheon, E.-J.; Bai, D.-S.; Lee, Y.H.; Koo, B.-H. Stress and heart rate variability: A meta-analysis and review of the literature. Psychiatry Investig. 2018, 15, 235. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kreibig, S.D. Autonomic nervous system activity in emotion: A review. Biol. Psychol. 2010, 84, 394–421. [Google Scholar] [CrossRef]
- Lotte, F.; Bougrain, L.; Cichocki, A.; Clerc, M.; Congedo, M.; Rakotomamonjy, A.; Yger, F. A review of classification algorithms for EEG-based brain--computer interfaces: A 10 year update. J. Neural Eng. 2018, 15, 31005. [Google Scholar] [CrossRef] [Green Version]
- Gruzelier, J.H. EEG-neurofeedback for optimising performance. I: A review of cognitive and affective outcome in healthy participants. Neurosci. Biobehav. Rev. 2014, 44, 124–141. [Google Scholar] [CrossRef]
- Thibault, R.T.; MacPherson, A.; Lifshitz, M.; Roth, R.R.; Raz, A. Neurofeedback with fMRI: A critical systematic review. Neuroimage 2018, 172, 786–807. [Google Scholar] [CrossRef] [Green Version]
- Naseer, N.; Hong, K.-S. fNIRS-based brain-computer interfaces: A review. Front. Hum. Neurosci. 2015, 9, 3. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Meißner, M.; Oll, J. The promise of eye-tracking methodology in organizational research: A taxonomy, review, and future avenues. Organ. Res. Methods 2019, 22, 590–617. [Google Scholar] [CrossRef]
- Calvo, M.G.; Nummenmaa, L. Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cogn. Emot. 2016, 30, 1081–1106. [Google Scholar] [CrossRef] [PubMed]
- Schuller, B.W. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Commun. ACM 2018, 61, 90–99. [Google Scholar] [CrossRef]
- Parsons, T.D. Virtual Reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Front. Hum. Neurosci. 2015, 9, 660. [Google Scholar] [CrossRef] [Green Version]
- Ekman, P. Basic Emotions. Handb. Cogn. Emot. 1999, 45–60. [Google Scholar] [CrossRef]
- Russell, J.A.; Mehrabian, A. Evidence for a three-factor theory of emotions. J. Res. Pers. 1977, 11, 273–294. [Google Scholar] [CrossRef]
- Calvo, R.A.; D’Mello, S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 2010, 1, 18–37. [Google Scholar] [CrossRef]
- Valenza, G.; Greco, A.; Gentili, C.; Lanata, A.; Sebastiani, L.; Menicucci, D.; Gemignani, A.; Scilingo, E.P. Combining electroencephalographic activity and instantaneous heart rate for assessing brain–heart dynamics during visual emotional elicitation in healthy subjects. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 2016, 374, 20150176. [Google Scholar] [CrossRef] [Green Version]
- Valenza, G.; Lanata, A.; Scilingo, E.P. The role of nonlinear dynamics in affective valence and arousal recognition. IEEE Trans. Affect. Comput. 2012, 3, 237–249. [Google Scholar] [CrossRef]
- Zangeneh Soroush, M.; Maghooli, K.; Setarehdan, S.K.; Motie Nasrabadi, A. A Review on EEG Signals Based Emotion Recognition. Int. Clin. Neurosci. J. 2018, 4, 118–129. [Google Scholar] [CrossRef]
- Kory Jacqueline, D. Sidney Affect Elicitation for affective Computing. In The Oxford Handbook of Affective Computing; Oxford University Press: New York, NY, USA, 2014; pp. 371–383. [Google Scholar]
- Ekman, P. The directed facial action task. In Handbook of Emotion Elicitation and Assessment; Oxford University Press: New York, NY, USA, 2007; pp. 47–53. [Google Scholar]
- Harmon-Jones, E.; Amodio, D.M.; Zinner, L.R. Social psychological methods of emotion elicitation. Handb. Emot. Elicitation Assess. 2007, 25, 91–105. [Google Scholar]
- Roberts, N.A.; Tsai, J.L.; Coan, J.A. Emotion elicitation using dyadic interaction task. Handb. Emot. Elicitation Assess. 2007, 01, 106–123. [Google Scholar]
- Nardelli, M.; Valenza, G.; Greco, A.; Lanata, A.; Scilingo, E.P. Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans. Affect. Comput. 2015, 6, 385–394. [Google Scholar] [CrossRef]
- Kim, J. Emotion Recognition Using Speech and Physiological Changes. Robust Speech Recognit. Underst. 2007, 29, 265–280. [Google Scholar]
- Soleymani, M.; Pantic, M.; Pun, T. Multimodal emotion recognition in response to videos (Extended abstract). In Proceedings of the ACII 2015: International Conference on Affective Computing and Intelligent Interaction, Xi’an, China, 21–24 September 2015; Volume 3, pp. 491–497. [Google Scholar]
- Jang, D.P.; Kim, I.Y.; Nam, S.W.; Wiederhold, B.K.; Wiederhold, M.D.; Kim, S.I. Analysis of physiological response to two virtual environments: Driving and flying simulation. CyberPsychol. Behav. 2002, 5, 11–18. [Google Scholar] [CrossRef] [PubMed]
- Marín-Morales, J.; Higuera-Trujillo, J.L.; Greco, A.; Guixeres, J.; Llinares, C.; Scilingo, E.P.; Alcañiz, M.; Valenza, G. Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci. Rep. 2018, 8, 13657. [Google Scholar] [CrossRef] [PubMed]
- Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G.; The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 2009, 6, e1000097. [Google Scholar] [CrossRef] [Green Version]
- Meehan, M.; Razzaque, S.; Insko, B.; Whitton, M.; Brooks, F.P. Review of four studies on the use of physiological reaction as a measure of presence in stressful virtual environments. Appl. Psychophysiol. Biofeedback 2005, 30, 239–258. [Google Scholar] [CrossRef]
- Wilhelm, F.H.; Pfaltz, M.C.; Gross, J.J.; Mauss, I.B.; Kim, S.I.; Wiederhold, B.K. Mechanisms of virtual reality exposure therapy: The role of the behavioral activation and behavioral inhibition systems. Appl. Psychophysiol. Biofeedback 2005, 30, 271–284. [Google Scholar] [CrossRef] [Green Version]
- Gorini, A.; Griez, E.; Petrova, A.; Riva, G. Assessment of the emotional responses produced by exposure to real food, virtual food and photographs of food in patients affected by eating disorders. Ann. Gen. Psychiatry 2010, 9, 30. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Philipp, M.C.; Storrs, K.R.; Vanman, E.J. Sociality of facial expressions in immersive virtual environments: A facial EMG study. Biol. Psychol. 2012, 91, 17–21. [Google Scholar] [CrossRef] [PubMed]
- Parsons, T.D.; Courtney, C.G.; Dawson, M.E. Virtual reality Stroop task for assessment of supervisory attentional processing. J. Clin. Exp. Neuropsychol. 2013, 35, 812–826. [Google Scholar] [CrossRef] [PubMed]
- Pallavicini, F.; Cipresso, P.; Raspelli, S.; Grassi, A.; Serino, S.; Vigna, C.; Triberti, S.; Villamira, M.; Gaggioli, A.; Riva, G. Is virtual reality always an effective stressors for exposure treatments? Some insights from a controlled trial. BMC Psychiatry 2013, 13, 52. [Google Scholar] [CrossRef] [Green Version]
- Felnhofer, A.; Kothgassner, O.D.; Hetterle, T.; Beutl, L.; Hlavacs, H.; Kryspin-Exner, I. Afraid to be there? Evaluating the relation between presence, self-reported anxiety, and heart rate in a virtual public speaking task. Cyberpsychol. Behav. Soc. Netw. 2014, 17, 310–316. [Google Scholar] [CrossRef]
- Hartanto, D.; Kampmann, I.L.; Morina, N.; Emmelkamp, P.G.M.; Neerincx, M.A.; Brinkman, W.-P. Controlling social stress in virtual reality environments. PLoS ONE 2014, 9, e92804. [Google Scholar] [CrossRef]
- McCall, C.; Hildebrandt, L.K.; Bornemann, B.; Singer, T. Physiophenomenology in retrospect: Memory reliably reflects physiological arousal during a prior threatening experience. Conscious. Cogn. 2015, 38, 60–70. [Google Scholar] [CrossRef]
- Notzon, S.; Deppermann, S.; Fallgatter, A.; Diemer, J.; Kroczek, A.; Domschke, K.; Zwanzger, P.; Ehlis, A.C. Psychophysiological effects of an iTBS modulated virtual reality challenge including participants with spider phobia. Biol. Psychol. 2015, 112, 66–76. [Google Scholar] [CrossRef]
- Hildebrandt, L.K.; Mccall, C.; Engen, H.G.; Singer, T. Cognitive flexibility, heart rate variability, and resilience predict fine-grained regulation of arousal during prolonged threat. Psychophysiology 2016, 53, 880–890. [Google Scholar] [CrossRef]
- Higuera-Trujillo, J.L.; Marín-Morales, J.; Rojas, J.C.; López-Tarruella-Maldonado, J. Emotional maps: Neuro architecture and design applications. 6th International Forum Design as a Processes. Syst. Des. Beyond Process. Think. 2016, 677–685. [Google Scholar] [CrossRef] [Green Version]
- Bian, Y.; Yang, C.; Gao, F.; Li, H.; Zhou, S.; Li, H.; Sun, X.; Meng, X. A framework for physiological indicators of flow in VR games: Construction and preliminary evaluation. Pers. Ubiquitous Comput. 2016, 20, 821–832. [Google Scholar] [CrossRef]
- Shiban, Y.; Diemer, J.; Brandl, S.; Zack, R.; Mühlberger, A.; Wüst, S. Trier Social Stress Test in vivo and in virtual reality: Dissociation of response domains. Int. J. Psychophysiol. 2016, 110, 47–55. [Google Scholar] [CrossRef] [PubMed]
- Chirico, A.; Cipresso, P.; Yaden, D.B.; Biassoni, F.; Riva, G.; Gaggioli, A. Effectiveness of Immersive Videos in Inducing Awe: An Experimental Study. Sci. Rep. 2017, 7, 1–11. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zou, H.; Li, N.; Cao, L. Emotional response—Based approach for assessing the sense of presence of subjects in virtual building evacuation studies. J. Comput. Civ. Eng. 2017, 31, 4017028. [Google Scholar] [CrossRef]
- Breuninger, C.; Sláma, D.M.; Krämer, M.; Schmitz, J.; Tuschen-Caffier, B. Psychophysiological reactivity, interoception and emotion regulation in patients with agoraphobia during virtual reality anxiety induction. Cognit. Ther. Res. 2017, 41, 193–205. [Google Scholar] [CrossRef]
- van’t Wout, M.; Spofford, C.M.; Unger, W.S.; Sevin, E.B.; Shea, M.T. Skin conductance reactivity to standardized virtual reality combat scenes in veterans with PTSD. Appl. Psychophysiol. Biofeedback 2017, 42, 209–221. [Google Scholar] [CrossRef]
- Banaei, M.; Hatami, J.; Yazdanfar, A.; Gramann, K. Walking through Architectural Spaces: The Impact of Interior Forms on Human Brain Dynamics. Front. Hum. Neurosci. 2017, 11, 1–14. [Google Scholar] [CrossRef]
- Anderson, A.P.; Mayer, M.D.; Fellows, A.M.; Cowan, D.R.; Hegel, M.T.; Buckey, J.C. Relaxation with immersive natural scenes presented using virtual reality. Aerosp. Med. Hum. Perform. 2017, 88, 520–526. [Google Scholar] [CrossRef]
- Chittaro, L.; Sioni, R.; Crescentini, C.; Fabbro, F. Mortality salience in virtual reality experiences and its effects on users’ attitudes towards risk. Int. J. Hum. Comput. Stud. 2017, 101, 10–22. [Google Scholar] [CrossRef]
- Biedermann, S.V.; Biedermann, D.G.; Wenzlaff, F.; Kurjak, T.; Nouri, S.; Auer, M.K.; Wiedemann, K.; Briken, P.; Haaker, J.; Lonsdorf, T.B.; et al. An elevated plus-maze in mixed reality for studying human anxiety-related behavior. BMC Biol. 2017, 15, 125. [Google Scholar] [CrossRef]
- Tsai, C.-F.; Yeh, S.-C.; Huang, Y.; Wu, Z.; Cui, J.; Zheng, L. The Effect of Augmented Reality and Virtual Reality on Inducing Anxiety for Exposure Therapy: A Comparison Using Heart Rate Variability. J. Healthc. Eng. 2018, 2018, 27–36. [Google Scholar] [CrossRef] [PubMed]
- Kisker, J.; Gruber, T.; Schöne, B. Behavioral realism and lifelike psychophysiological responses in virtual reality by the example of a height exposure. Psychol. Res. 2019, 1–14. [Google Scholar] [CrossRef] [PubMed]
- Gromer, D.; Reinke, M.; Christner, I.; Pauli, P. Causal Interactive Links Between Presence and Fear in Virtual Reality Height Exposure. Front. Psychol. 2019, 10, 141. [Google Scholar] [CrossRef] [PubMed]
- Zimmer, P.; Wu, C. Same same but different? Replicating the real surroundings in a virtual Trier Social Stress Test (TSST-VR) does not enhance presence or the psychophysiological stress response. Physiol. Behav. 2019, 212, 112690. [Google Scholar] [CrossRef]
- Lin, J.; Cao, L.; Li, N. Assessing the influence of repeated exposures and mental stress on human wayfinding performance in indoor environments using virtual reality technology. Adv. Eng. Inform. 2019, 39, 53–61. [Google Scholar] [CrossRef]
- Schweizer, T.; Renner, F.; Sun, D.; Becker-Asano, C.; Tuschen-Caffier, B. Cognitive processing and regulation modulates analogue trauma symptoms in a Virtual Reality paradigm. Cognit. Ther. Res. 2019, 43, 199–213. [Google Scholar] [CrossRef]
- Kim, Y.; Moon, J.; Sung, N.-J.; Hong, M. Correlation between selected gait variables and emotion using virtual reality. J. Ambient Intell. Humaniz. Comput. 2019, 8, 1–8. [Google Scholar] [CrossRef]
- Uhm, J.-P.; Lee, H.-W.; Han, J.-W. Creating sense of presence in a virtual reality experience: Impact on neurophysiological arousal and attitude towards a winter sport. Sport Manag. Rev. 2019. [Google Scholar] [CrossRef]
- Takac, M.; Collett, J.; Blom, K.J.; Conduit, R.; Rehm, I.; De Foe, A. Public speaking anxiety decreases within repeated virtual reality training sessions. PLoS ONE 2019, 14, e0216288. [Google Scholar] [CrossRef] [Green Version]
- Stolz, C.; Endres, D.; Mueller, E.M. Threat-conditioned contexts modulate the late positive potential to faces—A mobile EEG/virtual reality study. Psychophysiology 2019, 56, e13308. [Google Scholar] [CrossRef]
- Granato, M.; Gadia, D.; Maggiorini, D.; Ripamonti, L.A. An empirical study of players’ emotions in VR racing games based on a dataset of physiological data. Multimed. Tools Appl. 2020, 1–30. [Google Scholar] [CrossRef]
- Bălan, O.; Moise, G.; Moldoveanu, A.; Leordeanu, M.; Moldoveanu, F. An investigation of various machine and deep learning techniques applied in automatic fear level detection and acrophobia virtual therapy. Sensors 2020, 20, 496. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Reichenberger, J.; Pfaller, M.; Mühlberger, A. Gaze Behavior in Social Fear Conditioning: An Eye-Tracking Study in Virtual Reality. Front. Psychol. 2020, 11, 1–12. [Google Scholar] [CrossRef] [PubMed]
- Huang, Q.; Yang, M.; Jane, H.-A.; Li, S.; Bauer, N. Trees, grass, or concrete? The effects of different types of environments on stress reduction. Landsc. Urban Plan. 2020, 193, 103654. [Google Scholar] [CrossRef]
- Shu, L.; Xie, J.; Yang, M.; Li, Z.; Li, Z.; Liao, D.; Xu, X.; Yang, X. A review of emotion recognition using physiological signals. Sensors 2018, 18, 2074. [Google Scholar] [CrossRef] [Green Version]
- Greco, A.; Valenza, G.; Citi, L.; Scilingo, E.P. Arousal and valence recognition of affective sounds based on electrodermal activity. IEEE Sens. J. 2016, 17, 716–725. [Google Scholar] [CrossRef]
Implicit Technique | Biometric Signal Measured | Sensor | Features | Psychological or Behavioural Construct Inferred |
---|---|---|---|---|
EDA (electro dermal activity) | Changes in skin conductance | Electrodes attached to fingers, palms or soles | Skin conductance response, tonic activity and phasic activity | Attention and arousal [80] |
HRV (heart rate variability) | Variability in heart contraction intervals | Electrodes attached to chest or limbs or optical sensor attached to finger, toe or earlobe | Time domain, frequency domain, non-linear domain | Stress, anxiety, arousal and valence [81,82] |
EEG (electroencephalogram) | Changes in electrical activity of the brain | Electrodes placed on scalp | Frequency band power, functional connectivity, event-related potentials | Attention, mental workload, drowsiness, fatigue, arousal and valence [83,84] |
fMRI (functional magnetic resonance imaging) | Concentrations of oxygenated vs. deoxygenated haemoglobin in the blood vessels of the brain | Magnetic resonance signal | blood-oxygen-level dependent | Motor execution, attention, memory, pain, anxiety, hunger, fear, arousal and valence [85] |
fNIRS (functional near-infrared spectroscopy) | Concentrations of oxygenated vs. deoxygenated haemoglobin in the blood | Near-infrared light placed on scalp | blood-oxygen-level dependent | Motor execution, cognitive task (mental arithmetic), decision-making and valence [86] |
ET (eye-tracking) | Corneal reflection and pupil dilation | Infrared cameras point towards eyes | Eye movements (gaze, fixation, saccades), blinks, pupil dilation | Visual attention, engagement, drowsiness and fatigue [87] |
FEA (facial expression analysis) | Activity of facial muscles | Camera points towards face | Position and orientation of head. Activation of action units | Basic emotions, engagement, arousal and valence [88] |
SER (speech emotion recognition) | Voice | Microphone | Prosodic and spectral features | Stress, basic emotions, arousal and valence [89] |
No | Author | Emotion | Signals | Features | Data Analysis | Subjects | HMD | VR Stimuli | Stimuli Comparison | Dataset Availability |
---|---|---|---|---|---|---|---|---|---|---|
1 | Jang et al. (2002) [104] | Arousal | HRV, EDA | HR, HRV frequency domain, SCL, ST | t-test | 11 | VFX3D | 3D flying and driving simulator | No | No |
2 | Meehan et al. (2005) [107] | Arousal | HRV, EDA | HR, SC, ST | t-test | 67 | Not reported | 3D training room vs. pit room | No | No |
3 | Wilhelm et al. (2005) [108] | Anxiety | HRV, EDA | HR, SC | ANOVA, correlations | 86 | Not reported | 3D height exposure | Partially (with a different real dataset) | No |
4 | Gorini et al. (2010) [109] | Anxiety | HRV, EDA | HR, SC | ANOVA | 30 (20 with food disorders) | Not reported | 3D photo and real food catering | VR vs. photo vs. real | No |
5 | Philipp et al. (2012) [110] | Valence | EMG | EMG | ANOVA | 49 | Virtual Research V8 | 3D room with IAPS pictures projected | No | No |
6 | Parsons et al. (2013) [111] | Arousal | HRV, EDA | HR, SC | ANOVA | 50 | eMagin Z800 | 3D high-mobility wheeled vehicle with Stroop task | No | No |
7 | Pallavicini et al. (2013) [112] | Stress | HRV, EMG, RSP | HR, SC, RR | ANOVA | 39 | Vuzix VR Bundle | 3D classroom | No | No |
8 | Peperkorn et al. (2014) [43] | Fear | HRV, EDA | HR, SC | ANOVA | 96 (48 spider-phobic) | eMagin Z800 | 3D virtual lab with time-varying threat (spiders and snakes) | No | No |
9 | Felnhofer et al. (2014) [113] | Anxiety | HRV | HR | ANOVA | 75 (30 high anxiety) | eMagin Z800 | 3D lecture hall | No | No |
10 | Hartanto et al. (2014) [114] | Stress | HRV | HR | MANOVA | 24 healthy subjects | eMagin Z800 | 3D stressful social environment | No | No |
11 | McCall et al. (2015) [115] | Arousal | HRV, EDA | HR, SC | Cross-correlations | 306 | NVIS nVisor SX60 | 3D room with time-varying threat (explosions, spiders, gunshots, etc.) | No | No |
12 | Felnhofer et al. (2015) [54] | Arousal | EDA | SCL | ANOVA | 120 | Sony HMZ-T1 3D | 3D park with 5 variations (joy, sadness, boredom, anger and anxiety) | No | No |
13 | Notzon et al. (2015) [116] | Anxiety | HRV, EDA | HR, SC | ANOVA | 83 (42 spider-phobic) | eMagin Z800 | 3D virtual lab with spiders | No | No |
14 | Hildebrandt et al. (2016) [117] | Arousal | HRV, EDA | RMSSD, SC | Regression | 300 | NVIS nVisor SX60 | 3D room with time-varying threats (explosions, spiders, gunshots, etc.) | No | No |
15 | Higuera-Trujillo et al. (2016) [118] | Stress | EDA | SCR | Kruskall–Wallis Test and correlations | 12 | Oculus Rift DK2 | 3D rooms (neutral, stress and calm) | No | No |
16 | Bian et al. (2016) [119] | Arousal | HRV, EMG, RSP | HR, LF, HF, LF/HF, RR, RS | Regression | 36 | Oculus Rift DK2 | 3D Flight simulator | No | No |
17 | Shiban et al. (2016) [120] | Stress | HRV, EDA | HR, SC | ANOVA | 45 | NVIS nVisor SX60 | 3D Trier Social Stress Test | No | No |
18 | Chirico et al. (2017) [121] | Awe | HRV, EDA, EMG | HF, VLF, SC | ANOVA | 42 | Samsung Gear VR | 360° neutral and awe videos | Immersive vs. non-immersive | No |
19 | Zou et al. (2017) [122] | Arousal | HRV, EDA | HRV time domain (AVNN, SDNN…) and frequency domain (LF, HF…), SC, SCL, SCR | t-test | 40 | Oculus Rift DK2 | 3D fire evacuation | No | No |
20 | Breuninger et al. (2017) [123] | Arousal | HRV, EDA | HR, HF, SC | t-test | 51 (23 agoraphobics) | TriVisio VR Vision | 3D car accident | No | No |
21 | van’t Wout et al. (2017) [124] | Stress | EDA | SCR | MANOVA | 44 veterans (19 with PTSD) | eMagin Z800 | 3D combat-related and classroom-related | No | No |
22 | Banaei et al. (2017) [125] | Arousal, Valence | EEG | PSD, ERSPs | MANOVA | 17 | Samsung Gear VR | 3D rooms | No | No |
23 | Anderson et al. (2017) [126] | Stress | HRV, EDA | LF, HF, LF/HF, SC | MANOVA | 18 | Oculus Rift DK2 | 360° indoor vs. natural panoramas | No | No |
24 | Chittaro et al. (2017) [127] | Arousal | HRV | HR, LF, HF, LF/HF | ANOVA | 108 | Sony HMZ-T1 3D | 3D cemetery and park | No | No |
25 | Higuera-Trujillo et al. (2017) [64] | Pleasantness | HRV, EDA | HF, SCR | Mann–Whitney U tests and correlations | 100 | Samsung Gear VR | 3D, 360° and real retail store | real vs. 3D VR vs. 360° VR | No |
26 | Biedermann et al. (2017) [128] | Anxiety | HRV, EDA, RSP | HR, SC, RR | ANOVA | 100 | HTC Vive | Mixed reality (3D VR with real-world elements) | No | Yes |
27 | Tsai et al. (2018) [129] | Anxiety | HRV | HRV time domain (HR, RMSSD…) and frequency domain (HF, LF…) | ANOVA | 30 | eMagin Z800 | 3D VR claustrophobic environments | Augmented reality vs. VR | Upon request |
28 | Marín-Morales et al. (2018) [105] | Arousal, Valence | EEG, HRV | PSD and functional connectivity, HRV Time (HR, RMSSD…), frequency (HF, LF…) and non-linear (SD1, SD2, Entropy…) domain | SVM | 60 | Samsung Gear VR | 360° virtual rooms | No | Upon request |
29 | Kisker et al. (2019) [130] | Arousal | HRV | HR | t-test, correlations and regressions | 30 | HTC Vive | 3D exposure to a high height | No | No |
30 | Gromer et al. (2019) [131] | Fear | HRV, EDA | HR, SC | ANOVA | 49 (height-fearful) | HTC Vive | 3D forest | No | Yes |
31 | Zimmer et al. (2019) [132] | Stress | HRV, salivary | HR, salivary cortisol responses, salivary alpha amylase | ANOVA | 50 | Oculus Rift DK2 | 3D Trier Social Stress Test | Replication of a real study | No |
32 | Lin et al. (2019) [133] | Stress | EDA, Navigation | SC, travel distance, travel time | Mann–Whitney U | 60 | HTC Vive | 3D, building on fire | No | No |
33 | Schweizer et al. (2019) [134] | Stress | HRV, EDA | HR, SC | t-test and correlations | 80 | TriVisio VR Vision | 3D neutral and trauma-related scene | No | No |
34 | Kim et al. (2019) [135] | Calm, sadness and joy | Gait Patterns | Step count, gait speed, foot plantar pressure | ANOVA | 12 | HTC Vive | 360° emotion-related videos | No | No |
35 | Uhm et at. (2019) [136] | Arousal | EEG | PSD | MANOVA | 28 | Samsung Gear VR | 360° sport videos | No | No |
36 | Takac et al. (2019) [137] | Anxiety | HRV | HR | ANOVA | 19 | Oculus Rift | 3D rooms with public audience | No | No |
37 | Marín-Morales et al. (2019) [65] | Arousal, Valence | HRV, EEG | PSD and functional connectivity, HRV Time (HR, RMSSD…), frequency (HF, LF…) and non-linear (SD1, SD2, Entropy…) domain | SVM | 60 | HTC Vive | 3D art museum | Real museum vs. 3D museum | Upon request |
38 | Stolz et al. (2019) [138] | Fear | EEG | ERPs | ANOVA | 29 | Oculus Rift | 3D room with angry avatars | No | No |
39 | Granato et al. (2020) [139] | Arousal, Valence | HRV, EDA, EMG, RSP | HR, SC, SCL, SCR, EMG, RR | SVM, RF, Gradient Boosting, Gaussian Process Regression | 33 | Oculus Rift DK2 | 3D video games | No | Yes |
40 | Bălan et al. (2020) [140] | Fear | HRV, EDA, EEG | HR, SC, PSD | kNN, SVM, RF, LDA, NN | 8 | HTC Vive | 3D acrophobia game | No | No |
41 | Reichenberger et al. (2020) [141] | Fear | Eye-tracking | Fixation counts, TTFF | ANOVA, t-test | 53 (26 socially anxious) | HTC Vive | 3D room with angry avatars | No | Upon request |
42 | Huang et al. (2020) [142] | Stress | EDA | SCL | MANOVA | 89 | Oculus Rift DK2 | 360° built vs. natural environments | No | Yes |
Type of Validation | % of Papers | Number of Papers |
---|---|---|
No validation | 83.33% | 35 |
Real | 7.14% | 3 |
Format | 7.14% | 3 |
Immersivity | 2.38% | 1 |
Previous datasets | 2.38% | 1 |
Replication | 2.38% | 1 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Marín-Morales, J.; Llinares, C.; Guixeres, J.; Alcañiz, M. Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing. Sensors 2020, 20, 5163. https://doi.org/10.3390/s20185163
Marín-Morales J, Llinares C, Guixeres J, Alcañiz M. Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing. Sensors. 2020; 20(18):5163. https://doi.org/10.3390/s20185163
Chicago/Turabian StyleMarín-Morales, Javier, Carmen Llinares, Jaime Guixeres, and Mariano Alcañiz. 2020. "Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing" Sensors 20, no. 18: 5163. https://doi.org/10.3390/s20185163