Cognitive Load Estimation in VR Flight Simulator
Abstract
:Introduction
VR Flight simulator framework
AI agent’s evasive maneuvers
AI agent 1: No guidance
Agent 3: AI Agent trained using reinforcement learning algorithm
- Distance between PA and AI agent dt: dmin<dt<dmax. This condition ensures that evader is within the attack range.
- Deviation angle in degrees μ< μmax.
- Aspect angle in degrees η< ηmax
- ▪
- Distance from pursuer is more than the chase range of the missile.
- ▪
- Angle between the line-of-sight vector and the aircraft heading is larger than the maximum predefined angle.
- ▪
- Aspect angle, which is the angle between pursuer’s longitudinal axis and the line joining from pursuer’s tail to agent’s nose, is larger than the predefined angle.
User Study: Study of Pilot’s interactions with aircraft
Mission preparation
Participants
Procedure
Data Processing and analysis
Ocular parameters
a. Pupil dilation dynamics
b. Gaze fixation analysis
c. Gaze distribution analysis
EEG parameters
d. EEG Task load index (TLI)
e. Task engagement index
Pilot control parameters
- Duty cycle (DC): DC denotes the total percentage of time participant uses his/her controls. DC increases as the task demands higher control as given in Equation 6.
- Aggressiveness: Aggressiveness describes how rapid are the control inputs. Aggressiveness is measured in terms of rate of change of pilot stick inputs (Equation 7). Increase in aggressiveness correlates with more random and abrupt control inputs; which is in turn related to higher task demands.
Results and Discussions
Ocular parameter analysis
a. Pupil dynamics analysis
b. Fixation analysis
c. Gaze distribution analysis
EEG Analysis
a. Task load Index:
b. Task engagement index
Inter-pilot variability
- We measured pilot’s interaction with aircraft using his/her actions on the inceptor control through PIW metric. We further correlated PIW with ocular and EEG parameters-based metrics discussed in section 3.
- From the ocular parameter analysis, we found that variations in pupil dilation is an important cognitive load parameter. Strong position correlation is observed between increase in pupil diameter and PIW.
- We also noticed significant increase in fixation rate and reduction in length of fixation with increase in PIW. Results of the study corroborates the statement made in the previous section. The task reported in this article demands pilot to simultaneously process many flight parameters such as altitude, airspeed, orientation, distance from the target aircraft and number of remaining missiles to take decision to fire the missile. Decision-making process becomes more complex with increase in task difficulty; thereby resulting in an increased fixation rate.
- We observed that TLI shows strong positive correlation with increasing pilot activity. More importantly, we observed an inverse relation between TLI and TEI with increase in task difficulty. This negative association may be attributed to be due to nature of the task. TEI, by definition, explores information gathering, visual processing, vigilance and attention allocation. Decrease in TEI with PIW may be attributed to be due to decrease in availability of attentional resources and the amount of information processing due to pilot’s focus on tracking the target. This, in general, led to reduced peripheral activities such as visual scanning which is also evident from a reduced NNI with PIW (refer Figure 11).
- Initially, we assumed each data point as an independent observation and carried out Pearson correlation. These results were substantiated with repeated measure correlation. Table 2 shows the comparison between Pearson correlation coefficient (r) and repeated measure correlation (rmcorr) for all the cognitive load parameters reported in the paper. It can be observed from Table 2 that r and rmcorr values are comparable for parameters where strong correlation is observed.
Conclusions
Ethics and Conflict of Interest
Acknowledgements
References
- Arjun, S., G. S. R. Reddy, A. Mukhopadhyay, S. Vinod, and P. Biswas. 2021. Evaluating visual variables in a virtual reality environment. 34th British HCI Conference, 34, 11–22. [Google Scholar]
- Babu, M. D., D. V. JeevithaShree, G. Prabhakar, K. P. S. Saluja, A. Pashilkar, and P. Biswas. 2019. Estimating pilots cognitive load from ocular parameters through simulation and inflight studies. Journal of Eye Movement Research 12, 3: 3. [Google Scholar] [CrossRef] [PubMed]
- Bakdash, J. Z., and L. R. G Marusich. 2017. Repeated measures correlation. Frontiers in psychology 8: 456. [Google Scholar] [CrossRef] [PubMed]
- Berka, C., D. J. Levendowski, M. N. Lumicao, A. Yau, G. Davis, V. T. Zivkovic, R. E. Olmstead, P. D. Tremoulet, and P. L. Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviat Space Environ Med 78, 5 Suppl: B231–44. [Google Scholar] [PubMed]
- Bellenkes, A. H., C. D. Wickens, and A. F. Kramer. 1997. Visual scanning and pilot expertise: the role of attentional flexibility and mental model development. Aviat Space Environ Med 68, 7: 569–79. [Google Scholar] [PubMed]
- Biella, M., M. Wies, R. Charles, N. Maille, B. Berberian, and J. Nixon. 2017. How eye tracking data can enhance human performance in tomorrow’s cockpit. Results from a flight simulation study in FUTURE SKY SAFETY, in Proceedings of Joint AIAA and Royal Aeronautical Society Fall Conference on Modeling and Simulation for ATM. 13–15. [Google Scholar]
- Boeing. 2016. QF-16 Full scale aerial target. https://www.boeing.com/defense/support/qf-16/index.page (accessed on 29.4.2023).
- Borghini, G., G. Vecchiato, J. Toppi, L. Astolfi, A. Maglione, and R. Isabella. 2012. Assessment of mental fatigue during car driving by using high resolution EEG activity and neurophysiologic indices. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society (San Diego, CA: IEEE). 6442–6445. [Google Scholar]
- Chandrasekaran, K., V. Theningaledathil, and A. Hebbar. 2021. Ground based variable stability flight simulator. Aviation 25, 1 (Apr. 2021): 22–34. [Google Scholar] [CrossRef]
- Chaouachi, M., and C. Frasson. 2010. Exploring the Relationship between Learner EEG Mental Engagement and Affect. In: Aleven, V., Kay, J., Mostow, J. (eds) Intelligent Tutoring Systems. ITS 2010. Lecture Notes in Computer Science, vol 6095. Springer, Berlin, Heidelberg. [Google Scholar] [CrossRef]
- Coelli, S., R. Sclocco, R. Barbieri, G. Reni, C. Zucca, and A. M. Bianchi. 2015. EEG-based index for engagement level monitoring during sustained attention. In 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Milan: IEEE). 1512–1515. [Google Scholar]
- Di Nocera, F., M. Camilli, and M. Terenzi. 2007. A Random glance at the flight deck: Pilot’s scanning strategies and the real-time assessment of mental workload. Journal of cognitive engineering and decision making. [Google Scholar] [CrossRef]
- VIVE Developers. 2022. Eye and facial tracking SDK. https://developer-express.vive.com/resources/vive-sense/eye-and-facialtracking-sdk/ (accessed on 26.8.2022).
- EASA. 2021. EASA approves the first virtual reality (VR) based flight simulation training device. https://www.easa.europa.eu/newsroom-and-events/press-releases/easa-approves-first-virtual-reality-vr-based-flight-simulation (accessed on 18.8.2021).
- Emotiv. 2023. How are band powers calculated? (Accessed on 29/3/2023). https://www.emotiv.com/knowledge-base/how-band-powers-are-calculated/.
- Freeman, F. G., P. J. Mikulka, M. W. Scerbo, L. J. Prinzel, and K. Clouatre. 2000. Evaluation of a psychophysiologically Controlled Adaptive Automation System, Using Performance on a Tracking Task. Appl Psychophysiol Biofeedback 25, 02: 103–15. [Google Scholar] [CrossRef]
- Gevins, A., and M. E. Smith. 2003. Neurophysiological measures of cognitive workload during human–computer interaction. Theor. Issues Ergon. Sci.: vol. 4, no. 1–2, pp. 113–131. [Google Scholar]
- Galy, E., M. Cariou, and C. Mélan. 2012. What is the relationship between mental workload factors and cognitive load types? International Journal of Psychophysiology 83, 3: 269–275. [Google Scholar] [CrossRef]
- Hanson, C., J. Schaefer, J. J. Burken, D. Larson, and M. Johnson. 2014. Complexity and pilot workload metrics for the evaluation of adaptive flight controls on a fullscale piloted aircraft. NASA/TM-2014-216640. [Google Scholar]
- Hassan, M. A., and E. A. Mahmoud. 2015. A Comparison between Windowing FIR Filters for Extracting the EEG Components. Journal of Biosensors &Amp; Bioelectronics 06, 04. [Google Scholar] [CrossRef]
- Hebbar, P. A., K. Bhattacharya, G. Prabhakar, P. Biswas, and A. A. Pashilkar. 2021. Correlation Between Physiological and Performance-Based Metrics to Estimate Pilot’s Cognitive Workload. Frontiers in Psychology 12: pp 954. [Google Scholar] [CrossRef]
- Helliar, R. 2022. Measuring latency from radar interfacing to display. https://www.unmannedsystemstechnology.com/feature/measuring-latency-from-radar-interfacing-to-display/ (accessed on 29/8/2022).
- Holm, A., K. Lukander, J. Korpela, M. Sallinen, and K. M. I. Müller. 2009. Estimating Brain Load from the EEG. Sci. World J.: vol. 9, pp. 639–651. [Google Scholar] [CrossRef]
- Nowak, W., A. Hachol, and H. Kasprzak. 2008. Time-frequency analysis of spontaneous fluctuation of the pupil size of the human eye. Optica Applicata 38, 2: 469–480. https://search.ebscohost.com/login.aspx?direct=true&db=asx&AN=43828825&site=eds-live (accessed on February 6, 2023).
- Li, Y. F., J. P. Shi, W. Jiang, W. G. Zhang, and Y. Lyu. 2022. Autonomous maneuver decision-making for a UCAV in short-range aerial combat based on an MSDDQN algorithm. Def. Technol. 18: 1697–1714. [Google Scholar] [CrossRef]
- Liu, Y., K. Li, L. Chen, and Y. Liang. 2019. Novel augmented proportional navigation guidance law for midrange autonomous rendezvous. Acta Astronautica Vol 162: pp-526-535. [Google Scholar] [CrossRef]
- Liu, W., G. Andrade, J. Schulze, N. Doran, and K. E. Courtney. 2022. Using Virtual Reality to Induce and Assess Objective Correlates of Nicotine Craving: Paradigm Development Study. JMIR Serious Games 15;10, 1: e32243. [Google Scholar] [CrossRef] [PubMed] [PubMed Central]
- Liu, J. C., K. A. Li, S. L. Yeh, and S. Y. Chien. 2022. Assessing Perceptual Load and Cognitive Load by Fixation-Related Information of Eye Movements. Sensors (Basel) 22, 3: 1187. [Google Scholar] [CrossRef] [PubMed] [PubMed Central]
- Mitchell, D. G., K. A. Brian, and S. S. John. 1998. A flight investigation of pilot-induced oscillation due to rate limiting. In IEEE Aerospace Conference. [Google Scholar] [CrossRef]
- Mukhopadhyay, A., V. K. Sharma, P. G. Tatyarao, A. K. Shah, A. M. C. Rao, P. R. Subin, and P. Biswas. 2023. A comparison study between XR interfaces for driver assistance in take over request. Transportation Engineering, (11), 100159, Elsevier, ISSN: 2666-691X. ISSN 2666-691X. [Google Scholar] [CrossRef]
- Walter, K., and P. Bex. 2021. Cognitive load influences oculomotor behavior in natural scenes. Scientific Reports 11, no. 1: 1–12. [Google Scholar] [CrossRef] [PubMed]
- Oberhauser, M., D. Dreyer, R. Braunstingl, and I. Koglbauer. 2018. What’s real about virtual reality flight simulation? Comparing the fidelity of a virtual reality with a conventional flight simulation environment. Aviation Psychology and Applied Human Factors 8, 1: 22–34. [Google Scholar] [CrossRef]
- Palinko, O., A. L. Kun, and A. Shyrokov. 2010. Estimating cognitive load using remote eye tracking in a driving simulator. In Proceedings of Symposium on Eye-Tracking Research & Applications, ETRA 2010, Austin, Texas, USA. [Google Scholar]
- Pedrotti, M., M., A. Mirzaei, A. Tedescho, J. Chardonnet, F. Merienne, and et al. 2014. Automatic stress classification with pupil diameter analysis. International Journal of Human-Computer Interaction. Taylor & Francis. 30, (3), 220–236. [Google Scholar] [CrossRef]
- Petkar, H., S. Dande, R. Yadav, Y. Zeng, and T. A. Nguyen. 2009. A pilot study to assess designer’s mental stress using eye gaze system and electroencephalogram. In Proceedings of the ASME 2009 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, San Diego, California, USA. [Google Scholar]
- Pope, A. T., E. H. Bogart, and D. S. Bartolome. 1995. Biocybernetic system evaluates indices of operator engagement in automated task. Biol. Psychol. 40: 187–195. [Google Scholar] [CrossRef]
- Prinzel, L., F. Freeman, M. Scerbo, P. Mikulka, and A. Pope. 2003. Effects of a Psychophysiological System for Adaptive Automation on performance, workload, and the Event-Related Potential P300 Component. Hum. Factors: vol. 45, no. 4, pp. 601–613. [Google Scholar] [CrossRef]
- R Core Team. 2017. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing. Available online at: https://www.R-project.org/.
- Robinson, T. 2018. https://www.aerosociety.com/news/wearable-cockpits-the-ultimate-human-machine-interface (accessed on 18/8/2021).
- Rowen, A., M. Grabowski, and J. Rancy. 2019. Through the Looking Glass(es): Impacts of Wearable Augmented Reality Displays on Operators in a Safety-Critical System. in IEEE Transactions on Human-Machine Systems 49, 6: 652–660. [Google Scholar] [CrossRef]
- Marshall, S. P. 2002. The Index of Cognitive Activity: Measuring cognitive workload. In Proceedings of the IEEE 7th Conference on Human Factors and Power Plants, pp.7-7, Scottsdale, USA. [Google Scholar] [CrossRef]
- Škvareková, I., P. Pecho, V. Ažaltovic, and B. Kandera. 2020. Number of saccades and fixation duration as indicators of pilot. In Transportation Research Procedia 51: 67–74. [Google Scholar] [CrossRef]
- Stinton, D. 1996. Flying Qualities and Flight Testing of the Airplane. American Institute of Aeronautics and Astronautics, Inc.: p. 265. [Google Scholar]
- Szafir, D., and B. Mutlu. 2012. Designing adaptive agents that monitor and improve user engagement. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Austin Texas USA: ACM. [Google Scholar] [CrossRef]
- Tobii XR Devzone. 2022. HTC Vive pro eye development guide. https://vr.tobii.com/sdk/develop/unity/gettingstarted/vive-pro-eye/ (accessed on 26/8/2022).
- Vora, J., S. Nair, A. K. Gramopadhye, A. T. Duchowski, B. J. Melloy, and B. Kanki. 2002. Using virtual reality technology for aircraft visual inspection training: presence and comparison studies. Applied Ergonomics 33: 559–570. [Google Scholar] [PubMed]
- Wang, Z., H. Li, H. Wu, and Z. Wu. 2020. Improving Maneuver Strategy in Air Combat by Alternate Freeze Games with a Deep Reinforcement Learning Algorithm. Mathematical Problems in Engineering 2020, 1: 1–17. [Google Scholar] [CrossRef]
- Williams, N. S., G. M. McArthur, D. B. Wit, G Ibrahim, and N. A. Badcock. 2020. A validation of Emotiv EPOC Flex saline for EEG and ERP research. PeerJ 8: e9713. [Google Scholar] [CrossRef] [PubMed] [PubMed Central]
- Xie, J., G. Xu, J. Wang, M. Li, C. Han, and Y. Jia. 2016. Effects of mental load and fatigue on steady-state evoked potential based brain computer interface tasks: a comparison of periodic flickering and motion-reversal based visual attention. PLoS ONE 11: e0163426. [Google Scholar] [CrossRef]
- Zhang, J., Z. Yin, and R. Wang. 2017. Design of an Adaptive Human-Machine System Based on Dynamical Pattern Recognition of Cognitive TaskLoad. Front. Neurosci.: vol. 11. [Google Scholar] [CrossRef]
© 2023 by the authors. This article is licensed under a Creative Commons Attribution 4.0 International License.
Share and Cite
Hebbar, P.A.; Vinod, S.; Shah, A.K.; Pashilkar, A.A.; Biswas, P. Cognitive Load Estimation in VR Flight Simulator. J. Eye Mov. Res. 2022, 15, 1-16. https://doi.org/10.16910/jemr.15.3.11
Hebbar PA, Vinod S, Shah AK, Pashilkar AA, Biswas P. Cognitive Load Estimation in VR Flight Simulator. Journal of Eye Movement Research. 2022; 15(3):1-16. https://doi.org/10.16910/jemr.15.3.11
Chicago/Turabian StyleHebbar, P Archana, Sanjana Vinod, Aumkar Kishore Shah, Abhay A Pashilkar, and Pradipta Biswas. 2022. "Cognitive Load Estimation in VR Flight Simulator" Journal of Eye Movement Research 15, no. 3: 1-16. https://doi.org/10.16910/jemr.15.3.11
APA StyleHebbar, P. A., Vinod, S., Shah, A. K., Pashilkar, A. A., & Biswas, P. (2022). Cognitive Load Estimation in VR Flight Simulator. Journal of Eye Movement Research, 15(3), 1-16. https://doi.org/10.16910/jemr.15.3.11