Hidden Semi-Markov Models-Based Visual Perceptual State Recognition for Pilots
Abstract
:1. Introduction
2. Related Work
2.1. Pilot Visual Scanning Strategy
2.2. Application of HSMM
3. Model Structure for Pilot State Identification
3.1. cSPADE-Based Sequence Pattern Mining
Algorithm 1. The cSPADE algorithm |
1. 2. Enumerate-Frequent(S): 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. Enumerate-Frequent (T); 15. delete S; |
3.2. Pilot Perceptual Behavior Modeling
3.2.1. Visual Perception Based on HSMM Identification of Pilots
- (1)
- Suppose the hidden-state sequence is a first-order Markov chain:
- (2)
- The probability of an observation occurring in the system, conditional on the state being at moment t, is:
- (3)
- The probability that the system resides in state Si at moment t for time d is:
3.2.2. A Forward–Backward Solution Algorithm for the HSMM
4. Experimental Design and Analysis of Results
4.1. Experimental Equipment
4.2. Task Setting and Data Collection
4.2.1. Task Setting
4.2.2. Data Collection
4.3. Analysis of Results
5. Discussion
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Kearney, P.; Li, W.-C.; Lin, J.J. The impact of alerting design on air traffic controllers’ response to conflict detection and resolution. Int. J. Ind. Ergon. 2016, 56, 51–58. [Google Scholar] [CrossRef]
- Allsop, J.; Gray, R. Flying under pressure: Effects of anxiety on attention and gaze behavior in aviation. J. Appl. Res. Mem. Cogn. 2014, 3, 63–71. [Google Scholar] [CrossRef]
- Martinez-Marquez, D.; Pingali, S.; Panuwatwanich, K.; Stewart, R.A.; Mohamed, S. Application of eye tracking technology in aviation, maritime, and construction industries: A systematic review. Sensors 2021, 21, 4289. [Google Scholar] [CrossRef] [PubMed]
- Naeeri, S.; Kang, Z.; Mandal, S.; Kim, K. Multimodal Analysis of Eye Movements and Fatigue in a Simulated Glass Cockpit Environment. Aerospace 2021, 8, 283. [Google Scholar] [CrossRef]
- Zaki, M.J. Sequence mining in categorical domains: Incorporating constraints. In Proceedings of the Ninth International Conference on Information and Knowledge Management, McLean, VA, USA, 6–11 November 2000; pp. 422–429. [Google Scholar]
- Sullivan, J.; Yang, J.H.; Day, M.; Kennedy, Q. Training simulation for helicopter navigation by characterizing visual scan patterns. Aviat. Space Environ. Med. 2011, 82, 871–878. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Fitts, P.M.; Jones, R.E. Eye Fixation of Aircraft Pilots, III. Frequency, Duration, and Sequence Fixations When Flying Air Force Ground Controlled Approach System; AF-5967, Air Materiel Command Wright-Patterson AFB OH: Dayton, OH, USA, 1949. [Google Scholar]
- Lefrancois, O.; Matton, N.; Gourinat, Y.; Peysakhovich, V.; Causse, M. The role of Pilots’ Monitoring Strategies in Flight Performance. In Proceedings of the EAAP32, European Association for Aviation Psychology Conference, Amsterdam, The Netherlands, 27–28 September 2023. [Google Scholar]
- Bellenkes, A.H.; Wickens, C.D.; Kramer, A. Visual scanning and pilot expertise: The role of attentional flexibility and mental model development. Aviat. Space Environ. Med. 1997, 68, 569–579. [Google Scholar] [PubMed]
- Kasarskis, P.; Stehwien, J.; Hickox, J.; Aretz, A.; Wickens, C. Comparison of expert and novice scan behaviors during VFR flight. In Proceedings of the 11th International Symposium on Aviation Psychology, Columbus, OH, USA, 5–8 March 2001. [Google Scholar]
- Lefrançois, O.; Matton, N.; Causse, M. Improving Airline Pilots’ Visual Scanning and Manual Flight Performance through Training on Skilled Eye Gaze Strategies. Safety 2021, 7, 70. [Google Scholar] [CrossRef]
- Fox, E.B.; Sudderth, E.B.; Jordan, M.I.; Willsky, A.S. An HDPHMM for systems with state persistence. In Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, 5–9 July 2008; pp. 312–319. [Google Scholar]
- Xiao, H.; Zeng, H.; Jiang, W.; Zhou, Y.; Tu, X. HMM-TCN-based health assessment and state prediction for robot mechanical axis. Int. J. Intell. Syst. 2022, 37, 10476–10494. [Google Scholar] [CrossRef]
- Xiao, R.; Yu, X.; Shi, R.; Zhang, Z.; Yu, W.; Li, Y.; Chen, G.; Gao, J. Ecosystem health monitoring in the Shanghai-Hangzhou Bay Metropolitan Area: A hidden Markov modeling approach. Environ. Int. 2019, 133, 105170. [Google Scholar] [CrossRef] [PubMed]
- Wang, X.; Geng, X.; Wang, J.; Tamura, S. A comparative research on g-hmm and tss technologies for eye movement tracking analysis. An enhanced Hidden Semi-Markov model for anics in Medicine and Biology. J. Mech. Med. Biol. 2021, 21, 2140023. [Google Scholar] [CrossRef]
- Su, Y.; Lloyd, L.; Chen, X.; Chase, J.G. Latency mitigation using applied HMMs for mixed reality-enhanced intuitive teleoperation in intelligent robotic welding. Int. J. Adv. Manuf. Technol. 2023, 126, 2233–2248. [Google Scholar] [CrossRef]
- Zhu, K.; Liu, T. Online Tool Wear Monitoring via Hidden Semi-Markov Model with Dependent Durations. IEEE Trans. Ind. Inform. 2017, 14, 69–78. [Google Scholar] [CrossRef]
- Yang, W.; Chen, L. Machine condition recognition via hidden semi-Markov model. Comput. Ind. Eng. 2021, 158, 107430. [Google Scholar] [CrossRef]
- Wang, K.J.; Lin, C.J.; Tadesse, A.A.; Woldegiorgis, B.H. Modeling of human–robot collaboration for flexible assembly—A hidden semi-Markov-based simulation approach. Int. J. Adv. Manuf. Technol. 2023, 126, 5371–5389. [Google Scholar] [CrossRef]
- Manoharan, G. Sivaoutlier detection in multivariate datasets. J. Intell. Fuzzy Syst. 2022, 1–7. [Google Scholar]
- Tian, X.; Wei, G.; Wang, J. Target Location Method Based on Compressed Sensing in Hidden Semi Markov Model. Electronics 2022, 11, 1715. [Google Scholar] [CrossRef]
- Khaleghei, A.; Makis, V. Model parameter estimation and residual life prediction for a partially observable failing system. Nav. Res. Logist. (NRL) 2015, 62, 190–205. [Google Scholar] [CrossRef]
- Li, F.; Zheng, W.X.; Xu, S. Stabilization of discrete-time hidden semi-Markov jump singularly perturbed systems with partially known emission probabilities. IEEE Trans. Autom. Control. 2021, 67, 4234–4240. [Google Scholar] [CrossRef]
- Liu, Q.; Xu, S.; Lu, C.; Yao, H.; Chen, H. Early Recognition of Driving Intention for Lane Change Based on Recurrent Hidden Semi-Markov Model. IEEE Trans. Veh. Technol. 2020, 69, 10545–10557. [Google Scholar] [CrossRef]
- Lin, M.; Wanqing, S.; Chen, D.; Zio, E. Evolving Connectionist System and Hidden Semi-Markov Model for Learning-Based Tool Wear Monitoring and Remaining Useful Life Prediction. IEEE Access 2022, 10, 82469–82482. [Google Scholar] [CrossRef]
N1 | N2 | N3 | P/% | R/% | |
---|---|---|---|---|---|
HMM | 72 | 18 | 15 | 80.00 | 82.76 |
HSMM | 89 | 10 | 9 | 93.55 | 91.58 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Gao, L.; Wang, C.; Wu, G. Hidden Semi-Markov Models-Based Visual Perceptual State Recognition for Pilots. Sensors 2023, 23, 6418. https://doi.org/10.3390/s23146418
Gao L, Wang C, Wu G. Hidden Semi-Markov Models-Based Visual Perceptual State Recognition for Pilots. Sensors. 2023; 23(14):6418. https://doi.org/10.3390/s23146418
Chicago/Turabian StyleGao, Lina, Changyuan Wang, and Gongpu Wu. 2023. "Hidden Semi-Markov Models-Based Visual Perceptual State Recognition for Pilots" Sensors 23, no. 14: 6418. https://doi.org/10.3390/s23146418
APA StyleGao, L., Wang, C., & Wu, G. (2023). Hidden Semi-Markov Models-Based Visual Perceptual State Recognition for Pilots. Sensors, 23(14), 6418. https://doi.org/10.3390/s23146418