Two Hours in Hollywood: A Manually Annotated Ground Truth Data Set of Eye Movements During Movie Clip Watching
Abstract
:Introduction
Methods
Eye movement definitions
Labelling procedure
Results
Basic statistics
Evaluation of classification algorithms
Sample-level F1 | Event-level F1 | ||||||
Model | F1 average | Fixation | Saccade | SP | Fixation | Saccade | SP |
1D CNN-BLSTM Startsev et al. (2019a) | 0.787 | 0.872 | 0.827 | 0.680 | 0.808 | 0.946 | 0.588 |
sp tool + Startsev and Dorr (2019) | 0.755 | 0.853 | 0.816 | 0.617 | 0.820 | 0.905 | 0.516 |
REMoDNaV (Dar et al., 2019) | 0.748 | 0.779 | 0.755 | 0.622 | 0.784 | 0.931 | 0.615 |
sp_tool (Agtzidis et al., 2016b) | 0.703 | 0.819 | 0.815 | 0.616 | 0.587 | 0.900 | 0.483 |
(Dorr et al., 2010) | 0.685 | 0.832 | 0.796 | 0.373 | 0.821 | 0.884 | 0.403 |
(Larsson et al., 2015) | 0.647 | 0.796 | 0.803 | 0.317 | 0.807 | 0.886 | 0.274 |
(Berg et al., 2009) | 0.601 | 0.824 | 0.729 | 0.137 | 0.845 | 0.826 | 0.243 |
I-VMP San Agustin (2010) | 0.564 | 0.726 | 0.688 | 0.564 | 0.503 | 0.563 | 0.338 |
I-KF Sauter et al. (1991) | 0.523 | 0.816 | 0.770 | – | 0.748 | 0.803 | – |
I-VDT Komogortsev and Karpov (2013) | 0.504 | 0.813 | 0.700 | 0.136 | 0.557 | 0.559 | 0.263 |
I-HMM Salvucci and Anderson (1998) | 0.480 | 0.811 | 0.720 | – | 0.646 | 0.700 | – |
I-DT Salvucci and Goldberg (2000) | 0.473 | 0.803 | 0.486 | – | 0.744 | 0.802 | – |
I-VT Salvucci and Goldberg (2000) | 0.432 | 0.810 | 0.705 | – | 0.520 | 0.555 | – |
I-VVT Komogortsev and Karpov (2013) | 0.390 | 0.751 | 0.705 | 0.247 | 0.061 | 0.555 | 0.023 |
I-MST Goldberg and Schryver (1995) | 0.385 | 0.793 | 0.349 | – | 0.590 | 0.576 | – |
Discussion
Data set statistics
Combination with other data sets
Conclusion
Ethics and Conflict of Interest
Acknowledgments
References
- Agtzidis, I., M. Startsev, and M. Dorr. 2016a. In the pursuit of (ground) truth: A hand-labelling tool for eye movements recorded during dynamic scene viewing. In 2016 IEEE Second Workshop on Eye Tracking and Visualization (ETVIS); pp. 65–68. [Google Scholar] [CrossRef]
- Agtzidis, I., M. Startsev, and M. Dorr. 2016b. Smooth pursuit detection based on multiple observers. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, ETRA ’16; pp. 303–306. [Google Scholar] [CrossRef]
- Agtzidis, I., M. Startsev, and M. Dorr. 2019. 360-degree video gaze behaviour: A ground-truth data set and a classification algorithm for eye movements. In Proceedings of the 27th ACM International Conference on Multimedia, MM ’19; pp. 1007–1015. [Google Scholar] [CrossRef]
- Alers, H., J. A. Redi, and I. Heynderickx. 2012. Examining the effect of task on viewing behavior in videos using saliency maps. In Human Vision and Electronic Imaging XVII. volume 8291, pp. 1–9. International Society for Optics and Photonics. [Google Scholar] [CrossRef]
- Andersson, R., L. Larsson, K. Holmqvist, M. Stridh, and M. Nyström. 2017. One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms. Behavior Research Methods 49, 2: 616–637. [Google Scholar] [CrossRef] [PubMed]
- Bahill, A. T., M. R. Clark, and L. Stark. 1975. The main sequence, a tool for studying human eye movements. Mathematical Biosciences 24, 3: 191–204. [Google Scholar] [CrossRef]
- Berg, D. J., S. E. Boehnke, R. A. Marino, D. P. Munoz, and L. Itti. 2009. Free viewing of dynamic stimuli by humans and monkeys. Journal of Vision 9, 5: 1–15. [Google Scholar] [CrossRef] [PubMed]
- Dar, A. H., A. S. Wagner, and M. Hanke. 2019. REMoDNaV: Robust eye movement detection for natural viewing. bioRxiv. [Google Scholar] [CrossRef]
- David, E. J., J. Gutiérrez, A. Coutrot, M. P. Da Silva, and P. L. Callet. 2018. A dataset of head and eye movements for 360 ◦ videos. In Proceedings of the 9th ACM Multimedia Systems Conference, MMSys ’18; pp. 432–437. [Google Scholar] [CrossRef]
- Dorr, M., T. Martinetz, K. R. Gegenfurtner, and E. Barth. 2010. Variability of eye movements when viewing dynamic natural scenes. Journal of Vision 10, 10: 28–28. [Google Scholar] [CrossRef]
- Georgescu, A. L., B. Kuzmanovic, L. Schilbach, R. Tepest, R. Kulbida, G. Bente, and K. Vogeley. 2013. Neural correlates of “social gaze” processing in high-functioning autism under systematic variation of gaze duration. NeuroImage: Clinical 3: 340–351. [Google Scholar] [CrossRef]
- Goldberg, J. H., and J. C. Schryver. 1995. Eye-gaze-contingent control of the computer interface: Methodology and example for zoom detection. Behavior Research Methods, Instruments, & Computers 27, 3: 338–350. [Google Scholar] [CrossRef]
- Hanke, M., N. Adelhöfer, D. Kottke, V. Iacovella, A. Sengupta, F. R. Kaule, R. Nigbur, A. Q. Waite, F. Baumgartner, and J. Stadler. 2016. A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation. Scientific data 3: 160092. [Google Scholar] [CrossRef] [PubMed]
- Henderson, J. M., and W. Choi. 2015. Neural correlates of fixation duration during real-world scene viewing: Evidence from fixation-related (FIRE) fMRI. Journal of Cognitive Neuroscience 27, 6: 1137–1145. [Google Scholar] [CrossRef]
- Hessels, R. S., D. C. Niehorster, M. Nyström, R. Andersson, and I. T. C. Hooge. 2018. Is the eye-movement field confused about fixations and saccades? A survey among 124 researchers. Royal Society Open Science 5, 8: 180502. [Google Scholar] [CrossRef]
- Hooge, I. T. C., R. S. Hessels, and M. Nyström. 2019. Do pupil-based binocular video eye trackers reliably measure vergence? Vision Research 156: 1–9. [Google Scholar] [CrossRef] [PubMed]
- Hooge, I. T. C., D. C. Niehorster, M. Nyström, R. Andersson, and R. S. Hessels. 2017. Is human classification by experienced untrained observers a gold standard in fixation detection? Behavior Research Methods 50, 5: 1864–1881. [Google Scholar] [CrossRef] [PubMed]
- Hooge, I. T. C., M. Nyström, T. Cornelissen, and K. Holmqvist. 2015. The art of braking: Post saccadic oscillations in the eye tracker signal decrease with increasing saccade size. Vision Research 112: 55–67. [Google Scholar] [CrossRef]
- Itti, L., and R. Carmi. 2009. Eye-tracking data from human volunteers watching complex video stimuli. [Google Scholar] [CrossRef]
- Jiang, L., M. Xu, T. Liu, M. Qiao, and Z. Wang. 2018. DeepVS: A deep learning based video saliency prediction approach. Edited by V. Ferrari, M. Hebert, C. Sminchisescu and Y. Weiss. In Computer Vision–ECCV 2018. pp. 625–642. [Google Scholar] [CrossRef]
- Komogortsev, O. V. 2014. Eye movement classification software. http://cs.txstate.edu/~ok11/emd_offline.html.
- Komogortsev, O. V., D. V. Gobert, S. Jayarathna, D. H. Koh, and S. M. Gowda. 2010. Standardization of automated analyses of oculomotor fixation and saccadic behaviors. IEEE Transactions on Biomedical Engineering 57, 11: 2635–2645. [Google Scholar] [CrossRef]
- Komogortsev, O. V., and A. Karpov. 2013. Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades. Behavior Research Methods 45, 1: 203–215. [Google Scholar] [CrossRef]
- Larsson, L., M. Nyström, R. Andersson, and M. Stridh. 2015. Detection of fixations and smooth pursuit movements in high-speed eye-tracking data. Biomedical Signal Processing and Control 18: 145–152. [Google Scholar] [CrossRef]
- Larsson, L., M. Nyström, and M. Stridh. 2013. Detection of saccades and postsaccadic oscillations in the presence of smooth pursuit. IEEE Transactions on Biomedical Engineering 60, 9: 2484–2493. [Google Scholar] [CrossRef] [PubMed]
- Leboran, V., A. Garcia-Diaz, X. R. Fdez-Vidal, and X. M. Pardo. 2017. Dynamic whitening saliency. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 5: 893–907. [Google Scholar] [CrossRef]
- Lencer, R., and P. Trillenberg. 2008. Neurophysiology and neuroanatomy of smooth pursuit in humans. Brain and Cognition 68, 3: 219–228. [Google Scholar] [CrossRef]
- Luna, B., K. R. Thulborn, M. H. Strojwas, B. J. McCurtain, R. A. Berman, C. R. Genovese, and J. A. Sweeney. 1998. Dorsal cortical regions subserving visually guided saccades in humans: an fMRI study. Cerebral Cortex 8, 1: 40–47. [Google Scholar] [CrossRef] [PubMed]
- Mathe, S., and C. Sminchisescu. 2012. Dynamic eye movement datasets and learnt saliency models for visual action recognition. In European Conference on Computer Vision; pp. 842–856. [Google Scholar] [CrossRef]
- Meyer, C. H., A. G. Lasker, and D. A. Robinson. 1985. The upper limit of human smooth pursuit velocity. Vision Research 25, 4: 561–563. [Google Scholar] [CrossRef]
- Ohlendorf, S., A. Sprenger, O. Speck, V. Glauche, S. Haller, and H. Kimmig. 2010. Visual motion, eye motion, and relative motion: a parametric fMRI study of functional specializations of smooth pursuit eye movement network areas. Journal of Vision 10, 14: 21–21. [Google Scholar] [CrossRef] [PubMed]
- Petit, L., and J. V. Haxby. 1999. Functional anatomy of pursuit eye movements in humans as revealed by fMRI. Journal of Neurophysiology 82, 1: 463–471. [Google Scholar] [CrossRef]
- Salvucci, D. D., and J. R. Anderson. 1998. Tracing eye movement protocols with cognitive process models. In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society; pp. 923–928. [Google Scholar]
- Salvucci, D. D., and J. H. Goldberg. 2000. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications, ETRA ’00; pp. 71–78. [Google Scholar] [CrossRef]
- San Agustin, J. 2010. Off-the-shelf gaze interaction. PhD thesis, IT-Universitetet i København. [Google Scholar]
- Santini, T., W. Fuhl, T. Kübler, and E. Kasneci. 2016. Bayesian identification of fixations, saccades, and smooth pursuits. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, ETRA ’16; pp. 163–170. [Google Scholar] [CrossRef]
- Sauter, D., B. J. Martin, N. Di Renzo, and C. Vomscheid. 1991. Analysis of eye tracking movements using innovations generated by a Kalman filter. Medical and Biological Engineering and Computing 29, 1: 63–69. [Google Scholar] [CrossRef] [PubMed]
- Schenk, S., P. Tiefenbacher, G. Rigoll, and M. Dorr. 2016. SPOCK: A smooth pursuit oculomotor control kit. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems; pp. 2681–2687. [Google Scholar] [CrossRef]
- Sestieri, C., V. Pizzella, F. Cianflone, G. L. Romani, and M. Corbetta. 2008. Sequential activation of human oculomotor centers during planning of visually-guided eye movements: a combined fMRI-MEG study. Frontiers in Human Neuroscience 2. [Google Scholar] [CrossRef]
- Startsev, M., I. Agtzidis, and M. Dorr. 2019a. 1D CNN with BLSTM for automated classification of fixations, saccades, and smooth pursuits. Behavior Research Methods 51, 2: 556–572. [Google Scholar] [CrossRef]
- Startsev, M., I. Agtzidis, and M. Dorr. 2019b. Characterizing and automatically detecting smooth pursuit in a large-scale ground-truth data set of dynamic natural scenes. Journal of Vision 19, 14: 10–10. [Google Scholar] [CrossRef]
- Startsev, M., and M. Dorr. 2019. Improving the state of the art in eye movement event detection via trainable label correction. In The 20th European Conference on Eye Movements: Abstract book, ECEM ’19; pp. 135–135. [Google Scholar] [CrossRef]
- Startsev, M., and M. Dorr. 2020. Supersaliency: A novel pipeline for predicting smooth pursuit-based attention improves generalisability of video saliency. IEEE Access 8: 1276–1289. [Google Scholar] [CrossRef]
- Steil, J., M. X. Huang, and A. Bulling. 2018. Fixation detection for head-mounted eye tracking based on visual similarity of gaze targets. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA ’18; pp. 23:1–23:9. [Google Scholar] [CrossRef]
- Thibaut, M., C. Delerue, M. Boucart, and T.-H.-C. Tran. 2016. Visual exploration of objects and scenes in patients with age-related macular degeneration. Journal Français d’Ophtalmologie 39, 1: 82–89. [Google Scholar] [CrossRef]
- Tseng, P.-H., I. G. M. Cameron, G. Pari, J. N. Reynolds, D. P. Munoz, and L. Itti. 2013. High-throughput classification of clinical populations from natural viewing eye movements. Journal of Neurology 260, 1: 275–284. [Google Scholar] [CrossRef] [PubMed]
- Vidal, M., K. Pfeuffer, A. Bulling, and H. W. Gellersen. 2013. Pursuits: eye-based interaction with moving targets. In Proceedings of the 2013 CHI Conference Extended Abstracts on Human Factors in Computing Systems; pp. 3147–3150. [Google Scholar] [CrossRef]
- Wang, W., J. Shen, F. Guo, M.-M. Cheng, and A. Borji. 2018. Revisiting video saliency: A large-scale benchmark and a new model. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR); pp. 4894–4903. [Google Scholar] [CrossRef]
- Zemblys, R., D. C. Niehorster, O. Komogortsev, and K. Holmqvist. 2018. Using machine learning to detect events in eye-tracking data. Behavior Research Methods 50, 1: 160–181. [Google Scholar] [CrossRef] [PubMed]
© 2020 by the authors. This article is licensed under a Creative Commons Attribution 4.0 International License.
Share and Cite
Agtzidis, I.; Startsev, M.; Dorr, M. Two Hours in Hollywood: A Manually Annotated Ground Truth Data Set of Eye Movements During Movie Clip Watching. J. Eye Mov. Res. 2020, 13, 1-16. https://doi.org/10.16910/jemr.13.4.5
Agtzidis I, Startsev M, Dorr M. Two Hours in Hollywood: A Manually Annotated Ground Truth Data Set of Eye Movements During Movie Clip Watching. Journal of Eye Movement Research. 2020; 13(4):1-16. https://doi.org/10.16910/jemr.13.4.5
Chicago/Turabian StyleAgtzidis, Ioannis, Mikhail Startsev, and Michael Dorr. 2020. "Two Hours in Hollywood: A Manually Annotated Ground Truth Data Set of Eye Movements During Movie Clip Watching" Journal of Eye Movement Research 13, no. 4: 1-16. https://doi.org/10.16910/jemr.13.4.5
APA StyleAgtzidis, I., Startsev, M., & Dorr, M. (2020). Two Hours in Hollywood: A Manually Annotated Ground Truth Data Set of Eye Movements During Movie Clip Watching. Journal of Eye Movement Research, 13(4), 1-16. https://doi.org/10.16910/jemr.13.4.5