For HLTHPSYC 715 Research Methods in Health Psychology, three postgraduate students plus the course coordinator viewed the images and completed the exam question. Their eye gaze data and scores are summarised in Table 1
. The results indicate that the course coordinator spent longer continuous periods of time on the model summary in particular, as well as the coefficients table than the students who tended to shift their gaze more frequently. The course coordinator also spent the longest total time on the task, with students spending less than the 12 min that they should have allocated to the question.
These results suggest that the students may not be as familiar with where to look for the relevant numbers that are necessary to answer the question. The main areas the students missed out on marks were from (i) not reporting the coefficients table sufficiently and (ii) incorrectly copying numbers from the model summary and ANOVA tables. This suggests that more time should be spent teaching students to look at these tables in particular ways, and which numbers to use.
As a control experiment, for MEDIA 327/222 Comics and Visual Narrative, we chose pages from a 1939 Superman story that, aside from the half-page title panel, obeyed a regular eight-panel page (Figure 2
). To this we added examples of contemporary comics that, while observing elements of the iconic grid, also presented specific issues such as reading across a double-page (Figure 3
). We chose these because we were especially interested to examine the tension between reading and looking at images. Reading includes the linear and disciplined reading of captions and speech balloons as well as following the conventional Z-shaped navigation through the page. By contrast, scanning often involves looking at the whole page, or parts of the page, before reading properly starts. This is akin to the scanpaths mentioned in the section on art above. As James Elkins has noted [34
], this play between reading and looking takes place in any visual medium, but it is especially interesting in a medium where a set of differentiated units that form a serial narrative are all available to view at once.
The early Superman comics (Figure 3
) produced the most disciplined readings. The first page is a half-page title panel with four symmetrical panels beneath it, while page two is a regular grid of eight symmetrical panels. Here, the large title panel immediately drew the eye, but it was interesting to see how viewers moved around the visual element of the large panel before moving to the text, with two of the readers even using the image of Superman on the left-hand side of the panel to scan a vector up to the first panel of page two and down to the second panel of page one before alighting on the first text box. Even when the reading had properly begun, though, it was interesting to see the play between reading the textual component of the panel and scanning the visual component, with readers clearly setting up rhythms as they moved between the two planes of signification. It was also notable how, even in this relatively straightforward story, readers often moved backwards to revisit material as a result of acquiring new information in the panel they had just read.
This ebb and flow was even more pronounced in the double-page taken from The Homeland Directive (Figure 4
). Here the reader is asked to follow a strip across both pages before moving down. It is also a highly dynamic page as the majority of it depicts an attempted assassination in a hotel room. The requirement to move across the page did not cause a problem for viewers, but what was noticeable was that without having already established a reading rhythm and without a strong opening panel (such as the titular panel in the Superman comic), many readers started close to the middle of the double page before moving towards the start point in the top left-hand corner, with one viewer actually spending some time digesting the final panel (bottom right) before moving to the top of the left-hand page. Additionally, of interest here is the fact that the pages were not text-heavy. While the Z-movement is one way of establishing reading direction, the positioning of text is another key component used to direct the movement of the eye. In the absence of this component, the scanpaths became much more pronounced and also idiosyncratic, although it was possible to detect circular movements around sections of four of five panels as different viewers sought to make sense of the co-presence of elements. Using eye-tracking technology for the comics course provided an excellent heuristic device and an important pedagogical tool in that we can now show students exactly what people do when they “read” comics. It perfectly illustrates the ebb and flow across current and previous elements within a series, the play between textual and visual components within a panel, the difference between reading and looking (and often the priority of looking), the irregularity and non-linear navigation of the page and the significance of all the elements being co-present and available to be “read” simultaneously. It was also really useful for getting students to think about the page or the double-page as a single aesthetic unit that, much like a painting, sets up dynamic relationships among elements that can all be seen together. It thus provides an important tool in both the study of the medium and the creation of comics.
For CM20216 Designing Interactive Systems, it was surprising to see how casual users simply overlook interface elements such as buttons if they are not clearly marked or if they are buried in visual clutter. Gaze recordings showed this quite nicely and prepared aspiring user interface designers. Experts, on the other hand, have their own systematic way of looking at user interfaces, and again, this could be useful for students to learn from. The contrast between baseline, novice gaze recordings and expert gaze recordings was used as a learning tool or diagnostic exercise, with the option of examining the gaze behaviour of students who have learned from recordings to see what tips they have picked up. The gaze tracking videos of users performing tasks on well-designed and ill-designed web user interfaces helped explain principles of interaction design with the use of relevant examples of good and bad practice, underlining the positive pedagogical efficacy of this eye tracking technology. This ability to reflect on teaching practice was also noted for the comics course, where the lecturer felt that the structural composition of various images affected eye behaviour, and that this could be discussed as part of the lecture. While there were aspects of invariant behaviour, there were also some highly personalised strategies adopted to inspect images and the interaction between text and images, as with interactive systems, was highly complex, but revealed interesting interplay between nonlinear (pictorial) and linear (textual) searching and reading.
This was also the case for the HLTHPSYC 715 Research Methods in Health Psychology, where students struggled to read and understand complex charts (Figure 5
) and did not properly focus on the correct numerical information, contrasting greatly with the expert’s eye behaviour (Figure 6
). The video showing novices’ mistakes and the expert’s examination of the images was an important and effective teaching tool for the lecturer, clearly demonstrating how students could improve their understanding of the data by adopting, in a mindful manner, the expert’s repeated eye movements—horizontally with very little vertical distractions.
In PSYCH 303 Cognitive Science, the lecturer played back the video to students to show the principle of change blindness. Visual attention is a central topic of this course. Integrating eye movement recordings within the teaching of this topic improved pedagogical practice by adding a directly experiential element. This was especially useful in the context of discussing topics such as inattentional blindness and change blindness. The former refers to the dramatic failures of perception that can be seen when an individual’s attention is engaged on another task. Inattentional blindness appears to be a major component of the driver distraction effects caused by conversing on hands-free or hand-held cell phones while driving. Students remarked on how one can verbally describe this phenomenon, but the video provided a direct and dramatic experiential element to this phenomenon (see Table 2
), so that one could “feel” how it is possible to overlook something by following the movements of groups of people. Importantly one could also see that, from the eye tracking, subjects actually looked at the gorilla in the room, but did not register it, because they were paying attention to the actions of the group, passing the basketball to each other.
In ARTHIST 231/331 Framing the Viewer, eye tracking technology revealed how viewers adopted various nonlinear scanpaths for viewing images and how long they dwelled on these aspects; it was also important to see what was ignored. The patterns of the scanpaths focussed on Picasso’s Guernica
(1935) (Figure 7
) show a very consistent pattern of oscillatory rhythms from left to right among the four novices recorded. Remarkably, all four individuals ignored the same dark areas, in-between figures, and fixated on facial details and areas of higher luminosity. However, the expert also had more varied and configual (that is, general) viewing habits which included peripheral and darker areas and relationships that pick out a deeper understanding of structural possibilities beyond the obviously salient features. These findings are consistent with Zangemeister et al. (1995) [35
], who studied the eye behaviour of art experts and novices, and found that those with little experience of viewing artworks tended to move their eyes across shorter distances, particularly when viewing abstract art, compared to those more acquainted with art who moved their eyes over the whole picture. This is also what artists tend to do [36
], which suggests that configual viewing takes in the broader aspects of a composition. This coheres well with results presented by Cela-Conde et al. which show that experts process configural and global shapes and forms rather than fixating on particular details [37
]. The expert is able to focus on particular fine-grained structures when required, while working memory keeps the global representation in mind. In Rudolph Arnheim’s gestalt psychology, the mind naturally seeks integrated structures as a form of visual behaviour, and vision automatically fixes the centre and periphery, verticals and horizontals and various kinds of shape recognition as “perception centres”, reading relationships between them: “motifs like rising and falling, dominance and submission, weakness and strength, harmony and discord, struggle and conformance” [38
]. In other words, the organisation of complex concepts becomes analogous with the geometric arrangement of perceptual cues in the artwork. For Arnheim, the end result of art is to establish a pleasing order or balance of stresses—an aesthetic concept superimposed over a perceptual tendency identified as “natural”. What needs to be avoided at all costs are conditions of imbalance, where “the artistic statement becomes incomprehensible. The ambiguous pattern allows no decision on which of the possible configurations is meant” [38
In addition to repeated scanpaths and dwell times, vision has built into it recursive patterns, often spirals, which create order and also erratic variations, sometimes striking out a random path in pursuit of an eye catching detail. This created a rhythmic set of dynamics, somewhere between chaos and order. However, those works which are not abstract, and which do not encourage free-viewing, control eye behaviour in more predictable ways. Our examination of eye behaviour of these different works shows how, broadly speaking, we can have controlled, purposeful eye behaviour which is looking for something to match to a prior search term, or more capricious and involuntary eye behaviour.
Alfred Yarbus, a Russian psychologist, showed that the scanpaths followed depend on the task that the observer has to perform [39
]. For example, viewers were asked different questions about a famous painting An Unexpected Visitor
painted by Ilya Repin in 1884. One question was “how old are the people in this painting?”. This caused rapid inspection of faces across the canvas. Another question was “how poor are the people in this painting?” which caused eye sight to be directed at the furnishing and clothes depicted.
Thus, although we are fond of praising conscious and efficient viewing, this may not be the same as purposeless and pleasurable visual observation where the viewer is discovering new structures and relationships suggested by the artworks. When we played back the expert’s ways of viewing configually and inspecting normally ignored areas, students were able to adopt more critical, but no less pleasurable ways of engaging with and learning how artists structure visuals to communicate information. It is perhaps useful to use the terms “consecutive vision” to denote many dwell times and focal points and “simultaneous vision” for a general view of the whole scene. Large areas of relatively empty space play their part in information-rich areas of scenes and images, either as a way to provide peripheral framing or boundary areas, or as resting places for absorbing or reflecting on information.
Finally, we used a famous abstract expressionist painting by Jackson Pollock (Figure 8
). Figure 8
shows three novices with the different colours with the numbers showing the order by which each novice looked at things (1–45). We tend to understand vision, particularly that employed in looking at art, as a so-called free viewing experience. We tend to understand that individual differences in looking at things are common, but our team recorded the eye behaviour of individuals looking at the image revealing a remarkable consistency. Particularly striking are the areas ignored by vision. When played back to students, they learned that they could possibly try and look at low contrast areas as well as high contrast areas in order to get a more global and configural view of the image. It is also interesting that vision adopts various complex dynamics to inspect an image, and importantly this vision is only a reflection of the nonlinear dynamic aspects of thought. It is almost as if we believe that we think like a text from start to finish in sequential order but in fact our thought, our stream of consciousness, continually goes back and forth and jumbles up thought and sensations, as we have seen examples of in all the disciplines participating in this project. Students learn the importance of mindful visual searching as a great enhancement and demonstration by looking
, in addition to instructions. This layered learning across the senses enforces and increases learning and aid memory and retrieval. This kind of multisensory learning should appeal to different abilities, learning styles and cultural backgrounds. This is somewhat different from comics, where stable anchor points were provided by the text areas.
Eye tracking played an important role in increasing students’ engagement and provided a sense of agency viewing images that they feel can be explored rather than passively received (Table 3
). From these comments, and many others that repeated similar points, we can conclude that the shift to visual learning using eye tracking in these different disciplinary contexts supports and consolidates learning in a way that can capture the imagination: “seeing through others’ eyes”, and in ways that can be easily recalled (Table 4
). Exposure to the working methods and behaviour of experts, and importantly, the study of mistakes made by novices (Table 2
and Table 5
) provided immediate knowledge of different viewing methods and showed students what to avoid in their own engagements with visual material in the different disciplines.
These recordings were also used for lectures in an honours/advanced course of 10 students.
The aim of the project was to use scientific technology, demonstrating the latest vision science, for four undergraduate courses and two postgraduate courses in multiple disciplines and faculties. Studying the eye behaviour of viewers looking at technical charts, graphs, statistics, computer design interfaces, artworks and comics increased teachers’ understanding of how students learn, and what they tend to find difficult, while students in their questionnaires found these videos engaging and illuminating. Learning where to look for the most important information in images, and how to adopt methods of effective visual searching are quickly absorbed lessons. The project illustrates that not only do teachers use these videos to reflect on teaching practice but also the novelty of the images increases student curiosity, engagement and learning directly, and through experiential demonstration.