Autobiographical Recall Triggers Visual Exploration

Abstract

Autobiographical recall is thought to rely on the ability to generate a visual image of the remembered event. Neuropsychological studies suggest a relationship between deterioration in visual mental imagery and autobiographical distortions, while neuroimaging data similarly implicate visual brain areas in autobiographical recall. However, neither approach provides evidence about whether autobiographical retrieval is associated with visual exploration, or not. Our paper aimed to provide such evidence one way or the other. Using an eye tracking system, we recorded eye movements of 40 participants during autobiographical recall and during a control condition in which participants had to count aloud. In both conditions, the participants had to look at a blank screen while their gaze location was recorded by the eye-tracker. Autobiographical recall triggered a lower number of fixations and reduced their duration. In contrast, the number, duration, and amplitude of saccades increased compared to the control condition. Our data suggest that autobiographical recall is characterized by visual processing.

Introduction

It has been widely suggested that autobiographical memories, or memories of personal events, are dominated by visual mental imagery. Brewer (1996), for instance, proposed that many specific autobiographical memories come to mind in the form of visual images and that, further, the retrieval of a visual image is a core feature of autobiographical memory, distinguishing events that are remembered from those that are simply known. In line with this idea, Conway (2009) proposed that visual images are the main format of specific autobiographical memories, characterizing moments of conscious experience (Conway, 2009).

If visual mental imagery dominates autobiographical memory, then it follows that, brain lesions leading to a loss of the ability to generate these images will give rise to retrograde amnesia. Indeed, patients with occipital lobe damage show an inability, or a strikingly reduced ability, to construct visual images of events that occurred prior to their brain injury (Greenberg & Rubin, 2003). These patients had difficulties copying from a picture or a model, drawing from memory, generating scenes and rating their vividness (Greenberg & Rubin, 2003). In the same vein, Ogden (1993) reported the case of a patient with bilateral occipital damage that demonstrated extensive amnesia of memories from his premorbid period. The few memories he could recall were typically dominated by nonvisual elements such as olfactory, auditory (particularly music), and tactile elements. A similar case was reported by Brown and Chobor (1995) regarding a patient with right frontal and left occipital damage who showed visual agnosia and an extensive retrograde amnesia. Yet another patient with damage in the right occipital lobe along with the occipitotemporal junction showed, alongside manifested retrograde amnesia, difficulties recognizing common objects from memory as well as difficulty generating and manipulating visual images (Greenberg et al., 2005). Although he was able to remember names and dates encountered before his illness, the patient exhibited great difficulties remembering specific episodes associated with those facts. Taken together, these neuropsychological data support the idea that visual mental imagery plays a key role in autobiographical memory.

Complementing this evidence from neuropsychological studies, a substantial body of neuroimaging literature shows activation of visual brain areas during autobiographical recall. A significant work in this field comes from Viard et al. (2007) who found activation of bilateral lingual gyri, the left angular gyrus, and the bilateral precuneus gyrus, during autobiographical retrieval. This finding suggests a critical role of temporoparietal junction in autobiographical retrieval, a role that may be attributed to the involvement of this brain region in saccadic control (e.g., Rawley & Constantinidis, 2009). Critically, the bilateral lingual gyrus is related to the encoding and recall of complex visual stimuli (Gilboa et al. 2004), the left angular gyrus to imagery and spatial processing in the context of episodic memory (Viard et al., 2007), and the precuneus to visual mental imagery (Rubin, 2005). Furthermore, the fMRI studies show consistent occipital activation during recollection of specific autobiographical events (e.g., Piolino, Desgranges, & Eustache, 2009).

As reviewed above, neuropsychological studies suggest a relationship between deterioration in visual mental imagery and retrograde amnesia, while neuroimaging data further implicate visual brain areas in autobiographical recall. However, neither approach provides details about how visual images may be retrieved during autobiographical recall. At a more basic level, evidence about whether autobiographical retrieval is associated with visual exploration is still lacking. Our paper aimed to provide such evidence using eye movement tracking. This instrument is widely acknowledged as a reliable index of both whether individuals are performing active visual exploration and of the way in which the direction of their eyes are transferred from one spatial location to another. Eye movements are also believed to reveal memory processing, a claim that finds its origin in the study of Loftus and Mackworth (1978) who recorded participants’ eye movement while they were exposed to scenes in which an item was either consistent or inconsistent with the semantic context of the scene. The authors (Loftus & Mackworth, 1978) found longer fixation durations for inconsistent than for consistent items, revealing influences of early knowledge on eye movements. Consistent with these findings, eye movements were found to reveal memory for previous occurrence of various types of items, such as images of famous and non-famous faces (Althoff & Cohen, 1999) and novel scenes (Ryan, Althoff, Whitlow & Cohen, 2000). In these studies, previously-exposed items were viewed with fewer fixations than were novel items, revealing memory for previous occurrence of these items (for a review, see, Hannula et al., 2010). Although primarily dealing with semantic or short term memory, these studies support our attempt to use eye movement measurement as a reliable tool for revealing features of autobiographical processing.

With the recording of eye movements, we are able to determine whether autobiographical retrieval is associated with visual processing or not. In light of neuropsychological and functional imaging data, suggesting a relationship between visual mental imagery and autobiographical memory, we hypothesize that autobiographical retrieval triggers more visual exploration than a control condition that does not involve autobiographical recall.

Method

Participants

Forty graduate/undergraduate students or employees at the University of Lille 3 took part in the study. They were native French speakers with normal or corrected-to normal vision and hearing. Exclusion criteria were a history of neurological, psychiatric, or learning disorders. Informed consent was obtained from all participants in accordance with the principles laid down by the Helsinki Declaration. Demographic and mnemonic characteristics of participants are summarized in

Table 1. Demographic and mnemonic characteristics of participants. 

Age in years	25.88 (7.11)
Years of schooling	14.17 (4.42)
Episodic memory	13.80 (2.83)
Working memory forward span	7.25 (1.39)
Working memory backward span	5.45 (1.30)

Note. Standard deviations are given between brackets. In the episodic memory task (Van der Linden et al., 2004), the participants had to retain 16 words, each of which described an item that belonged to a different semantic category. After a 20 s distraction phase, the participants had to recall as many words as possible, and the maximum score was 16. In the working memory assessment, participants had to repeat string of numbers in the same order (i.e., forward spans) or in the inverse order (i.e., backward spans). These tasks were assessed as in the autobiographical research investigating general memory performance

. It is worthy of note that in the span task, the examiner orally presented a string of random single digits at the rate of approximately 1 per second. At the end of the string, the participants had to repeat the string verbatim, in the same order (i.e., forward spans) or in the inverse order (i.e., backward spans).

Apparatus

An iView X^TM HED from SMI^® was used as a headmounted monocular eye tracker. This system is based on the dark pupil system by which the eye is illuminated by an infrared light. An infrared camera, attached to a hat allowing free head movement, watches the eye. The pupil absorbs the infrared light and appears as a dark ellipse, and an image-analysis software determines where the center of the pupil is located. This system requires a calibration process for each participant, allowing the relationship between the position of the eye, as followed by the camera, and a gaze point in space to be established. The calibration procedures required defining 5 respective points (center, up-left, up-right, down-left, down-right) which the participants had to watch on a 21-inch flat blank-displayed screen. Once the calibration was done satisfactorily, an accurate estimation of the gaze location could be acquired. The sampling rate of this system is 50 Hz and its gaze position accuracy is 0.5° - 1.0° (typ.) according to the manufacturer.

Procedures

In order to assess the relationship between autobiographical recall and visual exploration, two experimental conditions were compared in a repeated-measures design. In the control condition, the participants were asked to count from 1 “in their own time” until the experimenter said “Stop”. The latter signal was provided after 2 minutes of counting. In the autobiographical condition, the participants were asked to generate an autobiographical event in response to the following instruction: ‘‘recount in detail an event in your life’’ (For the same procedure, see, El Haj, Fasotti, & Allain, 2012a; El Haj, Postal, & Allain, 2012b; El Haj, Clément, Fasotti, & Allain, 2013). Two-minutes were allocated to generate autobiographical memories, and the duration was made clear so the participants could structure their memories accordingly. The time limit was adopted to avoid bias such as repetition of memories or distractibility (for a similar free recall procedure, see, El Haj et al., 2012a, 2012b, 2013). Autobiographical recall was recorded using a Galaxy Note® for later transcription. In the control condition and during autobiographical recall, the participants were free to explore all parts of the blank-displayed screen, and asked not to look outside the screen, all the while their gaze location was recorded by the eye tracker. The distance between participants and the screen was approximately 45 to 70 cm, and behind the screen there was a blank white wall displaying no visual stimuli.

The “control” and “autobiographical” conditions were randomly counterbalanced across participants so half of the participants began with the first session, and the other half with the second session. During both conditions, the participants were told that these procedures aimed to control visual and auditory aspects of a subsequent cover task. The cover task, lasting 20s, consisted of a visual and auditory free-exploration-and-description of a drawing depicting several faces.

Before the experiment, and in order to control their mental visual imaging ability, the participants were asked to form a mental image of six objects (car, cup of tea, book, chair, glove, and door) and to determine whether each object was bigger or smaller than a shoebox.

Saccades were identified as eye accelerations that exceeded 2000°/sec² until 35% of peak deceleration with a minimum duration of ≥ 9 msec. Segments not meeting these criteria were considered as fixations (for another study using the same criteria, see, Ross et al., 1998). Blinks were identified by the typical loss of corneal reflection and were automatically excluded from the data. Gazes were automatically eliminated when the horizontal deviation of gaze exceeded 2° (a minority of our dataset, 5%).

The mean of all variables (number of fixations, fixation duration, number of saccades, duration of saccades, and amplitude of saccades) was calculated for each participant. Means for all the participants were compared between the “control” and “autobiographical” conditions with pairwise t-tests after checking for normal distribution of data with Kolmogorov-Smirnov tests. When reporting t-tests values, we also reported effects size by Cohen’s d criteria (1988) (0.20=small, 0.50=medium, 0.80=large).

Results

Less fixations and more saccades during autobiographical recall.

Results are summarized in

Table 2. Eye movement indicators during autobiographical recall and during the control condition in which the participants had to count aloud from 1. 

	Autobiographical recall	Control
Number of fixations	134.18 (45.46)	180.38 (86.19)
Total duration of fixations in msec	58137 (29544)	79771 (15558)
Number of saccades	139.83 (59.43)	104.85 (42.84)
Total duration of saccades in msec	12555 (6875)	8775 (3988)
Amplitude of saccades in degree	412.13 (215.35)	288.68 (234.94)

Note. Standard deviations are given between brackets

. Autobiographical retrieval triggered a lower number of fixations, t(39) = 2.86, p < .01, Cohen’s d = .67, and significantly reduced fixation duration, t(39) = 5.08, p < .001, Cohen’s d = .92, compared to the control condition. The autobiographical condition triggered, however, a greater number of saccades, t(39) = 3.06, p < .01, Cohen’s d = .67, duration of saccades, t(39) = 3.47, p = .001, Cohen’s d = .67, and amplitude of saccades, t(39) = 3.03, p < .00, Cohen’s d = .55, than the control condition.

Discussion

The aim of this paper was to examine the assumption that autobiographical recall is associated with visual exploration. Using eye tracking, we found that autobiographical recall triggered a lower number and a reduced duration of fixations, together with a larger number, duration, and amplitude of saccades than a control condition requiring no autobiographical recall. These data suggest that visual exploration is characteristic of autobiographical memory.

Neuropsychological cases suggest a relationship between deterioration in visual mental imagery and autobiographical memory distortion (Greenberg et al., 2005), while neuroimaging studies show activation of visual brain areas during autobiographical recall (e.g., Viard et al., 2007). However, neither approach has been able to demonstrate whether autobiographical memory is associated with visual exploration. Our work provides solid support for this claim. Using eye tracking, we found that autobiographical recall triggered less fixations and more saccades than during the control condition. As neither condition implicated external visual stimuli (i.e., the participants had to watch a blank screen), fixations could be interpreted in terms of visual inactivity, while saccades could be attributed to visual exploration. Because the blank screen involved no stimulation, the visual exploration is likely to be activated by internal processing, here, autobiographical memory generation. In line with this view, our participants produced more saccadic movements when generating autobiographical events than in the control condition, which may be interpreted in terms of visual exploration, as activated by autobiographical recall. Autobiographical recall was also characterized by longer durations and amplitude of saccades; visual exploration is likely to cross more distance in autobiographical recall than in the control condition. Taken together, these saccadic movements can be interpreted in terms of visual exploration as triggered by autobiographical recall.

The relationship between autobiographical recall and visual search can be understood by emphasizing the link between fixations and saccades. According to Ferreira et al. (2008), fixations, or looking at nothing, reflect the attempt of the visual system to find and activate stored memory representations. In other words, fixations reflect the attempt to activate a memory representation, as derived from visual and linguistic cues (i.e., in our work, the verbal instructions). When part of the memory representation is reactivated, the eyes move to the location in which the item originally appeared, and this fixation enhances subsequent memory retrieval. In this sense, fixations can be interpreted as an index of memory activation, serving to facilitate retrieval of further information. The theory of Ferriera et al. (2008) relies on empirical evidence showing that looking at a “now-empty” location facilitates retrieval of information that was previously exposed in that location (e.g., Hollingworth, 2006; Henderson, & Hollingworth, 1999; Hollingworth & Henderson, 2002). Thus, this literature relates fixations to the presence of items that were reliably exposed in the same location of that fixation, which was not the case in the present work. Our participants looked at a blankscreen exposing no previously-studied items, and their fixations could not be reliably interpreted in terms of retrieving a particular item. The theory of Ferreira et al., (2008), relating fixations to previously-exposed items, should be hence interpreted with caution since the autobiographical construction of our participants referred to uncontrolled encoding events.

In our study, when participants generated autobiographical memories they showed longer durations and amplitude of saccades than in the control condition. These visual processes are likely to reflect visual reconstruction of the relived autobiographical event. From a psychological viewpoint, reliving of personal events relies heavily on the ability to generate a visual scene corresponding to perceptual features present during the original event (Brewer, 1996). Visual mental imagery should improve the recall of specific details and the subjective experience of remembering (Greenberg and Rubin, 2003). Visual mental imagery is hence an important feature of autobiographical recall, allowing the phenomenological reliving of the past. Although we tend to relate visual exploration, as observed in autobiographical recall, to the reconstruction of visual elements of the relived events, it would be of interest to investigate how this exploration contributes to memory specificity. According to Conway (2005), autobiographical memories are typically retrieved in a generative fashion, that is, from semantic knowledge (e.g., when I was a child) to detailed specific events (e.g., my father once took me to the sea side in his car). Specific autobiographical events are usually characterized by perceptual, affective, and spatiotemporal contextual detail, and their reliving allows the episodic phenomenological experience (Tulving, 2002). Specific autobiographical events are hence likely to involve more visual elements than semantic ones. It would be of interest to investigate whether the former type of memory involves more visual exploration than the latter. Using reliable autobiographical tasks, allowing disentangling of semantic and specific autobiographical memories (e.g., the TEMPau task, Piolino et al., 2006), future research will be able to elaborate on this issue.

Another suggestion for future research is to extend the procedures to normal and pathological aging populations. Healthy elderly and Alzheimer’s Disease patients tend to show significant reduction in autobiographical recall (El Haj et al., 2011, 2012a, 2012b, 2013). It would be of interest to investigate whether such patterns can be associated with reduction in visual exploration. The same thing can be said for Schizophrenia patients, showing autobiographical memory deficits (e.g., McLeod, Wood, & Brewin, 2006) and abnormalities of eye-movement patterns during association between items and their locations (Williams et al., 2010).

As mentioned earlier, a substantial body of research suggests the recording of eye movements is a reliable tool for monitoring semantic memory (Althoff & Cohen, 1999; Loftus & Mackworth, 1978; Ryan, Althoff, Whitlow, & Cohen, 2000). Another group of studies suggest the same for spatial memory. In these studies (Ryan et al., 2000; Ryan & Cohen, 2004), a greater number of fixations and saccades were observed for scenes where items had been removed than for intact scenes, revealing the effects of relational memory on eye movement. In line with these results, a body of research has shown that people fixate on a blank location if a relevant visual stimulus was previously exposed in that region of space (Ferreira et al., 2008; Hoover & Richardson, 2008; Richardson, & Spivey, 2000). Eye movement is also found to be a useful tool to monitor memory for temporal relations. In a study by Ryan and Villate (2009), participants were successively exposed to items that were later simultaneously presented. Participants tended to inspect the items in the order that matched the originally temporal exposure. Taken together, these results indicate that memory for different aspects of experience guide eye movement behavior. Our present paper extends this assumption to autobiographical memory, since the previous literature was primarily concerned with item memory. Our work also shows how useful the monitoring of eye movements can be for addressing autobiographical memory. By using eye movements to assess this memory, rather than merely relying on participants’ subjective reliving, we may gain access to autobiographical recall under circumstances in which verbal narration cannot be reliably obtained, such as in the case of patients with severe amnesia (for the same view, see Hannula et al., 2010).

In closing, by providing solid evidence for the classic notion that autobiographical recall is characterized by visual processing, our paper paves the way for new behavioural and neuroscientific evaluations of autobiographical memory and invites further consideration of the nature of memory functioning.