Fixation and saliency during search of natural scenes: The case of visual agnosia

Neuropsychologia 47 (2009) 1994–2003

Contents lists available at ScienceDirect

Neuropsychologia journal homepage: www.elsevier.com/locate/neuropsychologia

Fixation and saliency during search of natural scenes: The case of visual agnosia Tom Foulsham a,∗ , Jason J.S. Barton b,c,d , Alan Kingstone d , Richard Dewhurst a , Geoffrey Underwood a a

School of Psychology, University of Nottingham, UK Department of Medicine (Neurology), University of British Columbia, Canada c Department of Ophthalmology and Vision Sciences, University of British Columbia, Canada d Department of Psychology, University of British Columbia, Canada b

a r t i c l e

i n f o

Article history: Received 15 July 2008 Received in revised form 12 January 2009 Accepted 10 March 2009 Available online 18 March 2009 Keywords: Agnosia Eye movements Scene perception Visual search Saliency

a b s t r a c t Models of eye movement control in natural scenes often distinguish between stimulus-driven processes (which guide the eyes to visually salient regions) and those based on task and object knowledge (which depend on expectations or identification of objects and scene gist). In the present investigation, the eye movements of a patient with visual agnosia were recorded while she searched for objects within photographs of natural scenes and compared to those made by students and age-matched controls. Agnosia is assumed to disrupt the top-down knowledge available in this task, and so may increase the reliance on bottom-up cues. The patient’s deficit in object recognition was seen in poor search performance and inefficient scanning. The low-level saliency of target objects had an effect on responses in visual agnosia, and the most salient region in the scene was more likely to be fixated by the patient than by controls. An analysis of model-predicted saliency at fixation locations indicated a closer match between fixations and low-level saliency in agnosia than in controls. These findings are discussed in relation to saliency-map models and the balance between high and low-level factors in eye guidance. © 2009 Elsevier Ltd. All rights reserved.

1. Introduction When humans look at a scene, they scan it with a series of fixations directed at different elements of the scene. The distribution of these fixations is not random, but at the same time they are not stereotyped: since the seminal studies of Yarbus (1967), it has been clear that there is great variability between observers in their scanpaths. Understanding the factors that guide the distributions of these fixations and generate this variability is important to our understanding of how natural scene perception is accomplished. How is the scanning of scenes affected by deficits in object recognition? Visual agnosia is a neuropsychological impairment in which the patient is unable to recognize objects by sight, despite normal visual acuity and semantic knowledge (Farah, 1990; Humphreys & Riddoch, 1987; Riddoch & Humphreys, 1987). Time-honoured distinctions have been drawn between those whose agnosia can be attributed to problems in perceiving shape and form (apperceptive agnosics) and those whose perception is intact but disconnected from semantic associations (associative agnosics; Lissauer, 1890); in addition there are more recent descriptions of an integrative form, consisting of failure to combine elements into a coherent

∗ Corresponding author at: Department of Psychology, University of British Columbia, 2509–2136 West Mall, Vancouver, Canada V6T 1Z4. E-mail address: [email protected] (T. Foulsham). 0028-3932/$ – see front matter © 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.neuropsychologia.2009.03.013

object, as typified by subject HJA (Behrmann, 2003; Riddoch & Humphreys, 1987). The scanning of scenes containing natural objects by patients with visual agnosia is of interest given current distinctions being drawn between factors guiding fixation patterns, particularly between top-down and bottom-up guidance (Hayhoe & Ballard, 2005; Henderson, 2003). Top-down guidance refers to the influence on fixation patterns of a variety of observer- or taskdependent factors, such as task instructions (Dewhurst & Crundall, 2008), task goals (Foulsham & Underwood, 2007; Yarbus, 1967), prior knowledge (Althoff & Cohen, 1999), or perceptual expertise (Reingold, Charness, Pomplun, & Stampe, 2001; Underwood, Chapman, Brocklehurst, Underwood, & Crundall, 2003). In a search task, for example, the goal of the observer (to find an item) causes them to move their eyes towards targets or target-similar distractors (van Zoest, Donk, & Theeuwes, 2004; Wolfe, Butcher, Lee, & Hyle, 2003). Top-down search is therefore directed more to ‘relevant’ items in a scene. Bottom-up guidance, on the other hand, refers to the influence of stimulus properties or features such as colour, contrast and motion, properties that are not dependent on the observer or task: bottom-up search is directed towards conspicuous or ‘salient’ items. Several models of eye movement control have been proposed to explain and predict the location and duration of fixations made during perception. Much of this work is found in the literature on reading and eye movements although there are related models which can also be applied to visual search and scene perception

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003

(Engbert, Nuthmann, Richter, & Kliegl, 2005; Findlay & Walker, 1999; Logan, 1996; Reichle, Rayner, & Pollatsek, 2003). Findlay and Walker’s (1999) framework specifies a system where two metrics are programmed for every saccade: WHEN the saccade should occur, and WHERE it should be directed. Normally this system regulates how long we look at objects, and the location our eyes select as the target of upcoming saccades to ensure optimal visual scanning. A key question is whether the operation of this system is affected by the defective object processing in visual agnosia. Could this lead to an extension of fixations durations or time spent dwelling upon objects which is comparable to increased activity in the WHEN pathway of Findlay and Walker’s model? The location of the next saccade within Findlay and Walker’s (1999) framework is determined by the greatest saliency peak represented on a topographic map. In the past, quantifying this bottom-up saliency was difficult, although various statistics suggested that people were drawn to edges and areas of contrast (Mannan, Ruddock, & Wooding, 1996; Reinagel & Zador, 1999). Recent work has generated a well-specified model that combines visual features (colour, intensity and orientation contrast) at various spatial scales to produce an overall spatial representation of the bottom-up saliency of the different parts of a scene (Itti & Koch, 2000). The model assumes that attention and therefore fixations will be attracted towards the peaks of high saliency in this map, and thus predicts search patterns in which people move fixations and/or covert attention between scene elements in their descending order of saliency, predictions supported by data from human search times while looking for camouflaged military vehicles in outdoor scenes (Itti & Koch, 2000). The model also performs better than chance at predicting eye movements when scenes are freeviewed or inspected in preparation for a memory test (Foulsham & Underwood, 2008; Parkhurst, Law, & Niebur, 2002). In other search tasks however, bottom-up saliency is not a good predictor of fixation distribution. Chen and Zelinsky (2006) found that when top-down and bottom-up signals were placed in competition, top-down guidance dominated, presumably by filtering out irrelevant items based on known target features and peripheral vision. The stimuli used in this study were greyscale, photorealistic objects arranged in a circular array, which removed real-life expectations about the probable locations of objects. Nevertheless, in more realistic search tasks, Foulsham and Underwood (2007) and Henderson, Brockmole, Castelhano, and Mack (2007) also found that the saliency-map model did not predict eye movements. In Foulsham and Underwood (2007) (see also Underwood & Foulsham, 2006; Underwood, Foulsham, van Loon, Humphreys, & Bloyce, 2006) participants viewed a set of natural photographs, some of which contained key objects which had been ranked by the saliency model, under several task conditions. When viewing the same scenes for a memory test, visual saliency had an effect on how early and how often these objects were inspected. However, when these objects became targets in visual search the effects of saliency were eliminated and targets were found quickly regardless of their saliency. Henderson et al. (2007) recorded the eye movements of observers searching for people in outdoor scenes and found that the saliency-map model did not predict these movements. As a consequence of such results Torralba, Oliva, Castelhano, and Henderson (2006) proposed a contextual guidance model of real-world search which biases eye movements with global scene features or scene “gist”. The implication of these studies is that for the healthy observer top-down knowledge “overrides” or weights a saliency-based guidance system, and that this information is available early, perhaps within the first fixation on a scene. This top-down knowledge might be in the form of expectations concerning target features or locations (as in models proposed by Navalpakkam & Itti, 2005; Rao, Zelinsky, Hayhoe, & Ballard, 2002; Zelinsky, Zhang, Yu, Chen, &

1995

Samaras, 2005), or it might consist of recognizing global scene properties and gist preattentively and using these to guide the eyes. What happens when the ability to recognize object or global scene properties is absent or impaired? Under such circumstances it might be more difficult to implement top-down guidance. A person with visual agnosia might not be able to override a bottom-up saliency-based system, due to their inability to link raw visual input to top-down knowledge. If so, their fixation patterns should conform more closely to the predictions of the saliency-map model: that is, they should show a significant impact of the salience of stimulus properties on visual search in situations where normal subjects do not display such effects. Such a finding would extend the applicability of the saliency-map model to more naturalistic situations and support the tentative suggestion that such bottom-up effects are present but normally over-ridden by top-down considerations. In this study we report on the fixation patterns made during visual search by a patient with general visual agnosia. Despite good visual acuity and peripheral fields, indicating functioning low-level vision, and apparently normal voluntary saccades, she is unable to recognize even simple three-dimensional abstract forms, or line drawings of common objects. How would such a patient distribute fixations in a scene filled with objects, particularly when given the instruction to search for a certain category of object? We hypothesized that, with her deficit, there should be less top-down guidance in this task than with normal controls. If visual saliency is computed earlier than, or independent from, object recognition, then saliency would be predicted to have an effect on eye movements. Furthermore, with reduced top-down bias, the eye movements produced might be closer to a raw saliency map than those made by normal controls. 2. Method 2.1. Case description CH is a 63-year-old right-handed woman with slowly progressive visual difficulties over a period of 6 years. She first noted trouble with reading, especially large type, although she could still write. Subsequently she had difficulty recognizing the faces of her friends, relying on their voices instead. She had problems locating household objects, for example in the refrigerator or on the kitchen counter, and often misreached for items like light switches and cups. She confused navy with black but otherwise believed her colour vision to be normal. She later developed more problems navigating in familiar surroundings and required an escort on her visits to the clinic. Her visual acuity was good (20/25 for single letters) and her visual fields were full, confirmed by Goldmann perimetry. Saccades and pursuit eye movements were normal. Neuropsychological evaluation showed normal general knowledge, expressive vocabulary and comprehension. She was able to write to dictation, with occasional spelling errors, but could not read words, proceeding instead by deciphering one letter at a time. Digit span was 6 forwards and 4 backwards, and she performed in the average range on all tests of auditory verbal learning and memory. Abstract verbal reasoning abilities were average to superior. Visual tasks were severely impaired, with difficulty on line bisection, cancellation and search tasks. Her object recognition was slow and effortful, with severe impairments on the Boston Naming Test: with line drawings she frequently failed or misidentified objects, often focussing on small details (key = “see the O”, chair = “grass. . .something to sit on”). She had difficulty naming three-dimensional shapes (cylinder = “cone”, cube = “all these angles, octagon”, pyramid = “triangles in it, a tent”), although she could name two-dimensional figures like squares and circles. She interpreted the Cookie theft picture as “a woman washing dishes, something spilling, it must be a restaurant”. There was complete disorganization in copying or spontaneous drawing of a flower (Fig. 1). Further perceptual tasks confirmed severe visual problems. Benton line orientation test showed 20% accuracy, in the severely deficient range, but curvature discrimination was normal. Her ability to judge the spatial configuration of dot patterns was impaired, scoring 56% correct with two dots and 22% correct with four dots (chance = 33% correct). Her ability to judge whether a triangle was symmetric or not was at chance (56% correct), a task controls do with 100% accuracy. Her ability to distinguish famous faces from anonymous ones was poor (d = 0.12, versus d for controls > 2.5). Imagery for famous faces was borderline for facial features (13/16) and impaired on overall face shape (11/16), with the lower limit of normal performance being 82% accuracy (Barton & Cherkasova, 2003). A cerebral perfusion scan with technetium injection at 4 years after onset showed marked hypoperfusion of the posterior parietal and temporal lobes. CT

1996

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003 Control subjects were tested using similar, head-mounted Eyelink eye trackers, an Eyelink I (for YCs; sample rate 250 Hz) and an Eyelink II (for AMCs; sample rate 500 Hz). In both cases, a fixed viewing distance of 60 cm was used with a chin-rest to restrict movement. 2.4. Stimuli The stimuli for this experiment were 48 colour photographs of interior scenes (Foulsham & Underwood, 2007). Half of these pictures contained a target object, a piece of fruit, among other everyday objects and items of furniture. Four fruits (apple, pear, orange and lemon) were used with equal frequency. Fruit were chosen as they are members of a clear category that was known to all subjects and easily recognizable. They also share smooth contours and uniform colouring meaning that they are often not highly salient. The target object was approximately the same size in each picture and was located the same distance from the centre of the screen, where fixation would begin. All scenes contained approximately the same amount of clutter. The photographs were displayed on a monitor with a resolution of 1024 × 768 pixels, giving an image that subtended 37◦ × 28◦ of visual angle. The setup for the younger control subjects involved a slightly smaller screen, resulting in an image that was 31◦ × 25◦ . The 24 pictures with targets were classed as containing medium- or low-saliency targets on the basis of predictions from the Itti and Koch (2000) model of bottom-up attention. Fig. 2 shows one of the target pictures, along with a saliency map and the predicted fixations until target selection. The source code of this model is freely available (http://www.ilab.usc.edu/) and the software was used to produce saliency maps for all the stimuli. Specific features of this model are discussed elsewhere (Foulsham & Underwood, 2008; Itti & Koch, 2000). Standard parameters were used, resulting in a focus of attention (which we consider a “model fixation”) of radius 64 pixels (approximately 1.5◦ ) that shifted to points in the scene in their order of decreasing saliency. To rate the saliency of a target fruit relative to other scene elements in each picture, we calculated the number of model fixations that occurred before the target was itself selected as a point of fixation. Those targets selected after 5–10 model fixations were considered to be of medium saliency, while targets selected after 10 or more model fixations were considered to be of low saliency. In order to make the task more challenging, no targets were highly salient (selected after fewer than five model fixations) but the most salient region in the scene was recorded for further analysis. The decision to exclude targets that were highly salient was motivated by a desire to separate knowledge-driven and saliency-driven factors in eye guidance. In this case, if subjects can saccade efficiently to the target they must be doing so based on their knowledge of the target and not saliency. Similarly, frequent inspection of the most salient region must be due to bottom-up factors. For results regarding the inspection of highly salient objects, see Underwood and Foulsham (2006) and Underwood et al. (2006). Saliency maps used by the model to direct attention were also produced. Fig. 2 shows one of the target pictures with model-predicted fixations, the combined output from all feature maps, and the final saliency map which receives further processing in order to favour a few salient peaks and reduce all other points to zero. To assess the correlation between fixation patterns and saliency we used the first of these maps in order to assess fixations across the whole image, and we will consider these maps to be representative of the general saliency approach.

Fig. 1. Copying of a flower by CH. She makes two attempts to copy the design at top, both showing a lack of perceptual organization. scan 6 years after onset of symptoms revealed some general sulcal prominence with occipital predominance of enlargement of the lateral ventricles, consistent with a diagnosis of posterior cortical atrophy (Benson, Davis, & Snyder, 1988). 2.2. Control participants Data from students performing this experiment have been previously reported elsewhere (Foulsham & Underwood, 2007). These control participants (‘younger controls’, YCs) were 15 student volunteers (7 females) with normal vision. Participants ranged in age from 19 to 30 years old. Age is known to have an impact on visual search and oculomotor behaviour (Rabbitt, 1965; Scialfa, Thomas, & Joffe, 1994) and so we also tested a group of 10 age-matched controls (AMCs; 5 females). These participants ranged in age from 58 to 76 years old, with a mean age of 65, making them comparable with patient CH. All reported themselves as healthy, had normal or corrected-to-normal vision (all but three wore glasses) and had visited an optician in the last 3 years. A short questionnaire confirmed that none of these participants reported difficulties in finding or identifying objects around the home or in recognizing the faces of friends or family, and that they had no other history of visual problems. 2.3. Apparatus An Eyelink 1000 eyetracker (http://www.sr-research.com) was used to record eye movements with CH. This system uses a desktop-mounted camera to track the pupil image and corneal reflection, with a sampling rate of 1000 Hz. The subject’s head was placed in a chin-rest and head frame to minimize head movements, viewing a screen placed 60 cm from the corneal surface, with dim background lighting.

2.5. Procedure The system was first calibrated using a nine-dot grid, spanning 31◦ horizontally and 23 vertically (26◦ × 21 for the YCs), and this procedure repeated as many times as was necessary to achieve a spatial error of less than 0.5◦ . CH performed calibration well, although she sometimes found it difficult to maintain focus on the fixation dot without an experimenter pointing to it on the screen. Despite this, her performance during calibration and validation procedures indicates that she was able to make accurate, voluntary saccades to simple visual targets (a single dot on a blank screen). Before testing the task was explained and CH demonstrated that she had intact semantic knowledge of different types of fruit, confirming that she suffers from a visual agnosia rather than a multimodal deficit in semantic memory. A short set of practice trials was then given, using pictures which were not part of the experimental set, with instructions to answer as quickly and accurately as possible the question “Is there a piece of fruit in the scene?” It was stressed that the participant did not need to identify the fruit and that not all pictures would contain the target. The response was a verbal “yes” or “no” that the investigator recorded by pressing one of two keys on a keyboard. Young controls responded themselves by pressing the key. This key press terminated the trial. Following practice trials, subjects viewed all 48 experimental stimuli in a random order. Before each trial, a dot appeared in the centre of the screen for the subject to fixate. This procedure was used to correct for any drift in the spatial position of the signal, and it also confirmed that CH was able to locate and saccade to a single dot on a blank screen. Her saccadic accuracy in this case can be inferred from the minimal offset between target and gaze location (M = 0.44◦ ). 2.6. Analysis For behavioural performance, we calculated accuracy in terms of hits and false alarms and calculated a d measure of discriminative power for the absence or pres-

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003

1997

threshold of 4000◦ /s2 . Periods of fixation were identified as epochs between the offset of one saccade and the onset of the next, with a minimum duration of 100 ms. Few fixations are shorter than this cut-off, and it is assumed that these shorter fixations are caused by noise in the saccadic system and do not reflect useful cognitive processing (Rayner, 1998). For each fixation we recorded its duration and its x- and y-coordinates. Our subsequent analyses were divided into three categories. First, we made general observations about the number of fixations, their duration, and the amplitude of saccades (fixation shifts) during each trial. The second and third categories were more specific analyses aimed at evaluating the influence of top-down and bottom-up guidance on search patterns. To examine the influence of top-down versus bottom-up effects at the target location, we first measured the proportion of trials on which the subject fixated the target. We also examined whether subjects were more likely to fixate the target if it was of medium saliency than of low saliency. As a second more dynamic measure, we calculated how many fixations occurred before the target was fixated, and again assessed whether this was affected by target saliency. If CH’s fixation patterns are guided more by bottom-up effects of saliency than by top-down effects from target knowledge, then she should fixate the target less often, show effects of target saliency, and require more fixations before fixating the target. To examine the influence of bottom-up guidance, we can also examine fixation behaviour at the most salient region in the scene. We calculated the proportion of trials on which the most salient object was fixated, and the proportion of all fixations that were directed to this object versus the target object. Did CH fixate this region more often than the control participants? Conversely, we also analysed the saliency at each fixation location to produce a closer comparison of the fixation patterns made and the visual saliency of the stimuli. If the relationship linking fixation and saliency differs between the patient and controls then we might expect saliency at CH’s fixation locations to be higher, on average. In each case the data of CH can be related to the distribution of values provided by the control groups, and quantified by a z-score. We were most concerned with the effect of agnosia, rather than normal aging, so this statistic was computed for the comparison between CH and age-matched controls only, although in most cases the results were the same for the younger controls. The degree to which age had an impact on the results is also of interest, so the two control groups were compared using independent samples t tests, or with group as a between-subjects factor in a mixed ANOVA with target saliency. Contrasts between low- and medium-saliency trials within the single subject CH were made using the non-parametric Wilcoxon signed ranks test or the 2 -test (for nominal data).

3. Results 3.1. Task performance What was the effect of visual agnosia on performance of the naturalistic search task? CH’s overall accuracy was 62.5%, significantly worse than control subjects (YCs = 92%, AMCs = 95%, z = 6.9; p < .001; Fig. 3). While she detected both low saliency (8/12 hit rate) and medium saliency targets (9/12), this was offset by a relatively high false alarm rate (11/24), resulting in a low d of 0.65. Control subjects rarely made false alarms, leading to a much higher mean d (YCs = 3.6, AMCs = 3.8, z = 4.9; p < .001). Furthermore, although she was not required to name targets or other objects, she did so on several occasions, but usually with a misidentification (e.g. she called

Fig. 2. An example stimulus with predictions from the saliency model. The top panel shows an example scene from the experiment, overlaid with the model fixations predicted by the saliency-map model, shown as yellow rings connected by red lines with arrows showing the sequence of fixations. The target fruit is a pear, which is selected on the fifth model fixation. Features are combined in a centre-surround fashion, normalized and then summed to give a representation of the conspicuity of each point (middle panel). The final saliency map for this scene is shown in the bottom panel. In both cases brighter areas indicate higher saliency. The most salient element in this scene is the blue binder, which was selected for the first model fixation.

ence of a target fruit, using signal detection theory. Given that the responses made by CH and the age-matched controls were verbal and logged by the experimenter, reaction times will not be analysed in detail. To analyse the eye movement data, we first identified saccades from the raw samples using SR-Research’s Eyelink DataViewer software. A saccade was identified using a motion threshold of 0.15◦ , a velocity threshold of 30◦ /s and an acceleration

Fig. 3. Proportion correct for CH and the control groups. The overall proportion of correct responses is shown for controls as a box plot with the median and the 25th and 75th percentile (whiskers show 95th and 5th percentiles).

1998

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003

Fig. 4. Examples of the fixation behaviour of CH (top) and controls. CH’s fixations from a single trial are represented by yellow circles linked by lines showing saccades. The fixation locations of all control participants when looking at this scene are shown below, for YCs (middle) and AMCs (bottom).

a brightly coloured book “a watermelon”). The mean d of the two control groups did not differ from each other (t23 < 1). 3.2. General ocular motor statistics (Table 1) Fig. 4 provides an illustrative example of the difference between the scanning patterns of CH and control subjects. In the example scene, CH makes a large number of saccades and fixations and dwells on colourful and salient regions such as the lamp, but completely misses the target. Control participants (bottom panels)

fixate the target and occasionally other objects, and avoid blank areas and surfaces. Our subsequent analyses quantified these observations across all trials. The accuracy data showed that CH found the task difficult, so we can ask how this was reflected in the eye movements she made. It might be that the patient required more shifts of attention or longer fixations than controls in order to make the present/absent judgement, and this might have been moderated by the saliency of the target. Looking at global eye movement measurements should also reveal the impact of deficient object recognition on saccade generation. As the display was terminated when the target was found, the number of fixations per trial reflects the efficiency of visual search. CH made far more fixations than controls (means, CH = 36.7, YCs = 6.6, AMCs = 12.3, z = 4.6, p < .001). The control groups were compared in a mixed factorial ANOVA with trial type as the within-subjects factor. The control groups were reliably different (F(1,23) = 16.6, MSE = 27.2, p < .0001); AMCs made more fixations. Trial type also had an effect (F(2,46) = 36.9, MSE = 7.7, p < .0001) with more fixations occurring in target-absent trials than either low(t(24) = 5.0) or medium-saliency (t(24) = 5.6) target-present trials (post hoc t-tests, both p < .0001). There was also a reliable difference between low and medium saliency trials with fewer fixations in trials with a medium saliency target (t(24) = 5.629, p < .0001). The interaction was also significant (F(2,46) = 5.07, MSE = 7.7, p = .01); the AMCs had more fixations than YCs in all trial types (all at least p < .03), but more so in the target-absent trials. While CH’s data suggested similar differences between different trial types, these were not significant when tested at the trial level. Fixation duration can reflect the processing achieved in each fixation (Rayner, 1998). We calculated the mean duration of all fixations with the exception of the first and last and any that were on the target, so as to exclude processing time associated with responding. CH’s fixation durations were slightly longer than those of controls, although they were within the normal range (overall means, CH = 230.2 ms; YCs = 183.8 ms; AMCs = 205.0, z = 0.83, p = .20). There was a trend for mean fixation duration to be shorter in the young controls than in the older group (t(23) = 1.9, p = .075). CH also made smaller saccades (4.5◦ on average across trial types) than control subjects (YCs, mean = 9.1◦ ; AMCs = 9.5◦ , z = 3.88, p < .001). The two control groups did not differ from each other (t(23) < 1). A potential problem with these analyses is that, as CH made many more eye movements, the differences seen might be due to a change in saccade dynamics over time. It is known that as viewing progresses, inspection becomes more fine-grained, leading to shorter saccades and longer durations (Yarbus, 1967). Could the differences seen above be due simply to the fact that CH makes more, rather than qualitatively different saccades? To address this we conducted a further analysis into the early part of viewing by excluding all events beyond the 5th saccade. In terms of fixation duration, CH’s mean duration during early viewing remained slightly greater than that of controls (CH = 224.2 ms; YCs = 181.9 ms; AMCs = 185.0, z = 1.48, p = .07). Using only these data, however, the trend for longer fixations in the older age group disappeared (t(23) < 1). Looking at the mean saccade amplitude of the first five saccades, the difference between CH and the normal controls remained (CH = 3.65◦ ; YCs = 9.41◦ ; AMCs = 9.66◦ , z = 3.82, p = .0001). There was no difference between the two control groups (t(23) < 1). The controls tended to make slightly larger saccades in the first part of the trial than later on during viewing (hence the larger mean when only looking at the first five saccades), but CH actually made smaller saccades from the start. The difference is even more striking when the distribution of saccade amplitudes made by each group during the whole trial is compared (Fig. 5). The two control groups are highly similar, and they show a fairly flat distribution with a similar proportion of short- and medium-length saccades. CH, however, shows many more short saccades, with the majority being less than 5◦ . It is

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003

1999

Table 1 Measures reflecting global eye movement performance in CH and the control groups. Mean values are shown, with standard deviations (across trials for CH and across subjects in the control groups) in parentheses. CH Low Number of fixations per trial 31.75 (19.3) Fixation duration (ms) 253.56 (56.0) 4.07 (3.9) Saccadic amplitude (◦ )

Young controls

Age-matched controls

Medium

Target absent

Low

Medium

Target absent

Low

Medium

Target absent

32.33 (19.2) 247.09 (55.9) 4.75 (9.5)

41.38 (22.2) 210.13 (25.2) 4.66 (7.7)

5.06 (1.0) 176.19 (28.4) 8.59 (1.3)

4.43 (0.6) 172.83 (31.3) 8.49 (1.6)

8.45 (3.8) 193.13 (26.7) 10.27 (2.0)

8.91 (3.16) 208.31 (30.5) 9.12 (1.3)

7.68 (2.71) 205.51 (39.6) 9.18 (1.4)

16.38 (8.18) 203.1 (29.5) 9.92 (1.4)

interesting to note that CH’s distribution is more like the positively skewed, long tailed distribution commonly seen when people freely view images (and which is less skewed in search; Tatler, Baddeley, & Vincent, 2006). The control subjects were more likely to saccade towards areas of interest from further away, which can be seen as a signature of guided eye movement behaviour in scenes. To summarize global scan parameters, CH made smaller saccades and more fixations during scene search. There were also indications that age had an effect; the older control group made slightly longer fixations, which may have been because they made more fixations and resorted to closer inspection in the later part of the trial. 3.3. Fixations directed at the target In Foulsham and Underwood (2007) target-directed eye movements were analysed to see how early and how often targets of different saliency were fixated. Although CH clearly had difficulty performing the task, did she look at the targets? The following measures looked only at those trials where there was a target. First, we examined the proportion of trials where the target object was fixated at least once (Fig. 6). While controls fixated targets on most trials (means, YCs = 77%, AMCs = 89%), CH did so on only 25% of the trials (z = 5.15, p < .001). A mixed ANOVA looked at the fixation of targets of different saliency. The difference between the control groups was not significant (F(1,23) = 2.33, MSE = 0.08, p = .141) and the within-subjects effect of saliency was marginally significant (F(1,23) = 4.55, MSE = 0.008, p = .044). There was no interaction (F(1,23) < 1). Those trials where controls did not fixate the target, but correctly detected it, can be attributed to their use of peripheral vision, but this was rare, particularly in the age-matched

Fig. 5. Saccade amplitude distributions for CH and the two control groups. The xaxis shows the upper-bound of 0.5◦ bins. Data points indicate the proportion of all saccades in each bin.

group. Control subjects were slightly more likely to fixate medium saliency targets, although this effect was small and was confined to the older group. While CH’s data suggest that she was twice as likely to fixate the target if it was medium rather than low saliency, this difference was not statistically significant (Chi-squared test, 2 = 0.89, p = .35). Given that CH rarely fixated the target, measures of how early she fixated the target are based on only a small number of trials. CH reached the target considerably later than the control subjects when it was medium saliency (mean ordinal fixation on target; CH = 12.75; YCs = 4.17, AMCs = 4.88; z = 5.50, p < .0001). When searching for low saliency targets CH did not differ from controls (CH = 7.5, YCs = 4.8, AMCs = 5.98, z = 0.79, p = .21). Comparing the control groups, older age-matched controls fixated targets marginally later than young controls (F(1,23) = 3.17, MSE = 3.24, p = .088), medium saliency targets were fixated earlier than low saliency targets (F(1,23) = 10.1, MSE = 0.94, p < .005) and there was no interaction (F(1,23) < 1). 3.4. Fixations directed at the most salient region Normal participants are efficient when searching for objects and are rarely distracted by salient but irrelevant regions (Chen & Zelinsky, 2006). Our control participants fixated the most salient region on a minority of trials, (means, YCs = 22%, AMCs = 36%; t(23) = 2.61, p < .05; Fig. 6). While fixation of the most salient region may have been higher than expected by chance (see analysis below), it was much less frequent than fixation of the target, as we would expect given the task, and it was also lower than the same measure in our patient. CH fixated the most salient region on 67% of all trials, almost twice as often as age-matched controls (z = 2.02, p < .05). This comparison is problematic, however: since CH made many more fixations per trial than the control group, there is a higher

Fig. 6. The proportion of trials where the target was fixated, as a function of its saliency. Error bars for the control groups indicate ± one standard error of the mean.

2000

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003

3.5. Saliency at fixation

Fig. 7. The proportion of all fixations landing in each of the regions of interest. Error bars for the control groups indicate ± one standard error of the mean.

likelihood that she would fixate the most salient region by chance, even if attention was not being guided by the saliency of scene elements. To correct for this confounding element, we performed an additional analysis. We calculated the frequency of fixations that landed on the two regions of interest—the target and the most salient object. Each region was the same size and occupied only. 0.025 of the total image area, so if fixations were distributed uniformly regardless of task or saliency this is the proportion of fixations we should expect on these regions. This value is used, with the total number of fixations made on pictures where this object was present, to attach a onetailed binomial proportion to the fixation probabilities and thus give an indication of whether the regions were fixated more often than chance. The data show an interaction between subject and regionof-interest (Fig. 7). The control groups fixated the target with a frequency 10 times higher than chance, whereas CH rarely fixated the low- and medium-salient targets. The binomial test showed that, in all cases, the proportion of fixations made by controls on the target was much greater than chance (all ps < .001). In the case of CH, the proportion of fixations on the medium target was also greater than the mean chance expectancy (p < .001) but the low saliency target was rarely fixated, at a frequency not greater than chance (p = .90). In all cases CH differed reliably from age-matched controls (all zs > 2, all ps < .05). CH also made more fixations on medium-than low-saliency targets (2 = 14.6, p < .0001), an effect also seen in both control groups (F(1,23) = 10.54, MSE = 0.004, p < .005), with no difference between the groups and no interaction (both Fs(1,23) < 1). Thus CH fixated the target less frequently than the controls, but did so more often when it was of higher saliency. The proportion of fixations on the most salient region showed a different pattern. The binomial test showed that all groups fixated the most salient region more often than the mean chance expectancy (all ps < .001), However, CH fixated this region far more often than the control participants (z = 2.14, p < .05). The control groups did not differ from each other (t(23) < 1). Again, we can ask whether these results were the same in the first five fixations. When looking at only this early part of viewing, CH fixated the most salient region on 0.067 of the first five fixations and this was within the normal range (YCs = 0.064; AMCs = 0.077, z < 1). These fixation probabilities were greater than the mean chance expectancy (all ps < .05).

The previous section suggested that not only was CH less likely to look at the target, she was actually biased towards salient regions. To look more fully at the relationship between saliency and fixation we examined the saliency value at fixated locations. Several researchers have demonstrated a correlational relationship between fixation locations and saliency by showing that the saliency value at fixated locations is higher than that at non-fixated or randomly selected locations (Foulsham & Underwood, 2008; Itti & Koch, 2000; Parkhurst et al., 2002). We performed a similar analysis here, using the saliency maps described in the method. For each fixation location, the saliency value at the corresponding location in the map was extracted. Maps were arbitrarily scaled to a fixed range of 0–100, so that constantly fixating the most salient point would give a mean saliency at fixation of 100. Assuming a uniform distribution of fixations throughout the scene, a guidance system that selected locations regardless of saliency would lead to a mean saliency at fixation equivalent to the mean of the whole map. Following Parkhurst et al., we therefore subtracted the overall map mean from each saliency value to produce a ‘chance adjusted’ saliency value for each fixation location. An unbiased guidance system would give a mean chance-adjusted saliency of zero, while guidance by saliency would be indicated by a significantly positive value. A potential problem with this approach is that fixations tend to be biased towards the centre of most displays and so comparison with an unbiased map mean will tend to overestimate the relationship between saliency and fixation (Foulsham & Underwood, 2008; Tatler, Baddeley, & Gilchrist, 2005). In this section we are interested in differences between patient and control groups rather than getting an absolute estimate of the effect of saliency. As such we will ignore the question of whether the saliency at fixation is meaningfully greater than chance and instead look for a difference between the groups. The chance-adjusted saliency was computed for all fixations with the exception of the first (which was necessarily in the centre) and any fixations that lay on target regions. Target fixations were excluded on the basis that they were specifically primed by the task and that the saliency at their locations was constrained by the experiment. The resulting values were averaged across all stimuli from the experiment, and the means are presented in Fig. 8. The mean, chance-adjusted saliency at fixations was larger for CH than for age-matched controls (z = 1.80, p < .05). The two control groups did not differ (t(23) < 1).

Fig. 8. The mean, chance-adjusted saliency at fixation. Values are averaged across all fixations and trials. Error bars show the standard error of the mean across subjects (in controls) and across trials (in CH).

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003

4. Discussion Consistent with the results from neuropsychological testing, CH was severely impaired at detecting objects in natural scenes. She was less accurate than normal control participants, who found the task easy: in contrast, CH’s low d confirmed a negligible ability to tell if a target was present or not. In other visual evaluations, her performance was similar to that of visual agnosics presented elsewhere (e.g. Humphreys & Riddoch, 1987). There were large differences in global fixation parameters between CH and the age-matched controls, with CH making more fixations and shorter saccades. Thus impairment in object perception had dramatic effects, not just on performance but also on the eye movements made during search. This supports a large role for top-down guidance by object information in scanning by normal populations during search and scene perception (Henderson, 2003), which allows controls to move quickly to the target. The higher frequency of fixations and the shorter saccades made by the patient may reflect the difficulty in guiding information acquisition when normal object recognition processes are not available. Assessment of CH’s low-level vision was normal; she was able to detect targets quickly in the peripheral field and within the range of normal observers, and she could make accurate saccades to targets in the calibration phase. It therefore seems likely that the differences found in global eye movements are due, not to bottom-up processing difficulties but a top-down recognition deficit. Short saccades and more numerous fixations, for example, might be found in this patient because she cannot recognize the information at fixation within a single glance. Bottom-up saliency of objects does not influence search in a number of paradigms (Chen & Zelinsky, 2006; Henderson et al., 2007; Underwood, Templeman, Lamming, & Foulsham, 2008). This is replicated in the current study, in that control participants rarely looked at the most salient region in the scene (see also Foulsham & Underwood, 2007). Was there any evidence for an influence of saliency on eye guidance in search? Control participants showed relatively few effects of target saliency. On the other hand, there was evidence that medium saliency targets were fixated earlier and slightly more often, especially in the older controls. The most salient region was fixated more often than we would expect from its size. However, given that we did not control the content of this region, we cannot be sure that it was fixated because of saliency, or for some other reason (because it was semantically interesting, for example). Instead, we were interested in whether agnosia led to different or more pronounced effects of saliency. CH was often better with targets of higher saliency in most of the measures taken: she was more accurate and quicker and more likely to fixate the target object, when it was of medium- rather than low-saliency. The implication is that with reduced top-down control due to poor recognition of parts of the scene saliency has a greater effect in guiding search. We found clear evidence that the most salient region captured CH’s fixations; while normal controls concentrated fixations on the relevant target, CH fixated the most salient region more than the relevant target, despite the fact that CH and the controls received the same instructions. CH looked at the most salient region much more than would be expected by chance and relative to normal performance. Our final analysis looked at the saliency across all fixations: these values were higher in CH than in controls, indicating that her fixations were a closer match to the saliency map than those of the controls. Why did agnosia lead to more saliency-driven eye movements? Theories of visual agnosia suggest that it is caused by an inability to integrate features into an object and link this percept to semantic knowledge. Normal individuals are able to access previously learned information, such as what items of fruit tend to look like and where they tend to occur, in response to a categori-

2001

cal label (fruit). In a difficult visual search, when targets cannot be found using peripheral vision alone, eye movements are directed to regions that match the learned feature information. For example, in Rao et al. (2002) a target template is represented as a vector of features (colours, luminance, oriented edges) at different spatial scales. The regions that show the greatest correlation with this vector are selected for further inspection. In Navalpakkam and Itti (2005) this top-down information is combined with a raw saliency map to guide eye movements. We hypothesize that in agnosia the target information is impoverished or inaccessible, which leads to less weighting of regions likely to contain the relevant target: the failure to generate maps of relevant locations may lead to search being tipped towards reliance on bottom-up saliency. Further research is needed to explore whether any features of the target are available with which to bias search, and this might be accomplished by using visual or exemplar targets rather than a categorical cue. Despite normal colour vision, there was no evidence in the present study that CH could bias her search towards regions of the appropriate colour. It remains to be seen how complex the target and the search array have to be lead to an agnosic deficit, and exploring this could reveal which object information is preserved in this disorder. Normal observers are also guided by the gist, context and layout of a scene and by their expectations of where an object will appear (Torralba et al., 2006). It is possible that CH’s perception of the global features of the scene was also impaired, and this may have contributed to her inability to find the target (although it would not necessarily have biased her towards salient regions). The current set of data can help us understand how gaze is controlled on the basis of both bottom-up saliency and topdown, knowledge-driven information. One possibility is that search instructions induce a mode of attention that completely overrides raw saliency and is instead determined only by one’s knowledge of the target. If this were the case then we might expect CH, whose top-down knowledge is incomplete, to veto saliency and perhaps make completely random or uniform eye movements. In fact, she was not just unable to locate the target; she was biased towards salient regions. This suggests that models where saccade targets are derived from the product of top-down information weighting a bottom-up saliency map are correct. In the case of visual agnosia, the top-down information is not available to modify the saliency map and thus visually salient, task irrelevant regions are selected more often than controls. The fact that there were some effects of target saliency in controls is also consistent with this notion: target regions that were also visually conspicuous would receive the highest priority. The most salient region would also be fixated sometimes, especially if it also resembled known target features. A final point of interest is that CH’s tendency to fixate salient regions was not found early in scene viewing, which is contrary to some research suggesting saliency is most potent at the beginning of inspection (Parkhurst et al., 2002). Some degree of top-down control, perhaps reflected in a tendency to make short saccades and dwell on objects, may have obscured the saliency bias earlier on in viewing. Assessing the degree to which the saliency model can predict fixations in this task is not completely straightforward. Even in controls, the most salient region was fixated more than a uniform chance expectancy, but this is likely to be too liberal a baseline as it assumes that fixation locations are independent and uniformly distributed across the scene. Similarly, the fact that saliency at fixation is higher than the scene average is not conclusive evidence that observers are selecting regions for saliency. Salient regions may have receive privileged inspection merely as a virtue of being likely to contain objects, rather than background, or due to their tendency to appear near the centre of the screen. For these reasons, we do not wish to make definitive estimates of the relationship between saliency and fixation in controls here but merely highlight that in several measures this relationship is stronger in visual agnosia.

2002

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003

To our knowledge, there are no previous investigations into the eye movement behaviour of general visual agnosics viewing complex images. However, several researchers have measured agnosic performance in simple visual search, without recording eye movements. CH’s poor search performance is consistent with previous reports of visual agnosia affecting conjunction search (Delvenne, Seron, Coyette, & Rossion, 2004; Humphreys, Riddoch, Quinlan, Price, & Donnelly, 1992), and extends these observations to search with natural scenes. Top-down effects on visual search may also be altered, which is consistent with our interpretation of the effects here. For example, Ballaz, Boutsen, Peyrin, Humphreys, and Marendaz (2005) showed that normal subjects were affected by the canonical orientation of a target object but a patient with general visual agnosia was not. Also relevant to the current study is recent research into eye movements in prosopagnosia. Several studies have shown that impaired knowledge of faces affects the eye movements made when processing this class of object (Barton, Radcliffe, Cherkasova, & Edelman, 2007; Le, Raufaste, Roussel, Puel, & Démonet, 2003). In this case, deficient top-down guidance may result in fewer fixations on relevant features, such as the eye region. The results presented here also provide an insight into the factors that govern eye movement parameters (for example fixation duration, saccade amplitude, and overall search efficiency). Although the focus of this paper is the relationship between fixations and saliency when top-down guidance is severely impaired, it is worthwhile to return to the implications of the present results for other theoretical models of eye-guidance. Models of eye movement control often specify thresholds for processing of fixated stimuli (Engbert et al., 2005; Findlay & Walker, 1999; Logan, 1996; Reichle et al., 2003). Within Findlay and Walker’s (1999) framework a saccade is generated upon completion of this processing at fixation, the speed and efficiency of which can be affected by extrafoveal stimuli vying for attention. As such there is a competitive balance between remaining fixated (modulated by the FIXATE centre) and generating a saccade (modulated by the MOVE centre); whichever of the two independent pathways governing FIXATE and MOVE receives the most activity determines respectively how long the eyes remain fixated and the location they go to next. It is of interest therefore to return to CH’s atypical scanning behaviour and saccade amplitudes. CH made much smaller saccades, even during the first part of scene viewing that was performed by all participants. This may reflect that, while controls rapidly extracted the information content available within the vicinity of a fixation, CH’s reduced visual efficiency limited the amount of data processed, leading to smaller saccades and multiple fixations on the same region or object. Here it may be argued that MOVE centre activity is restricted for longer than usual to within-object saccades. This can go some way to explaining the distribution of saccade amplitudes seen in CH, which was skewed towards smaller gaze shifts. There were also some differences between young controls and older age-matched controls. Older controls made more fixations, reflecting longer search times, and these were slightly longer in duration on average. It has long been known that aging can lead to longer response times in visual search (e.g. Rabbitt, 1965) and the present research extends this to a more naturalistic task. Other researchers have reported age-related changes in the number of fixations (Scialfa et al., 1994), saccadic accuracy and saccadic reaction time (Munoz, Broughton, Goldring, & Armstrong, 1998) that could explain some of the difference in search. Longer fixation durations in older participants have also been found (Scialfa & Joffe, 1997) which could in part be due to differences in the time to program and initiate the next saccade. Given that this study was concerned with the balance between bottom-up and top-down factors in overt attention it is also interesting to ask whether there are differences in this balance across age groups. Several authors have

suggested that older observers are more affected by distractors or visual clutter, which may indicate greater impact of irrelevant but salient objects (McPhee, Scialfa, Dennis, Ho, & Caird, 2004; Scialfa, Hamaluk, Skaloud, & Pratt, 1999; but see Kramer, Hahn, Irwin, & Theeuwes, 1999). While we did find that older observers fixated the most salient region in more trials, this may be due only to the fact that the older age-matched controls made more fixations overall. The higher frequency of fixations may indicate prolonged search in the older participants, but there was no difference in terms of the saliency at fixation. Overall, the evidence for a changing impact of saliency with age in our naturalistic search task seems modest. In sum, we have made novel observations regarding naturalistic visual search in visual agnosia, which suggest not just an increase in task difficulty but a shift in the balance between bottom-up and top-down guidance in scanning patterns. Loss of visual object knowledge may have severely constrained her ability to use topdown information to bias search, resulting in fixations that reflected saliency more than relevancy. Acknowledgements TF and RD were supported by a Universitas 21 travel award. JJSB was supported by a Canada Research Chair and Senior Scholar Award from the Michael Smith Foundation for Health Research. We thank Laurent Itti and colleagues for providing the saliency map program and George Malcolm for perceptual testing of CH. We are also grateful for the suggestions of two anonymous reviewers. References Althoff, R. R., & Cohen, N. J. (1999). Eye-movement-based memory effect: A reprocessing effect in face perception. Journal of Experimental Psychology-Learning Memory and Cognition, 25(4), 997–1010. Ballaz, C., Boutsen, L., Peyrin, C., Humphreys, G. W., & Marendaz, C. (2005). Visual search for object orientation can be modulated by canonical orientation. Journal of Experimental Psychology: Human Perception and Performance, 31(1), 20–39. Barton, J. J. S., & Cherkasova, M. (2003). Face imagery and its relation to perception and covert recognition in prosopagnosia. Neurology, 61(2), 220–225. Barton, J. J. S., Radcliffe, N., Cherkasova, M., & Edelman, J. A. (2007). Scan patterns during the processing of facial identity in prosopagnosia. Experimental Brain Research, 181, 199–211. Behrmann, M. (2003). Neuropsychological approaches to perceptual organization. In M. A. Peterson & G. Rhodes (Eds.), Perception of faces, objects and scenes: Analytic and holistic processes (pp. 295–334). Oxford: Oxford University Press. Benson, D. F., Davis, R. J., & Snyder, B. D. (1988). Posterior cortical atrophy. Archives of Neurology, 45(7), 789–793. Chen, X., & Zelinsky, G. J. (2006). Real-world visual search is dominated by top-down guidance. Vision Research, 46(24), 4118–4133. Delvenne, J., Seron, X., Coyette, F., & Rossion, B. (2004). Evidence for perceptual deficits in associative visual (prosop)agnosia: A single-case study. Neuropsychologia, 42, 597–612. Dewhurst, R., & Crundall, D. (2008). Training eye movements: Can training people where to look hinder the processing of fixated objects? Perception, 37 (11), 1729–1744. Engbert, R., Nuthmann, A., Richter, E. M., & Kliegl, R. (2005). SWIFT: A dynamical model of saccade generation during reading. Psychological Review, 112(4), 777–813. Farah, M. J. (1990). Visual agnosia: Disorders of object recognition and what they tell us about normal vision. Cambridge, MA: MIT Press. Findlay, J. M., & Walker, R. (1999). A model of saccade generation based on parallel processing and competitive inhibition. Behavioral and Brain Sciences, 22(4), 661–721. Foulsham, T., & Underwood, G. (2007). How does the purpose of inspection influence the potency of visual saliency in scene perception? Perception, 36, 11231138. Foulsham, T., & Underwood, G. (2008). What can saliency models predict about eye movements? Spatial and sequential aspects of fixations during encoding and recognition. Journal of Vision, 8(2), 1–17, 6. http://journalofvision.org/8/2/6 Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4), 188–194. Henderson, J. M. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7(11), 498–504. Henderson, J. M., Brockmole, J. R., Castelhano, M. S., & Mack, M. L. (2007). Visual saliency does not account for eye movements during visual search in real-world scenes. In R. van Gompel, M. Fischer, W. Murray, & R. W. Hill (Eds.), Eye movements: A window on mind and brain (pp. 537–562). Amsterdam: Elsevier. Humphreys, G. W., & Riddoch, M. J. (1987). To see but not to see: A case study of visual agnosia. Hillsdale, NJ: Erlbaum.

T. Foulsham et al. / Neuropsychologia 47 (2009) 1994–2003 Humphreys, G. W., Riddoch, M. J., Quinlan, P. T., Price, C. J., & Donnelly, N. (1992). Parallel pattern processing and visual agnosia. Canadian Journal of Psychology, 46(3), 377–416. Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40(10–12), 1489–1506. Kramer, A. F., Hahn, S., Irwin, D. E., & Theeuwes, J. (1999). Attentional capture and aging: Implications for visual search performance and oculomotor control. Psychology and Aging, 14(1), 135–154. Le, S., Raufaste, E., Roussel, S., Puel, M., & Démonet, J. (2003). Implicit face perception in a patient with visual agnosia? Evidence from behavioural and eye-tracking analyses. Neuropsychologia, 41, 702–712. Lissauer, H. (1890). Einfall von Seelenblindheit nebst einem Bintrag zur Theorie derselben. Arch fur Psychiatrie und Nervenkrankheiten, 2, 22. Logan, G. D. (1996). The CODE theory of visual attention: An integration of spacebased and object-based attention. Psychological Review, 103(4), 603–649. Mannan, S., Ruddock, K., & Wooding, D. (1996). The relationship between the locations of spatial features and those of fixations made during visual examination of briefly presented images. Spatial Vision, 10(10), 65–188. McPhee, L. C., Scialfa, C. T., Dennis, W. M., Ho, G., & Caird, J. K. (2004). Age differences in visual search for traffic signs during a simulated conversation. Human Factors, 46(4), 674–685. Munoz, D. P., Broughton, J. R., Goldring, J. E., & Armstrong, I. T. (1998). Age-related performance of human subjects on saccadic eye movement tasks. Experimental Brain Research, 121(4), 391–400. Navalpakkam, V., & Itti, L. (2005). Modeling the influence of task on attention. Vision Research, 45(2), 205–231. Parkhurst, D., Law, K., & Niebur, E. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research, 42(1), 107–123. Rabbitt, P. (1965). An age-decrement in the ability to ignore irrelevant information. Journal of Gerontology, 20, 233–237. Rao, R. P. N., Zelinsky, G. J., Hayhoe, M., & Ballard, D. (2002). Eye movements in iconic visual search. Vision Research, (42), 1447–1463. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422. Reichle, E. D., Rayner, K., & Pollatsek, A. (2003). The E-Z reader model of eyemovement control in reading: Comparisons to other models. Behavioral and Brain Sciences, 26(4), 445–526. Reinagel, P., & Zador, A. M. (1999). Natural scene statistics at the centre of gaze. Network-Computation in Neural Systems, 10(4), 341–350. Reingold, E. M., Charness, N., Pomplun, M., & Stampe, D. M. (2001). Visual span in expert chess players: Evidence from eye movements. Psychological Science, 12(1), 48–55.

2003

Riddoch, M. J., & Humphreys, G. W. (1987). A case of integrative visual agnosia. Brain, 110, 1431–1462. Scialfa, C. T., Hamaluk, E., Skaloud, P., & Pratt, J. (1999). Age differences in saccadic averaging. Psychology and Aging, 14, 695–699. Scialfa, C. T., & Joffe, K. M. (1997). Age differences in feature and conjunction search: Implications for theories of visual search and generalized slowing. Aging Neuropsychology and Cognition, 4(3), 227–246. Scialfa, C. T., Thomas, D. M., & Joffe, K. M. (1994). Age-differences in the useful fieldof-view—an eye-movement analysis. Optometry and Vision Science, 71(12), 736– 742. Tatler, B. W., Baddeley, R. J., & Gilchrist, I. D. (2005). Visual correlates of fixation selection: Effects of scale and time. Vision Research, 45(5), 643–659. Tatler, B. W., Baddeley, R. J., & Vincent, B. (2006). The long and the short of it: Spatial statistics at fixation vary with saccade amplitude and task. Vision Research, 46, 1857–1862. Torralba, A., Oliva, A., Castelhano, M. S., & Henderson, J. M. (2006). Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychological Review, 113(4), 766–786. Underwood, G., Chapman, P., Brocklehurst, N., Underwood, J., & Crundall, D. (2003). Visual attention while driving: Sequences of eye fixations made by experienced and novice drivers. Ergonomics, 46(6), 629–646. Underwood, G., & Foulsham, T. (2006). Visual saliency and semantic incongruency influence eye movements when inspecting pictures. Quarterly Journal of Experimental Psychology, 59(11), 1931–1949. Underwood, G., Foulsham, T., van Loon, E., Humphreys, L., & Bloyce, J. (2006). Eye movements during scene inspection: A test of the saliency map hypothesis. European Journal of Cognitive Psychology, 18(3), 321–343. Underwood, G., Templeman, E., Lamming, L., & Foulsham, T. (2008). Is attention necessary for object identification? Evidence from eye movements during the inspection of real-world scenes. Consciousness and Cognition, 17, 159–170. van Zoest, W., Donk, M., & Theeuwes, J. (2004). The role of stimulus-driven and goaldriven control in saccadic visual selection. Journal of Experimental PsychologyHuman Perception and Performance, 30(4), 746–759. Wolfe, J. M., Butcher, S. J., Lee, C., & Hyle, M. (2003). Changing your mind: On the contributions of top-down and bottom-up guidance in visual search for feature singletons. Journal of Experimental Psychology-Human Perception and Performance, 29(2), 483–502. Yarbus, A. L. (1967). Eye movements and vision. New York: Plenum. Zelinsky, G. J., Zhang, W., Yu, B., Chen, X., & Samaras, D. (2005). The role of top-down and bottom-up processes in guiding eye movements in visual search. In Paper presented at the Neural Information Processing Systems