- Email: [email protected]
The role of the vertical meridian in visual memory for objects
Neuropsychologia 40 (2002) 1873–1880
The role of the vertical meridian in visual memory for objects J. Hornak a , J. Duncan b , D. Gaffan a,∗ a
Deparment of Experimental Psychology, Oxford University, South Parks Rd., Oxford OX1 3UD, UK b MRC Cognition and Brain Sciences Unit, 15 Chaucer Rd., Cambridge CB2 2EF, UK Received 2 November 2001; received in revised form 16 May 2002; accepted 17 May 2002
Abstract It is widely believed that, in human and nonhuman primates, visual memories of objects are stored in the temporal lobe. Electrophysiological results in monkeys, however, indicate that when a visual scene contains two or more objects, with at least one object in each visual hemifield, neurons in the temporal lobe of each hemisphere respond only to the objects that are in the contralateral visual hemifield, and their activity is unaffected by the objects in the ipsilateral hemifield. Putting these two premises together predicts that object memory should fail, or at least suffer a substantial decrement, when an object is presented for learning and retention as part of such a scene, but crosses the vertical meridian between the learning trial and the retention test. The effect of this change should be much greater than the effect of an equal retinal translation that crosses the horizontal rather than the vertical meridian. An experiment with normal human subjects verified this prediction under conventional conditions of tachistoscopic viewing, with a single constant fixation spot. A further condition in the same experiment, however, tested the same retinal translations in a more naturalistic condition, where the retinal changes were produced by varying the position on the display screen of the fixation spot rather than of the objects. Here, there was no significant special effect of crossing the vertical meridian. We conclude that visual memories are not stored exclusively in the temporal lobe. © 2002 Elsevier Science Ltd. All rights reserved. Keywords: Vertical meridian; Memory; Object
1. Introduction The medial and inferior temporal lobe, including the perirhinal and entorhinal cortex and the hippocampus as well as the white matter of the anterior temporal stem, is indispensable for normal memory processing in both human and nonhuman primates, as is shown by the severe memory impairments which follow lesions in these structures [5,13,18]. The inference that the temporal lobe contains a memory system [23], or multiple memory systems [8], in which memories are processed and stored, is widely accepted. The perirhinal cortex is the most important of these temporal-lobe structures in memory for visually presented objects, which is the most extensively studied kind of memory in nonhuman primates [16,18]; furthermore, the function of the perirhinal cortex has been doubly dissociated in the monkey from the function of adjacent structures in the temporal lobe [3,8]. On this basis it has been suggested [8] that the perirhinal cortex is a memory system, differentiated from other memory systems in the temporal lobe, and specialised ∗ Corresponding author. Tel.: +44-1865-27-1349; fax: +44-1865-31-0447. E-mail address: [email protected] (D. Gaffan).
for the processing and storage of object memories. According to this view, memory for objects that are presented in complex visual scenes [9] is produced by the interaction within each temporal lobe between perirhinal cortex and the other memory systems of the temporal lobe, including the hippocampus [12]. Against the simplest version of this hypothesis of temporal-lobe memory systems for scene memory, however, is evidence that some of the effects of perirhinal cortex ablations in monkeys are perceptual deficits rather than memory deficits [2], and arguments that some or all of the memory deficits that are produced by perirhinal cortex ablations in monkeys can be seen as a secondary effect of the loss of high-level perceptual representations of objects, rather than as direct loss of memory storage [2,19]. The present experiment is concerned with one of the features of temporal-lobe function that seems most difficult to reconcile with the idea that memories of visual scenes are processed and stored exclusively or mainly within the temporal lobe, as the hypothesis of temporal-lobe memory systems suggests. This is the fact that the representation of complex visual scenes in the temporal lobes is divided at the vertical meridian of the visual field between the two hemispheres. When a monkey sees a single object that is presented against a large blank background, neurons in the temporal
0028-3932/02/$ – see front matter © 2002 Elsevier Science Ltd. All rights reserved. PII: S 0 0 2 8 - 3 9 3 2 ( 0 2 ) 0 0 0 7 0 - 2
1874
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
lobe in both hemispheres respond to the object and signal its identity, even when the object is confined to one visual hemifield, and even though there are slight differences in strength of response favouring the neurons in the temporal lobe contralateral to the object [4]. However, a single object presented against a large blank background is not a natural condition of object vision and memory. In a more naturalistic scene-like condition, when two or more objects are presented, with at least one object in each visual hemifield, the difference between the two temporal lobes in the objects they respond to is stark. Neurons in the temporal lobe of each hemisphere now respond only to the objects that are in the contralateral visual hemifield, and their activity is unaffected by the objects in the ipsilateral hemifield [4]. This poses a problem for understanding the memory of objects that are presented in this fashion, that is, in scenes having at least one object in each visual hemifield. Consider the case where an object, a constituent of a visual scene that contains other objects, is presented for learning in one visual hemifield, but is subsequently presented for a recognition memory test in the opposite hemifield. In the learning trial, neurons in only one temporal lobe respond to the object, and in the retention test trial, neurons in only the other temporal lobe respond to the object. All proposals as to the neural basis of object memory assume that the memory of an object is stored as a modification in neurons that respond to the presentation of the object [1,17], and indeed it is difficult to see how neurons that do not respond to the presentation of an object could either lay down or retrieve a memory of it. It thus follows from the electrophysiological data, that neurons in neither of the two temporal lobes have both the opportunity to lay down and also the opportunity to retrieve a memory of the target object in the case we are considering, where the target object in a scene is presented in opposite hemifields at learning and at the retention test. Therefore, if object memories are stored either exclusively or mainly as modifications of neurons within the temporal lobe, memory for the object should either fail or suffer a substantial decrement in this case. The present experiment tested this counter-intuitive prediction in normal human subjects. We administered two object recognition memory tasks. The trials were presented tachistoscopically, in order to control the retinal location of the presented objects. On every trial, both in learning and at the retention test, there were two different objects, one in each hemifield. Thus, each learning trial in these tasks presented two different objects, both to be remembered, and one in each visual hemifield. In each task a total of 128 objects were presented for learning in this fashion, and after all the objects had been presented for learning there was a retention test, at which the subjects had to recognise the objects they had seen in the learning trials by distinguishing them from novel foils. Each retention test trial presented one previously learned object together with one foil object that had not been seen before, one object in each hemifield. Some of the previously learned objects were presented at the retention test in the same location in
the visual field as they had occupied in the learning trials, in order to provide a baseline measure of object memory in these conditions. Other objects, however, were presented at the retention test in the hemifield opposite to that which they had occupied in the learning trials (“horizontal shift”). The prediction, derived above, is that memory for these objects should either fail or suffer a substantial decrement. To control for the possibility that a shift in an object’s retinal location between learning and retention test is by itself sufficient to produce a memory decrement, even when it is not a shift across the vertical meridian, a third set of objects was presented at retention in the same hemifield as that which they had occupied in the learning trials, but shifted within that hemifield (“vertical shift”). In the monkey, temporal cortex neurons that fail to respond to objects in the ipsilateral hemifield, in the conditions we have discussed, respond in the same conditions to objects in the contralateral hemifield almost independently of their location within that hemifield [4]. Thus, the main question to be asked in each task is whether horizontal shifts produce worse memory performance than vertical shifts. A further important aspect of the design of the two tasks was the means by which the shifts in the retinal location of objects were accomplished. To vary an object’s retinal location, one can either change the position of the object while maintaining the position of the subject’s eyes, or maintain the position of the object while changing the position of the subject’s eyes; one cannot change only retinal position and nothing else. In tachistoscopic experiments, and in electrophysiological studies, there is usually a single constant fixation spot in the centre of the display screen, and objects are shifted from one retinal location to another by changing the object’s position on the display screen. In the most usual natural condition, however, when stable complex scenes are viewed in free vision in one or more trials, changes in the retinal location of an object result from changes in the position of the subject’s point of fixation, not from changes in the position of the object in the scene. Figs. 1 and 2 show how the two tasks instantiated these two different means by which an object can shift its retinal position. In tasks using the Square template there was a single central fixation point which was in the same position in all trials, and objects were shifted across the visual field, between learning and retention testing, by moving the objects on the display screen. But in tasks using the Lozenge template, exactly the same shifts of objects’ positions in the visual field were accomplished by changing the position of the fixation spot, leaving the objects in an unchanged position on the display screen (the three pairs of learning and test trials using the Square template, that are illustrated in the leftmost two columns of Fig. 2, are identical, in terms of the retinal positions of the target and foil objects at learning and test, to the three pairs using the Lozenge template, that are illustrated in the rightmost two columns of Fig. 2). Though identical in terms of the retinal positions of objects, these tasks are quite different from each other in the
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
Fig. 1. The templates for displays of fixation spots and objects. Each task used either the Square template or the Lozenge template. Each trial in any task displayed a fixation spot in one of the positions shown as ‘+’ in the figure, and two objects in two of the positions shown as ‘䊊’. The two objects were always in two positions adjacent to the fixation spot, and always on diametrically opposite sides of it. Trials generated using these templates are illustrated in Fig. 2.
1875
natural memory performances they resemble. In the task with the Square template the subject repeatedly sees the same view of the apparatus from exactly the same fixation point, and memory in this task thus resembles the natural task of reconstructing in memory a complex scene in which one fixated only one object. The task using the Lozenge template requires the subject to fixate different positions in successive views, and memory in this task thus, because of the successive fixations at different points, resembles the natural task of reconstructing in memory a complex scene which one inspected with successive saccades to different fixation points. If the subjects possess the ability to construct a memory of the scene that is not hemifield specific, the Lozenge task should encourage the use of that strategy. Thus, we expected the effects of shifts between hemifields, if any, to be weaker in the task using the Lozenge template than in the task using the Square template.
Fig. 2. Some of the possible configurations of learning trials and test trials in tasks using the Square template (leftmost two columns) and the Lozenge template (rightmost two columns). In each case a learning trial for an object is shown next to the retention test for that object. For a description of all the possible configurations, see the text.
1876
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
Using these two templates we created two tasks, Square100 and Lozenge-100, so named because 100 ms was the tachistoscopic exposure duration of each trial, chosen to be short enough to prevent within-trial saccades. Each subject in an initial group of subjects performed both of these tasks. Subsequently a separate group of subjects performed a third task, Square-1500. Our purpose in the third task was to establish whether a memory decrement would be produced by a change of the position of an object on the screen, in the same displays as in Square-100, but without the correlated change in its position in retinal space. The Square-1500 task was the same as the Square-100 task but the exposure duration of each display of objects, whether in learning or in retention tests, was 1500 ms instead of 100 ms, chosen to be long enough to allow within-trial saccades. This was to give the subjects sufficient time to briefly fixate each of the two objects. To equate the overall difficulty of the Square-1500 task to that of the Square-100 task, we presented each object in fewer learning trials in Square-1500 than in Square-100, and we required the subjects to remember more objects in Square-1500 than in Square-100. For the sake of clarity it should be noted that there are two sets of retinal relationships between object positions in our tasks: (1) the simultaneous relationship, between the retinal positions of two different objects presented together in any single trial, is always this: one object is in the retinal quadrant that is diametrically opposite to the retinal quadrant occupied by the other object. That is to say, if one object is in the lower left then the other is in the upper right, and if one object is in the upper left then the other is in the lower right, and these two are the only possible pairs of retinal positions of simultaneously presented objects that ever occurred, both in learning trials and in retention tests, both with the Square template and with the Lozenge template; (2) the successive relationship, between the retinal position of an object at its learning trial and the retinal position of the same object at its subsequent retention test, can be either identical, or shifted; and if it shifts, the shift can be either horizontal, that is to say, either between upper left and upper right or between lower left and lower right, or vertical, that is to say, either between upper left and lower left or between upper right and lower right.
2. Methods 2.1. Subjects These were 30 adult volunteers, recruited from subject panels in Oxford and Cambridge. Eighteen of the subjects each performed both the Square-100 and the Lozenge-100 tasks, and the order in which they performed those two tasks was determined in a pseudo-random manner, double alternating in order of the subjects’ entry to the experiment. Twelve subjects performed only the Square-1500 task.
2.2. Apparatus and stimulus material The experiment was conducted on a notebook computer. The screen displayed a square background, which was white throughout the task. The background square was 147 mm × 147 mm. The objects were a pool of 512 coloured clipart images. The size of any object was 35 mm × 35 mm, though this could include some white background. When displayed in either the Square or Lozenge templates (Fig. 1), the corner of the object that was nearest to the fixation spot was 6 mm up or down and 6 mm left or right from the fixation spot. In the Square template the fixation spot was in the centre of the white background square. In the Lozenge template the four possible positions for the fixation spot were positioned at the centre of the four possible positions of objects in the Square template. The effect was that, given the design (below), the relationships of objects to fixation spots were identical in the Lozenge and Square templates. Figs. 1 and 2 are not to scale but they illustrate the layout schematically. The fixation spot in the experiment was a black dot 0.25 mm × 0.25 mm. Subjects viewed the display from a distance of about 400 mm. Tachistoscopic exposures were timed with reference to the screen refresh cycle, to avoid flicker. 2.3. Square-100 task This task is so called because it used the “Square” template and because each display lasted 100 ms. The template is illustrated in Fig. 1 and some of the possible stimulus configurations are illustrated schematically in the left side of Fig. 2 (see Section 2.2 for the actual dimensions of the display). The fixation spot (shown as ‘+’ in the figures) was always in the centre of the display in this task, and objects could appear in four possible positions (shown as ‘䊊’ in Fig. 1). The display on any trial in the task had objects in only two of the four possible positions for objects shown for the Square template in Fig. 1. On any learning trial two objects were presented, in diametrically opposite quadrants of the visual field to each other. In subsequent retention tests, each object that had been learned was presented again, paired with a novel object in the diametrically opposite quadrant, and the subject judged which of these two objects had been seen before. The subject spoke to indicate whether the upper or lower of the two objects at the retention test was the one that had been presented during the learning trials. The experimenter wrote down the subject’s responses for later analysis. There were three conditions for the retention test. The previously learned object could be presented for retention test either in the same quadrant as in learning (no change), or in the horizontally opposite quadrant to that which it had occupied in learning (horizontal shift), or in the vertically opposite quadrant to that which it had occupied in learning (vertical shift). Fig. 2 illustrates some trials. All the trials, both in learning and in retention tests, were tachistoscopic. The fixation spot appeared on the display screen to indicate that the next trial was ready, and the subject fixated the spot before touching the
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
keyboard to make the objects appear. The objects appeared for 100 ms and then blanked into the white background. Each object to be remembered was presented in four learning trials. All the objects to be remembered were presented once each in a random order, then a second time once each in a new random order, and so on until they had all been presented four times. The random orders were unique to each subject. Every learning trial presented two objects. Any object was presented in the same quadrant in all four of its learning trials, but the object in the opposite quadrant was not the same in all four learning trials. Each subject was given 128 objects to learn. These were taken at random, in a different random order for each subject, from a pool of 512 available objects. Of these 128 objects that were learned, 64 were then tested for retention with no change of position, 32 were tested with horizontal shift, and 32 with vertical shift. Both at retention tests and at learning trials, the four possible positions for objects were occupied equally often by objects that changed position between learning and retention, and by objects that did not change position. At each of the same four possible positions, the objects that changed position between learning and retention were half shifted vertically and half shifted horizontally. The 128 retention test trials were in a unique random order for each subject. Eighteen subjects performed the task. 2.4. Lozenge-100 task This was identical to the Square-100 task, except that it used the Lozenge template (Fig. 1). There were four possible positions for the fixation spot (each shown as ‘+’ Fig. 1). Whichever fixation spot was used in any trial, there were two objects in opposite quadrants, in the same positions relative to the fixation spot as they would have occupied in a trial in the Square task. Thus, for example, if the fixation spot used in a trial in the Lozenge-100 task was the fixation spot at the lower left, then the two objects presented at that trial would always be one at the left object position and one at the bottom object position in the Lozenge template. In all four of the learning trials with any object, the object was always in the same position on the screen, and the fixation spot was always in the same position, but the object in the opposite retinal quadrant was not the same in all four learning trials. At the retention test, the shifts of objects relative to the fixation spot were identical to those in the Square task, but were accomplished by moving the fixation spot rather than the object. The right half of Fig. 2 illustrates some trials. As in Square-100, 128 objects were learned by each subject and subsequently tested for retention, 64 with no change of position, 32 with horizontal shift, and 32 with vertical shift. The four possible fixation spots were used equally often. The four possible positions for objects at retention tests and at learning trials were occupied equally often by objects that changed their retinal position (position relative to the fixation spot) between learning and retention, and by objects that did not change position. Objects that underwent
1877
a horizontal shift of retinal position had to occupy either the top or bottom of the four possible positions for objects, and objects that underwent a vertical shift had to occupy either the left or right of the four possible positions for objects. Fig. 2 shows 3 out of the total of 16 possible combinations of the spatial configuration of the learning trials and the spatial configuration of the retention test for an object in the Lozenge task. These 16 possible spatial combinations were used equally often, each combination being applied to eight unique objects, and each of these 16 combinations corresponded to an identical combination of spatial configurations, in terms of retinal locations, for an equal number of objects in the Square-100 task. The same 18 subjects who performed Square-100 also performed Lozenge-100. Half the subjects performed Square-100 first then Lozenge-100, and the other half performed the tasks in the opposite order. For each subject, the 256 objects used in Lozenge-100 were those that were not used, for that subject, in Square-100. 2.5. Square-1500 task This was identical to the Square-100 task except for the following changes. The exposure duration of each display of objects, whether in learning or in retention tests, was 1500 ms instead of 100 ms, and the subjects were instructed to glance at each of the objects during this interval. Each object was presented in only one learning trial, instead of four. The subjects were given 256 objects to learn, instead of 128. Twelve further subjects, who had not performed the Square-100 and Lozenge-100 tasks, performed this Square-1500 task.
3. Results The results are shown in Fig. 3. One group of subjects performed both the Square-100 and the Lozenge-100 tasks.
Fig. 3. Results from the three tasks.
1878
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
Half of these subjects performed the Square-100 task before the Lozenge-100 task, and the others performed the tasks in the opposite order. Preliminary analyses of the results from these two tasks showed that this between-subjects factor, order of tasks, had no overall effect on performance and did not interact with any other factor, F < 1 in every case. Task order was therefore discarded from the analyses of these two tasks. Results from the vertical and horizontal shift conditions in Square-100 and Lozenge-100 were entered into an analysis with the factors shift type (two levels, vertical and horizontal) and task (two levels, Square-100 and Lozenge-100). The main effects of shift type and task were not significant (both F < 1). The interaction of shift type with task was significant (F = 7.187, d.f. 1, 17, P = 0.016). This interaction was further analysed by designed comparisons, using the appropriate pooled error terms from the interaction and the nonsignificant main effects. Within the Lozenge-100 task, there was no significant difference between horizontal and vertical shifts (t < 1). Within the Square-100 task, however, horizontal shifts produced significantly less accurate retention test performance than vertical shifts, as predicted (t = 2.34, d.f. 17, P = 0.016, one-tailed). Further, horizontal shifts in Square-100 produced significantly less accurate retention test performance than horizontal shifts in Lozenge-100, as predicted (t = 1.92, d.f. 17, P = 0.036, one-tailed), while performance after vertical shifts did not differ significantly between the two tasks (t = 1.55, d.f. 17, P = 0.140, two-tailed). In Square-100, any of the four possible object positions, either in learning trials or in retention tests, is equally likely to be occupied by an object that undergoes a horizontal shift as by an object that undergoes a vertical shift. In Lozenge-100, however, objects that undergo a horizontal shift of retinal position necessarily occupy the top or bottom object positions in the Lozenge template (Figs. 1 and 2) while objects that undergo a vertical shift necessarily occupy the left or right positions in the template. In principle, therefore, the effects of these shifts in the Lozenge template might need to be assessed against different baselines. This would be the case if the objects that underwent no change of retinal position produced better memory performance when they were in the top or bottom positions than when they were in the left or right positions, or vice versa. In fact, however, the mean retention accuracy for these two classes of no-change objects in the Lozenge-100 task were almost identical (78.3 and 78.5% correct at the retention test, respectively) and the difference between them did not approach statistical significance (t < 1). Results from Square-100 and Lozenge-100 were further analysed to assess the effect of change, averaging over horizontal and vertical shifts and comparing this average with performance on the no-change trials. Overall the effect of change was significant (F = 27.755, d.f. 1,17, P = 0.000) and the effect of change did not interact with task (F < 1). Designed comparisons derived from this analysis confirmed
that the effect of change was significant within each of the two tasks (smaller t = 3.828, d.f. 17, P = 0.001) and that performance in the retention test trials with no change did not differ between the two tasks (t < 1). The results from the free-vision task, Square-1500, are also shown in Fig. 3. This task was performed by a separate group of subjects, drawn from the same pool of subjects as those who performed the first two tasks. In Square-1500 there was no difference between vertical and horizontal shifts (F < 1) and no effect of change (F < 1). 4. Discussion In Square-100, memory accuracy was substantially impaired when an object changed from one visual hemifield to the other between learning and retention test, crossing the vertical meridian of the visual field in a horizontal shift. Performance in this condition was significantly worse than after a comparable shift vertically, within a visual hemifield. We can see only three possible explanations for this difference between horizontal and vertical shifts in the Square-100 task. First, it is conceivable that a horizontal change in the position of an object in the world, irrespective of its retinal position, might be more deleterious to memory than a vertical change of position in the world. An explanation along these lines is ruled out, however, by the failure of a horizontal–vertical difference to appear in free vision (Square-1500). Second, it is conceivable that some unforeseen properties of the stimulus objects that were used in the present tasks could make horizontal change on the retina more deleterious than vertical change. For example, if the quarter of an object that is nearest to the fixation point is more salient than the other quarters of the object, and if objects tend to be up–down symmetrical but not left–right symmetrical, then a horizontal shift will produce a bigger change in the appearance of an object than a vertical shift will. An explanation along these lines applies to Lozenge-100 in exactly the same way as to Square-100, however, and therefore is ruled out by the absence of a horizontal–vertical difference in Lozenge-100. Third, the only remaining possibility appears to be that the horizontal shift between visual hemifields in Square-100 was more deleterious to memory than the equivalent vertical shift within a hemifield because object memories in this task were, to a large extent, hemifield specific. This last was the hypothesis we derived in Section 1, where we saw that, because of the electrophysiological properties of temporal lobe neurons, visual memories stored within the temporal lobe should be expected to be hemifield specific. Our Square-100 task is similar in some ways to an earlier task used by Gratton et al. [14]. These authors reported, as our results confirm, that horizontal shifts between hemifields are more deleterious than vertical shifts within hemifields in tachistoscopic visual recognition memory with a single constant fixation spot. The effect reported by Gratton et al. seems less powerful than the effect we observed, and if so this
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
is consistent with the electrophysiological data (Section 1), since Gratton et al. presented only one object in the visual field on each trial. However, it is difficult to compare the two sets of results directly, since the baseline performance with no shift was at a much lower level in the earlier experiment than in the present experiment. This is probably due to the difference in stimulus materials, since the stimuli used by Gratton et al. were structures of straight-line segments that were tightly constrained. It is even possible that the stimulus material used by Gratton et al. should be thought of as testing spatial memory for straight lines, rather than object memory. A further difference between the two studies is that Gratton et al. had no condition corresponding to our Lozenge template. A hemifield-specific memory store cannot have been the only memory mechanism available in Square-100, since memory performance after horizontal shifts in this task was not at chance. This implies that, in addition to the hemifield-specific store of visual memories, the subjects in Square-100 also made partial use of some other memory store, in which information was integrated across the vertical meridian. Equally the results from Lozenge-100, where no significant horizontal–vertical difference was seen, imply that in that task the subjects made no significant use of a hemifield-specific store of visual memories, and were instead relying entirely on a memory store in which information was integrated across the vertical meridian. As outlined in Section 1, the different fixation requirements in Square-100 and Lozenge-100 were designed to induce these different strategies of memory performance. Results from both tasks, however, raise the question as to the nature of those object memories which are not hemifield specific, and therefore do not conform to the expected properties of visual memories that are stored within the temporal lobe. In principle, verbal encoding of visually presented objects could provide a memory that is not hemifield specific. However, it seems unlikely that this could be powerful enough to explain any substantial part of memory performance in the present tasks. If one takes any one of the stimulus objects from the present experiment and constructs, at leisure, a verbal description of it that is sufficiently precise to enable the object to be discriminated from any other object in the pool we used, the resulting description is so long as to be almost impossible to remember. Brief verbal labels, names, on the other hand, are of little use because there were always several examples in the pool fitting any such brief label: several eagles, several houses, several shields, and so on. Furthermore, subjects frequently commented that they could not identify (name) the objects in the tachistoscopic exposures. Evidence of a more positive kind comes from the memory decrement, relative to the baseline case of no retinal change, which was produced by both vertical and horizontal shifts in Lozenge-100. If memory in this task was name based, why did a retinal change produce a memory decrement? The most plausible explanation is that memory in Lozenge-100 was indeed visual, and that the memory decrement after a reti-
1879
nal shift in this task was produced by a change in the visual appearance of the object. In the shift conditions, and not in the baseline no-change condition, the quarter of the object that is nearest the fixation point and thus most prominent to the observer is not the same at the retention test as in learning, both in horizontal and vertical shifts. We conclude that the memories in the present tasks which were not hemifield specific, as well as those that were, were visual memories. We have already argued that visual memories which are not hemifield specific cannot be stored within the temporal lobe, however. This implies that visual perceptual information in the temporal lobe not only contributes to memory storage in the temporal lobe, but is also available to extra-temporal cortical areas, in which a visual representation that is not hemifield specific must be constructed and stored. A lesion in the medial temporal lobe, either bilateral [5] or unilateral [15], would thus impair not only temporal-lobe memory storage, but also extra-temporal memory storage in areas that receive visual information from the temporal lobe. Thus, the severe visual memory impairment that is produced by medial temporal lesions is not to be explained by assuming that all visual memories are stored within the temporal lobe. Behavioural experiments with monkeys have demonstrated that an object presented for remembering in one hemifield can lay down a memory trace in both hemispheres, so long as the forebrain commissures are intact, and can thus, be accessed at retrieval from either the original or the opposite hemifield [22]. However, in these experiments and others of this kind, one hemifield is entirely blank, since the optic chiasm is sagittally sectioned and one eye is occluded. In the light of the results of the present experiment, it will be important for future studies to investigate the neural basis of transfer between hemifields in the monkey when both hemifields contain objects. The electrophysiological data we described in Section 1 [4] indicate that in this condition, when both hemifields contain objects, the commissural input to the temporal lobe from the ipsilateral hemifield is suppressed. This may appear to imply that the only function of the commissural ipsilateral input to the temporal lobe is when the contralateral hemifield is blank, an implausible conclusion. To the contrary, there is another important function for this ipsilateral commissural input. This is to allow all the parts of a single object to control object identification in both temporal lobes when an object straddles the vertical meridian. This was shown by an experiment with monkeys in which the optic chiasm was intact throughout [7]. Here the monkeys learned to discriminate among objects which were each constructed so as to straddle the midline and to have distinctive and independent parts in both hemifields, abutted together to form a single object. For example, representing the parts of a single object by letters of the alphabet, the monkeys learned to discriminate the single object HXM from the single object OXW, where X is at fixation. After such training the monkeys could transfer the learning to a test with rotated objects, now discriminating MXH from
1880
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
WXO; and this transfer was prevented by section of the anterior commissure and the splenium of the corpus callosum [7]. Where could the putative extra-temporal visual memory store be located? Ample evidence in nonhuman primates indicates that prefrontal cortex is essential to normal memory, not just in working memory but in a wide range of memory functions [11,20,21]. Furthermore, evidence in nonhuman and human primates indicates that prefrontal cortical activity adapts flexibly to task demands, including memory demands [6]. For these reasons, a recent review has concluded [10] that “memory may be just one of the many different clever things that the prefrontal cortex as a whole is capable of doing, in collaboration with temporal cortex, when required.” However, it would be wrong to rule out the possible contribution of other cortical areas, parietal and even occipital, to the storage of memory traces. We conclude that memory traces are stored in widespread areas of cortex, each of which contributes to other cognitive functions as well as to memory, rather than in a specialised memory system.
Acknowledgements This research was supported by the UK Medical Research Council. References [1] Bogasz R, Brown MW, Giraud-Carrier C. Model of familiarity discrimination in the perirhinal cortex. Journal of Computational Neuroscience 2001;10:5–23. [2] Buckley MJ, Booth MCA, Rolls ET, Gaffan D. Selective perceptual impairments after perirhinal cortex ablation. Journal of Neuroscience 2001;21:9824–36. [3] Buckley MJ, Gaffan D, Murray EA. Functional double-dissociation between two inferior temporal cortical areas: perirhinal cortex versus middle temporal gyrus. Journal of Neurophysiology 1997;97:587–98. [4] Chelazzi L, Duncan J, Miller EK, Desimone R. Responses of neurons in inferior temporal cortex during memory-guided visual search. Journal of Neurophysiology 1998;80:2918–40. [5] Corkin S, Amaral DG, Gonzalez RG, Johnson KA, Hyman BT. H.M.s medial temporal lobe lesion: findings from magnetic resonance imaging. Journal of Neuroscience 1997;17:3964–79.
[6] Duncan J. An adaptive coding model of neural function in prefrontal cortex. Nature Reviews Neuroscience 2001;2:820–9. [7] Eacott MJ, Gaffan D. Interhemispheric transfer of visual learning in monkeys with intact optic chiasm. Experimental Brain Research 1989;74:348–52. [8] Gaffan D. Dissociated effects of perirhinal cortex ablation, fornix transection and amygdalectomy: evidence for multiple memory systems in the primate temporal lobe. Experimental Brain Research 1994;99:411–22. [9] Gaffan D. Scene-specific memory for objects: a model of episodic memory impairment in monkeys with fornix transection. Journal of Cognitive Neuroscience 1994;6:305–20. [10] Gaffan D. What is a memory system? Horel’s critique revisited. Behavioural Brain Research 2001;127:5–11. [11] Gaffan D, Harrison S. A comparison of the effects of fornix transection and sulcus principalis ablation upon spatial learning by monkeys. Behavioural Brain Research 1989;31:207–20. [12] Gaffan D, Parker A. Interaction of perirhinal cortex with the fornix-fimbria: memory for objects and object-in-place memory. Journal of Neuroscience 1996;16:5864–9. [13] Gaffan D, Parker A, Easton A. Dense amnesia in the monkey after transection of fornix, amygdala and anterior temporal stem. Neuropsychologia 2001;39:51–70. [14] Gratton G, Corballis PM, Jain S. Hemispheric organisation of visual memories. Journal of Cognitive Neuroscience 1997;9:92–104. [15] Hornak J, Oxbury S, Oxbury J, Iversen SD, Gaffan D. Hemifield-specific visual recognition memory impairments in patients with unilateral temporal lobe removals. Neuropsychologia 1997;35:1311–5. [16] Meunier M, Bachevalier J, Mishkin M, Murray EA. Effects on visual recognition of combined and separate ablations of the entorhinal and perirhinal cortex in rhesus monkeys. Journal of Neuroscience 1993;13:5418–32. [17] Miyashita Y, Chang HS. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature 1988;331:68–70. [18] Murray EA. What have ablation studies told us about the neural substrates of recognition memory? Seminars in the Neurosciences 1996;8:13–22. [19] Murray EA, Bussey TJ. Perceptual-mnemonic function of the perirhinal cortex. Trends in Cognitive Sciences 1999;3:142–51. [20] Parker A, Gaffan D. Interaction of frontal and perirhinal cortices in visual object recognition memory in monkeys. European Journal of Neuroscience 1998;10:3044–57. [21] Parker A, Gaffan D. Memory after frontal-temporal disconnection in monkeys: conditional and nonconditional tasks, unilateral and bilateral frontal lesions. Neuropsychologia 1998;36:259–71. [22] Ringo JL. The medial temporal lobe in encoding, retention, retrieval and interhemispheric transfer of visual memory in primates. Experimental Brain Research 1993;96:387–403. [23] Squire LR, Zola-Morgan S. The medial temporal lobe memory system. Science 1991;253:1380–6.
The role of the vertical meridian in visual memory for objects J. Hornak a , J. Duncan b , D. Gaffan a,∗ a
Deparment of Experimental Psychology, Oxford University, South Parks Rd., Oxford OX1 3UD, UK b MRC Cognition and Brain Sciences Unit, 15 Chaucer Rd., Cambridge CB2 2EF, UK Received 2 November 2001; received in revised form 16 May 2002; accepted 17 May 2002
Abstract It is widely believed that, in human and nonhuman primates, visual memories of objects are stored in the temporal lobe. Electrophysiological results in monkeys, however, indicate that when a visual scene contains two or more objects, with at least one object in each visual hemifield, neurons in the temporal lobe of each hemisphere respond only to the objects that are in the contralateral visual hemifield, and their activity is unaffected by the objects in the ipsilateral hemifield. Putting these two premises together predicts that object memory should fail, or at least suffer a substantial decrement, when an object is presented for learning and retention as part of such a scene, but crosses the vertical meridian between the learning trial and the retention test. The effect of this change should be much greater than the effect of an equal retinal translation that crosses the horizontal rather than the vertical meridian. An experiment with normal human subjects verified this prediction under conventional conditions of tachistoscopic viewing, with a single constant fixation spot. A further condition in the same experiment, however, tested the same retinal translations in a more naturalistic condition, where the retinal changes were produced by varying the position on the display screen of the fixation spot rather than of the objects. Here, there was no significant special effect of crossing the vertical meridian. We conclude that visual memories are not stored exclusively in the temporal lobe. © 2002 Elsevier Science Ltd. All rights reserved. Keywords: Vertical meridian; Memory; Object
1. Introduction The medial and inferior temporal lobe, including the perirhinal and entorhinal cortex and the hippocampus as well as the white matter of the anterior temporal stem, is indispensable for normal memory processing in both human and nonhuman primates, as is shown by the severe memory impairments which follow lesions in these structures [5,13,18]. The inference that the temporal lobe contains a memory system [23], or multiple memory systems [8], in which memories are processed and stored, is widely accepted. The perirhinal cortex is the most important of these temporal-lobe structures in memory for visually presented objects, which is the most extensively studied kind of memory in nonhuman primates [16,18]; furthermore, the function of the perirhinal cortex has been doubly dissociated in the monkey from the function of adjacent structures in the temporal lobe [3,8]. On this basis it has been suggested [8] that the perirhinal cortex is a memory system, differentiated from other memory systems in the temporal lobe, and specialised ∗ Corresponding author. Tel.: +44-1865-27-1349; fax: +44-1865-31-0447. E-mail address: [email protected] (D. Gaffan).
for the processing and storage of object memories. According to this view, memory for objects that are presented in complex visual scenes [9] is produced by the interaction within each temporal lobe between perirhinal cortex and the other memory systems of the temporal lobe, including the hippocampus [12]. Against the simplest version of this hypothesis of temporal-lobe memory systems for scene memory, however, is evidence that some of the effects of perirhinal cortex ablations in monkeys are perceptual deficits rather than memory deficits [2], and arguments that some or all of the memory deficits that are produced by perirhinal cortex ablations in monkeys can be seen as a secondary effect of the loss of high-level perceptual representations of objects, rather than as direct loss of memory storage [2,19]. The present experiment is concerned with one of the features of temporal-lobe function that seems most difficult to reconcile with the idea that memories of visual scenes are processed and stored exclusively or mainly within the temporal lobe, as the hypothesis of temporal-lobe memory systems suggests. This is the fact that the representation of complex visual scenes in the temporal lobes is divided at the vertical meridian of the visual field between the two hemispheres. When a monkey sees a single object that is presented against a large blank background, neurons in the temporal
0028-3932/02/$ – see front matter © 2002 Elsevier Science Ltd. All rights reserved. PII: S 0 0 2 8 - 3 9 3 2 ( 0 2 ) 0 0 0 7 0 - 2
1874
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
lobe in both hemispheres respond to the object and signal its identity, even when the object is confined to one visual hemifield, and even though there are slight differences in strength of response favouring the neurons in the temporal lobe contralateral to the object [4]. However, a single object presented against a large blank background is not a natural condition of object vision and memory. In a more naturalistic scene-like condition, when two or more objects are presented, with at least one object in each visual hemifield, the difference between the two temporal lobes in the objects they respond to is stark. Neurons in the temporal lobe of each hemisphere now respond only to the objects that are in the contralateral visual hemifield, and their activity is unaffected by the objects in the ipsilateral hemifield [4]. This poses a problem for understanding the memory of objects that are presented in this fashion, that is, in scenes having at least one object in each visual hemifield. Consider the case where an object, a constituent of a visual scene that contains other objects, is presented for learning in one visual hemifield, but is subsequently presented for a recognition memory test in the opposite hemifield. In the learning trial, neurons in only one temporal lobe respond to the object, and in the retention test trial, neurons in only the other temporal lobe respond to the object. All proposals as to the neural basis of object memory assume that the memory of an object is stored as a modification in neurons that respond to the presentation of the object [1,17], and indeed it is difficult to see how neurons that do not respond to the presentation of an object could either lay down or retrieve a memory of it. It thus follows from the electrophysiological data, that neurons in neither of the two temporal lobes have both the opportunity to lay down and also the opportunity to retrieve a memory of the target object in the case we are considering, where the target object in a scene is presented in opposite hemifields at learning and at the retention test. Therefore, if object memories are stored either exclusively or mainly as modifications of neurons within the temporal lobe, memory for the object should either fail or suffer a substantial decrement in this case. The present experiment tested this counter-intuitive prediction in normal human subjects. We administered two object recognition memory tasks. The trials were presented tachistoscopically, in order to control the retinal location of the presented objects. On every trial, both in learning and at the retention test, there were two different objects, one in each hemifield. Thus, each learning trial in these tasks presented two different objects, both to be remembered, and one in each visual hemifield. In each task a total of 128 objects were presented for learning in this fashion, and after all the objects had been presented for learning there was a retention test, at which the subjects had to recognise the objects they had seen in the learning trials by distinguishing them from novel foils. Each retention test trial presented one previously learned object together with one foil object that had not been seen before, one object in each hemifield. Some of the previously learned objects were presented at the retention test in the same location in
the visual field as they had occupied in the learning trials, in order to provide a baseline measure of object memory in these conditions. Other objects, however, were presented at the retention test in the hemifield opposite to that which they had occupied in the learning trials (“horizontal shift”). The prediction, derived above, is that memory for these objects should either fail or suffer a substantial decrement. To control for the possibility that a shift in an object’s retinal location between learning and retention test is by itself sufficient to produce a memory decrement, even when it is not a shift across the vertical meridian, a third set of objects was presented at retention in the same hemifield as that which they had occupied in the learning trials, but shifted within that hemifield (“vertical shift”). In the monkey, temporal cortex neurons that fail to respond to objects in the ipsilateral hemifield, in the conditions we have discussed, respond in the same conditions to objects in the contralateral hemifield almost independently of their location within that hemifield [4]. Thus, the main question to be asked in each task is whether horizontal shifts produce worse memory performance than vertical shifts. A further important aspect of the design of the two tasks was the means by which the shifts in the retinal location of objects were accomplished. To vary an object’s retinal location, one can either change the position of the object while maintaining the position of the subject’s eyes, or maintain the position of the object while changing the position of the subject’s eyes; one cannot change only retinal position and nothing else. In tachistoscopic experiments, and in electrophysiological studies, there is usually a single constant fixation spot in the centre of the display screen, and objects are shifted from one retinal location to another by changing the object’s position on the display screen. In the most usual natural condition, however, when stable complex scenes are viewed in free vision in one or more trials, changes in the retinal location of an object result from changes in the position of the subject’s point of fixation, not from changes in the position of the object in the scene. Figs. 1 and 2 show how the two tasks instantiated these two different means by which an object can shift its retinal position. In tasks using the Square template there was a single central fixation point which was in the same position in all trials, and objects were shifted across the visual field, between learning and retention testing, by moving the objects on the display screen. But in tasks using the Lozenge template, exactly the same shifts of objects’ positions in the visual field were accomplished by changing the position of the fixation spot, leaving the objects in an unchanged position on the display screen (the three pairs of learning and test trials using the Square template, that are illustrated in the leftmost two columns of Fig. 2, are identical, in terms of the retinal positions of the target and foil objects at learning and test, to the three pairs using the Lozenge template, that are illustrated in the rightmost two columns of Fig. 2). Though identical in terms of the retinal positions of objects, these tasks are quite different from each other in the
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
Fig. 1. The templates for displays of fixation spots and objects. Each task used either the Square template or the Lozenge template. Each trial in any task displayed a fixation spot in one of the positions shown as ‘+’ in the figure, and two objects in two of the positions shown as ‘䊊’. The two objects were always in two positions adjacent to the fixation spot, and always on diametrically opposite sides of it. Trials generated using these templates are illustrated in Fig. 2.
1875
natural memory performances they resemble. In the task with the Square template the subject repeatedly sees the same view of the apparatus from exactly the same fixation point, and memory in this task thus resembles the natural task of reconstructing in memory a complex scene in which one fixated only one object. The task using the Lozenge template requires the subject to fixate different positions in successive views, and memory in this task thus, because of the successive fixations at different points, resembles the natural task of reconstructing in memory a complex scene which one inspected with successive saccades to different fixation points. If the subjects possess the ability to construct a memory of the scene that is not hemifield specific, the Lozenge task should encourage the use of that strategy. Thus, we expected the effects of shifts between hemifields, if any, to be weaker in the task using the Lozenge template than in the task using the Square template.
Fig. 2. Some of the possible configurations of learning trials and test trials in tasks using the Square template (leftmost two columns) and the Lozenge template (rightmost two columns). In each case a learning trial for an object is shown next to the retention test for that object. For a description of all the possible configurations, see the text.
1876
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
Using these two templates we created two tasks, Square100 and Lozenge-100, so named because 100 ms was the tachistoscopic exposure duration of each trial, chosen to be short enough to prevent within-trial saccades. Each subject in an initial group of subjects performed both of these tasks. Subsequently a separate group of subjects performed a third task, Square-1500. Our purpose in the third task was to establish whether a memory decrement would be produced by a change of the position of an object on the screen, in the same displays as in Square-100, but without the correlated change in its position in retinal space. The Square-1500 task was the same as the Square-100 task but the exposure duration of each display of objects, whether in learning or in retention tests, was 1500 ms instead of 100 ms, chosen to be long enough to allow within-trial saccades. This was to give the subjects sufficient time to briefly fixate each of the two objects. To equate the overall difficulty of the Square-1500 task to that of the Square-100 task, we presented each object in fewer learning trials in Square-1500 than in Square-100, and we required the subjects to remember more objects in Square-1500 than in Square-100. For the sake of clarity it should be noted that there are two sets of retinal relationships between object positions in our tasks: (1) the simultaneous relationship, between the retinal positions of two different objects presented together in any single trial, is always this: one object is in the retinal quadrant that is diametrically opposite to the retinal quadrant occupied by the other object. That is to say, if one object is in the lower left then the other is in the upper right, and if one object is in the upper left then the other is in the lower right, and these two are the only possible pairs of retinal positions of simultaneously presented objects that ever occurred, both in learning trials and in retention tests, both with the Square template and with the Lozenge template; (2) the successive relationship, between the retinal position of an object at its learning trial and the retinal position of the same object at its subsequent retention test, can be either identical, or shifted; and if it shifts, the shift can be either horizontal, that is to say, either between upper left and upper right or between lower left and lower right, or vertical, that is to say, either between upper left and lower left or between upper right and lower right.
2. Methods 2.1. Subjects These were 30 adult volunteers, recruited from subject panels in Oxford and Cambridge. Eighteen of the subjects each performed both the Square-100 and the Lozenge-100 tasks, and the order in which they performed those two tasks was determined in a pseudo-random manner, double alternating in order of the subjects’ entry to the experiment. Twelve subjects performed only the Square-1500 task.
2.2. Apparatus and stimulus material The experiment was conducted on a notebook computer. The screen displayed a square background, which was white throughout the task. The background square was 147 mm × 147 mm. The objects were a pool of 512 coloured clipart images. The size of any object was 35 mm × 35 mm, though this could include some white background. When displayed in either the Square or Lozenge templates (Fig. 1), the corner of the object that was nearest to the fixation spot was 6 mm up or down and 6 mm left or right from the fixation spot. In the Square template the fixation spot was in the centre of the white background square. In the Lozenge template the four possible positions for the fixation spot were positioned at the centre of the four possible positions of objects in the Square template. The effect was that, given the design (below), the relationships of objects to fixation spots were identical in the Lozenge and Square templates. Figs. 1 and 2 are not to scale but they illustrate the layout schematically. The fixation spot in the experiment was a black dot 0.25 mm × 0.25 mm. Subjects viewed the display from a distance of about 400 mm. Tachistoscopic exposures were timed with reference to the screen refresh cycle, to avoid flicker. 2.3. Square-100 task This task is so called because it used the “Square” template and because each display lasted 100 ms. The template is illustrated in Fig. 1 and some of the possible stimulus configurations are illustrated schematically in the left side of Fig. 2 (see Section 2.2 for the actual dimensions of the display). The fixation spot (shown as ‘+’ in the figures) was always in the centre of the display in this task, and objects could appear in four possible positions (shown as ‘䊊’ in Fig. 1). The display on any trial in the task had objects in only two of the four possible positions for objects shown for the Square template in Fig. 1. On any learning trial two objects were presented, in diametrically opposite quadrants of the visual field to each other. In subsequent retention tests, each object that had been learned was presented again, paired with a novel object in the diametrically opposite quadrant, and the subject judged which of these two objects had been seen before. The subject spoke to indicate whether the upper or lower of the two objects at the retention test was the one that had been presented during the learning trials. The experimenter wrote down the subject’s responses for later analysis. There were three conditions for the retention test. The previously learned object could be presented for retention test either in the same quadrant as in learning (no change), or in the horizontally opposite quadrant to that which it had occupied in learning (horizontal shift), or in the vertically opposite quadrant to that which it had occupied in learning (vertical shift). Fig. 2 illustrates some trials. All the trials, both in learning and in retention tests, were tachistoscopic. The fixation spot appeared on the display screen to indicate that the next trial was ready, and the subject fixated the spot before touching the
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
keyboard to make the objects appear. The objects appeared for 100 ms and then blanked into the white background. Each object to be remembered was presented in four learning trials. All the objects to be remembered were presented once each in a random order, then a second time once each in a new random order, and so on until they had all been presented four times. The random orders were unique to each subject. Every learning trial presented two objects. Any object was presented in the same quadrant in all four of its learning trials, but the object in the opposite quadrant was not the same in all four learning trials. Each subject was given 128 objects to learn. These were taken at random, in a different random order for each subject, from a pool of 512 available objects. Of these 128 objects that were learned, 64 were then tested for retention with no change of position, 32 were tested with horizontal shift, and 32 with vertical shift. Both at retention tests and at learning trials, the four possible positions for objects were occupied equally often by objects that changed position between learning and retention, and by objects that did not change position. At each of the same four possible positions, the objects that changed position between learning and retention were half shifted vertically and half shifted horizontally. The 128 retention test trials were in a unique random order for each subject. Eighteen subjects performed the task. 2.4. Lozenge-100 task This was identical to the Square-100 task, except that it used the Lozenge template (Fig. 1). There were four possible positions for the fixation spot (each shown as ‘+’ Fig. 1). Whichever fixation spot was used in any trial, there were two objects in opposite quadrants, in the same positions relative to the fixation spot as they would have occupied in a trial in the Square task. Thus, for example, if the fixation spot used in a trial in the Lozenge-100 task was the fixation spot at the lower left, then the two objects presented at that trial would always be one at the left object position and one at the bottom object position in the Lozenge template. In all four of the learning trials with any object, the object was always in the same position on the screen, and the fixation spot was always in the same position, but the object in the opposite retinal quadrant was not the same in all four learning trials. At the retention test, the shifts of objects relative to the fixation spot were identical to those in the Square task, but were accomplished by moving the fixation spot rather than the object. The right half of Fig. 2 illustrates some trials. As in Square-100, 128 objects were learned by each subject and subsequently tested for retention, 64 with no change of position, 32 with horizontal shift, and 32 with vertical shift. The four possible fixation spots were used equally often. The four possible positions for objects at retention tests and at learning trials were occupied equally often by objects that changed their retinal position (position relative to the fixation spot) between learning and retention, and by objects that did not change position. Objects that underwent
1877
a horizontal shift of retinal position had to occupy either the top or bottom of the four possible positions for objects, and objects that underwent a vertical shift had to occupy either the left or right of the four possible positions for objects. Fig. 2 shows 3 out of the total of 16 possible combinations of the spatial configuration of the learning trials and the spatial configuration of the retention test for an object in the Lozenge task. These 16 possible spatial combinations were used equally often, each combination being applied to eight unique objects, and each of these 16 combinations corresponded to an identical combination of spatial configurations, in terms of retinal locations, for an equal number of objects in the Square-100 task. The same 18 subjects who performed Square-100 also performed Lozenge-100. Half the subjects performed Square-100 first then Lozenge-100, and the other half performed the tasks in the opposite order. For each subject, the 256 objects used in Lozenge-100 were those that were not used, for that subject, in Square-100. 2.5. Square-1500 task This was identical to the Square-100 task except for the following changes. The exposure duration of each display of objects, whether in learning or in retention tests, was 1500 ms instead of 100 ms, and the subjects were instructed to glance at each of the objects during this interval. Each object was presented in only one learning trial, instead of four. The subjects were given 256 objects to learn, instead of 128. Twelve further subjects, who had not performed the Square-100 and Lozenge-100 tasks, performed this Square-1500 task.
3. Results The results are shown in Fig. 3. One group of subjects performed both the Square-100 and the Lozenge-100 tasks.
Fig. 3. Results from the three tasks.
1878
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
Half of these subjects performed the Square-100 task before the Lozenge-100 task, and the others performed the tasks in the opposite order. Preliminary analyses of the results from these two tasks showed that this between-subjects factor, order of tasks, had no overall effect on performance and did not interact with any other factor, F < 1 in every case. Task order was therefore discarded from the analyses of these two tasks. Results from the vertical and horizontal shift conditions in Square-100 and Lozenge-100 were entered into an analysis with the factors shift type (two levels, vertical and horizontal) and task (two levels, Square-100 and Lozenge-100). The main effects of shift type and task were not significant (both F < 1). The interaction of shift type with task was significant (F = 7.187, d.f. 1, 17, P = 0.016). This interaction was further analysed by designed comparisons, using the appropriate pooled error terms from the interaction and the nonsignificant main effects. Within the Lozenge-100 task, there was no significant difference between horizontal and vertical shifts (t < 1). Within the Square-100 task, however, horizontal shifts produced significantly less accurate retention test performance than vertical shifts, as predicted (t = 2.34, d.f. 17, P = 0.016, one-tailed). Further, horizontal shifts in Square-100 produced significantly less accurate retention test performance than horizontal shifts in Lozenge-100, as predicted (t = 1.92, d.f. 17, P = 0.036, one-tailed), while performance after vertical shifts did not differ significantly between the two tasks (t = 1.55, d.f. 17, P = 0.140, two-tailed). In Square-100, any of the four possible object positions, either in learning trials or in retention tests, is equally likely to be occupied by an object that undergoes a horizontal shift as by an object that undergoes a vertical shift. In Lozenge-100, however, objects that undergo a horizontal shift of retinal position necessarily occupy the top or bottom object positions in the Lozenge template (Figs. 1 and 2) while objects that undergo a vertical shift necessarily occupy the left or right positions in the template. In principle, therefore, the effects of these shifts in the Lozenge template might need to be assessed against different baselines. This would be the case if the objects that underwent no change of retinal position produced better memory performance when they were in the top or bottom positions than when they were in the left or right positions, or vice versa. In fact, however, the mean retention accuracy for these two classes of no-change objects in the Lozenge-100 task were almost identical (78.3 and 78.5% correct at the retention test, respectively) and the difference between them did not approach statistical significance (t < 1). Results from Square-100 and Lozenge-100 were further analysed to assess the effect of change, averaging over horizontal and vertical shifts and comparing this average with performance on the no-change trials. Overall the effect of change was significant (F = 27.755, d.f. 1,17, P = 0.000) and the effect of change did not interact with task (F < 1). Designed comparisons derived from this analysis confirmed
that the effect of change was significant within each of the two tasks (smaller t = 3.828, d.f. 17, P = 0.001) and that performance in the retention test trials with no change did not differ between the two tasks (t < 1). The results from the free-vision task, Square-1500, are also shown in Fig. 3. This task was performed by a separate group of subjects, drawn from the same pool of subjects as those who performed the first two tasks. In Square-1500 there was no difference between vertical and horizontal shifts (F < 1) and no effect of change (F < 1). 4. Discussion In Square-100, memory accuracy was substantially impaired when an object changed from one visual hemifield to the other between learning and retention test, crossing the vertical meridian of the visual field in a horizontal shift. Performance in this condition was significantly worse than after a comparable shift vertically, within a visual hemifield. We can see only three possible explanations for this difference between horizontal and vertical shifts in the Square-100 task. First, it is conceivable that a horizontal change in the position of an object in the world, irrespective of its retinal position, might be more deleterious to memory than a vertical change of position in the world. An explanation along these lines is ruled out, however, by the failure of a horizontal–vertical difference to appear in free vision (Square-1500). Second, it is conceivable that some unforeseen properties of the stimulus objects that were used in the present tasks could make horizontal change on the retina more deleterious than vertical change. For example, if the quarter of an object that is nearest to the fixation point is more salient than the other quarters of the object, and if objects tend to be up–down symmetrical but not left–right symmetrical, then a horizontal shift will produce a bigger change in the appearance of an object than a vertical shift will. An explanation along these lines applies to Lozenge-100 in exactly the same way as to Square-100, however, and therefore is ruled out by the absence of a horizontal–vertical difference in Lozenge-100. Third, the only remaining possibility appears to be that the horizontal shift between visual hemifields in Square-100 was more deleterious to memory than the equivalent vertical shift within a hemifield because object memories in this task were, to a large extent, hemifield specific. This last was the hypothesis we derived in Section 1, where we saw that, because of the electrophysiological properties of temporal lobe neurons, visual memories stored within the temporal lobe should be expected to be hemifield specific. Our Square-100 task is similar in some ways to an earlier task used by Gratton et al. [14]. These authors reported, as our results confirm, that horizontal shifts between hemifields are more deleterious than vertical shifts within hemifields in tachistoscopic visual recognition memory with a single constant fixation spot. The effect reported by Gratton et al. seems less powerful than the effect we observed, and if so this
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
is consistent with the electrophysiological data (Section 1), since Gratton et al. presented only one object in the visual field on each trial. However, it is difficult to compare the two sets of results directly, since the baseline performance with no shift was at a much lower level in the earlier experiment than in the present experiment. This is probably due to the difference in stimulus materials, since the stimuli used by Gratton et al. were structures of straight-line segments that were tightly constrained. It is even possible that the stimulus material used by Gratton et al. should be thought of as testing spatial memory for straight lines, rather than object memory. A further difference between the two studies is that Gratton et al. had no condition corresponding to our Lozenge template. A hemifield-specific memory store cannot have been the only memory mechanism available in Square-100, since memory performance after horizontal shifts in this task was not at chance. This implies that, in addition to the hemifield-specific store of visual memories, the subjects in Square-100 also made partial use of some other memory store, in which information was integrated across the vertical meridian. Equally the results from Lozenge-100, where no significant horizontal–vertical difference was seen, imply that in that task the subjects made no significant use of a hemifield-specific store of visual memories, and were instead relying entirely on a memory store in which information was integrated across the vertical meridian. As outlined in Section 1, the different fixation requirements in Square-100 and Lozenge-100 were designed to induce these different strategies of memory performance. Results from both tasks, however, raise the question as to the nature of those object memories which are not hemifield specific, and therefore do not conform to the expected properties of visual memories that are stored within the temporal lobe. In principle, verbal encoding of visually presented objects could provide a memory that is not hemifield specific. However, it seems unlikely that this could be powerful enough to explain any substantial part of memory performance in the present tasks. If one takes any one of the stimulus objects from the present experiment and constructs, at leisure, a verbal description of it that is sufficiently precise to enable the object to be discriminated from any other object in the pool we used, the resulting description is so long as to be almost impossible to remember. Brief verbal labels, names, on the other hand, are of little use because there were always several examples in the pool fitting any such brief label: several eagles, several houses, several shields, and so on. Furthermore, subjects frequently commented that they could not identify (name) the objects in the tachistoscopic exposures. Evidence of a more positive kind comes from the memory decrement, relative to the baseline case of no retinal change, which was produced by both vertical and horizontal shifts in Lozenge-100. If memory in this task was name based, why did a retinal change produce a memory decrement? The most plausible explanation is that memory in Lozenge-100 was indeed visual, and that the memory decrement after a reti-
1879
nal shift in this task was produced by a change in the visual appearance of the object. In the shift conditions, and not in the baseline no-change condition, the quarter of the object that is nearest the fixation point and thus most prominent to the observer is not the same at the retention test as in learning, both in horizontal and vertical shifts. We conclude that the memories in the present tasks which were not hemifield specific, as well as those that were, were visual memories. We have already argued that visual memories which are not hemifield specific cannot be stored within the temporal lobe, however. This implies that visual perceptual information in the temporal lobe not only contributes to memory storage in the temporal lobe, but is also available to extra-temporal cortical areas, in which a visual representation that is not hemifield specific must be constructed and stored. A lesion in the medial temporal lobe, either bilateral [5] or unilateral [15], would thus impair not only temporal-lobe memory storage, but also extra-temporal memory storage in areas that receive visual information from the temporal lobe. Thus, the severe visual memory impairment that is produced by medial temporal lesions is not to be explained by assuming that all visual memories are stored within the temporal lobe. Behavioural experiments with monkeys have demonstrated that an object presented for remembering in one hemifield can lay down a memory trace in both hemispheres, so long as the forebrain commissures are intact, and can thus, be accessed at retrieval from either the original or the opposite hemifield [22]. However, in these experiments and others of this kind, one hemifield is entirely blank, since the optic chiasm is sagittally sectioned and one eye is occluded. In the light of the results of the present experiment, it will be important for future studies to investigate the neural basis of transfer between hemifields in the monkey when both hemifields contain objects. The electrophysiological data we described in Section 1 [4] indicate that in this condition, when both hemifields contain objects, the commissural input to the temporal lobe from the ipsilateral hemifield is suppressed. This may appear to imply that the only function of the commissural ipsilateral input to the temporal lobe is when the contralateral hemifield is blank, an implausible conclusion. To the contrary, there is another important function for this ipsilateral commissural input. This is to allow all the parts of a single object to control object identification in both temporal lobes when an object straddles the vertical meridian. This was shown by an experiment with monkeys in which the optic chiasm was intact throughout [7]. Here the monkeys learned to discriminate among objects which were each constructed so as to straddle the midline and to have distinctive and independent parts in both hemifields, abutted together to form a single object. For example, representing the parts of a single object by letters of the alphabet, the monkeys learned to discriminate the single object HXM from the single object OXW, where X is at fixation. After such training the monkeys could transfer the learning to a test with rotated objects, now discriminating MXH from
1880
J. Hornak et al. / Neuropsychologia 40 (2002) 1873–1880
WXO; and this transfer was prevented by section of the anterior commissure and the splenium of the corpus callosum [7]. Where could the putative extra-temporal visual memory store be located? Ample evidence in nonhuman primates indicates that prefrontal cortex is essential to normal memory, not just in working memory but in a wide range of memory functions [11,20,21]. Furthermore, evidence in nonhuman and human primates indicates that prefrontal cortical activity adapts flexibly to task demands, including memory demands [6]. For these reasons, a recent review has concluded [10] that “memory may be just one of the many different clever things that the prefrontal cortex as a whole is capable of doing, in collaboration with temporal cortex, when required.” However, it would be wrong to rule out the possible contribution of other cortical areas, parietal and even occipital, to the storage of memory traces. We conclude that memory traces are stored in widespread areas of cortex, each of which contributes to other cognitive functions as well as to memory, rather than in a specialised memory system.
Acknowledgements This research was supported by the UK Medical Research Council. References [1] Bogasz R, Brown MW, Giraud-Carrier C. Model of familiarity discrimination in the perirhinal cortex. Journal of Computational Neuroscience 2001;10:5–23. [2] Buckley MJ, Booth MCA, Rolls ET, Gaffan D. Selective perceptual impairments after perirhinal cortex ablation. Journal of Neuroscience 2001;21:9824–36. [3] Buckley MJ, Gaffan D, Murray EA. Functional double-dissociation between two inferior temporal cortical areas: perirhinal cortex versus middle temporal gyrus. Journal of Neurophysiology 1997;97:587–98. [4] Chelazzi L, Duncan J, Miller EK, Desimone R. Responses of neurons in inferior temporal cortex during memory-guided visual search. Journal of Neurophysiology 1998;80:2918–40. [5] Corkin S, Amaral DG, Gonzalez RG, Johnson KA, Hyman BT. H.M.s medial temporal lobe lesion: findings from magnetic resonance imaging. Journal of Neuroscience 1997;17:3964–79.
[6] Duncan J. An adaptive coding model of neural function in prefrontal cortex. Nature Reviews Neuroscience 2001;2:820–9. [7] Eacott MJ, Gaffan D. Interhemispheric transfer of visual learning in monkeys with intact optic chiasm. Experimental Brain Research 1989;74:348–52. [8] Gaffan D. Dissociated effects of perirhinal cortex ablation, fornix transection and amygdalectomy: evidence for multiple memory systems in the primate temporal lobe. Experimental Brain Research 1994;99:411–22. [9] Gaffan D. Scene-specific memory for objects: a model of episodic memory impairment in monkeys with fornix transection. Journal of Cognitive Neuroscience 1994;6:305–20. [10] Gaffan D. What is a memory system? Horel’s critique revisited. Behavioural Brain Research 2001;127:5–11. [11] Gaffan D, Harrison S. A comparison of the effects of fornix transection and sulcus principalis ablation upon spatial learning by monkeys. Behavioural Brain Research 1989;31:207–20. [12] Gaffan D, Parker A. Interaction of perirhinal cortex with the fornix-fimbria: memory for objects and object-in-place memory. Journal of Neuroscience 1996;16:5864–9. [13] Gaffan D, Parker A, Easton A. Dense amnesia in the monkey after transection of fornix, amygdala and anterior temporal stem. Neuropsychologia 2001;39:51–70. [14] Gratton G, Corballis PM, Jain S. Hemispheric organisation of visual memories. Journal of Cognitive Neuroscience 1997;9:92–104. [15] Hornak J, Oxbury S, Oxbury J, Iversen SD, Gaffan D. Hemifield-specific visual recognition memory impairments in patients with unilateral temporal lobe removals. Neuropsychologia 1997;35:1311–5. [16] Meunier M, Bachevalier J, Mishkin M, Murray EA. Effects on visual recognition of combined and separate ablations of the entorhinal and perirhinal cortex in rhesus monkeys. Journal of Neuroscience 1993;13:5418–32. [17] Miyashita Y, Chang HS. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature 1988;331:68–70. [18] Murray EA. What have ablation studies told us about the neural substrates of recognition memory? Seminars in the Neurosciences 1996;8:13–22. [19] Murray EA, Bussey TJ. Perceptual-mnemonic function of the perirhinal cortex. Trends in Cognitive Sciences 1999;3:142–51. [20] Parker A, Gaffan D. Interaction of frontal and perirhinal cortices in visual object recognition memory in monkeys. European Journal of Neuroscience 1998;10:3044–57. [21] Parker A, Gaffan D. Memory after frontal-temporal disconnection in monkeys: conditional and nonconditional tasks, unilateral and bilateral frontal lesions. Neuropsychologia 1998;36:259–71. [22] Ringo JL. The medial temporal lobe in encoding, retention, retrieval and interhemispheric transfer of visual memory in primates. Experimental Brain Research 1993;96:387–403. [23] Squire LR, Zola-Morgan S. The medial temporal lobe memory system. Science 1991;253:1380–6.