Anticipatory spatial representation of 3D regions explored by sighted observers and a deaf-and-blind-observer

Cognition 94 (2004) 19–37 www.elsevier.com/locate/COGNIT

Anticipatory spatial representation of 3D regions explored by sighted observers and a deaf-and-blind-observer Helene Intraub* Psychology Department, University of Delaware, Newark, DE 19716, USA Received 13 June 2003; accepted 23 October 2003

Abstract Viewers who study photographs of scenes tend to remember having seen beyond the boundaries of the view [boundary extension; J. Exp. Psychol. Learn. Mem. Cogn. 15 (1989) 179]. Is this a fundamental aspect of scene representation? Forty undergraduates explored bounded regions of six common (3D) scenes, visually or haptically (while blindfolded) and then the delimiting borders were removed. Minutes later they reconstructed boundary placement. Boundary extension occurred: mean areas were increased by 53% (vision) and by 17% (haptics). A deaf-and-blind woman (KC) haptically explored the same regions. Although a “haptic expert”, she too remembered having explored beyond the boundaries, with performance similar to that of the blindfolded-sighted. Boundary extension appears to be a fundamental aspect of spatial cognition. Possibly constrained by the “scope” of the input modality (vision . haptics), this anticipatory spatial representation may facilitate integration of successively perceived regions of the world irrespective of modality and the perceiver’s sensory history. q 2004 Elsevier B.V. All rights reserved. Keywords: Boundary extension; Vision; Haptics

1. Anticipatory representation of layout in a deaf-and-blind observer KC is a “haptic expert.” Touch and movement form the basis of her interactions with the world. A 25-year-old college student, she has been blind and deaf since early life1 due * Tel.: þ1-302-831-8012. E-mail address: [email protected] (H. Intraub). 1 KC’s lack of vision and audition was caused by a rare degenerative genetic disorder called Leber’s Syndrome. It is unknown if she ever had normal vision early in infancy, but whatever visual capability was present decreased rapidly causing her to rely upon haptics while exploring her world. At 2 12 years old, her electroretinogram (ERG) was flat. A hearing test at age 1 12 years revealed “low” hearing loss in one ear and “moderate” hearing loss in the other, which decreased rapidly. Her hearing impairment prevented her from learning spoken language. She was described as profoundly deaf-and-blind by the time she turned 5. 0022-2860/$ - see front matter q 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.cognition.2003.10.013

20

H. Intraub / Cognition 94 (2004) 19–37

to a rare genetic disorder called Leber’s Syndrome. She has relied on haptics to master activities that are usually accomplished primarily through vision or audition. Her language is American Sign Language (ASL), which she understands by lightly touching another signer’s moving hands. She communicates with non-ASL signers either through a translator or by typing into a computer with a Braille interface. She uses a long cane to aid navigation. Her perception of the objects and surfaces around her (i.e. scene perception) is carried out through haptic exploration with her hands. Needless to say, in contrast to KC, most individuals explore scenes predominantly through the visual modality. The purpose of this research is not to determine if KC is “as good” as sighted observers at remembering specific regions of scenes. Instead it is to determine if she is “as bad.” Although sighted individuals can be thought of as “visual experts” they are nevertheless prone to an interesting misrepresentation when remembering a close-up photograph of a scene; they tend to remember having seen beyond the edges of the view. This occurs whether or not an object was cropped by a boundary—viewers simply remember having seen more of the background, extrapolating expected layout. First reported by Intraub and Richardson (1989), this unidirectional spatial error is referred to as boundary extension. It occurs when memory is tested via recall, recognition, or reconstruction of the view (see Intraub, 2002), and occurs as rapidly as 1 s following stimulus offset (Intraub, Gottesman, Willey, & Zuk, 1996). Boundary extension does not appear to be a general visual memory error; but is specifically tied to pictures of scenes (Intraub, Gottesman, & Bills, 1998; also see Gottesman & Intraub, 2002, 2003; Legault & Standing, 1992), with tight close-ups yielding the most extreme amounts (Intraub, Bender, & Mangels, 1992; Intraub et al., 1998). Boundary extension, which occurs even when viewers try to guard against it (Intraub & Bodamer, 1993), has been documented in individuals ranging from 6 to 84 years of age (Seamon, Schlegel, Heister, Landau, & Blumenthal, 2002).2 Research on scene representation almost always focuses on the visual modality—and usually on perception of 2D pictures of scenes. Given the dominance of vision in most individual’s perception of scenes, this is understandable. However, this pervasive focus raises the risk of automatically attributing to visual cognition, characteristics of representation that in fact may be more universal—underlying other relevant modalities. Many studies involving populations with unusual sensory histories have demonstrated that this “visual bias” can lead to the characterization of mental events as visual, or specific activities as requiring a visual representation to be successfully accomplished, when in fact this is not the case. For example, in their study of language acquisition in the blind, Landau and Gleitman (1985) reported that a congenitally blind child acquired visual terms and then used them appropriately without ever having experienced visual input; Klatzky, Golledge, Loomis, and Cicinelli (1995) and Loomis et al. (1993) reported that congenitally blind participants’ performance was comparable to that of adventitiously blind or blindfolded-sighted 2 It’s important to note that boundary extension is one of many examples of anticipatory processes in perception and cognition (see Intraub, 2002). In the case of visual memory, it has been demonstrated under a variety of conditions that the last location of a moving object tends to be displaced farther along the path of motion—at its anticipated location rather than its actual location (i.e. “representational momentum”: Freyd & Finke, 1984; see Freyd, 1993), and that this occurs even when motion is simply indicated, as in the case of frozen motion photographs (Freyd, 1983; Futterweit & Beilin, 1994).

H. Intraub / Cognition 94 (2004) 19–37

21

participants on a variety of spatial and navigational tasks in spite of their lack of exposure to visual representation. In other research, the pattern of response times typically interpreted as indicating “mental rotation” (originally reported and discussed within the context of visual imagery, e.g. Cooper & Shepard, 1973; Shepard & Metzler, 1971) was found to occur when congenitally blind, adventitiously blind, and blindfolded observers, participated in analogous haptic tasks (e.g. Carpenter & Eisenberg, 1978; Marmor & Zaback, 1976). There are, of course, obvious differences among sensory modalities that make certain capabilities unique to a sense (e.g. color perception), and there are many situations in which the lack of vision places limitations on spatial ability (e.g. see Loomis et al., 1993; Rieser, Hill, Talor, Bradfield, & Rosen, 1992). Haptic analogs of some wellknown visual illusions have been found to yield similar effects to vision, but others have not (e.g. Suzuki & Arashida, 1992). In the case of boundary extension plausible arguments can be made to support predictions either way: (a) that boundary extension reflects unique characteristics of the visual system (and thus will not occur following haptic exploration) or (b) that it reflects universal characteristics of scene perception (irrespective of modality). These arguments will be discussed in the following sections. 1.1. Boundary extension in visual memory The world is continuous, but sensory input is not. The borders of the visual field limit the scope of each view (see Bingham, 1993; Gibson, 1950), and high acuity is restricted to the tiny foveal region (only about 28 of visual angle). Head and eye movements allow the viewer to obtain detailed information from different regions of a scene. This can be accomplished very rapidly, with eye movements occurring as quickly as 4 times per second. However, the movements themselves place another constraint on perception because while the eyes move, vision is suppressed and information must be retained in memory (see Irwin, 1991, 1993). Research suggests that this transsaccadic representation is not detailed and “picture-like” (e.g. Irwin, 1991, 1993; McConkie & Currie, 1996; Rayner & Pollatsek, 1992) although information about layout and attended features are likely to be retained (see Intraub, 1997; Rensink, 2000; Wolfe, 1999). Yet somehow, viewers perceive and remember a coherent and continuous world. Hochberg has argued that during visual scanning, expectations about unseen but highly predictable portions of a display are included in the mental representation (e.g. Hochberg, 1986). Intraub and her colleagues have proposed that boundary extension reflects the salience of these anticipatory projections—with viewers remembering a portion of the expected space as having been seen (e.g. Gottesman & Intraub, 2002; Intraub, 2002; Intraub et al., 1996). Given that the visual system must provide a continuous representation of a continuous world, it would seem to be adaptive to “ignore” the spurious boundaries associated with any one view and extrapolate beyond them. Anticipatory representation of the unseen space could then serve to facilitate integration of a view with subsequent views (e.g. by priming upcoming layout Gottesman, 2003; Sanocki & Epstein, 1997), and to facilitate detection of unexpected discontinuities in an upcoming view (e.g. as when the next view reveals an object on an otherwise homogeneous floor). The same anticipatory processes that allow for the integration of successive views during visual scanning, may similarly be elicited by any truncated view: for example,

22

H. Intraub / Cognition 94 (2004) 19–37

when viewing a photograph, a movie screen, a TV screen, or a view through a window (see Hochberg, 1986; and Intraub, 2002). These anticipatory projections beyond a view-border may draw upon the same processes that subserve amodal completion (e.g., Michotte, Thines, & Crabbe, 1964; Nakayama, et al., 1995) and amodal continuation (e.g., Kellman & Shipley, 1991; Kellman, Yin, & Shipley, 1998; Yin, Kellman & Shipley, 1997) as well as drawing upon real world knowledge. Anticipatory spatial representation may ultimately support computation of a cognitive map of explored space as successive views accumulate – each contributing its own extended region (e.g., see Hock and Schmelzkopf, 1980). If only a single view is presented and memory tapped, boundary extension will be observed. On the other hand, although it is tempting to generalize the results of picture memory studies to memory for views of real 3D scenes: it is possible that boundary extension may actually be a memory error associated with 2D representations of space. This is because pictures do not contain many of the cues inherent in a 3D view: e.g. stereopsis, motion parallax, the perception of objects and boundaries in direct relation to the viewer’s body, etc. These rich sources of information may serve to constrain mental representation under normal viewing conditions and prevent errors like boundary extension from occurring. If boundary extension were limited to memory for pictures, then clearly its relevance to scene perception would be disproved. Conversely, if boundary extension is fundamental to visual scene perception, then it should also be evident in memory for a truncated view of a 3D scene (like a view through a window). If instead, it reflects a fundamental characteristic of how we perceive our surrounding world, then in addition to vision, we would also expect to see anticipatory projection of layout in memory following haptic exploration of the same truncated views. 1.2. Vision vs. haptics Viewers clearly remember having seen beyond the edges of a photograph. But would they remember having felt beyond a small region of a scene that they manually explored (without vision)? In contrast to vision, which is a distal sense, with a small foveal region and large low-acuity periphery, touch is a contact sense with multiple high acutely regions (i.e. the 5 fingertips as 5 “fovea”) and a relatively small low-acuity periphery (e.g. the size of a hand). The span of the hands is much smaller than the span of the eyes. These differences may make it less likely that explored space and anticipated space would be readily confused in the observer’s mental representation. The notion that “touch teaches vision” has a long history in psychology—including the sense that haptic input can be used to test the reliability of certain visual cues (see Atkins et al., 2001). After haptically exploring a bounded region of a scene without vision (see Fig. 1b), a more veridical representation of the boundaries may be obtained—one that does not yield the large unidirectional (outward) errors that characterize boundary extension. Of course, haptic exploration involves both touch and movement. Memory for the extent of deliberate hand movements across a small bounded region may also serve to provide information that would prevent errors regarding explored spatial expanse. On the other hand, in spite of their differences, vision and haptics do share a common problem where it comes to scene perception; both must provide the perceiver with a coherent representation of a continuous environment that can be explored only a part at

H. Intraub / Cognition 94 (2004) 19–37

23

Fig. 1. Visual exploration (a) and haptic exploration (b) of the “toys” scene (all borders were removed prior to test).

a time. For William James, this observation was a central point in his argument against the theory that blind individuals represent the spatial world in a radically different way than do sighted individuals—pointing out that the apparent piecemeal nature of haptic input is no different than the “innumerable stoppings and startings of the eyeballs” during visual perception, and that mental representation likely melds these successive inputs similarly in

24

H. Intraub / Cognition 94 (2004) 19–37

both cases (James, 1890). O’Regan (1992) illustrates the similarity between visual and haptic explorations through an analogy. If one holds a bottle of water (without looking at it), one perceives a bottle even though the sensory receptors provide information in the shape of a hand. The lack of specific knowledge about the type of bottle cap is not troubling to the perceiver (any more than the lack of detail in the low-acuity periphery of vision). If it is necessary to determine the type of bottle cap—a hand movement (similar to a saccade in visual scanning) will bring that detail into focus. Mental representation of the environment does not require simultaneous representation of all details to yield a useful representation, but must provide a coherent representation beyond a given “view”: one into which new information can be incorporated as the perceiver “samples” new regions. The act of exploring may serve to raise expectations about space beyond an explored region: boundary extension may occur irrespective of the input modality.

2. Experiment Research on scene perception sometimes includes fairly wide-angle views of scenes, e.g. views of a kitchen, or an office that one could only see from a distance (e.g. Biederman, 1981; Hollingworth & Henderson, 1998), and it sometimes includes fairly close-up views, e.g. the kinds of views that one sees when one is in the kitchen, using the sink, or in an office, sitting at a desk looking down at a book and nearby desktop paraphernalia (e.g. Intraub et al., 1992; Palmer, 1975; Potter, 1976). In the present study, memory for close-up views such as the latter served as stimuli. This is because they can be readily explored either visually or haptically. In addition, in the case of visual perception it seems likely that size and area judgments would be most accurate in the space directly in front of a viewer (as opposed to a distant view through a window: see Previc, 1998, for a review of theories regarding perception of near vs. far space)—thus providing a conservative test of whether boundary extension occurs in memory for real space. Based on research with pictorial views, close-ups yield the greatest amount of boundary extension (e.g. Gottesman & Intraub, 2002; Intraub et al., 1992, 1998). To set view-boundaries and allow us to test memory for regions that we know were not actually perceived, participants explored each stimulus through a “window” (see Fig. 1). After studying six such views the boundaries were all removed, without changing the position of the objects. Minutes later, the participants returned and using the same modality as before, re-explored the space and indicated the remembered location of each boundary. If boundary extension occurs in memory for the visually explored regions directly in front of an observer, this would show that boundary extension is not limited to picture memory. If it occurs for sighted participants in both the vision and haptic conditions, this would suggest that it is a general aspect of visual scene representation, but it would not shed any light on the question of whether the haptic modality itself, like vision—is prone to this anticipatory error. This is because haptic exploration without any visual input is a very unusual situation for the sighted, and although they are blindfolded, they may still recruit visual resources (e.g. visual imagery) to remember the scenes (see Intraub et al., 1998 for research on visual imagery and boundary extension). To determine if someone whose normal mode of scene exploration is through the haptic modality would

H. Intraub / Cognition 94 (2004) 19–37

25

also remember having explored beyond the boundaries, KC participated in the haptic condition. Without vision or audition to provide additional input about surrounding scenes, her reliance on haptics is profound.

3. Alternative outcomes and their implications KC’s daily experience with haptics includes making fine distinctions with her hands (e.g. perception of objects and faces, Braille reading, and comprehension of ASL through touch). If boundary extension is a visual phenomenon, then KC should not be prone to it and instead may accurately represent the location of the boundaries—exhibiting only random errors (i.e. nudging a border inward or outward). In contrast, if boundary extension is a fundamental aspect of scene perception that is related to the integration of successively perceived regions of the world then she too should remember having touched beyond the boundaries. If she does, then a comparison of the magnitude of her boundary extension with that elicited from the blindfolded-sighted, and from viewers would be telling. Some possibilities include the following. If KC and the viewers show similar amounts of boundary extension, and this surpasses that observed in the haptic condition (blindfolded-sighted participants), this would suggest that the expanse of the anticipated region in memory is tied one’s dominant mode of scene exploration. If KC and the blindfolded-sighted participants show similar amounts of boundary extension, and it is less pronounced than that observed in the visual condition, this would suggest that spatial extrapolation is constrained to some extent by sensory modality: i.e. less extrapolation for haptics (a contact sense) than for vision (a distal sense). If KC’s extrapolated regions were very small relative to that of the blindfolded-sighted (e.g. the width of part of a finger tip), this would suggest that the amount of boundary extension is closely tied to the size of the sensory periphery of the modality, and that the blindfolded-sighted observers’ representations may have included the influence of visual imagery.

4. Method 4.1. Participants In addition to KC, 40 undergraduates also attending the University of Delaware took part in the experiment: 20 (14 females) in the haptic control condition, and 20 (12 females) in the visual control condition. 4.2. Stimuli Objects (3-5) were arranged on distinctive natural backgrounds in two rooms to create commonplace settings: 3 on the floor and 3 on tabletops. The central portions of the six scenes are shown in Fig. 2. Backgrounds included various tabletop surfaces, tile flooring, carpeting, and in the case of the “gym scene”, large exercise mats. It is important to note

26

H. Intraub / Cognition 94 (2004) 19–37

Fig. 2. Stimulus regions and their dimensions (photographs were cropped to approximate the view through the “window”).

that although these stimuli are 3D they do not differ in terms of border placement from stimuli used in the majority of prior studies with photographs: like those, the borders crop the expanse of the scene by occluding the view of the background surface and they do not crop any objects, thus ruling out the role of object completion as the cause of boundary extension (e.g. Intraub et al., 1992).

H. Intraub / Cognition 94 (2004) 19–37

27

A “window” exposed a truncated region of each (i.e. the background surface continued beyond the boundaries of the window). This was accomplished in the haptic condition by using a 300 tall wooden frame (with a surrounding support structure: see Fig. 1b). Adhesive putty was used to fix the objects in place to avoid accidental displacements during exploration. Haptic observers were instructed to explore the region within the window. In the visual condition, the windows were made by attaching black cloth to thin wooden strips: see Fig. 1a. The size of the window was the same in both conditions and revealed the same region for exploration. The critical portion of each scene and its size (length and width) is shown in Fig. 2. Objects were never far from the borders of the “view”, and there was always at least one object that was only about an inch (2.5 cm) from the edge. 4.3. Procedure All participants were told that we had created common scenes (like movie sets) in different rooms in the lab and that they would be exposed to a small region of each one. Their task was to remember the objects, background and layout of the region in the window. They all (including KC) wore vision-blocking goggles while they were lead to a marked location in front of each region. Once in place, those in the vision condition slid the goggles up to the top of their heads when cued by the experimenter and then studied the region for 30 s. They described the objects and named the ‘gist’ of the scene before them. They replaced the goggles and were then lead to the next region in the series. This prevented them from seeing the regions from multiple angles as they were guided into position; thus “viewpoint” was controlled as much as possible across modalities. Those in the haptic condition followed a similar routine, but remained blindfolded during the entire time. Once in position in front of a region, the experimenter placed the participant’s hands in the center of that region. Participants were asked to use both hands during the 30 s exploration phase and were allowed to explore in whatever natural manner they chose in order to best understand and remember the regions (see Klatzky, Lederman, & Reed, 1987). They too described the objects and the ‘gist’ of each of the scenes they partially explored, before being led to the next region in the series. Overt naming and “gist” description was included to allow us to determine if the haptic observers could indeed recognize the objects and their contextual relationships. Viewers were required to do so as well simply to keep the tasks as similar as possible between groups. KC was treated the same as the other haptic participants, except that instructions were translated by an ASL interpreter (as were KC’s verbal responses). Reminders to explore the full region “right up to the borders” were provided verbally for hearing participants and via a prearranged signal (a tap on the shoulder) for KC. [Note. We had tested KC’s ability to name objects and identify scene “gist” given a 30 s interval using a different set of regions almost a year earlier. Her descriptions were the same as those of the blindfolded observers—the only difference was that we asked her to slow down and use the full allotment of time. She clearly had no difficulty with the task]. After exploring the final scene, participants were escorted to a waiting area outside the rooms for about 5 min while the experimenter and an assistant removed the “windows”. With goggles in place, participants were again guided to each of the six locations, which

28

H. Intraub / Cognition 94 (2004) 19–37

they re-inspected (same order, same modality as before). They designated the remembered position of the original boundaries of each region, using a fingertip to specify the position of each border. The experimenter and an assistant placed the borders at the designated positions. In the visual condition, the black cloth borders were set down. In the haptic condition, heavy markers (made of bricks wrapped in duct tape) were set down. The latter markers were used because they were heavy enough not to be accidentally moved by the blindfolded participant, but would easily slide if the observer chose to move it. Participants in both conditions made any fine adjustments to the borders they felt necessary to allow an accurate reconstruction of the remembered region. A confidence rating of “sure,” “pretty sure,” or “not sure” was provided. Because observers were only free to reconstruct borders during the test, at debriefing, all participants were asked, “When you explored the scenes the second time, did you feel the urge to move the objects as well as the borders even though this was not what you were asked to do? Or did you feel that the objects were in the “correct” places?”

5. Results All participants readily identified all objects and there were no identification errors or differences in the conceptual descriptions of the regions (e.g. “gym scene” or “exercise room scene”; “bedroom scene” or “bedroom floor by the bed”). Boundary extension tended to be the rule rather than the exception. This can be seen in Fig. 3, which shows: (a) the mean percent area remembered for each scene in the visual and haptic groups (error bars signify the 0.95 confidence intervals), and (b) the corresponding percent area remembered by KC.

Fig. 3. Mean percent area of each region remembered by sighted participants in the visual and haptic conditions, and the percent area of each region remembered by KC. Error bars show the 0.95 confidence interval around each mean. (Boundary extension occurs when the mean remembered area is significantly greater than 100%; i.e. when 100% is not included in the confidence interval.)

H. Intraub / Cognition 94 (2004) 19–37

29

5.1. Visual and haptic conditions (sighted participants) As may be seen in Fig. 3, viewers remembered having seen significantly larger portions of the background than had actually been visible. In the most extreme case, viewers remembered having seen so much more of the background, that they increased the area, on average, 94% (“bedroom”). Boundary extension also occurred in the haptic condition. Blindfolded-sighted participants remembered having explored a greater expanse in all but the “bureau scene” (which will be discussed later). In the most extreme case they remembered having felt, on average, 29% more area (“gym”). Because extreme cases of boundary extension could serve to skew the group means, the median area that observers remembered for each scene in each condition was determined and is presented in Table 1; it reveals a similar pattern of results. The tendency to remember having seen or touched more of the background can been seen not only in the group means, but across participants: only one participant in the visual condition and two in the haptic condition did not yield a mean area that was greater than 100%. Collapsing over scenes, on average, viewers remembered having seen 53% more and haptic observers remembered having explored 17% more of the scenes than had actually been available for inspection. Both increases were significant (0.95 confidence intervals), as was the difference between them (tð38Þ ¼ 5:33; P , 0:001; two-tailed). Viewing the regions clearly resulted in more boundary extension than exploring them haptically. 5.2. KC’s memory KC’s remembered areas tended to be quite similar to those of the other haptic participants (blindfolded-sighted). As may be seen in Fig. 3, her boundary extension was greater than the haptic group mean for two regions, showed the same amount for three (including the “bureau” scene’s lack of boundary extension), and was less than the group mean in only a single case. In the most extreme case, KC remembered the boundaries as having included 46% more area (“toys”). Her rank with respect to the other haptic participants on each scene is shown at the bottom of Table 1. Clearly her life-long Table 1 Median % area remembered and KC’s rank (size of the remembered area) relative to the other haptic participants, for each scene Condition

Median % area Haptic (blindfolded-sighted) Vision

N

20 20

KC’s rank (size of remembered area)a KC and blindfolded-sighted 21 a

Scene Desk

Toys

Bureau

Bedroom

Sink

Gym

112 124

112 160

97 111

122 179

125 145

129 128

10

1

12

7

9

19

A rank of “1” indicates the individual (out of 21) who remembered having felt the greatest expanse of the scene: a rank of “21” indicates the individual who remembered having felt the smallest expanse.

30

H. Intraub / Cognition 94 (2004) 19–37

expertise with haptic exploration of scenes did not eliminate or minimize her tendency to remember having explored beyond the boundaries of a truncated “view” as compared with the blindfolded-sighted observers. In all cases, her remembered area was always significantly less than the mean area remembered in the visual condition. A good characterization of the movements made by KC and the other haptic participants, is that they carefully explored the scenes with both hands, examining each of the objects and exploring the space between them, and between the objects and the borders. As instructed, all participants limited their exploration to the area inside the window. 5.3. Length and width analysis To evaluate the “shape” of boundary extension, it is important to note that with few exceptions, the increase in remembered area reflected expansion of both the width and the length of the exposed regions. Fig. 4, shows the mean remembered width and length for each scene in both conditions and for KC. 5.4. Confidence and subjective report Although they consistently erred in reconstructing the studied “view,” as typically observed in previous boundary extension experiments (with pictures), participants in the current study were rather confident about their reconstructions of the explored space: they reported being “sure” or “pretty sure” on 95% of the vision trials and 87% of the haptic trials. KC provided similar confidence ratings, reporting being “sure” or “pretty sure,” on all but 1 scene (“place setting”). When asked whether they thought the experimenters had moved the objects with respect to one another in the scenes, no participant thought this was the case, although 60% thought that in 1 –2 scenes an object was moved. There was no trend in the nature of these “displacements”: sometimes they reported the object as being closer to the center, sometimes closer to the periphery, and sometimes they reported that it was in the same place but slighted rotated. KC’s response, like 40% of the other participants was that they were all in the same place as before. 5.5. The “bureau scene” Unlike the other scenes, the bureau scene failed to yield boundary extension—but only in the haptic condition. This was reflected in both the group mean, and in KC’s performance. It is important, therefore, to consider why this might be. Referring to the photograph in Fig. 2, a possible explanation is that the circular doily falling just within the borders, may have provided a strong cue during haptic exploration which resulted in accurate memory. Analysis of the individual responses, however, does not suggest great accuracy. Responses varied widely (80 – 116%), with about 1/2 of the participants increasing the area and 1/2 reducing it (9 and 11, respectively). KC’s performance reflects this, in that she increased the width, but decreased the length. A more plausible explanation is that the composition of objects within that region caused another type of spatial error to occur—an “alignment” error in memory. A single

H. Intraub / Cognition 94 (2004) 19–37

31

Fig. 4. Mean percent length and width remembered for each region by sighted participants in the visual and haptic conditions, and the percent length and width of each region remembered by KC. (Extension of length or width occurs when 100% is not included in the confidence interval.)

large, rectangular object (jewelry box) was misaligned with respect to the boundaries of the view. Many participants may have remembered the jewelry box has having been aligned with the borders (similar to a rod and frame error: e.g. Witkin & Asch, 1948). This possibility is reflected in KC’s hand movements. At test, when she first touched the back of the jewelry box—she happened to focus her hands on the left rear corner (the one farthest from the top border) and then appeared surprised when she pulled the boundary marker down and it hit the right rear corner (which due to the tilt, was closer to the upper boundary). She then began exploring the back of the box with respect to the boundary marker, tilting the marker slightly as if to increase the alignment. In contrast, this did not occur when she adjusted the left and right borders—and in fact she extended the width of the window in that dimension in spite of the presence of the doily. This possibility provides a reasonable explanation for why the discrepant outcome only occurred in

32

H. Intraub / Cognition 94 (2004) 19–37

the haptic condition: in the visual condition, viewers could see the entire jewelry box at once, as well as its orientation with respect to the table and with respect to themselves.3

6. Discussion A classic problem in scene perception is to explain how the perceiver comes to experience a continuous representation of the world when sensory input provides only successive, discrete samples. The present results support the idea that the mind may “solve” the problem, in part, by “ignoring” the spurious boundaries of a given “view” and extrapolating beyond the edges. Minutes after haptically exploring bounded regions of 3D scenes, KC and 20 blindfolded-sighted participants remembered having felt beyond the borders of the original region. KC’s border reconstructions were very similar to those of the blindfolded-sighted: her remembered area was the same or greater than the group mean in all but one case. On the face of it, one might be surprised that a “haptic expert” whose first language is tactile ASL and who is a Braille reader, would make such large, unidirectional errors in remembering the expanse of small regions in which the boundaries were so close to the objects. Frequently, the space between objects and borders was quite small (approximately 2.5 cm), requiring participants to contort their hands to explore it. This seemingly salient cue to border placement did not prevent boundary extension. KC and the other haptic participants remembered having felt more space between the edge of the border and the edge of the window. When asked to make final adjustments, they gently moved their hands back and forth between the border and the object, apparently testing the distance, and failed to note that minutes earlier, the space had been so small that they could not move their hands in this way at all. The author had a similar experience when attempting the task, in spite of having squeezed my hand into the small space during initial exploration; minutes later I remembered with great confidence having felt an unimpeded region between the border and the object. It is important to remember that in the visual condition, the viewers could clearly see the proximity of objects and borders in those same regions, yet the vision group tended to make the most expansive errors of all. This outcome is reminiscent of a similar “disregard” for a seemingly obvious cue to boundary placement (cropping of a main object by a picture boundary) observed in memory for close-up views in photographs (Intraub & Bodamer, 1993). What appears to be most salient during scene perception (irrespective of modality) is comprehension and representation of the scene—which always continues beyond the borders of a given view. Indeed, all views of the world, including those contained in photographs, depict only a part of a continuous space. An invariant property of the visual world (as absurd as it may at first sound) is that “wherever one looks, there is always 3 Further supporting this hypothesis, the same “window” was used in conjunction with two other scenes in other experiments conducted in this lab. When the window contained another large misaligned object (in this case a large notebook in a desk scene), once again boundary extension occurred in the visual condition but not the haptic condition. However, when the window contained an arrangement of smaller objects (objects from the “sink” scene in the present study: see Fig. 2) boundary extension occurred in both the visual and haptic conditions (Intraub et al., 2003).

H. Intraub / Cognition 94 (2004) 19–37

33

more.” This is true whether a viewer stands in the middle of a prairie and turns his or her head, or if the viewer stands inside a closet. Scene perception seems to draw on this invariant. However, in spite of the similarities between the results in the visual and haptic conditions, a striking difference was obtained with respect to the magnitude of the extrapolated space: visual exploration tended to yield much greater amounts of boundary extension than did haptic exploration of the same regions. This difference has now been replicated in several other experiments using these and other 3D scenes (Intraub, Morelli, & Turner, 2003). Why would observers, particularly those who rarely explore scenes without vision, make greater errors when they can see the regions? A possible explanation can be drawn from the initial premise, that to be adaptive, mental extrapolation must have some internally imposed limits (e.g. see Shepard, 1984). Without such constraints extrapolation would continue unabated, yielding a misleading and confusing interpretation of a view that is more likely to hamper comprehension and integration of successive views, than to facilitate it. The disparity in the amount of extrapolated space in the visual and haptic conditions suggests that boundary extension may be constrained, in part, by the scope of the input modality used to explore the region (i.e. its “reach”). During exploration, a small shift of the head and/or the eyes can bring a much larger new region into view than can a small shift in hand position. It seems reasonable that the amount of extrapolation would reflect this difference. Put another way, anticipatory representation must be large enough to facilitate integration of views in memory, but not so large as to be confusing or misleading. A possible alternative explanation, however, can be drawn from a confounding feature of the design. Because all participants in the present study used the same modality to explore and to reconstruct each region, the difference in magnitude between the visual and haptic conditions may not have been due to the input modality at all—but instead, to the test modality. Perhaps individuals tend to be more conservative when reconstructing spatial expanse in the haptic modality. Recent research, however, argues against this alternative (Intraub et al., 2003). When input modality and test modality were tested in a 2 £ 2 design, it was found that regardless of how memory was tested (via visual or haptic exploration of the scene), visual input yielded more boundary extension than did haptic input. The amount of extrapolated space transferred across modalities without changing. Input modality, not test modality, determined the amount of boundary extension. Does this mean that boundary extension reflects the same underlying processes in both modalities? The results are suggestive, but one must be cautious in assuming that similar error patterns indicate the same underlying cause. One way in which to approach this problem is illustrated by Millar and Al-Attar (2002). To determine if similarities between vision and haptics with respect to the Mueller-Lyer illusion reflect common underlying processes, in one experiment in their series, they set out to determine if a single instruction could be provided that would have a similar effect on the size of the illusion in both modalities. They found that an explicit instruction to use a body-centered reference and to ignore the fins of the Mueller-Lyer figure reduced the illusion to near zero in both modalities—a dramatic reduction in a very powerful illusion. The similar effect supports their contention that a common underlying mechanism is at least partially responsible for the illusion in the visual and haptic modalities.

34

H. Intraub / Cognition 94 (2004) 19–37

Some similarities between boundary memory following visual and haptic exploration have already been demonstrated. In both vision and haptics, boundary extension occurred in tightly constrained regions directly in front of the observer. The close proximity of objects and borders did not eliminate the effect—large magnitude errors were obtained in both conditions in spite of apparently obvious cues. In other research on cross-modal transference, the input modality determined the amount of boundary extension and was unaffected by whether the same or the alternative modality was used during test (Intraub et al., 2003). Future research can provide additional direct tests of commonality by varying factors that have been shown to affect visual boundary extension in predictable ways, and testing their effectiveness in memory using haptic analogues. For example in picture research, it has been shown that boundary extension: (a) is greatest for tight close-ups in which little background surrounds the objects, and decreases as the amount of background area between an object and the boundaries increases (Intraub et al., 1992), (b) is greater on the to-be-fixated side of a picture—indicating the dynamic nature of the representation and its responsiveness to planned actions (Intraub, Hoffman, Wetherhold, & Stoehs, 2003), and (c) boundary extension occurs whether or not an object is occluded by one of the boundaries (e.g. Intraub & Bodamer, 1993). These can all be tested using haptic analogues and visual conditions in 3D space.

7. Conclusions Both the visual and haptic modalities provide the perceiver with a succession of partial views of a continuous world—yet somehow the resulting mental representation reflects the coherence and continuity of surrounding space. The present research advances the theory that one of the means by which coherence is achieved is through the projection of anticipated layout (e.g. Intraub, 2002). Boundary extension occurred in memory for regions of real 3D scenes—irrespective of whether the regions had been explored using vision or touch. Most important, a “haptic expert”, who has been deaf and blind since early life, made the same type of error—and did so to a similar degree. Although clearly an “error” with respect to the true content of the explored region, the extended region in the mental representation possesses adaptive potential in that it reflects an invariant property of real world scenes—the continuity of layout. And this region may serve to aid comprehension and the integration of successive views. The projected region appears to be as much a part of the observer’s mental representation as were the objects and background that had actually been explored. Boundary extension is neither limited to picture memory, nor to the visual modality. Instead, it may be a fundamental component of spatial cognition—whether the perceiver’s developmental history is one in which vision or haptics has been the primary means of scene perception.

Acknowledgements I gratefully acknowledge the assistance of E. Renee Clement, Amanda Turner, and Chuck Ritter. This research was supported by NIMH Grant R01MH54688.

H. Intraub / Cognition 94 (2004) 19–37

35

References Atkins, J. E., Fiser, J., & Jacobs, R. A. (2001). Experience-dependent visual cue integration based on consistencies between visual and haptic percepts. Vision Research, 41, 449 –461. Biederman, I. (1981). On the semantics of a glance at a scene. In M. Kubory & J. R. Pomerantz (Eds.), Perceptual Organization. Hillsdale NJ: Erlbaum. Bingham, G. P. (1993). Optical flow from eye movement with head immobilized: Ocular occlusion beyond the nose. Vision Research, 33, 777–789. Carpenter, P. A., & Eisenberg, P. (1978). Mental rotation and the frame of reference in blind and sighted individuals. Perception and Psychophysics, 23, 117– 124. Cooper, L. A., & Shepard, R. N. (1973). Chronometric studies of the rotation of mental images. In W. G. Chase (Ed.), Visual information processing (pp. 75–166). New York: Academic Press. Freyd, J. J. (1983). The mental representation of movement when static stimuli are viewed. Perception and Psychophysics, 33, 575–581. Freyd, J. J. (1993). Five hunches about perceptual processes and dynamic representations. In D. E. Meyer, & S. Kornblum (Eds.), Attention and performance 14: Synergies in experimental psychology, artificial intelligence, and cognitive neuroscience (pp. 99 –119). Cambridge, MA: MIT Press. Freyd, J. J., & Finke, R. A. (1984). Representational momentum. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 126–132. Futterweit, L. R., & Beilin, H. (1994). Recognition memory for movement in photographs: A developmental study. Journal of Experimental Child Psychology, 57, 163–179. Gibson, J. J. (1950). The perception of the visual world. Boston, MA: Houghton Mifflin. Gottesman, C. V. (2003). Layout extrapolations as primes for spatial processing. Manuscript submitted for publication. Gottesman, C. V., & Intraub, H. (2002). Surface construal and the mental representation of scenes. Journal of Experimental Psychology: Human Perception and Performance, 28, 1–11. Gottesman, C. V., & Intraub, H. (2003). Constraints on spatial extrapolation in the mental representation of scenes: View-boundaries versus object-boundaries. Visual Cognition, 10, 875–893. Hochberg, J. (1986). Representation of motion and space in video and cinematic displays. In K. J. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance, 1 (pp. 22:1–22:64). New York: Wiley. Hock, H. S., & Schmelzkopf, K. F. (1980). The abstraction of schematic representations from photographs of realworld scenes. Memory and Cognition, 8, 543 –554. Hollingworth, A., & Henderson, J. M. (1998). Does consistent scene context facilitate object perception? Journal of Experimental Psychology: General, 127, 398–415. Intraub, H. (1997). The representation of visual scenes. Trends in the Cognitive Sciences, 1, 217 –221. Intraub, H. (2002). Anticipatory spatial representation of natural scenes: Momentum without movement? Visual Cognition, 9, 93–119. Special Issue: Representational momentum: New findings, new directions. Intraub, H., Bender, R. S., & Mangels, J. A. (1992). Looking at pictures but remembering scenes. Journal of Experimental Psychology: Learning, Memory and Cognition, 18, 180– 191. Intraub, H., & Bodamer, J. L. (1993). Boundary extension: Fundamental aspect of pictorial representation or encoding artifact? Journal of Experimental Psychology: Learning, Memory and Cognition, 19, 1387–1397. Intraub, H., Gottesman, C. V., & Bills, A. J. (1998). Effects of perceiving and imagining scenes on memory for pictures. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 186–201. Intraub, H., Gottesman, C. V., Willey, E. V., & Zuk, I. J. (1996). Boundary extension for briefly glimpsed pictures: Do common perceptual processes result in unexpected memory distortions? Journal of Memory and Language, 35, 118 –134. Special edition, entitled, Memory Illusions. Intraub, H., Hoffman, J. E., Wetherhold & Stoehs, S. (2003). Visual-spatial representation shifts in anticipation of a planned eye movement. Paper presented at the Forty-Third Annual Meeting of the Psychonomic Society, Vancouver, BC, Canada.

36

H. Intraub / Cognition 94 (2004) 19–37

Intraub, H., Morelli, F., & Turner, A (2003). Anticipatory representation of 3D scenes following visual or haptic exploration. Manuscript in preparation. Intraub, H., & Richardson, M. (1989). Wide-angle memories of close-up scenes. Journal of Experimental Psychology: Learning, Memory and Cognition, 15, 179–187. Irwin, D. E. (1991). Information integration across saccadic eye movements. Cognitive Psychology, 23, 42–456. Irwin, D. E. (1993). Perceiving an integrated visual world. In D. E. Meyer, & S. Kornblum (Eds.), Attention and performance XIV: Synergies in experiment psychology, artificial intelligence, and cognitive neuroscience (pp. 121–142). Cambridge, MA: MIT Press. James, W. (1890) (Vol. II). The principles of psychology, New York: Holt and Company. Kellman, P. J., & Shipley, T. F. (1991). A theory of visual interpolation in object perception. Cognitive Psychology, 23, 141–221. Kellman, P. J., Yin, C., & Shipley, T. F. (1998). A common mechanism for illusory and occluded object completion. Journal of Experimental Psychology: Human Perception and Performance, 24, 859–869. Klatzky, R. L., Golledge, R. G., Loomis, J. M., & Cicinelli, J. G. (1995). Performance of blind and sighted persons on spatial tasks. Journal of Visual Impairment and Blindness, 89, 70 –82. Klatzky, R. L., Lederman, S. J., & Reed, C. (1987). There’s more to touch than meets the eye: The salience of attributes for haptics with and without vision. Journal of Experimental Psychology: General, 116, 356–369. Landau, B., & Gleitman, L. R. (1985). Language and experience. Cambridge, MA: Harvard University Press. Legault, E., & Standing, L. (1992). Memory for size of drawings and of photographs. Perceptual and Motor Skills, 75, 121. Loomis, J. M., Klatzky, R. L., Golledge, R. G., Cicinelli, J. G., Pellegrino, J. W., & Fry, P. A. (1993). Nonvisual navigation by blind and sighted: Assessment of path integration ability. Journal of Experimental Psychology: General, 122, 73–91. Marmor, G. S., & Zaback, L. A. (1976). Mental rotation by the blind: Does mental rotation depend on visual imagery? Journal of Experimental Psychology: Human Perception and Performance, 2, 515–521. McConkie, G. W., & Currie, C. B. (1996). Visual stability across saccades whole viewing complex pictures. Journal of Experimental Psychology: Human Perception and Performance, 22, 563–581. Michotte, A., Thines, G., & Crabbe, G. (1964). Amodal completion. In A. Michotte, A. G. Thines, A. Costall, & G. Butterworth (Eds.), Michotte’s experimental phenomenology of perception. Hillsdale, NJ: Lawrence Erlbaum. Millar, S., & Al-Attar, Z. (2002). The Mueller-Lyer illusion in touch and vision: Implications for multisensory processes. Perception and Psychophysics, 64, 353–365. Nakayama, K., He, Z. J., & Shimojo, S. (1995). Visual surface representation: A critical link between lower level and higher level vision. In S. M. Kosslyn, & D. N. Osherson (Eds.), Visual cognition, 2 (pp. 1–70). Cambridge, MA: MIT Press. O’Regan, J. K. (1992). Solving the real mysteries of visual perception: The world as an outside memory. Canadian Journal of Psychology, 46, 461–488. Palmer, S. E. (1975). The effects of contextual scenes on the identification of objects. Memory and Cognition, 3, 519–526. Potter, M. C. (1976). Short-term conceptual memory for pictures. Journal of Experimental Psychology: Human Learning and Memory, 2, 509–522. Previc, F. H. (1998). The neuropsychology of 3-D space. Psychological Bulletin, 124, 123–164. Rayner, K., & Pollatsek, A. (1992). Eye movements and scene perception. Canadian Journal of Psychology, 46, 342–376. Rensink, R. A. (2000). Seeing, sensing, and scrutinizing. Vision Research, 40, 1469– 1487. Rieser, J. J., Hill, E. W., Talor, C. R., Bradfield, A., & Rosen, S. (1992). Visual experience, visual field size, and the development of nonvisual sensitivity to the spatial structure of outdoor neighborhoods explored by walking. Journal of Experimental Psychology: General, 121, 210– 221. Sanocki, T., & Epstein, W. (1997). Priming spatial layout of scenes. Psychological Science, 8, 374–378.

H. Intraub / Cognition 94 (2004) 19–37

37

Seamon, J. G., Schlegel, S. E., Heister, P. M., Landau, A. M., & Blumenthal, B. F. (2002). Misremembering pictured objects: People of all ages demonstrate the boundary extension illusion. American Journal of Psychology, 115, 151–167. Shepard, R. N. (1984). Ecological constraints on internal representation: Resonant kinematics of perceiving, imagining, thinking, and dreaming. Psychological Review, 91, 417 –447. Shepard, R. N., & Metzler, J. (1971). Mental rotation of three dimensional objects. Science, 171, 701 –703. Suzuki, K., & Arashida, R. (1992). Geometrical haptic illusions revisited: Haptic illusions compared with visual illusions. Perception and Psychophysics, 52, 329 –335. Witkin, H. A., & Asch, S. E. (1948). Studies in space orientation. IV. Further experiments on perception of the upright with displaced visual fields. Journal of Experimental Psychology, 38, 762–782. Wolfe, J. M. (1999). Inattentional amnesia. In V. Coltheart (Ed.), Fleeting memories: Cognition of brief visual stimuli (pp. 71–94). Bradford books series in cognitive psychology, Cambridge, MA: MIT Press. Yin, C., Kellman, P. J., & Shipley, T. F. (1997). Surface completion complements boundary interpolation in the visual integration of partly occluded objects. Perception, 26, 1459–1479.