About turn: How object orientation affects categorisation and mental rotation

Neuropsychologia 49 (2011) 3758–3767

Contents lists available at SciVerse ScienceDirect

Neuropsychologia journal homepage: www.elsevier.com/locate/neuropsychologia

About turn: How object orientation affects categorisation and mental rotation Branka Milivojevic a,b,c,∗ , Jeff P. Hamm a , Michael C. Corballis a a

Department of Psychology, University of Auckland, New Zealand Experimental and Developmental Psychology, Utrecht University, The Netherlands c Child and Adolescent Psychiatry, University Medical Centre Utrecht, The Netherlands b

a r t i c l e

i n f o

Article history: Received 24 May 2011 Received in revised form 14 September 2011 Accepted 19 September 2011 Available online 28 September 2011 Keywords: Object orientation Mental rotation Categorisation Letters Numbers ERPs

a b s t r a c t High-density ERPs evoked by rotated alphanumeric characters were examined to determine how neural processing is affected by stimulus orientation during letter/digit classifications and during mirror/normal discriminations. The former task typically produces response times that are unaffected by stimulus orientation while the latter is thought to require mental rotation. Sensitivity to orientation was first observed around 100–140 ms and this effect was attributed to differences in low-level features between vertical and oblique orientations. Subsequently, character misorientation amplified the N170, a neural marker of object classification, between 160 and 220 ms. Top-down processing is reflected in the ERPs beginning at 280–320 ms and this time range may reflect binding of ventral and dorsal stream information. In the case of mirror-normal discrimination these top-down processes can lead to mental rotation between 340 and 700 ms. Therefore, although neural processing reflects object orientation, these effects do not translate into increases in reaction-times or impaired accuracy for categorisation, and precede those that do in the mental-rotation task. © 2011 Elsevier Ltd. All rights reserved.

1. Introduction Many familiar objects are immediately recognisable regardless of their orientation in the world. We can recognise a chair, or a bicycle, or letters of the alphabet independently of how they are oriented relative to one’s point of view. Of course, we can also see how they are oriented, suggesting that the process of recognition may be differentiated from that of visual perception, at some level at least. Information about orientation and identity of objects is thought to be processed by the dorsal and the ventral pathways, respectively, and the question of how (or when) this information is integrated is central to the “binding” problem. A case in point is what happens when we make decisions about an object’s left–right parity (e.g. whether it is normal or mirror-reversed). These types of decisions require alignment between the object and our own egocentric frame of reference. For example, deciding whether a shoe is the left or the right one requires either physical or mental rotation of the shoe into alignment with our feet, or the feet with the shoe. The same holds for any object class that has a well-defined left–right orientation, such as alphanumeric characters, which can be readily recognised as “backward” if they have been mirror-reversed (Cooper & Shepard, 1973; Shepard & Metzler,

∗ Corresponding author. Present address: Experimental Psychology, Rm. 17.17, Van Unnik Gebouw, Utrecht University, Heidelberglaan 2, 3584 CS Utrecht, The Netherlands. Tel.: +31 30 253 3092. E-mail address: [email protected] (B. Milivojevic). 0028-3932/$ – see front matter © 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.neuropsychologia.2011.09.034

1971) – but only if they are presented at upright. Rotated characters require mental rotation to their canonical upright before we can notice if they are normal or backward. This suggests that object identity must be extracted before information about object orientation can be determined. Although this appears to be a fairly logical conclusion, there is evidence that object recognition can also be affected by changes in stimulus orientation. So, for example, face recognition is worse when faces are inverted (Yin, 1969) and discriminating between objects within a semantic category (subordinate-level decisions) is also affected (while between-category or basic-level decisions are largely unaffected, Hamm & McMullen, 1998). Furthermore, previous experience with objects at a given orientation also seems to enhance recognition, while pre-cuing the upcoming orientation of the stimulus does not result in this benefit (McMullen, Hamm, & Jolicoeur, 1995), suggesting that viewpoint-dependent representations are also stored (Hayward & Tarr, 1997; Jolicoeur, Snow, & Murray, 1987). This has led some authors to suggest that there are multiple routes to object recognition – some based on viewpoint-dependent configuration while others are based on viewpoint-independent feature extraction (Jolicoeur, 1990). Nevertheless, these object-recognition costs, if they occur, would need to be resolved before mental rotation begins. Furthermore, the evidence that changes in object orientation can affect subordinate-level categorisation (e.g. this is a collie), but not basic-level categorisation (e.g. this is a dog) or superordinatelevel categorisation (e.g. this is an animal/quadruped; Hamm & McMullen, 1998) hint that orientation may affect perceptual

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

processing of familiar objects in subtle ways, not necessarily evident in behavioural measures, such as reaction times, naming latencies, or accuracy, which reflect combined contribution of multiple, interacting, stages of neural processing. Furthermore, these observations are consistent with the notion that an observer’s goal determines whether perception of object orientation will influence their cognition. Although mental rotation and orientation-dependent object recognition have received considerable attention in the literature, this study attempts to provide a more comprehensive overview of how orientation affects neural processing before and during mental rotation, and to dissociate the effects that can be attributed to mental-rotation task demands from those that result from differences in the orientation of the object. For that reason we compare ERPs evoked by two tasks: a letter-digit discrimination task and a mirror-normal discrimination task with alphanumeric characters. We assume that both tasks require a similar degree of visual processing of letters and digits required for character recognition, but that the mirror-normal task would elicit additional mental rotation processes. Based on results reported by Heil, Rauch, and Hennighausen (1998), we expect that mirror-normal discrimination will elicit linear increases in parietal negativity commonly associated with mental rotation (Heil, 2002; Milivojevic, Hamm, & Corballis, 2009b; Milivojevic, Johnson, Hamm, & Corballis, 2003; Peronnet & Farah, 1989; Wijers, Otten, Feenstra, Mulder, & Mulder, 1989) while letter-digit categorisation will not. We aim to extend those results by using a high-density montage which enables us to investigate the effects of orientation on multiple ERP components that may vary in spatial topographies. We focus the analysis on the time period preceding the time interval thought to be involved in mental rotation in order to examine how orientation affects neural processing independently of mental rotation and at what point in the course of visual processing the effects of orientation become specific to the mental rotation task. The answers may bear on the roles of the ventral stream which underlies perception of identity, and the dorsal stream, which mediates perception of orientation, and reveal how and when the information carried by these streams is integrated. This should provide a temporal window during which the “binding problem” is solved by the brain. We chose letters and digits as stimuli because they are highly familiar objects that have a clearly defined (and meaningful) left–right orientation. Categorisation of these characters into letters and digits does not appear to be systematically affected by 2D rotation (Corballis & Nagourney, 1978; Corballis, Zbrodoff, Shetzer, & Butler, 1978) while decisions about left–right orientation (i.e. whether they are normal or backward) require rotation back to the canonical upright. Orientation affects both types of tasks, however, when the characters have been mirror-reversed. For categorisation, mirror-reversed characters impose an additional 20 or so milliseconds to classify (Corballis & Nagourney, 1978), while for mirror-normal discrimination, the extra time taken to respond to backward characters is closer to around 200 ms. For mirror-normal discriminations, the reaction-time cost of character reversal is correlated with individuals’ mental rotation rates, while that does not seem to be the case for letter-digit categorisation. These results suggest that mirror-reversed characters are “flipped” out of the picture plane following the initial “in-plane” rotation. This additional flip brings them into alignment with their canonical forms in our mental representation, and is used to verify that the character is indeed a backward representation of the one stored in memory (Hamm, Johnson, & Corballis, 2004; Kung & Hamm, 2010). In terms of in-plane orientation, two types of orientationrelated effects are of interest: effects related to mental rotation and effects preceding mental rotation. While mental rotation is expected to induce linear increases in parietal negativity with

3759

larger angular rotation from upright (Milivojevic et al., 2009b), the relationship between orientation and object recognition can be described in terms of a combined linear and quadratic trends. In this sense, modelling the ERP data as linear and quadratic functions may be particularly insightful. Indeed, ERP evidence suggests that such linear–quadratic relationship can also be used to describe effects of orientation on amplitude and latency of the N170 evoked by faces (Jacques & Rossion, 2007) and alphanumeric characters (Milivojevic, Corballis, & Hamm, 2008; Milivojevic, Johnson, et al., 2003). Inversion of faces and scenes further increases activity levels outside of typically face- and scene-responsive areas (Epstein, Higgins, Parker, Aguirre, & Cooperman, 2006; Haxby et al., 1999, 2001), which suggests that visual processing of at least some kinds of rotated familiar shapes requires additional neural resources. Less systematic effects of orientation have also been reported for both the P1 (Boutsen, Humphreys, Praamstra, & Warbrick, 2006; Itier & Taylor, 2004; Taylor, 2002) and the P2 components (Boutsen et al., 2006; Milivojevic, Clapp, Johnson, & Corballis, 2003; Muthukumaraswamy, Johnson, & Hamm, 2003) although it is as yet unknown whether these effects can be described in terms of either linear or quadratic (or combination of linear and quadratic) functions. Considerably more is known about how mental rotation is carried out by the brain. A large number of neuroimaging studies have identified a network of regions including parietal regions (Alivisatos & Petrides, 1997; Cohen et al., 1996; Harris et al., 2000; Jordan, Heinze, Lutz, Kanowski, & Jäncke, 2001; Jordan, Schadow, Wuestenberg, Heinze, & Jäncke, 2004; Jordan, Wüstenberg, Heinze, Peters, & Jäncke, 2002; Koshino, Carpenter, Keller, & Just, 2005; Milivojevic, Hamm, & Corballis, 2009a; Podzebenko, Egan, & Watson, 2002; Richter, Ugurbil, Georgopoulos, & Kim, 1997; Seurinck, Vingerhoets, Vandemaele, Deblaere, & Achtenb, 2005), ventral stream regions such as the inferior temporal gyrus (ITG; Koshino et al., 2005), lateral occipital cortex (Podzebenko et al., 2002) and area MT (Cohen et al., 1996), and higher order premotor regions (Jordan et al., 2001, 2002; Lamm, Windischberger, Leodolter, Moser, & Bauer, 2001; Podzebenko et al., 2002; Richter et al., 2000) that have been associated with mental rotation tasks. It seems, however, that only areas in dorso-lateral fronto-parietal network are actively involved in mental rotation per se (Milivojevic et al., 2009a) while other regions probably subserve other cognitive processes involved in mental-rotation tasks. These processes might include pattern and object recognition, recognition of stimulus orientation, visuospatial working memory and attention, decision making, motor planning and motor output. The extent of the cortical network that has been identified as playing a role in mental-rotation tasks no doubt reflects the synthesis of these cognitive processes. Nevertheless, the fronto-parietal mental-rotation network is also likely to underlie the ERP correlates of mental rotation (Milivojevic et al., 2009a, 2009b) which are characterised as linear increases in centro-parietal negativity between approximately 400 and 800 ms (Heil, 2002; Milivojevic et al., 2009b; Peronnet & Farah, 1989; Wijers et al., 1989), and which last longer for larger angular departures from upright (Hamm et al., 2004; Milivojevic, Johnson, et al., 2003). As EEG is particularly well suited for analysis of sequential processing stages due to its high temporal resolution, we used it to investigate how orientation, either in the picture plane or as mirror-reversal, affects neural processing associated with object recognition and mental rotation. 2. Methods 2.1. Participants Eighteen neurologically normal volunteers were recruited from students and faculty at the University of Auckland for approximately 2 h of participation. All had

3760

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

normal or corrected-to-normal vision and were right-handed (LQ range: 60–100, mean 94.28), as determined by Edinburgh Handedness Inventory (Oldfield, 1971). The procedures were approved by the University of Auckland Human Subjects Ethics Committee, and all participants gave their informed consent to participate in the experiment. Four were excluded from analysis due to ocular and/or movement artefact during recording. Thus, 14 participants (7 women) were included in the final analysis. They ranged in age from 18 years to 47 years, with a mean of 27.57 years. 2.2. Visual displays Four uppercase letters (R, F, L, and P) and four digits (2, 4, 5, and 7) were printed in black 72-point Arial font and presented on white background in their normal and mirror-reversed versions, at four different angular displacements from upright: 0◦ , ±60◦ , ±120◦ and 180◦ . At upright, the characters subtended a vertical visual angle of 2◦ and a horizontal visual angle of 1.45◦ , on average (range: 1.3–1.8◦ ). Manipulation of visual displays was performed using Microsoft Office Picture Editor (MS). Stimuli were displayed on an SVGA computer monitor (1024 × 768 pixel resolution; 60 Hz refresh rate) from a distance of 57 cm. Stimulus presentation was controlled using E-Prime v1.1.4.1 (Psychology Software Tools, Pittsburgh, PA, USA). TTL pulses generated via the parallel port of the display computer provided synchronization of stimulus events with EEG acquisition. Millisecond timing routines for the visual displays and pulse generation were conducted as outlined in the E-Prime User Guide (Schneider, Eschmann, & Zuccolotto, 2002). 2.3. Tasks The participants performed two tasks: a mirror-normal discrimination (mirrornormal or MN task) task and a letter-digit discrimination (letter-digit or LD task) task. In the MN task, the participants pressed the left mouse button when they saw normal characters and the right mouse button when they saw mirror-reversed characters. In the LD task, the left mouse button corresponded to letters and the right mouse button to digits. For both tasks, the participants were instructed to respond as quickly and as accurately as possible.

response was made. Immediately after stimulus offset another trial would begin (with fixation-only screen). 2.5. EEG apparatus Electrical Geodesics Inc. 128-channel Ag/AgCl electrode nets (Tucker, 1993) were used. EEG was recorded continuously (250 Hz sampling rate; 0.1–100 Hz analogue band-pass) with Electrical Geodesics Inc. amplifiers (200 M input impedance) and acquisition software running on a Macintosh G4 computer with a 16-bit analogue-to-digital conversion card bit. Electrode impedances were below 50 k (range 30–50 k), an acceptable level for this system (Ferree, Luu, Russell, & Tucker, 2001). EEG was acquired using a common vertex (Cz) reference. 2.6. Pre-processing Pre-processing was performed with custom (in-house) software. Since the participants were instructed to withhold blinks and other movements until after they have made a response, we wished to use segments that are as short as possible, to avoid losing too many trials to artefact rejections. For this reason, the EEG files were segmented with respect to event triggers in 1000 ms epochs including a 200 ms prestimulus baseline and 800 ms post-stimulus epoch. This time period was chosen as it captures the time period preceding the response for upright alphanumeric characters in the MN task, which is used as the “baseline” condition for ERPs at all other orientations. Only the trials on which the participants responded correctly were included in the analyses. Eye-movement correction was made on all segments using the method of Jervis, Nichols, Allen, Hudson, and Johnson (1985). The corrected data from each subject were then averaged to produce a total of 16 ERPs (two tasks, two character versions (mirrored or normal), and four orientations). DC offsets were calculated from the pre-stimulus baseline and removed from all waveforms. The individual waveforms were digitally filtered with a band-pass filter for 0.01–30 Hz range using a bi-directional 3 Pole Butterworth filter (Alarcon, Guy, & Binnie, 2000). Averaged and filtered ERPs were re-referenced to the average reference off-line.

3. Results 2.4. Procedure

3.1. Behavioural data At the beginning of the experiment, participants first performed two practice blocks, one for each task. Since people usually find the MN task quite difficult there were more practice trials for the MN task (48 trials) than for the LD task (16 trials). After practice, participants performed four blocks of 256 trials (two for each task). Task order was randomised across blocks and order of stimulus conditions was randomised within the blocks. Clockwise and counter-clockwise stimulus rotations were considered equivalent, and, as in the study by Milivojevic, Johnson, et al. (2003), twice as many stimuli were presented for the 0◦ and 180◦ orientations, compared to those rotated by 60◦ or 120◦ either in clockwise or in counter-clockwise direction. ERP and RTs were estimated by collapsing trials with “equivalent” orientations. Thus, 128 stimuli were presented at each orientation for each task, 64 in their normal and 64 in their mirrored version. Each trial began with a fixation-only screen presented for 1000 ms, after which a stimulus would appear on the screen and remain there for 10 s or until a participant responded, whichever was sooner. Participants were instructed to keep looking at the stimulus, avoid eye movements, and withhold blinking until after the

RTs for accurate responses and accuracy, as percent correct, were analysed with a 2 × 2 × 4 repeated-measures ANOVA with task, character version and orientation as factors. The mean RTs and accuracy as a function of task, version and orientation are plotted in Fig. 1. Huynh–Feldt ε value correction was used to correct for sphericity violations associated with repeated-measures effects (Huynh & Feldt, 1976). For reaction times, significant main effects of task (F(1,13) = 23.79, p < 0.001) and character version (F(1,13) = 25.82, p < 0.001) were observed. A significant interaction between character version and task was also observed (F(1,13) = 16.55, p = 0.001) and could be attributed to a considerably smaller, although still

Fig. 1. Reaction times and accuracy as a function of character orientation, version and task.

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

significant, effect of stimulus version for the LD task (mean: 22.8 ms, SE: 6.76, p = 0.005) than for the MN task (mean: 218.8 ms, SE: 47.40, p < 0.001). A main effect of stimulus orientation was also observed (F(3,39) = 17.51, p = 0.001, ε = 0.372), as was the orientation-bytask interaction (F(3,39) = 18.26, p < 0.001, ε = 0.394). Simple-effects analysis indicated that the RTs for the LD task did not systematically vary with character orientation (F(3,39) = 2.53, p = 0.083, ε = 0.848), while the RTs for the MN task did (F(3,39) = 18.00, p = 0.001, ε = 0.379). The effects of orientation for the MN task could be predominantly attributed to a significant linear (F(1,13) = 20.02, p = 0.001) trend that accounted 94.46% of variance, although quadratic (F(1,13) = 6.75, p = 0.022) and cubic (F(1,13) = 5.58, p = 0.034) trends, which accounted for 4.95% and 0.58% of variance, respectively, were also significant. The slope of the linear component of the RT function was 2.76 ms/degree, indicating that the rate of mental rotation was approximately 363◦ /s. In terms of accuracy, participants performed the LD task (96.6% correct) more accurately than the MN task (93.4% correct, F(1,13) = 15.11, p = 0.002). Accuracy was also significantly higher for normal (95.7%) than for mirror-reversed (94.3%) characters (F(1,13) = 4.98, p = 0.044). Furthermore, the main effect of orientation was significant (F(3,39) = 16.95, p < 0.001, ε = 0.667), as was the task-by-orientation interaction (F(3,39) = 17.34, p < 0.001, ε = 0.895). Simple effects analysis revealed significant effects of stimulus orientation on the MN task (F(3,39) = 21.32, p < 0.001, ε = 0.720), and no effects of orientation for the LD task (F(3,39) < 1, ε = 0.799). The effects of orientation on accuracy of MN decisions were characterised by a significant linear trend (F(1,13) = 38.49, p < 0.001) that accounted for 93.37% of variance, and a significant quadratic trend (F(1,13) = 5.57, p = 0.035) that accounted for 5.48% of variance. 3.2. EEG results As discussed earlier, linear and quadratic trends are particularly interesting for an investigation of orientation-related effects before and during mental rotation. To identify time periods during which ERPs show effects that have a linear and/or quadratic relationship to stimulus orientation, we computed linear and quadratic ERP-trends for both tasks (LD and MN), collapsed across character versions (normal and mirror-reversed). We then used a combination of global field power (GFP; Lehmann & Skrandies, 1984) and microstate plots (global-dissimilarity scores, GDS; Lehmann & Skrandies, 1980) to determine the time periods during which ERPs showed large-amplitude effects (based on increase in GFP amplitudes) which also showed stable topographies (based on low GDS scores; see Fig. 2).1 The linear and quadratic effects of orientation, which were evident for both tasks, coincided with P1, N170 and P2, corresponding to three time windows of interest: 96–136 ms, 160–216 ms and 248–320 ms, respectively. Clear effects of orientation were evident for the MN task in the late 340–700 ms time period, although they differed slightly depending on character version (c.f. Hamm et al., 2004). Less clear effects were observed for the LD task between 340 and 480 ms. The choice

1 Global field power is a reference-free measure of variability across the scalp at each point in time, and is useful for visualising when increases in amplitudes are happening, irrespective of where on the scalp such effects may occur. Global dissimilarity scores are a reference-free measure of similarity (or correlations) between two sequential time points. In principle, they can range from 0 to 2, where 0 indicates scalp topographies are identical and 2 indicates scalp topographies have reversed in polarity between two adjacent time points. For the purposes of display, we scaled the GDS scores to enable us to visualise them on the same plot as the GFP. Periods of low GDS scores suggest periods of stable cortical source activity, and will be referred to as “microstates”.

3761

of electrodes for analysis was based on the distribution of the effects of interest. See Fig. 2 for electrodes used in reference to the remainder of the montage as well as the distribution of the components of interest and the late-parietal orientation effects. For P1 and N170 we used occipito-temporal clusters, for P2, we used parieto-occipital clusters and for the late parietal component we used parietal-electrode clusters. Average amplitudes over P1, N170 and P2 time windows of interest, within each cluster of interest (presented in Fig. 3) were subjected to a 2 × 2 × 4 × 2 repeated measures ANOVA with task, character version, orientation and hemisphere as factors. As the reaction times for the LD task were considerably shorter than those for the MN task, we were unable to use the same time window for the analysis of the LP component for both tasks. For this reason, the late parietal component was analysed separately for each task using a repeated-measures ANOVAs with character version, orientation and hemisphere as factors. Violations of the sphericity assumption were corrected using Huynh–Feldt ε value correction (Huynh & Feldt, 1976). To keep the results brief, we report only the effects of relevance to the main questions of interests for this study. Our aim was to determine how departures from the canonical orientation affect visual processing of highly familiar objects. Since we are dealing with alphanumeric characters, their canonical orientation is defined in terms of both in-plane orientation, and left–right orientation. In that sense we are equally interested in how the factors of orientation and version (normal vs. backward) affect neural processes. We were also interested in how task demands modulate processing of objects and their orientations, and in that sense, interactions between task and either of the two orientation factors (orientation and version) become of particular interest, but not so much the effects of task itself. We also included hemisphere as a factor, and differences between hemispheres was deemed of interest if they interacted with either of the orientation factors. A summary of significant effects is presented in Table 1. We found significant effects of orientation for P1, N170 and P2 components. Character version also had a significant effect on N170 and P2 components, and interacted significantly with orientation in both cases. Significant task-by-orientation-by-version was also observed for the P2 task. Interpretable effects of version and orientation, as well as an interaction between these two factors and hemisphere were only observed for the MN task as no simple effects reached significance for the LD task (p ≥ 0.059). 3.2.1. Simple effects of in-plane orientation We characterised the effects of orientation in terms of linear, quadratic and cubic polynomial trends. Effects of orientation attributable to a purely linear trend were only expected from time periods related to mental rotation proper, as it would suggest an increase in neural activity with the increase in angular rotation away from upright. Effects characterised by a purely quadratic trend would indicate that stimuli presented at oblique orientations (±60◦ and ±120◦ ) differed from those aligned to the vertical ones (0◦ and 180◦ ), and could be expected from time periods related to low-level visual processing before object recognition. A combination of linear and quadratic trend components, depending of course on both the direction and the amplitude of such effects, may resemble the nonlinear function frequently associated with object-recognition deficits commonly reported in the literature (e.g. Jolicoeur, 1985, 1990; Lawson & Jolicoeur, 2003). Percentages of variance explained by significant trends components are presented in Table 2. As shown there, the effects of orientation on P1 are explained by a quadratic trend, while both quadratic and linear trends components underlie the effects of orientation on the N170. The contribution of the linear trend for the mirrorreversed characters is smaller than that for the normal characters

3762

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

Fig. 2. (A) Global field power and microstate plots for grand-average ERPs for mirror-normal (MN) and letter-digit (LD) tasks collapsed across character-version and in-plane orientations (top), and GFP and microstate plots for linear (middle) and quadratic (bottom) orientation effects for the two tasks. (B) Topographic distributions of P1, N170 and P2 ERP components, alongside electrode clusters used in the analysis. (C) Topographic distributions of linear orientation effects for MN and LD tasks, alongside electrode clusters used in the analysis (bottom row).

since mirror-reversal results in a significant increase in N170 amplitude when characters are presented upright (p < 0.001). The effects of orientation differ for the two tasks over the P2 component – with predominantly linear and quadratic increases in P2 amplitudes for the MN task, and mainly quadratic effects for the LD task. Effects of orientation over the P2 component were also characterised in terms of a cubic trend, but since we had no prior hypotheses with regard to cubic-trend contribution, we attribute these effects to noise. No significant orientation effects were observed for the late parietal component for the LD task. For the MN task, however, significant linear trend accounted for almost all orientation-related variance.

3.2.2. Simple effects of left–right orientation Next, we wished to characterise the interactions between character version (i.e. normal vs. mirrored left–right orientation) and any of the other factors used in the analysis (task, orientation and hemisphere for P1, N170 and P2 components; and orientation and hemisphere for the LP component for two tasks separately). For the P1 component, three-way task-by-version-by-hemisphere interaction (F(1,13) = 9.82, p = 0.008) was also significant. Simple effects analysis of how character version affected this component indicated that mirror-reversed characters elicited larger

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

3763

Fig. 3. Mean amplitudes over time windows of interest collapsed across electrodes within clusters of interest, as a function of task, hemisphere, character version and orientation. Error bars represent standard error of the mean.

amplitudes than normal characters, but only for the LD task and only over the right hemisphere (p = 0.016). For the N170, there was a significant task-by-orientation interaction (F(3,39) = 3.2, p = 0.046, ε = 0.796). This effect can be characterised by larger N170 amplitudes for the category task at 60

and 180◦ orientations. Since these effects do not appear systematic, we did not consider them of interest. For the P2 component, a significant version-by-hemisphere interaction was also observed (F(1,13) = 4.72, p = 0.049) and was characterised by significant effects of version on the right

3764

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

Table 1 Summary of significant main effects of character orientation (orientation or version), and significant interactions of interest. Component

Effect

P1

Orientation Task × version × hemisphere

F(3,39) = 10.1, p < 0.001, ε = 1 F(1,13) = 9.82, p = 0.008

N170

Orientation Version (mirror > normal) Version × orientation Task × orientation

F(3,39) = 29.62, p < 0.001, ε = 0.994 F(1,13) = 40.66, p < 0.001 F(3,39) = 4, p = 0.018, ε = 0.887 F(3,39) = 3.2, p = 0.046, ε = 0.796

P2

Orientation Orientation × hemisphere Version × hemisphere Task × orientation Version × orientation Task × orientation × version

F(3,39) = 27.47, p < 0.001, ε = 1 F(3,39) = 27.47, p < 0.001, ε = 1 F(1,13) = 4.72, p = 0.049 F(3,39) = 6.84, p = 0.001, ε = 1 F(3,39) = 5.4, p = 0.004, ε = 0.984 F(3,39) = 4.24, p = 0.013, ε = 0.946

Version × orientation × hemisphere Version (mirror > normal) Orientation Version × hemisphere Version × orientation × hemisphere

F(3,39) = 3.16, p = 0.048, ε = 0.794 F(1,13) = 25.82, p < 0.001 F(3,39) = 19.30, p < 0.001, ε = 0.989 F(1,13) = 6.13, p = 0.028 F(3,39) = 7.32, p = 0.001, ε = 1

Late parietal LD task MN task

(p = 0.003), but not the left (p = 0.835), with increases in P2 amplitudes for mirror-reversed characters. For the mirror-normal task during the late-parietal time window, the interaction between character version and hemisphere (F(1,13) = 6.13, p = 0.028) was significant, as was the version-byorientation-by-hemisphere interaction (F(3,39) = 7.32, p = 0.001, ε = 1). Pair-wise comparisons indicated that the pattern of significant differences between normal and mirrored characters was different between the hemispheres. On the left, mirror-reversed characters elicited lower amplitudes than the normal characters for 0, 60 and 120 orientations (p ≤ 0.018), but not when the characters were fully inverted (p = 0.583). On the right, significant differences between the versions were only evident at 120◦ (p = 0.015). Closer inspection of Fig. 3 suggests that these results can be attributed to stronger effects of orientation over the left than the right hemisphere for normal stimuli, and relatively weaker effects of orientation for mirror-reversed stimuli over the left than the right hemisphere. This pattern of results can be explained by a change in hemispheric dominance over time whereby both hemispheres are involved in the early phase of rotation, while only the left hemisphere is associated with the later phase, as we have suggested previously (Milivojevic et al., 2009b). 4. Discussion To characterise how and when changes in orientation of familiar objects affect neural processing, we compared ERPs evoked by

letters and digits presented at various orientations while participants performed mirror-normal discriminations (which require mental rotation) and letter-digit discriminations (which do not). This study presents evidence, summarised in Table 3, that the ERPs are affected by changes in object-orientation at multiple time periods, suggestive of orientation effects at four distinct perceptual and cognitive stages. At first, orientation affects low-level visual processing in a purely bottom-up manner – divorced from any effects of object identity. At the second stage, match between the visual input and mental representation is affected. Subsequently, task requirements begin to interact with perceptual processing, and in the case of mirror-normal discrimination, they lead to mental rotation, the fourth, and final, stage of orientation-dependent processing. 4.1. Low-level visual effects The earliest processing stage we examined was in the range of the P1 visually evoked potential. These early effects, characterised by an increase in amplitude in response to stimuli presented at oblique orientations, were distributed bilaterally over occipitotemporal electrodes with a distribution similar to that of the P1 component. Given the early latency of these effects and the fact that there is no differentiation between upright and inverted stimuli, effects of orientation at this visual-processing stage area likely to reflect low-level visual differences between vertical and oblique orientations possibly related to the oblique effect (Appelle, 1972).

Table 2 Summary of orientation effects. Percentage variance explained by significant linear, quadratic and cubic trend components for normal and mirrored characters during mirror-normal and letter-digit tasks. Component

Task

Character version

P1

Both

Both

N170 P2

Both Mirror-normal Letter-digit

LP * **

p < 0.05. p < 0.001.

Mirror-normal Letter-digit

Trend components Linear

Quadratic

Cubic

n.s.

99.97%**

n.s.

**

Normal Mirrored

**

33.03% 5.5%*

65.32% 94.24%**

n.s. n.s.

Normal Mirrored Normal Mirrored

73.99%** 70.08%** 10.6%* n.s.

15.48%* n.s. 88.34%** n.s.

10.53%* n.s. n.s. 45.11%**

Both Both

98.85%** n.s.

n.s. n.s.

n.s. n.s.

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

3765

Table 3 Summary of the when (time period) and the how (orientation effects and task dependence) of neural and behavioural sensitivity to object orientation and corresponding perceptual or cognitive stages. Measure

Time period

Orientation effects

Task dependence

Perceptual or cognitive process

ERPs

96–136 ms 160–216 ms 248–320 ms 340–700 ms

Quadratic Quadratic + linear Quadratic + linear Linear for MN task

No Marginal Yes Yes

Perceptual: oblique effect Perceptual and cognitive: matching visual input to internal representation Cognitive: recognition of object orientation and preparation for mental rotation Cognitive: beginning of mental rotation

RTs

750–1400 ms

Linear for MN task

Yes

Cognitive: duration of mental rotation

4.2. Effects on matching to internal template The N170 is the ERP signature of the activity of networks specialised for processing of particular object categories (Allison, Puce, Spencer, & McCarthy, 1999; Haxby et al., 2001; Rossion et al., 2000), and in that sense it likely represents the ERP correlate of object classification. Increases in activity-levels of such networks, or spread of activity outside of the primary network, has been related to increased difficulty in making a match between visual input and internal (mental) representations of the objects in question (Haxby et al., 2001). Therefore, increased N170 amplitudes probably represent more increased difficultly in matching the uncommon viewpoint of the character and the canonically defined mental representation. We observed increases in N170 amplitudes in response to both changes in 2D orientation and mirror-reversal, indicating that both types of orientation changes have a negative effect on such neural classification. Similar effects have been reported during a colourdiscrimination task with rotated letters and digits (Milivojevic et al., 2008), indicating that they are not specific to explicit objectrecognition demands (although participants probably recognised the characters despite not needing to do so). These results replicate findings from two of our previous studies (Milivojevic et al., 2008; Milivojevic, Johnson, et al., 2003) and suggest that alphanumeric characters are processed more efficiently when presented at their canonical orientation (i.e. upright and non-mirror reversed), an effect that may be a consequence of more perceptual experience with letters and numbers in this format.

the top–bottom axis. Amplitude increase would be characterised by an increase with inversion in addition to an increase with misalignment of the top–bottom axis from the vertical. Although no behavioural correlates of these effects were observed with the current stimulus set consisting of over-learned alphanumeric characters, this interpretation would fit well with the observation that recognition of stimuli may be easier when the stimuli are inverted than when the stimuli are presented at orientations around the 120◦ point. The shape of the orientation-effects for the LD task may be related to the nonlinear function commonly associated with orientation-dependent object-naming (Jolicoeur, 1985). 4.4. Mental rotation to egocentric reference frame Mental rotation was only elicited by the mirror-normal task, as indicated by both behavioural and ERP data. As previously established, the ERP effects of mental rotation were manifest in terms of a reduction in parietal positivity between 350 and 700 ms, which in this context is interpreted as an increase in a latent negative component, temporally coincident, but functionally independent from the P3 associated with stimulus recognition (Heil, 2002). Our results also replicate the finding by Hamm et al. (2004) that, subsequent to mental rotation within the picture plane to the upright, mirror-reversed characters are then flipped out of the picture plane, into alignment with canonically oriented mental representation. This observation is based on consistently lower (i.e. more negative) amplitudes in response to mirror-reversed stimuli, compared to normal stimuli, at all stimulus in-plane orientations.

4.3. Recognising object orientation

5. Conclusion

P2 amplitude differentiates the mirror-normal and the letterdigit tasks. Cognitive processes marked by the P2, at least in the mirror-normal task, provide information regarding stimulus orientation primarily in reference to the egocentrically-defined canonical position, i.e. normal, upright characters. Modulation of the P2 amplitude in the mirror-normal task probably represents the degree to which characters are rotated away from their canonical orientations, knowledge of which is necessary for subsequent mental rotation back to the canonical orientations. If this is the case, then the integration of ventral and dorsal stream information must occur either before, or more likely, during this time period. An interesting observation is that the P2 increases, for the mirror-normal task at least, precede the occurrence of mental rotation. Kung and Hamm (2010) suggested that mental rotation may not be used every time a stimulus is seen at a small angular departure from upright. If that is the case, the P2 effects may reflect the time period at which the decision to undergo effortful mental rotation is made. Note, this is not to imply that the decision to employ mental rotation is a conscious one. If this is the case, then the modulation of P2 amplitude for the letter-digit task may also index the recognition of stimulus orientation, which may depend on both the top–bottom axis inversion and any additional changes in orientation, such as a tilt away from

This study examined the temporal sequence of visual processing of rotated objects before and during mental rotation. Our results can be interpreted in terms of the following sequence of events leading to mental rotation: (1) feature extraction, (2) object recognition, (3) determining the object’s orientation, (4) determining which way to rotate to the upright, and (5) actual mental rotation to the upright. Mental rotation proper would be then followed by left–right parity discrimination which also requires mental rotation out of the picture plane for alphanumeric characters, which have a clearly defined left–right orientation. Although the current study does not explore cortical localisation of mental rotation processes, our previous results suggest that linear increases in ERP effects for the mental rotation task probably originate from the fronto-parietal network associated with mental rotation (Milivojevic et al., 2009a, 2009b). Given that the topographies of the preceding orientation-related effects differ from those involved in mental rotation, it is unlikely that they also reflect activity of the fronto-parietal mental-rotation network. Instead they are likely to reflect the activity of the regions involved in visual feature processing, object recognition and recognition of object orientation. The results suggest that at least three out of the four prerotation stages are orientation dependent, one of which is also

3766

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767

task-dependent. According to our interpretation, the visual system is initially sensitive to low-level orientation features, such as the orientation of the main axis of elongation. With time, the identity of the object has an increasing influence on orientation sensitivity, which may be attributable to perceptual expertise with upright alphanumeric characters (Milivojevic et al., 2008). Subsequently, the goal of the perceptual experience begins affecting orientationsensitivity of visual processing areas. This may mark the time point at which dorsal and ventral stream information is integrated, and is a prerequisite cognitive stage before mental rotation to upright can be initiated.

Acknowledgments This research was supported by the University of Auckland Research Grant (project number 3607199). Author BM was supported by a Top Achievers Doctoral Scholarship administered by Tertiary Education Commission of New Zealand. We would like to thank Dr Maarten Boksem for helpful comments regarding earlier versions of the manuscript. We would also like to thank the two anonymous reviewers whose comments greatly improved this manuscript.

References Alarcon, G., Guy, C. N. & Binnie, C. D. (2000). A simple algorithm for digital threepole Butterworth filter of arbitrary cut-off frequency: Application to digital electroencephalography. Journal of Neuroscience Methods, 104, 35–44. Alivisatos, B. & Petrides, M. (1997). Functional activation of the human brain during mental rotation. Neuropsychologia, 35, 111–118. Allison, T., Puce, A., Spencer, D. D. & McCarthy, G. (1999). Electrophysiological studies of human face perception: I. Potentials generated in occipitotemporal cortex by face and non-face stimuli. Cerebral Cortex, 9, 415–430. Appelle, S. (1972). Perception and discrimination as a function of stimulus orientation: The oblique effect in man and animals. Psychological Bulletin, 78(4), 266–278. Boutsen, L., Humphreys, G. W., Praamstra, P. & Warbrick, T. (2006). Comparing neural correlates of configural processing in faces and objects: An ERP study of the Thatcher illusion. NeuroImage, 32(1), 352–367. Cohen, M. S., Kosslyn, S. M., Breiter, H. C., DiGirolamo, G. J., Thompson, W. L., Anderson, A. K., et al. (1996). Changes in Cortical Activity During Mental Rotation: A mapping study using functional magnetic resonance imaging. Brain, 119, 89–100 (research). Cooper, L. A. & Shepard, R. N. (1973). Chronometric studies of the rotation of mental images. In W. G. Chase (Ed.), Visual information processing (pp. 75–176). New York: Academic Press. Corballis, M. C. & Nagourney, B. A. (1978). Latency to categorize disoriented alphanumeric characters as letters or digits. Canadian Journal of Psychology, 23, 186–188. Corballis, M. C., Zbrodoff, N. J., Shetzer, L. I. & Butler, P. B. (1978). Decisions about identity and orientation of rotated letters and digits. Memory & Cognition, 6, 98–107. Epstein, R. A., Higgins, J. S., Parker, W., Aguirre, G. K. & Cooperman, S. (2006). Cortical correlates of face and scene inversion: A comparison. Neuropsychologia, 44, 1145–1158. Ferree, T. C., Luu, P., Russell, G. S. & Tucker, D. M. (2001). Scalp electrode impedance, infection risk, and EEG data quality. Clinical Neurophysiology, 112(3), 536–544. Hamm, J. P., Johnson, B. W. & Corballis, M. C. (2004). One good turn deserves another: An event-related brain potential study of rotated mirror-normal letter discriminations. Neuropsychologia, 42, 810–820. Hamm, J. P. & McMullen, P. A. (1998). Effects of orientation on the identification of rotated objects depend on the level of identity. Journal of Experimental Psychology: Human Perception and Performance, 24(2), 413–426. Harris, I. M., Egan, G. F., Sonkkila, C., Tochon-Danguy, H. J., Paxinos, G. & Watson, J. D. G. (2000). Selective right parietal lobe activation during mental rotation: A parametric PET study. Brain, 123, 65–73. Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L. & Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539), 2425–2430 (see comment). Haxby, J. V., Ungerleider, L. G., Clark, V. P., Schouten, J. L., Hoffman, E. A. & Martin, A. (1999). The effect of face inversion on activity in human neural systems for face and object perception. Neuron, 22(1), 189–199. Hayward, W. G. & Tarr, M. J. (1997). Testing conditions for viewpoint invariance in object recognition. Journal of Experimental Psychology: Human Perception and Performance, 23, 1511–1521. Heil, M. (2002). The functional significance of ERP effects during mental rotation. Psychophysiology, 39, 535–545.

Heil, M., Rauch, M. & Hennighausen, E. (1998). Response preparation begins before mental rotation is finished: Evidence from event-related brain potentials. Acta Psychologica, 99(2), 217–232. Huynh, H. & Feldt, L. S. (1976). Estimation of the Box correction for degrees of freedom from sample data in randomized block and split-plot designs. Journal of Educational Statistics, 1, 69–82. Itier, R. J. & Taylor, M. J. (2004). Effects of repetition learning on upright, inverted and contrast-reversed face processing using ERPs. NeuroImage, 21, 1518–1532. Jacques, C. & Rossion, B. (2007). Early electrophysiological responses to multiple face orientations correlate with individual discrimination performance in humans. NeuroImage, 36(3), 863–876. Jervis, B. W., Nichols, M. J., Allen, E. M., Hudson, N. R. & Johnson, T. E. (1985). The assessment of two methods for removing eye movement artefact from the eeg. Electroencephalography and clinical Neurophysiology, 61, 444–452. Jolicoeur, P. (1985). The time to name disoriented objects. Memory & Cognition, 13, 289–303. Jolicoeur, P., Snow, D. & Murray, J. E. (1987). The time to identify disoriented letters: Effects of practice and font. Canadian Journal of Psychology, 41(3), 303–316. Jolicoeur, P. (1990). Identification and disoriented objects: A dual systems theory. Mind & Language, 5, 387–410. Jordan, K., Heinze, H. J., Lutz, K., Kanowski, M. & Jäncke, L. (2001). Cortical activations during the mental rotation of different visual objects. NeuroImage, 13, 143–152. Jordan, K., Schadow, J., Wuestenberg, T., Heinze, H. J. & Jäncke, L. (2004). Different cortical activations for subjects using allocentric or egocentric strategies in a virtual navigation task. NeuroReport, 15, 135–140. Jordan, K., Wüstenberg, T., Heinze, H. J., Peters, M. & Jäncke, L. (2002). Women and men exhibit different cortical activation patterns during mental rotation tasks. Neuropsychologia, 40, 2397–2408. Koshino, H., Carpenter, P. A., Keller, T. A. & Just, M. A. (2005). Interactions between the dorsal and the ventral pathways in mental rotation: An fMRI study. Cognitive, Affective, & Behavioral Neuroscience, 5, 54–66 (research). Kung, E. & Hamm, J. P. (2010). A model of rotated mirror/normal letter discriminations. Memory & Cognition, 38(2), 206–220. Lamm, C., Windischberger, C., Leodolter, U., Moser, E. & Bauer, H. (2001). Evidence for premotor cortex activity during dynamic visuospatial imagery from singletrial functional magnetic resonance imaging and event-related slow cortical potentials. NeuroImage, 14(2), 268–283. Lawson, R. & Jolicoeur, P. (2003). Recognition thresholds for plane-rotated pictures of familiar objects. Acta Psychologica, 112(1), 17–41. Lehmann, D. & Skrandies, W. (1980). Reference-free identification of components of checkerboard-evoked multichannel potential fields. Electroencephalography and Clinical Neurophysiology, 48, 609–621. Lehmann, D. & Skrandies, W. (1984). Spatial analysis of evoked potentials in man – A review. Progress in Neurobiology, 23, 227–250. McMullen, P. A., Hamm, J. P. & Jolicoeur, P. (1995). Rotated object identification with and without orientation cues. Canadian Journal of Psychology, 49, 133–149. Milivojevic, B., Clapp, W. C., Johnson, B. W. & Corballis, M. C. (2003). Turn that frown upside down: ERP effects of thatcherization of misorientated faces. Psychophysiology, 40(6), 967–978. Milivojevic, B., Corballis, M. C. & Hamm, J. P. (2008). Orientation sensitivity of the N1 evoked by letters and digits. Journal of Vision, 8(10) doi:10.1167/1168.1110.1111 Milivojevic, B., Hamm, J. P. & Corballis, M. C. (2009a). Functional neuroanatomy of mental rotation. Journal of Cognitive Neuroscience, 21(5), 945–959. Milivojevic, B., Hamm, J. P. & Corballis, M. C. (2009b). Hemispheric dominance for mental rotation: It is a metter of time. NeuroReport, 20(17), 1507–1512. Milivojevic, B., Johnson, B. W., Hamm, J. P. & Corballis, M. C. (2003). Non-identical neural mechanisms for two types of mental transformation: Event-related potentials during mental rotation and mental paper folding. Neuropsychologia, 41, 1345–4356. Muthukumaraswamy, S. D., Johnson, B. W. & Hamm, J. P. (2003). A high-density ERP comparison of mental rotation and mental size transformation. Brain and Cognition, 52, 271–280. Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh Inventory. Neuropsychologia, 9, 97–113. Peronnet, F. & Farah, M. J. (1989). Mental rotation: An event-related potential study with a validated mental-rotation task. Brain and Cognition, 9, 279–288. Podzebenko, K., Egan, G. F. & Watson, J. D. G. (2002). Widespread dorsal stream activation during a parametric mental rotation task, revealed with functional magnetic resonance imaging. NeuroImage, 15, 547–558 (research). Richter, W., Somorjai, R., Summers, R., Jarmasz, M., Menon, R. S., Gati, J. S., et al. (2000). Motor area activity during mental rotation studied by time-resolved single-trial fMRI. Journal of Cognitive Neuroscience, 12, 310–320 (research). Richter, W., Ugurbil, K., Georgopoulos, A. P. & Kim, S. G. (1997). Time-resolved fMRI of mental rotation. NeuroReport, 8, 3697–3702 (research). Rossion, B., Dricot, L., Devolder, A., Bodart, J. M., Crommelinck, M., De Gelder, B., et al. (2000). Hemispheric asymmetries for whole-based and part-based face processing in the human fusiform gyrus. Journal of Cognitive Neuroscience, 12(5), 793–802. Schneider, W., Eschmann, A. & Zuccolotto, A. (2002). E-Prime user’s guide. Pittsburgh, PA: Psychology Software Tools, Inc. Seurinck, R., Vingerhoets, T. G., Vandemaele, P., Deblaere, K. & Achtenb, E. (2005). Trial pacing in mental rotation tasks. NeuroImage, 25, 1187–1196. Shepard, R. N. & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(972), 701–703.

B. Milivojevic et al. / Neuropsychologia 49 (2011) 3758–3767 Taylor, M. J. (2002). Non-spatial attentional effects on P1. Clinical Neurophysiology, 113(12), 1903–1908. Tucker, D. M. (1993). Spatial sampling of head electrical fields: The geodesic sensor net. Electroencephalography and Clinical Neurophysiology, 87, 154–163.

3767

Wijers, A. A., Otten, L. J., Feenstra, S., Mulder, G. & Mulder, L. J. M. (1989). Brain potentials during selective attention, memory search, and mental rotation. Psychophysiology, 26, 452–467. Yin, R. K. (1969). Looking at upside down faces. Journal of Experimental Psychology, 81, 141–145.