The amodal system for conscious word and picture identification in the absence of a semantic task

NeuroImage 49 (2010) 3295–3307

Contents lists available at ScienceDirect

NeuroImage j o u r n a l h o m e p a g e : w w w. e l s e v i e r. c o m / l o c a t e / y n i m g

The amodal system for conscious word and picture identification in the absence of a semantic task Leen Van Doren a, Patrick Dupont a, Sophie De Grauwe a, Ronald Peeters b, Rik Vandenberghe a,c,⁎ a b c

Cognitive Neurology Laboratory, Experimental Neurology Section, K.U. Leuven, Belgium Radiology Department, University Hospital Gasthuisberg, Belgium Neurology Department, University Hospitals Leuven, Herestraat 49, 3000 Leuven, Belgium

a r t i c l e

i n f o

Article history: Received 13 August 2009 Revised 19 November 2009 Accepted 1 December 2009 Available online 29 December 2009 Keywords: Semantic fMRI Reading Naming Short-term memory Amodal

a b s t r a c t Previous studies using explicit semantic tasks, such as category or similarity judgments, have revealed considerable neuroanatomical overlap between processing of the meaning of words and pictures. This result may have been influenced by the semantic executive control required by such tasks. We examined the degree of overlap while minimizing semantic executive demands. In a first fMRI experiment (n = 28), we titrated word (35.3 ms, SD = 9.6) and picture presentation duration (50.7 ms, SD = 15.8) such that conscious stimulus identification became a stochastic process, with a 50% chance of success. Subjects had to indicate by key press whether or not they had been able to identify the stimulus. In a second fMRI experiment (n = 19), the identification runs were followed by a surprise forced-choice recognition task and events were sorted on the basis of subsequent memory retrieval success rather than a subjective consciousness report. For both words and pictures, when stimulus processing exceeded the conscious identification threshold, the left occipitotemporal sulcus (OTS), intraparietal sulcus, inferior frontal junction, and middle third of the inferior frontal sulcus (IFS) were more active than when subjects had been unable to identify the stimulus. For both words and pictures, activity in two of these regions, IFS and OTS, predicted subsequent memory retrieval success. A Bayesian comparison revealed that the effective connectivity between IFS and the word- or picture-specific systems was mainly mediated via its connections with OTS. The amodal nature of left OTS and IFS involvement in word and picture processing extends to tasks with minimal semantic executive demands. © 2009 Elsevier Inc. All rights reserved.

Introduction The substrate for shared versus input modality-specific processing of meaning of words versus pictures has received a great deal of attention, both in patient lesion studies (Rapp et al., 1993; Chertkow et al., 1997; Rogers and McClelland, 2004) and in functional imaging of the intact brain (Vandenberghe et al., 1996; Wagner et al., 1997; Buckner et al., 2000; Thierry and Price, 2006; Vandenbulcke et al., 2006, 2007). Candidate regions involved in amodal processing, i.e., processing of meaning of concrete words as well as pictures, are the ventral occipitotemporal cortex (Buckner et al., 2000), the posterior third of the left middle temporal gyrus (Chertkow et al., 1997; Vandenbulcke et al., 2007; Kircher et al., 2009), the anterior temporal pole (Hodges et al., 1992; Vandenberghe et al., 1996; Rogers and McClelland, 2004), the left inferior frontal sulcus (Wagner et al., 1997), the anterior inferior frontal gyrus (Goldberg et al., 2007), and the left middle frontal gyrus (Demb et al., 1995; Vandenberghe et al., 1996). ⁎ Corresponding author. Neurology Department, University Hospitals Leuven, Herestraat 49, 3000 Leuven, Belgium. Fax: +32 (0)16 3444285. E-mail address: [email protected] (R. Vandenberghe). 1053-8119/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.neuroimage.2009.12.005

Many tasks that have been used to study commonality between cognitive processing of words and pictures have required at least some degree of explicit task-related retrieval of the associations and meaning of the referent. This is true for relatively demanding tasks with a high decision weighting component such as the Pyramids and Palm Trees test (Howard and Patterson, 1992; Vandenberghe et al., 1996) but also for relatively easy tasks that require subjects to explicitly retrieve well-known and typical features of the referent such as living–nonliving judgments (Wagner et al., 1997), sound, or colour matching (Garrard and Carroll, 2006), etc. Task-related deployment of strategies for retrieving, comparing, or deciding about semantic properties and associations may partly account for the commonality in activity pattern between words and pictures in these studies, in particular in the prefrontal cortex. Other neuroimaging studies of amodal processing have made use of automatic semantic priming (Rissman et al., 2003; Sachs et al., 2008; Kircher et al., 2009; Sass et al., 2009) and combined words and pictures cross-modally (Kircher et al., 2009; Sass et al., 2009). During speeded lexical decision, the left posterior middle and superior temporal cortex shows a semantic relatedness effect during automatic priming (Copland et al., 2003; Gold et al., 2006) for word–word pairs

3296

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

and for picture–word pairs (Sachs et al., 2008; Kircher et al., 2009; Sass et al., 2009). Subliminal (masked) priming removes the strategic semantic effects altogether (Forster, 1998; Holcomb and Grainger, 2006; Diaz and McCarthy, 2007; Grainger and Holcomb, 2009). Neuroimaging studies of subliminal (masked) priming have reported differential effects in left fusiform cortex for word–word pairs comparing same versus different words (Dehaene et al., 2001; Qiao et al., 2010) and in the left posterior fusiform cortex for picture– picture pairs comparing repeated versus unrepeated pictures (Eddy et al., 2007). Cross-modal effects or within-study overlap between word–word and picture–picture priming effects have not been reported in neuroimaging studies of subliminal priming until now to the best of our knowledge. Prefrontal supraliminal priming effects (suppression or enhancement) have been reported at various locations distributed over the lateral convexity (Wagner et al., 1997; Copland et al., 2003; Rissman et al., 2003; Raposo et al., 2006; Race et al., 2009). This variability may relate to differences in the balance between automatic versus strategic component processes (Copland et al., 2003; Gold et al., 2006), but it could also be a consequence of the relatively small effect sizes and the inherent inability of fMRI to capture the time dependency of priming effects. For the first time, we applied a method to the study of amodal processing that stems from the consciousness research field (Kanwisher, 2001; Bar et al., 2001; Marois et al., 2004; Carmel et al., 2006; WileniusEmet et al., 2004). At brief stimulus durations, conscious stimulus identification is a probabilistic process. For each individual and each input modality, we selected a duration of word or picture presentation so that subjects reported conscious identification of the stimulus only with a probability of 50% across items (Marois et al., 2004; Carmel et al., 2006; Wilenius-Emet et al., 2004) and also within items across subjects. In this way, sensory input was matched between events that were associated with conscious processing and events in which processing remained subliminal. We tried to detect zones of amodal cognitive processing during supra- versus subliminal word and picture identification. In a previous study of speeded picture identification (Bar et al., 2001; Bar et al., 2006), activity in mid and anterior fusiform cortex correlated with the subjects' level of confidence that they had been able to identify the pictures correctly, ranging from zero (no stimulus perceived) to five (fully confident about stimulus identity), with also orbitofrontal involvement (Bar et al., 2006). In that study, confidence was manipulated by repeating the same stimuli up to 6 times. Only pictures were used (Bar et al., 2001; Bar et al., 2006). Our cross-modal design allowed us to examine whether these fusiform and other regions are shared between words and pictures during supra- versus subthreshold processing while sensory characteristics and prior exposure were strictly matched between supra- and subliminally processed stimuli. How does conscious processing of a word or picture differ from subliminal processing? According to the Theory of Visual Attention (TVA) (Bundesen, 1990; Bundesen et al., 2005; Bundesen and Habekost, 2008), conscious processing takes place when a perceptual entity is selected among other competing entities and gains access to visual short-term memory (VSTM) according to a winner-takes-all principle. From VSTM, the consciously perceived unit is made available (“broadcasted”) to other cognitive brain systems, such as declarative memory (Baars, 1988); 2002) or phonological retrieval (Jackendoff, 2007) systems. When we consciously identify a stimulus, the surface features of that stimulus and its visual form become available for subsequent explicit retrieval, as well as the identity of the referent. The central question in this study is whether similar brain networks are involved for word and picture input when stimuli gain access to consciousness. Conscious stimulus identification may in its turn trigger explicit lexical– or associative–semantic processes but the short stimulus durations and the task instruction in our study directed attentional resources towards perceptual identification and away from downstream processes of explicit semantic elaboration.

We determined which regions became activated when perceptual processing exceeded the threshold of conscious identification on the basis of a subjective consciousness report (Frith et al., 1999; Beck et al., 2001; Bar et al., 2001; Haynes et al., 2005). We dealt with the inherent subjectivity of this measure (Frith et al., 1999) in two ways: first, we also included foil stimuli consisting of nonexisting chimeras. Subjects were instructed to respond negatively to the foil stimuli, i.e., in the same way as to stimuli that they had not been able to identify. When the false-positive response rate to foil stimuli was too high, we excluded the run. In case of pictures this task is very similar to the object decision test (Riddoch and Humphreys, 1993), a classical neuropsychological task that probes the structural description level of processing. According to the Hierarchical Interactive Theory (HIT) (Humphreys and Forde, 2001), the activation of a structural description of the object is necessary for object identification and distinct from the associative–semantic level (Humphreys and Forde, 2001). At the structural description level, visual percepts are matched with mnemonic presentations of real-life entities in a way that is invariant for viewpoint, size, orientation, etc., and generalizes across different exemplars of a same entity (Humphreys and Forde, 2001). As a second measure to circumvent the subjectivity of a consciousness report, we only retained regions that also fulfilled a second, more objective criterion: following runs of subjective consciousness report, we conducted a surprise forced-choice yes/no recognition task and sorted events on the basis of subsequent memory retrieval success rather than subjective consciousness report. Only regions that stringently fulfilled both criteria, an association with a positive consciousness report and with successful encoding, conjointly for words and for pictures, were retained as amodal zones of supraversus subliminal processing. To further verify the level and accuracy of stimulus identification and also to evaluate how the extent of the amodal activations obtained in the first and second experiment related to those obtained in more classical tasks, we conducted a third, overt naming/reading fMRI experiment. Materials and methods Subjects Twenty-eight healthy native Dutch speakers (15 women and 13 men, between 19 and 29 years of age) participated in the first fMRI experiment, which was based on the subjective consciousness report, and 8 additional subjects (3 women and 5 men, age range 19– 25 years) in a control experiment. Nineteen other subjects (12 women and 7 men, between 19 and 35 years of age) participated in a third fMRI experiment, the subsequent memory retrieval experiment. Four other subjects (1 man and 3 women, between 21 and 23 years of age) participated in a fourth, overt naming/reading fMRI experiment. All participants were strictly right-handed (Oldfield, 1971), free of psychotropic and vasoactive medication, and without neurological or psychiatric history. They all gave written informed consent in accordance with the Declaration of Helsinki. The experiments were approved by the Ethics Committee, University Hospitals Leuven. Stimuli and tasks First experiment Visual stimuli were presented foveally and projected from a Barco 6400i LCD projector (1024 × 768 pixels) at a refresh rate of 75 Hz onto a screen 36 cm in front of the subject's eyes. Subjects viewed the screen using a mirror attached to the head coil. Each trial consisted of a forward mask (duration 200 ms), a written word (duration xword ms) or a picture (duration xpicture ms) and a backward mask (1700 ms − xword or xpicture, respectively), followed by a fixation point (duration 350 ms) (Fig. 1A). Subjects had to indicate by key press whether or not they had been able to identify the

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

3297

Fig. 1. Stimuli and tasks. (A) First experiment and subsequent memory retrieval experiment. For each subject, run, and modality, the variable x is titrated so as to reach a proportion of approximately 50% positive responses. (B) Overt naming/reading experiment with silent gap acquisition. The subsequent response window was 1000 ms during which scanning was interrupted (silent gap) and the fixation point was turned green. During this silent gap, the subjects had to read or name the presented stimulus aloud. The variable x was titrated so as to reach 50% correct naming/reading responses. Abbreviations: ITI: intertrial interval; TA: acquisition time.

stimulus. The variables xword and xpicture were titrated for each individual, each modality (word versus picture) and each run so as to reach a proportion of approximately 50% affirmative responses per individual and modality. The intertrial interval was 2250 ms. For each individual, the starting values for the word and picture durations were determined on the basis of the percentage positive responses during a prior test run on the scanner table using a training set of stimuli that was not used for fMRI. After each of the fMRI runs, we again checked the percentage positive reports per modality. If it was below 37.5% or above 62.5%, the presentation duration for that modality was lengthened or shortened, respectively, with 13.3 ms (screen refresh rate was 75 Hz) in the next run. Each run contained 8 catch trials (4 foil word and 4 foil picture trials). During these trials, we presented chimeric stimuli that did not refer to a real-world entity. They were constructed by dividing a word or a picture in halves and by combining the word or picture halves into chimeric word or chimeric picture stimuli, preserving word length and continuity of picture outline, respectively. Since these foil stimuli did not belong to the real world, subjects had to respond negatively to the foils (similarly to the object decision task (Riddoch and Humphreys, 1993)). Subjects therefore had to respond negatively when they had not been able to identify the stimulus and also when they had identified the stimulus as an entity that did not belong to the real world (in case of the chimeric foils). Runs in which subjects gave

an affirmative response to more than 3 out of 8 chimeric stimuli were not included in the primary analysis because this false-positive response rate may indicate that subjects responded affirmatively even in the absence of basic-level stimulus identification. Stimuli were pseudorandomly chosen for each run and each subject out of a pool of 240 concrete entities (108 biological, 132 nonbiological). Each entity was used during one run only per individual. In each run, it was used once as a word and once as a picture. Half of the entities were shown first as a word and in one of the later trials as a picture and half vice versa. For each entity, the modality of first presentation was counterbalanced across subjects. Word length was two to six letters (mean = 4.5, SD = 1.0) and 1 to 3 syllables (mean = 1.4, SD = 0.5), word size 0.79 to 3.17 visual degrees. The base-10 logarithms of the word frequencies (per million) ranged from 0.0 to 3.0 (mean = 1.2, SD = 0.7) (Baayen et al., 1993). Picture size was 0.95 to 3.97 visual degrees. The mean luminance of the stimuli was 101 cd/m2. Pictures were taken from the Snodgrass–Vanderwart set (Snodgrass and Vanderwart, 1980) but only if their names did not exceed an upper limit of 6 letters in length. The pictures were supplemented by outlines of concrete entities that were closely similar in appearance to the Snodgrass– Vanderwart set. The forward and backward masks were chosen pseudorandomly out of a pool of 40 different masks constructed by scrambling the pictures.

3298

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

In addition, each run contained 40 null events and 40 mask-only events per run. Together with the word (n = 40) and the picture trials (n = 40) as well as the catch trials (n = 8), this resulted in a total of 168 trials per run. Subjects were instructed to respond negatively to the mask-only trials. A negative key press therefore was required when subjects had been unable to identify the word or picture, when the stimulus consisted of a chimera, and in mask-only trials. Each run of the first experiment consisted of 195 volumes. Twenty subjects underwent 6 runs, 4 subjects underwent 5 runs, and 4 subjects underwent 4 runs. One week after the fMRI session, 27 of the subjects were behaviorally re-tested (one dropout for study-unrelated reasons). We presented the same runs as during the fMRI experiment with exactly the same stimulus durations (Fig. 1A). The subjects had to select a button depending on whether or not they were able to identify the stimulus, as in the fMRI experiment. Following the key press, they had to pronounce the target word aloud. This provided an additional estimate of the degree to which the stimulus durations employed during fMRI were compatible with accurate stimulus identification. In a control experiment in 8 additional subjects, the pictures consisted of color photographs of single concrete entities against an empty background instead of outlines. The entities were partly different from those used in the main experiment. This experiment also contained a manipulation of stimulus familiarity (not reported here). Masks were constructed by scrambling the pixels of the photographs. Each subject underwent 3 fMRI runs. The experiment was otherwise identical to the above experiment. Subsequent memory retrieval experiment In the second experiment we sorted trials on the basis of subsequent memory retrieval success rather than subjective consciousness report. Nineteen naive subjects underwent 3 fMRI runs identical in design to those of the first experiment. These runs were followed by a surprise subsequent episodic memory retrieval task. A foveal word or picture was presented for 500 ms and subjects had to indicate by key press whether the stimulus had been presented in the preceding fMRI runs. Half of the stimuli were from the preceding fMRI runs (“old”) and half were novel. Stimuli in this experiment were taken from the same pool as was used in the first experiment and were pseudorandomly allocated to be used either during the fMRI runs and as old stimuli during the subsequent memory retrieval phase or as novel stimuli during the episodic memory retrieval phase only. For each entity this was counterbalanced across subjects. Overt naming/reading experiment Subjects were instructed to read the word or name the picture aloud during a silent gap of 1000 ms during which the fixation point turned green (Fig. 1B). In case they had not been able to identify the stimulus, they had to pronounce the word “blanco.” The setup was otherwise identical to that of the first experiment. As in the first experiment, stimulus durations xword and xpicture were titrated per individual from run to run aiming for 50% correct naming/reading responses. A jittered interval between 125 and 965 ms (with steps of 120 ms) preceded the presentation of a forward mask. The fixation period that followed the backward mask was jittered between 1075 and 235 ms with steps of 120 ms so that the sum of the first and the second period was always 1200 ms. All subjects underwent 6 runs, consisting of 174 volumes each.

images (EPI) with blood oxygenation level-dependent (BOLD) contrast using parallel imaging (GRAPPA). EPIs (TR/TE= 2000/30 ms) comprised 36 axial slices acquired continuously in ascending order covering the entire cerebrum (voxel size = 2.75 × 2.75 × 3.75 mm3). For each experiment, the first 6 volumes of each run were discarded to allow the MRI signal to reach steady state. Structural images were acquired during the same session using a high-resolution T1-weighted sequence (TR/TE= 9.6/4.6 ms, voxel size = 0.98 × 0.98 × 1.2 mm3). Foam pillows were used to minimize head movements. In the overt naming/reading experiment, we used a clustered volume acquisition technique (Edmister et al., 1999). This acquisition creates a silent gap in between two consecutive scans during which the subjects responded overtly (Fig. 1B). In this event-related fMRI scheme, repetition time (TR) was 3000 ms and acquisition time (TA) was 2000 ms, leaving a brief period (1000 ms) without image acquisition in between two consecutive scans. Data analysis Behaviorally, the reaction times of the identification responses to pictures or words were analysed using repeated-measures analysis of variance (ANOVA) with input modality (words or pictures) and response (affirmative or negative) as within-subject factors. We also conducted a separate ANOVA with order of presentation of the entity (first versus second presentation of the entity) and input modality (words or pictures) as factors. In the forced-choice recognition phase of the subsequent memory retrieval experiment, the true-positive response rate as well as d′ (MacMillan and Creelman, 1990) were compared between old items that subjects had identified during the preceding identification runs and old items that they had failed to identify using a Student's t test. We also compared the true-positive response rate to old items that subjects had failed to identify during the preceding identification runs with the false-positive rate to novel items (Student's t test). The imaging data were analyzed with Statistical Parametric Mapping SPM2 (Wellcome Trust Centre for Neuroimaging, University College London, UK). After correction for differences in slice acquisition time, the EPI volumes were realigned and resliced and an average functional image was created. The structural image was co-registered with that mean volume and subsequently normalised to the Montreal Neurological Institute (MNI) T1 template (Friston et al., 1995). The same normalisation matrix was applied to the functional images, which were then spatially smoothed using a Gaussian filter with a kernel size of 5 × 5 × 7 mm3 full width of half maximum. For the statistical analysis of the time series, we applied a high-pass filter of (1/128) Hz. The event-related response was modeled using a canonical hemodynamic response function consisting of a mixture of two gamma functions (Friston et al., 1999). Statistical inference was corrected for intrinsic autocorrelations. A statistical parametric map of the t statistic for the parameter estimates was generated and subsequently transformed to a Z map. First experiment In the first experiment, we restricted the analysis of the fMRI data to those entities to which approximately half of the subjects (range, 37% to 63%) responded affirmatively in both the word and the picture format. In this way, the stimuli associated with a positive consciousness report and those associated with a negative response were matched over subjects. The remaining trials were modeled separately but were not included in the main analysis. For every individual, we calculated the following key contrasts:

Image acquisition All fMRI experiments were conducted on a 3-T Philips Achieva system (Best, The Netherlands) equipped with an 8-channel head volume coil. Functional images were obtained using T2⁎ echo-planar

1. conjunction analysis (Nichols et al., 2005): (positive − negative consciousness report for words) and (positive − negative consciousness report for pictures) 2. main effect of input modality: words − pictures and inversely

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

3. interaction effect: (positive − negative report for words) − (positive − negative report for pictures) and inversely 4. conjunction analysis (Nichols et al., 2005): (negative consciousness report for words − mask-only trials) and (negative consciousness report for pictures − mask-only trials). The individual contrast images were weighted for the total number of runs per subject and were then entered into a secondlevel, random-effects analysis. At the second level, we examined for each of the a priori comparisons whether, on average, the contrast images revealed significant differences (one-sample t test). The significance level for whole-brain analyses was set at a voxel-level inference threshold of uncorrected P b 0.001 combined with a clusterlevel inference of P b 0.05 corrected for the whole brain volume (Poline et al., 1997). In a secondary analysis, we included all trials and all runs. The control experiment with colored photographs was analysed by means of a fixed-effects analysis. Our main contrast was equivalent to contrast 1. The analysis was restricted to volumes of interest which corresponded to the significant clusters obtained in contrast 1 of the first experiment. The significance threshold was set at P b 0.05 corrected for these volumes of interest (small volume correction). Subsequent memory retrieval experiment In the subsequent memory retrieval experiment, events from the identification runs were sorted on the basis of subsequent memory retrieval success. The main contrast was: 5. conjunction analysis (Nichols et al., 2005): (words that subjects subsequently retrieved successfully − words that subjects failed to subsequently retrieve) and (pictures that subjects subsequently retrieved successfully − pictures that subjects failed to subsequently retrieve) The individual contrast images were weighted for the total number of runs per subject and were then entered into a secondlevel, random-effects analysis. At the second level, we examined whether, on average, the contrast images revealed significant differences (one-sample t test). Since we were interested in amodal regions that fulfilled a dual criterion (associated with a positive consciousness report as well as predictive of subsequent memory retrieval success), the analysis of contrast 5 was restricted to the significant clusters obtained in contrast 1 of the first experiment. We tested for a significant effect of subsequent memory retrieval in these volumes using a significance threshold of P b 0.05 corrected for these volumes of interest (small volume correction). Naming/reading experiment The overt naming/reading experiment was analysed by means of a fixed-effects analysis. We calculated the following contrasts: 6. (accurately named pictures+accurately read words)−(pictures+ words to which the subject responded “blanco”) 7. main effect of input modality: words − pictures and inversely 8. interaction effect: (accurately read words − words to which the subjects responded “blanco”) − (accurately named pictures − pictures to which the subject responded “blanco”) and inversely Dynamic causal modelling To test how the amodal system was connected with the input modality-specific systems, we formally compared different dynamic causal models (Friston et al., 2003). Amodal nodes were included when they were significantly activated in contrast 1 (conjoint activity during positive versus negative consciousness report for both words and pictures) as well as in contrast 5 (conjoint activity during subsequent memory retrieval success for both words and pictures). Modality-specific nodes were included when they were differentially active for words compared to pictures (contrast 2 and its inverse, first

3299

experiment), excluding activations that were due to a differential decrease. Each node contained the same voxels for all subjects. For a detailed representation of the models that we compared, we refer to Fig. 2. The models we compared differed in how the amodal system was connected with the modality-specific systems. For each amodal node, we constructed a model in which that node only was connected with the modality-specific systems (Figs. 2A and B). We also included a model where all amodal nodes were connected equally to the input modality-specific systems (Fig. 2C). The number of models therefore that we compared equalled the number of amodal nodes plus one. All other parameters were matched between the models (Fig. 2). Input modality-specific nodes that were located in occipital cortex received input from their specific modality, amodal nodes that were located in occipital cortex received input from both modalities (Fig. 2). The time series that were used for the DCM were extracted from the pooled data from the first experiment as well as the identification runs of the second experiment (n = 47). At the group level, the models were formally compared by means of the group Bayes factor (which is the product of the Bayes factors of each subject), which is based on the Akaike (AIC) and the Bayesian information criterion (BIC) (Penny et al., 2004). The group Bayes factor is a ratio of model evidences based on the data and is used to compare models. It takes into account the complexity of the models. To evaluate between-subject consistency, we also calculated in how many individuals DCM provided positive evidence (defined as an individual Bayes factor N 3) for a specific model compared to the other models (Stephan et al., 2007). Results Behavioral data First experiment In the first experiment, our titration procedure yielded an average word presentation duration across runs and subjects of 35.3 ms (SD = 9.6) and an average picture presentation duration of 50.7 ms (SD = 15.8). Thirty-four out of 156 runs were excluded because subjects responded false-positively to more than 3 out of 8 catch trials in these runs, indicating that in these runs subjects produced a positive key press even when they had failed to identify the stimuli at the basic level. Subjects responded false-positively to 0.6% (SD = 0.5) of all mask-only events. Seventy-two out of 240 real-world entities were identified by approximately half of the subjects in both the word and the picture format. We restricted the analysis to these entities so that the contrast between positive and negative report was matched for stimuli over subjects. When subjects gave a positive consciousness report, reaction times were significantly shorter compared to when they gave a negative report (F(1,27) = 14.1, P = 0.001) (Table 1). Subjects responded faster to words than to pictures (F(1,27) = 14.5, P = 0.001). There were no interaction effects (F(1,27) = 0.06, P = 0.816) (Table 1). The positive response rate did not differ significantly between first and second presentations of an entity (F(1,27) = 3.32, P = 0.08) or between words and pictures (F(1,27) = 2.11, P = 0.16). There was no interaction between repetition of an entity and input modality (F(1,27) = 0.28, P = 0.60) (first presentation of an entity, word modality: positive response rate 51.7% (SD = 6.4), first presentation of an entity, picture modality: 54.5% (SD = 8.1), second presentation of an entity, word modality: 50.6% (SD = 6.8); second presentation of an entity, picture modality: 52.1% (SD = 8.4)). In the behavioral retest study, in which the same stimulus durations were used as in the fMRI study, a positive key press in response to a word trial was associated with 90.6% (SD = 8.9) correct reading and a negative key press with 2.6% (SD = 8.2) correct reading. A positive key

3300

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

Fig. 2. Three DCM models tested. (A) First model: the connections between the amodal and the modality-specific system go exclusively via IFS. (B) Second model: the connections between the amodal and the modality-specific system go exclusively via OTS. (C) Third model: each of the two amodal nodes are connected with the two modality-specific systems. Legend: Lines marked with ID: connections that are modulated by consciousness report (positive versus negative) according to the model tested. Lines marked with P: connections that are modulated picture modality according to the model tested. Lines marked with W: connections that are modulated by word modality. Dashed lines: connections that are modulated by word input modality (W) and consciousness report (ID). Dash-dotted lines: connections that are modulated by picture input modality (W) and consciousness report (ID). OTS: left occipitotemporal sulcus; IFS: left inferior frontal sulcus; STS: left superior temporal sulcus; FO: left frontal operculum; mog: middle occipital gyrus; coll: collateral sulcus.

Table 1 First, control, and second experiment: reaction times (in ms) (mean and SD).

Positive report for words Negative report for words Positive report for pictures Negative report for pictures Mask events (word duration) Mask events (picture duration)

First experiment

Control experiment

Second experiment

Mean

SD

Mean

SD

Mean

SD

752.9 811.8 782.3 838.7 638.5 638.8

134.7 128.3 122.8 119.5 86.2 89.5

744.9 740.5 782.2 815.0 640.4 642.2

121.0 104.4 134.1 111.4 74.1 73.3

804.6 866.9 806.0 912.8 703.5 715.3

105.8 126.1 124.7 120.5 90.1 96.0

press to a picture trial was associated with 81.8% (SD = 14.1) correct naming and a negative key press with 2.0% correct naming (SD = 5.1). In the color photograph experiment, behavioral findings were very similar to those of the main experiment except for the fact that reaction times did not differ for words between positive and negative reports (Table 1). This finding is difficult to interpret and may be a false-negative due to the small sample size (n = 8). Subsequent memory retrieval experiment In the identification runs of the subsequent memory retrieval experiment, our titration procedure yielded an average word

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

presentation duration of 30.6 ms (SD = 9.2) and an average picture presentation duration of 49.7 ms (SD = 11.1) across runs and subjects. Reaction times were significantly faster when subjects responded affirmatively compared to negatively (F(1,18) = 13.0, P = 0.002) (Table 1). Subjects responded faster to words than to pictures (F(1,18) = 7.6, P = 0.013), without interaction effects. In the subsequent memory retrieval phase, the true-positive response rate differed significantly between old stimuli to which subjects had responded positively in the preceding fMRI runs (54.3%, SD = 15.2) and old stimuli to which subjects had responded negatively (23.1%, SD = 13.4) (Student's t(18) = 9.0, P b 0.001). This was confirmed by analysis of the sensitivity measure d′ (MacMillan and Creelman, 1990) (positive response: d′ = 1.11, SD = 0.68; negative response: d′ = 0.21, SD = 0.36, Student's t(18) = 9.40, P b 0.001). We also determined whether old stimuli to which subjects had given a negative consciousness report were recognized differently from novel items. The true-positive response rate for old stimuli to which subjects had given a negative consciousness report (23.1%, SD = 13.4, see above) was significantly higher than the false-positive response rate for novel stimuli (18.8%, SD = 14.3) (t(18) = 2.23, P = 0.039). Overall, the true-positive recognition rate, i.e., the proportion of old stimuli that subjects recognized correctly, was 38.7% (SD = 13.9) for words and 38.6% (SD 12.9) for pictures. The false-positive recognition rate, i.e., the proportion of novel stimuli to which subjects responded as if they had been presented in the previous fMRI runs, was 18.0% (SD 16.4) for words and 19.7% (SD 13.0) for pictures. Overt naming/reading experiment In the overt naming/reading experiment, we titrated stimulus duration so as to reach 50% accurate naming or reading. Across runs and subjects, the average word presentation duration was 45.5 ms (SD 9.9) and the average picture presentation duration was 77.0 ms (SD 12.2). Subjects read the words correctly in 53.6% (SD 19.2) of all word trials, read the words incorrectly in 17.0% (SD 10.2), and responded “blanco” in 28.8% (SD 20.9) of all word trials, with response omissions in the remainder of the word trials. Fifty-eight percent of errors consisted of substitution, addition, or omission of one to three letters, most often in the middle of the word (45.3%) and less frequently at the end (30.5%) or the beginning of the word (24.2%). About 9.8% of errors were semantic paralexias. The remainder of word errors did not bear any apparent relationship with the target word. Pictures were named correctly in 49.5% (SD 9.7) of all trials. Subjects named the pictures incorrectly in 31.5% (SD 14.1) and responded “blanco” in 18.5% (SD 15.2) of all trials. About 45.4% of picture naming errors were based on perceptual similarity without semantic relatedness, 28.5% of errors could be explained by both perceptual similarity and semantic relatedness, 4.3% of picture naming errors were semantically related to the target word, without perceptual similarity. In 6.3% of the picture naming errors, subjects provided a name at the superordinate level. The remainder of picture errors did not bear any apparent relationship with the target picture. Imaging results First experiment Regardless of input modality (conjunction contrast 1), words or pictures, a positive consciousness report was associated with significantly higher activity in the middle third of the left inferior frontal sulcus (IFS) (Figs. 3A and C), the left inferior frontal junction (IFJ) (Figs. 3B and C), the border between the posterior and the middle third of the left occipitotemporal sulcus (OTS) (Figs. 3C and D) and the middle third of the left intraparietal sulcus (IPS) (Figs. 3C and E) (Table 2). A secondary analysis in which we included all events and all runs strictly confirmed these findings.

3301

As for the input modality-specific effects (contrasts 2 and inverse), in comparison to pictures, words activated the posterior third of the left superior temporal sulcus (STS) and the left frontal operculum (FO) (Table 2; Fig. 4). The FO activation (Fig. 5, yellow) lay clearly inferior to the IFS and IFJ areas reported above (Fig. 5, green). Several other regions such as left and right inferior parietal lobule (Fig. 5, yellow) responded more strongly to words than to pictures, but these differences were purely due to a differential decrease compared to baseline. Pictures activated the middle occipital gyrus (mog) bilaterally as well as the left collateral sulcus (Table 2). In a whole-brain analysis, we did not obtain a significant interaction between input modality and conscious stimulus identification (contrasts 3 and inverse). In left FO (interaction effect −57, 12, 15, Z = 2.66, uncorrected (uncorr.) voxel-level P b 0.005) and left STS (interaction effect − 60, − 51, 6, Z = 2.84, uncorr. voxel-level P b 0.005) word-specific responses tended to be larger when subjects were able to consciously identify the stimuli (Fig. 4). The same was true for picture-specific responses in the right mog (interaction effect 42, −81, 12, Z = 3.96, uncorr. voxel-level P b 0.001). Compared to mask-only trials, when subjects gave a negative consciousness report to a word or a picture trial, left OTS and left IFJ were significantly activated for words and for pictures (Figs. 3B and D), as well as the anterior cingulate sulcus and the anterior insula bilaterally (Table 2). To examine IFS with higher sensitivity, we conducted a small volume correction using the IFS cluster that was obtained from contrast 1 as volume of interest. Activity in IFS was significantly higher during a negative consciousness report compared to mask-only trials both for words and for pictures (− 45, 33, 21, Z = 2.98, corr. P = 0.024). In the control experiment with colored photographs, the contrast between a positive and a negative consciousness report yielded very similar results, apart from IPS. When subjects gave a positive instead of a negative response, the left OTS (−45, −45, −18, Z = 3.18, ext. 8, corr. P b 0.05), IFS (− 45, 33, 15, Z = 3.35, ext. 9, corr. P b 0.01), and IFJ (−42, 12, 30, Z = 3.84, ext. 16, corr. P b 0.01) were significantly activated for both words and pictures. Subsequent memory retrieval experiment Activity in two clusters that were associated with a positive consciousness report for both words and pictures (derived from conjunction contrast 1) predicted subsequent memory retrieval success in an amodal manner, conjointly for both words and pictures (conjunction contrast 5): OTS (−54, − 57, −12, Z = 2.98, ext. 5, corrected P b 0.05) and IFS (− 42, 36, 18, Z = 2.67, ext. 15, corrected P = 0.05) (Fig. 5, orange outline). IPS and IFJ activity did not predict subsequent memory retrieval success, even if we lowered the threshold to an uncorrected P b 0.05. Overt naming/reading experiment When subjects responded correctly to the words or the pictures (contrast 6), left IFS, IFJ, and OTS were all significantly more active than when they responded “blanco” (IFS: − 48, 39, 9, Z = 5.16, ext. = 179, corr. P b 0.001, IFJ: −48, 9, 27, Z = 4.19; OTS: − 54, −54, −12, Z = 6.15, ext. = 197, corr. P b 0.001), together with several other activations (Fig. 5, blue outline). In contrast with the first experiment (contrast 2 and inverse, Figs 4 and 5, yellow), the overt naming/reading experiment did not reveal a main effect of input modality (contrast 7 and inverse) in left FO and the posterior third of left STS, even when we lowered the threshold to uncorrected P b 0.001. This was due to the fact that these regions were activated both during word and during picture trials compared to mask-only events (conjunction analysis (Nichols et al., 2005): − 48, 9, 21, Z = 6.33, corr. voxel-level P b 0.001 and −57, −39, 3, Z = 6.44, corr. voxel-level P b 0.001, respectively). We did not find any significant interactions between input modality and naming/reading success.

3302

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

Dynamic causal modelling The outcome of the comparison between the 3 models (Fig. 2) was highly in favor of model 2 (Fig. 2B) above model 1 or 3: the group Bayes factor for model 2 compared to the model 1 or 3 was more than 10768 and 10522, respectively (Penny et al., 2004). The proportion of

subjects with positive evidence in favor of model 2 versus subjects with positive evidence in favor of one of the other models was 45:2 and 47:0, respectively (Stephan et al., 2007). It was therefore much more likely that the effective connectivity between the amodal and the modality-specific regions went principally via OTS. In the winning

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307 Table 2 Details of the significant activity foci obtained in the first experiment. Region

x

y

z

Z

Extent

Conjunction contrast 1: (positive report − negative report) (W & and P) L IFS − 42 36 15 4.09 23 L IFJ − 45 9 33 3.97 22 L OTS − 51 − 57 − 15 4.29 33 L IPS − 39 − 51 57 3.78 27 Contrast 2: W − P L STS − 63 − 51 9 3.85 32 L FO − 51 12 18 4.26 25 L IPL − 60 − 42 42 4.01 47 R IPL 51 − 39 51 4.33 126 Contrast 2, inverse: P − W L mog − 45 − 81 15 4.72 194 R mog 51 − 69 −6 6.02 795 L coll. s. − 27 − 48 − 15 4.68 84 Conjunction contrast 4: (negative report − mask-only) (W & and P) L IFJ − 45 6 33 4.50 40 L OTS − 45 − 60 −9 3.94 40 Anterior cingulate 3 18 48 4.13 77 L anterior insula − 33 21 9 4.41 37 R anterior insula 42 24 3 4.84 66

Corr. P b0.05 b0.05 b0.01 b0.01 b0.005 b0.05 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.005 b0.001 b0.005 b0.001

Abbreviations: IPL: inferior parietal lobule; coll. s.: collateral sulcus; L: left; R: right; W: words; P: pictures; extent: unit = 3 × 3 × 3 mm3 voxels; corr. P first expt: cluster-level inference corrected for whole brain search volume.

model, there were no direct connections between IFS and the modality-specific nodes. There was a significant modulatory effect (strength = 0.05) of positive consciousness report on the forward connection from OTS to IFS and only a weak modulatory effect on the feedback connection (strength = 0.01).

Discussion Two regions, the middle third of the left IFS (Figs. 3A and C) and the left occipitotemporal sulcus (Figs. 3C and D), were activated in an amodal manner when a stimulus, word or picture, gained access to consciousness. In left OTS, and to a lesser degree also in IFS, this effect consisted of an enhancement of a response that was also present for subliminal stimuli. Activation in these two regions predicted subsequent memory retrieval success for both words and pictures (Fig. 5, orange outline). With regards to the initial research questions, our results imply that the functional contribution of the middle third of left IFS to amodal processing is not limited to conditions that require explicit semantic strategic retrieval. With regards to left OTS, subliminal effects (Dehaene et al., 2001; Eddy et al., 2007) and effects of conscious recognition (Bar et al., 2001) occurred in a same region independently of input modality, words, or pictures. The subjectivity of the response decision (Frith et al., 1999) in the first experiment cannot account for our findings. We monitored the response bias by means of the false-positive response rate to foil stimuli and rejected runs when this rate exceeded a preset threshold. Furthermore, in the subsequent memory retrieval experiment, events were sorted on the basis of subsequent memory retrieval rather than subjective consciousness report and this more objective sorting criterion yielded essentially the same results in IFS and OTS.

3303

Neither can differences in difficulty between trials with versus without conscious identification account for the effects. We included only those items in our primary analysis that were identified by approximately half of the participants. In this way, we were able to exclude item-related differences (such as stimulus familiarity or visual complexity) between trials that did and trials that did not lead to conscious identification across subjects. Furthermore, if anything, trials in which subjects failed to identify the stimulus could be considered more difficult than those in which subjects succeeded, yet activity was higher in the latter type (Fig. 3). Could subvocal phonological retrieval during picture processing have given rise to apparently amodal effects in left IFS during consciousness report? This can be ruled out for two reasons: in the first experiment, words but not pictures caused activation of the left posterior superior temporal sulcus (STS) and of the frontal operculum (FO) compared to baseline. Even when we lowered the significance threshold to an uncorrected P b 0.05 in the first experiment, left STS and FO were not activated during picture events compared to maskonly trials (Fig. 4). Had the subjects named the pictures silently, left STS and FO would be activated also during picture processing, as was the case during the overt naming and reading experiment. Furthermore, many previous neuroimaging studies of phonological processing as well as a lesion mapping study (Fiez et al., 2006) implicate the left FO rather than the IFS in phonological output retrieval (Paulesu et al., 1993; Fiez and Petersen, 1998; Price, 1998; Poldrack et al., 1999; Nixon et al., 2004). The absence of any silent naming/reading response in STS or FO can probably be accounted for by the high perceptual demands and the fast presentation rate which may have directed attentional resources during the picture trials towards perceptual processing rather than lexical retrieval. The OTS focus (Figs. 3C and D and 6) is near or identical to the visual word form area (VWFA) (Cohen et al., 2000; Dehaene et al., 2001; Cohen et al., 2002; Dehaene et al., 2002; Cohen et al., 2003; Gaillard et al., 2006; Vinckier et al., 2007). The anatomical relationship between the OTS cluster and the VWFA activity peaks from the literature is depicted in Fig. 6. While there is a certain degree of variability in the exact location of the activity peaks from the literature, the original focus reported by Cohen et al. (2000) falls well within our OTS cluster along the x, y, and z dimensions. In left OTS, activity is robustly enhanced during conscious identification compared to subliminal processing. In a previous study (Dehaene et al., 2001), left OTS showed an approximately 90% higher response when words were unmasked compared to subliminal (masked) words. Our findings also confirm previous evidence that the role of VWFA is not limited to word processing exclusively and that it is activated to a same degree for pictures as for words (Price and Devlin, 2004; Ben-Shachar et al., 2007). At a neuronal level, similar computations may be performed for the two types of stimuli or the apparent amodality may be a consequence of the relatively low resolution of fMRI: neurons with preference for word or for picture stimuli may be intermixed in this region creating an apparently amodal effect on fMRI responses. According to TVA, a visuoperceptual unit is consciously identified and selected for more elaborate processing when it wins the competition for entry into VSTM (Bundesen, 1990; Bundesen et al., 2005; Desimone and Duncan, 1995). Access to VSTM is required for

Fig. 3. First experiment: conjunction contrast: (positive − negative consciousness report for words) and (positive − negative consciousness report for pictures). Threshold: voxellevel uncorrected P b 0.001 (green). (A) IFS: Z map and time–activity curve. Time–activity curves show responses in each of the 6 conditions averaged over all 28 subjects and all 23 voxels belonging to this cluster. (B) IFJ: Z map and time–activity curves for each of the conditions averaged over all 28 subjects and all 22 voxels belonging to this cluster. (C) Z map of the conjunction projected on the Population-Average, Landmark and Surface based (PALS) (Van Essen, 2005) atlas using Computerized Anatomical Reconstruction and Editing Toolkit (side view and ventral view) (CARET, Washington University School of Medicine, Department of Anatomy and Neurobiology, http://brainmap.wustl.edu). (D) OTS: Z map and time–activity curves averaged for each of the conditions over all 28 subjects and all 27 voxels belonging to this cluster. (E) IPS: Z map and time–activity curves averaged for each of the conditions over all 28 subjects and all 33 voxels belonging to this cluster. Legend: Dark blue: Positive consciousness report for words. Cyan: Negative consciousness report for words. Red: Positive consciousness report for pictures. Magenta: Negative consciousness report for pictures. Black: Mask-only (solid: duration of interval between masks equal to that of the word events; dotted: duration of interval between masks equal to that of the picture events).

3304

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

Fig. 4. First experiment. Words minus pictures thresholded at uncorr. voxel-level P b 0.001. (A) Activity map projected on the PALS brain using CARET. (B) Left frontal operculum: Z map and time–activity curves for each of the 6 conditions averaged over all 28 subjects and all 25 voxels belonging to this cluster. (C) Left posterior STS: Z map and time-activity curves for each of the 6 conditions averaged over all 28 subjects and all 32 voxels belonging to this cluster. Same conventions as in Fig. 3.

subjects to be aware of object identity and also enhances episodic memory encoding, a combined effect that we observed in our study in the middle third of IFS and in OTS. IFS and OTS may mediate access to short-term memory enabling a more advanced level of processing associated with a higher likelihood of conscious processing and subsequent memory retrieval. Single neuron recording studies provide evidence that both inferior temporal cortex and prefrontal cortex are involved in short-term memory but with some fundamental distinctions. For instance, neuronal activity in inferior temporal cortex is more sensitive to intervening stimuli than prefrontal activity (Miller, 2000). Alternatively, the IFS activation may relate to cognitive

Fig. 5. Summary of the main results superimposed on the PALS atlas using CARET. Green: conjunction contrast: positive minus negative consciousness report for both words and pictures conjointly. Thresholded at uncorrected P b 0.001 (voxel-level inference). Orange outline: conjunction contrast: voxels within these clusters that are predictive of subsequent memory retrieval success for both words and pictures. Thresholded at P b 0.05 corrected for the volume of interest. Yellow: Contrast 2: words minus pictures. Thresholded at uncorrected P b 0.001 (voxel-level inference). Dark blue outline: Contrast 6: (accurately named pictures + accurately read words) − (pictures + words to which the subject responded “blanco”). Thresholded at uncorrected P b 0.001 (voxel-level inference).

control induced by perceptual difficulty (Duncan and Owen, 2000). IFS activation may reflect successful cognitive control allowing perceptual identification as opposed to a failure of cognitive control leading to lack of perceptual identification. Exactly how IFS exerts cognitive control to enhance success of perceptual identification remains to be specified in more detail, e.g., at the neuronal level. This explanation is akin to that proposed by Bar et al. (2001, 2003, 2006) regarding the interaction between orbitofrontal and ventral occipitotemporal cortex in stimulus identification. As a third possible explanation, IFS activity may reflect implicit semantic processing, i.e., a spread of activation to nearby semantic nodes whenever the entity is evoked regardless of task demands (Collins and Loftus, 1975; Neely, 1977; Ruff et al., 2008). According to Gabrieli et al. (1998), the middle third of IFS is involved in semantic working memory at the “crossroad” between semantic knowledge and episodic memory processing and mediates access to preexisting knowledge, integrating perception and interpretation of the current stimulus with other knowledge (Wagner et al., 1999). Repetition priming effects in this region are influenced by the task performed during the priming phase, semantic versus nonsemantic (Demb et al., 1995; Wagner et al., 2000). Our results demonstrate that IFS is also involved when semantic executive demands are minimal in the absence of a semantic task. The term “semantic working memory” may not be entirely appropriate for the type of processing subjects had to do in our experiment. The amodal areas that we obtained during conscious identification and subsequent memory retrieval overlapped with the activity pattern seen during classical picture naming and word reading but the activity pattern in our study was much more restricted. It was also

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

3305

Fig. 6. Neuroanatomical relationship between OTS cluster obtained in the current experiments and visual word form area coordinates from the literature. (A) Sagittal section (x = − 41 mm). This section provides a see-through projection of all activity peaks from x = − 33 to − 48 mm. This panel provides an accurate view of how the OTS cluster relates to activity peaks from the literature along the y and z dimensions. (B) Coronal section (y = − 59 mm). This section provides a see-through projection of all activity peaks from y = − 48 to − 69 mm. It provides an accurate view of how the OTS cluster relates to the activity peaks along the x and z dimensions. (C) Axial section (z = − 7 mm). This section provides a see-through projection from z = − 22 to 8 mm. It provides an accurate view of how the cluster relates to the activity peaks along the x and y dimensions. Legend: Green area: OTS cluster obtained from the conjunction contrast 1 thresholded at an uncorr. P b 0.001; orange area: voxels within the OTS cluster where activity predicts subsequent memory retrieval success (conjunction contrast 5); cyan circle (− 42, − 57, − 6): hemifield-independent response to words minus rest (Cohen et al., 2000) and visually presented words minus nonword stimuli (Gaillard et al., 2006); green circle (− 48, − 60, − 12) and triangle (− 44, − 52, − 16): masked words minus baseline (Dehaene et al., 2001); yellow circle (− 39, − 57, − 9): (words and consonant letter strings) minus checkerboards (Cohen et al., 2002); yellow triangle (− 33, − 69, − 6): words minus consonant letter strings (Cohen et al., 2002); magenta (− 41, − 61, 8): visually presented words minus rest (Dehaene et al., 2002); red circle (− 42, − 63, − 15): (words and consonant letter strings) minus fixation or checkerboards (Cohen et al., 2003); dark blue (− 42, − 51, − 3): letter strings minus checkerboards (Gaillard et al., 2006); black (− 40, − 48, − 22): increase with increasing proximity to words (Vinckier et al., 2007).

more restricted than the activity pattern obtained in studies of amodal processing that used tasks with higher semantic executive demands (Vandenberghe et al., 1996; Vandenbulcke et al., 2007). Prefrontal activations in particular were far less extensive than in previous experiments that made use of explicit semantic tasks. This indirectly suggests that some of the amodal prefrontal activations, e.g., in the middle frontal gyrus or in anterior inferior frontal gyrus, as seen in previous studies, may have been contingent on the level of semantic executive control required. As our experiments, however, demonstrate, this is not the case for the middle third of the left IFS. The absence of another region, the left anterior temporal pole, may be due to lack of sensitivity of fMRI in this region (Devlin et al., 2000) or it may point to a functional distinction between IFS and the anterior temporal pole. The anterior temporal pole is principally found to be activated when multiple representations must be combined in a novel way, for instance, during semantic association tasks with multiple items (Vandenberghe et al., 1996), comprehension of sentences (Vandenberghe et al., 2002), or stories (Fletcher et al., 1995). The left anterior temporal pole may play a role as an associative structure, binding distributed semantic representations in a combinatorial fashion (Vandenbulcke et al., 2005). The absence of anterior temporal cortex in our study may be due to the fact that we presented single stimuli and that subjects did not have to actively retrieve associations. Another region that has been typically found in studies of amodal processing but is missing in the current study is the posterior third of the left middle temporal gyrus (GTm) (Vandenberghe et al., 1996; Vandenbulcke et al., 2007). A possible explanation is that this area is involved in retrieval of associative–semantic knowledge beyond the mere identification of words or the pictures. If the task demands explicit naming, category or similarity judgments of words or pictures, its activity level may be higher than when simple identification is required (Ruff et al., 2008). To define the position of the amodal regions vis-à-vis the modality-specific systems, we explicitly modeled the effective connectivity between the amodal zones (IFS and OTS) and the modality-specific systems and compared different models. Direct feedforward connections from the two input modality-specific systems towards IFS would be in agreement with a role of IFS as a convergence zone onto which input from the input modality-specific systems converges. This convergence onto IFS might be associated with a higher probability of conscious processing (Baars, 1988; Damasio, 1989; Baars, 2002). According to this hypothesis, IFS serves

as a hub receiving afferents from the input modality-specific systems and sending efferents to other cognitive systems, such as the episodic memory system. As a second possibility, IFS may influence modalityspecific systems through its interactions with OTS rather than via direct effects on the modality-specific systems. A Bayesian comparison between different dynamic causal models provided strong arguments in favor of the latter model. To conclude, left IFS and OTS are involved in cognitive processing of words and pictures in an amodal manner, independent of semantic executive demands. We postulate that these amodal regions mediate access to short-term memory, enhancing cognitive control, detailed perceptual analysis, and episodic memory encoding. Acknowledgments This work was supported by Research Foundation Flanders (FWO) [G.0076.02 and G0668.07 to R.V.], K.U.Leuven [OT/04/41, OT/08/056, EF/05/014 to R.V.], and Federaal Wetenschapsbeleid belspo [IUAP P6/ 29]. References Baars, B., 1988. A cognitive theory of consciousness. Cambridge University Press. Cambridge, UK. Baars, B., 2002. The conscious access hypothesis: origins and recent evidence. Trends Cogn. Sci. 6, 47–52. Baayen, H., Piepenbrock, R., Rijn, H.V., 1993. The CELEX lexical database (CD-ROM). Linguistic Data Consortium. PA, Philadelphia. Bar, M., 2003. A cortical mechanism for triggering top-down facilitation in visual object recognition. J. Cogn. Neurosci. 15, 600–609. Bar, M., Kassam, K., Ghuman, A., Boshyan, J., Schmid, A., Dale, A., Hamalainen, M., Marinkovic, K., Schacter, D., Rosen, B., Halgren, E., 2006. Top-down facilitation of visual recognition. Proc. Nat. Acad. Sci. 103, 449–454. Bar, M., Tootell, R., Shacter, D., Greve, D., Fischl, B., Mendola, J., Rosen, B., Dale, A., 2001. Cortical mechanisms specific to explicit visual object recognition. Neuron 29, 529–535. Beck, D., Rees, G., Frith, C., Lavie, N., 2001. Neural correlates of change detection and change blindness. Nat. Neurosci. 4, 645–650. Ben-Shachar, M., Dougherry, R., Deutsch, G., Wandell, B., 2007. Differential sensitivity to words and shapes in ventral occipito-temporal cortex. Cereb. Cortex 17, 1604–1611. Buckner, R., Koutstaal, W., Shacter, D., Rosen, B., 2000. Functional MRI evidence for a role of frontal and inferior temporal cortex in amodal components of priming. Brain 123, 620–640. Bundesen, C., 1990. A theory of visual attention. Psychol. Rev. 97, 523–547. Bundesen, C., Habekost, T., 2008. Principles of visual attention. Linking Mind and Brain. Oxford University Press, Oxford, UK.

3306

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307

Bundesen, C., Habekost, T., Kyllingsbaek, S., 2005. A neural theory of visual attention: Bridging cognition and neurophysiology. Psychol. Rev. 112, 291–328. Carmel, D., Lavie, N., Rees, G., 2006. Conscious awareness of flicker in humans involves frontal and parietal cortex. Curr. Biol. 16, 907–911. Chertkow, H., Bub, D., Deaudon, C., Whitehead, V., 1997. On the status of object concepts in aphasia. Brain Lang. 58, 203–232. Cohen, L., Dehaene, S., Naccache, L., Lehericy, S., Dehaene-Lambertz, G., Henaff, M., Michel, F., 2000. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split brain patients. Brain 123, 291–307. Cohen, L., Lehericy, S., Chochon, F., Lemer, C., Rivaud, S., Dehaene, S., 2002. Languagespecific tuning of visual cortex? Functional properties of the visual word form area. Brain 125, 1054–1069. Cohen, L., Martinaud, O., Lemer, C., Lehéricy, S., Samson, Y., Obadia, M., Slachevsky, A., Dehaene, S., 2003. Visual word recognition in the left and right hemispheres: anatomical and functional correlates of peripheral alexias. Cereb. Cortex 13, 1313–1333. Collins, A., Loftus, E., 1975. A spreading–activation theory of semantic processing. Psychol. Rev. 82, 407–428. Copland, D., deZubicaray, G., McMahon, K., Wilson, S., Eastburn, M., Chenery, H., 2003. Brain activity during automatic semantic priming revealed by event-related functional magnetic resonance imaging. NeuroImage 20, 302–310. Damasio, A., 1989. Time-locked multiregional retroactivation: a systems-level proposal for the neural substrates of recall and recognition. Cognition 33, 25–50. Dehaene, S., Leclech, G., Poline, J., Bihan, D.L., Cohen, L., 2002. The visual word form area: a prelexical representation of visual words in the fusiform gyrus. Neuroreport 13, 321–325. Dehaene, S., Naccache, L., Cohen, L., LeBihan, D., Mangin, J., Poline, J., Rivere, D., 2001. Cerebral mechanisms of word masking and unconscious repetition priming. Nat. Neurosci. 4, 752–758. Demb, J., Desmond, J., Wagner, A., Vaidya, C., Glover, G., Gabrieli, J., 1995. Semantic encoding and retrieval in the left inferior prefrontal cortex: a functional MRI study of task difficulty and process specificity. J. Neurosci. 15, 5870–5878. Desimone, R., Duncan, J., 1995. Neural mechanisms of selective visual attention. Ann. Rev. Neurosci. 18, 193–222. Devlin, J., Russell, R., Davis, M., Price, C., Wilson, J., Moss, H., Matthews, P., Tyler, L., 2000. Susceptibility-induced loss of signal: comparing PET and fMRI on a semantic task. Neuroimage 11, 589–600. Diaz, M., McCarthy, G., 2007. Unconscious word processing engages a distributed network of brain regions. J. Cogn. Neurosci. 19, 1768–1775. Duncan, J., Owen, A., 2000. Common regions of the human frontal lobe recruited by diverse cognitive demands. Trends Neurosci. 23, 475–483. Eddy, M., Schnyer, D., Schmid, A., Holcomb, P., 2007. Spatial dynamics of masked picture repetition effects. Neuroimage 34, 1723–1732. Edmister, W., Talavage, T., Ledden, P., Weisskoff, R., 1999. Improved auditory cortex imaging using clustered volume acquisitions. Hum. Brain Mapp. 7, 89–97. Fiez, J., Petersen, S., 1998. Neuroimaging studies of word reading. Proc. Natl. Acad. Sci. U.S.A. 95, 914–921. Fiez, J., Tranel, D., Seager-Frerichs, D., Damasio, H., 2006. Specific reading and phonological processing deficits are associated with damage to the left frontal operculum. Cortex 42, 624–643. Fletcher, P., Happe, F., Frith, U., Baker, S., Dolan, R., Frackowiak, R., Frith, C., 1995. Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension. Cognition 57, 109–128. Forster, K., 1998. The pros and cons of masked priming. J. Psycholinguist. Res. 27, 203–232. Friston, K., Harrison, L., Penny, W., 2003. Dynamic causal modelling. NeuroImage 19, 1273–1302. Friston, K., Holmes, A., Worsley, K., Poline, J., Frith, C., Heather, J., Frackowiak, R., 1995. Statistical parametric maps in functional imaging: a general approach. Hum. Brain Mapp. 2, 189–210. Friston, K., Zarahn, E., Josephs, O., Henson, R., Dale, A., 1999. Stochastic designs in eventrelated fMRI. NeuroImage 10, 607–619. Frith, C., Perry, R., Lumer, E., 1999. The neural correlates of conscious experience: an experimental framework. Trends Cogn. Sci. 3, 105–114. Gabrieli, J., Poldrack, R., Desmond, J., 1998. The role of left prefrontal cortex in language and memory. Proc. Natl. Acad. Sci. U.S.A. 95, 906–913. Gaillard, R., Naccache, L., Pinel, P., Clémenceau, S., Volle, E., Hasboun, D., Dupont, S., Baulac, M., Dehaene, S., Adam, C., Cohen, L., 2006. Direct intracranial, fMRI and lesion evidence for the causal role of left inferotemporal cortex in reading. Neuron 50, 191–204. Garrard, P., Carroll, E., 2006. Lost in semantic space: a multi-modal, nonverbal assessment of feature knowledge in semantic dementia. Brain 129, 1152–1163. Gold, B., Balota, D., Jones, S., Powell, D., Smith, C., Andersen, A., 2006. Dissociation of automatic and strategic lexical–semantics: functional magnetic resonance imaging evidence for differing roles of multiple frontotemporal regions. J. Neurosci. 26, 6523–6532. Goldberg, R., Perfetti, C., Fiez, J., Schneider, W., 2007. Selective retrieval of abstract semantic knowledge in left prefrontal cortex. J. Neurosci. 27, 3790–3798. Grainger, J., Holcomb, P., 2009. Watching the word go by: on the time-course of component processes of visual word recognition. Lang. Linguist. Compass 1, 128–156. Haynes, J., Driver, J., Rees, G., 2005. Visibility reflects dynamic changes of effective connectivity between V1 and fusiform cortex. Neuron 46, 811–821. Hodges, J., Patterson, K., Oxbury, S., Funnell, E., 1992. Semantic dementia: progressive fluent aphasia with temporal lobe atrophy. Brain 115, 1783–1806.

Holcomb, P., Grainger, J., 2006. On the time course of visual word recognition: an eventrelated potential investigation using masked repetition priming. J. Cogn. Neurosci. 18, 1631–1643. Howard, D., Patterson, K., 1992. Pyramids and Palm Trees: A Test of Semantic Access from Pictures and Words. Thames Valley Publishing. Humphreys, G., Forde, E., 2001. Hierarchies, similarity and interactivity in object recognition. Behav. Brain Sci. 24, 453–476. Jackendoff, R., 2007. Language, Consciousness, Culture. Essays on Mental Structure. MIT Press, Cambridge, MA. Kanwisher, N., 2001. Neural events and perceptual awareness. In: Dehaene, S. (Ed.), The Cognitive Neuroscience of Consciousness. Elsevier Science publishers, Amsterdam, pp. 89–113. Kircher, T., Sass, K., Sachs, O., Krach, S., 2009. Priming words with pictures: neural correlates of semantic associations in a cross-modal priming task using fMRI. Hum. Brain Mapp. 30, 4116–4128. MacMillan, N., Creelman, C., 1990. Response bias: characteristics of detection theory, threshold theory and “non-parametric” indexes. Psychol. Bull. 107, 403–413. Marois, R., Yi, D., Chun, M., 2004. The neural fate of consciously perceived and missed events in the attentional blink. Neuron 41, 465–472. Miller, E., 2000. The prefrontal cortex and cognitive control. Nat. Rev. Neurosci. 1, 59–65. Neely, J., 1977. Semantic priming and retrieval from lexical memory: roles of inhibitionless spreading activation and limited-capacity attention. J. Exp. Psychol. Gen. 106, 226–254. Nichols, T., Brett, M., Andersson, J., Wager, T., Poline, J., 2005. Valid conjunction inference with the minimum statistic. Neuroimage 25, 653–660. Nixon, P., Lazarova, J., Hodinott-Hill, I., Gough, P., Passingham, R., 2004. The inferior frontal gyrus and phonological processing: an investigation using rTMS. J. Cogn. Neurosci. 16 (2), 289–300. Oldfield, R., 1971. The assessment and analysis of handedness: the Edinburgh Inventory. Neuropsychologia 9, 97–113. Paulesu, E., Frith, C., Frackowiak, R., 1993. The neural correlates of the verbal component of working memory. Nature 362, 342–345. Penny, W., Stephan, K., Mechelli, A., Friston, K., 2004. Comparing dynamic causal models. NeuroImage 22, 1157–1172. Poldrack, R., Wagner, A., Prull, M., Desmond, J., Glover, G., Gabrieli, J., 1999. Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex. NeuroImage 10, 15–35. Poline, J., Worsley, K., Evans, A., Friston, K., 1997. Combining spatial extent and peak intensity to test for activations in functional imaging. NeuroImage 5, 83–96. Price, C., 1998. The functional anatomy of word comprehension and production. Trends Cogn. Sci. 2, 281–288. Price, C., Devlin, J., 2004. The pro and cons of labelling a left occipitotemporal region ‘visual word form area’. Neuroimage 22, 477–479. Qiao, E., Vinckier, F., Szwed, M., Naccache, L., Valabrégue, R., Dehaene, S., Cohen, L., 2010. Unconsciously deciphering handwriting: subliminal invariance for handwritten words in the visual word form area. Neurimage 49, 1786–1799. Race, E., Shanker, S., Wagner, A., 2009. Neural priming in human frontal cortex: multiple forms of learning reduce demands on the prefrontal executive system. J. Cogn. Neurosci. 21, 1766–1781. Raposo, A., Moss, H., Stamatakis, E., Tyler, L., 2006. Repetition suppression and semantic enhancement: an investigation of the neural correlates of priming. Neuropsychologia 44, 2284–2295. Rapp, B., Hillis, A., Caramazza, A., 1993. The role of representations in cognitive theory: more on multiple semantics and the agnosias. Cogn. Neuropsychol. 10, 235–249. Riddoch, M., Humphreys, G., 1993. Birmingham Object Recognition Battery. Lawrence Erlbaum Associates Ltd, Hove, UK. Rissman, J., Eliassen, J., Blumstein, S.E., 2003. An event-related fMRI investigation of implicit semantic priming. J. Cogn. Neurosci. 15, 1160–1175. Rogers, T., McClelland, J., 2004. Semantic Cognition. A Parallel Distributed Approach. MIT Press, Cambridge, MA. Ruff, I., Blumstein, S., Myers, E., Hutchison, E., 2008. Recruitment of anterior and posterior structures in lexical–semantic processing: an fMRI study comparing implicit and explicit tasks. Brain Lang. 105, 41–49. Sachs, O., Weis, S., Zellagui, N., Huber, W., Zvyagintsev, M., Mathiak, K., Kircher, T., 2008. Automatic processing of semantic relations in fMRI: neural activation during semantic priming of taxonomic and thematic categories. Brain Res. 1218, 194–205. Sass, K., Krach, S., Sachs, O., Kircher, T., 2009. Lion–tiger–stripes: neural correlates of indirect semantic priming across processing modalities. Neuroimage 45, 224–236. Snodgrass, J., Vanderwart, M., 1980. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity and visual complexity. J. Exp. Psychol. Hum. Learn. Mem. 6, 174–215. Stephan, K., Marshall, J., Penny, W., Friston, K., Fink, G., 2007. Inter-hemispheric integration of visual processing during task-driven lateralization. J. Neurosci. 28, 3512–3522. Thierry, G., Price, C., 2006. Dissociating verbal and nonverbal conceptual processing in the human brain. J. Cogn. Neurosci. 18, 1018–1028. Vandenberghe, R., Nobre, A., Price, C., 2002. The response of left temporal cortex to sentences. J. Cogn. Neurosci. 14, 550–560. Vandenberghe, R., Price, C., Wise, R., Josephs, O., Frackowiak, R., 1996. Functional anatomy of a common semantic system for words and pictures. Nature 383, 254–256. Vandenbulcke, M., Peeters, R., Dupont, P., Van Hecke, P., Vandenberghe, R., 2007. Word reading and posterior temporal dysfunction in amnestic mild cognitive impairment. Cereb. Cortex 17, 542–551.

L. Van Doren et al. / NeuroImage 49 (2010) 3295–3307 Vandenbulcke, M., Peeters, R., Fannes, K., Vandenberghe, R., 2006. Knowledge of visual attributes in the right hemisphere. Nat. Neurosci. 9, 964–970. Vandenbulcke, M., Peeters, R., Hecke, P.V., Vandenberghe, R., 2005. Anterior temporal laterality in primary progressive aphasia shifts to the right. Ann. Neurol. 58, 362–370. Van Essen, D.C., 2005. A population-average, landmark- and surface-based atlas of human cerebral cortex. Neuroimage 28, 635–662. Vinckier, F., Dehaene, S., Jobert, A., Dubus, J., Sigman, M., Cohen, L., 2007. Hierarchical coding of letter strings in the ventral stream: dissecting the inner organization of the visual word-form system. Neuron 55, 143–156.

3307

Wagner, A., Desmond, J., Demb, J., Glover, G., Gabrieli, J., 1997. Semantic repetition priming for verbal and pictorial knowledge: a functional MRI study of left inferior prefrontal cortex. J. Cogn. Neurosci. 9, 714–726. Wagner, A., Koutstaal, W., Maril, A., Shacter, D., Buckner, R., 2000. Task-specific repetition priming in left inferior prefrontal cortex. Cereb. Cortex 10, 1176–1184. Wagner, A., Koutstaal, W., Shacter, D., 1999. When encoding yields remembering: insights from event-related neuroimaging. Phil. Trans. R. Soc. Lond. B 354, 1307–1324. Wilenius-Emet, M., Revonsuo, A., Ojanen, V., 2004. An electrophysiological correlate of human visual awareness. Neurosci. Lett. 354, 38–41.