- Email: [email protected]
Neural correlates of narrative shifts during auditory story comprehension
NeuroImage 47 (2009) 360–366
Contents lists available at ScienceDirect
NeuroImage j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / y n i m g
Neural correlates of narrative shifts during auditory story comprehension Carin Whitney a,⁎, Walter Huber b, Juliane Klann b,c, Susanne Weis d, Sören Krach e, Tilo Kircher e a
Department of Psychology, University of York, York, YO10 5DD, UK Section Neurolinguistics, Department of Neurology, RWTH Aachen University, Germany c Central Service Facility “Functional Imaging” at the ICCR-BIOMAT, RWTH Aachen University, Germany d Section Clinical Neuropsychology, Department of Neurology, RWTH Aachen University, Germany e Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany b
a r t i c l e
i n f o
Article history: Received 29 October 2008 Revised 31 March 2009 Accepted 8 April 2009 Available online 17 April 2009
a b s t r a c t The ability to segment continuous linguistic information online into larger, meaningful units is a key element in narrative comprehension. Narrative shifts, i.e. transitions between individual units, are postulated to continuously update the mental situation model. Their cerebral correlates, however, have hardly been investigated. Under highly naturalistic conditions this study seeks to identify the neural correlates of implicit processing of narrative shifts during continuous speech comprehension. 16 male native German speakers listened passively to a German novella for 23 min while BOLD signal was recorded with fMRI. Text comprehension was tested in a short post-scan interview asking for critical episodes of the story. Narrative shifts were defined on the basis of a macropropositional analysis. Compared to listening to text passages of the narrative that neither contained narrative shifts nor structurally similar linguistic control events (i.e., sentence boundaries), narrative shifts evoked increased BOLD signal changes in the right temporal gyrus, precuneus and posterior/middle cingulate cortex bilaterally. When narrative shifts were contrasted with sentence boundaries, activation in the right precuneus and cingulate cortex remained significant. The results strengthen the relevance of medial parietal structures for natural language comprehension. More precisely, the precuneus and posterior cingulate appear to be the neural substrate for updating mental story representations and can be regarded as critical parts of a more complex, distributed neural network underlying story comprehension. © 2009 Elsevier Inc. All rights reserved.
Introduction The majority of psycholinguistic neuroimaging studies are designed to investigate a single, highly specific component of language functioning that is isolated from other confounding processes, which might co-occur in more naturalistic conditions. Such paradigms reduce the ecological validity of the experiment, and it remains debatable to which degree the investigation of isolated processes contributes to the understanding of the same process in a more complex environment. It is also questionable in how far this approach is suitable for the investigation of implicit variables underlying complex linguistic performance such as text comprehension, which requires a more naturalistic setting as can be offered with traditional fMRI designs. In the last decade, a different approach has been introduced by presenting or recording continuous, more natural linguistic data during fMRI. In the first of these studies, recurrent expressive linguistic events within the continuous speech output during spontaneous picture description (e.g., hesitation pauses or segments
⁎ Corresponding author. Fax: +44 1904 433181. E-mail address: [email protected] (C. Whitney). 1053-8119/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.neuroimage.2009.04.037
of graded complexity of linguistic encoding) were determined posthoc for individual speakers and taken as input for subsequent eventrelated analyses (Kircher et al., 2000, 2004). A similar approach was used to study continuous movie and story perception, e.g. during video-clip viewing of everyday activities (Zacks et al., 2001), watching of commercial movies (Bartels and Zeki, 2004; Golland et al., 2007; Hasson et al., 2004) or story comprehension in children (Karunananayaka et al., 2007; Schmithorst et al., 2006). However, most studies that utilized continuous fMRI recording concentrated on a corroboration of standard fMRI findings with the application of this novel technique or with newly developed fMRI data analysis tools such as independent component analysis (ICA) (e.g., Karunananayaka et al., 2007; Schmithorst et al., 2006). For example, Zacks et al. (2001) applied post-hoc event-related analyses in order to explore event boundaries during active and passive video-clip viewing in a relatively naturalistic task. Participants were asked to segment video clips of everyday activities into natural and meaningful units by pressing a handheld button during fMRI scanning (active viewing condition). The resulting temporal event structure of each participant was taken to analyze the continuous fMRI signal under the active viewing condition and a passive version, during which no unit boundaries were marked. Remarkably, similar neural activations were found for both conditions in a bilateral
C. Whitney et al. / NeuroImage 47 (2009) 360–366
posterior network (precuneus, cuneus, inferior and superior temporal sulci, fusiform gyrus) signalling an implicit parsing capacity during which a temporally evolving environment was automatically junked into meaningful units. It should be noted, however, that the assumed capacity of parsing was purely operationally derived in this study. It remained unclear to what extent parsing relied on external visual cues or on underlying processes of semantic interpretation. Furthermore, one might object that these attempts of studying naturalistic comprehension lack theoretically based predictions on the cognitive and/or linguistic nature of the cues that trigger such common, timelocked human brain activity. Semantic theories of online story comprehension may provide a framework in which boundaries between natural and meaningful units can be specified. In particular, the notion of “narrative shift” appears to be promising. Most text comprehension models agree that narratives are represented as mental models of the situations being described in the text (e.g., ‘event-indexing model’ Zwaan and Radvansky, 1998; ‘mental model’ Johnson-Laird, 1983; ‘situation model’ van Dijk and Kintsch, 1983; ‘structure building framework’ Gernsbacher, 1997; for a review see Mar, 2004). Individuals constantly attempt to integrate the incoming linguistic information into a propositional text base which is then integrated with their world knowledge into a situation model. Development and changes in such a model occur when narrative units or ‘events’ fail to overlap in one or more of their constituent features such as character, time, action or location (e.g., Johnson-Laird, 1983; Rich and Taylor, 2000; Zwaan et al., 1995). Macropropositions form hereby the basis for such narrative events, which abstract from the text base and represent the ‘gist of the text’ (Kintsch and van Dijk, 1978) or ‘fact units’ (Huber, 1990). They are confined by knowledge or schemata (Schank and Abelson, 1977) of the described situation, on the basis of which irrelevant or redundant textual information is deleted, a sequence of individual propositions is generalized, or new propositions are constructed by means of inference. The same rules can be iteratively applied to the resulting macrostructure, creating a mental representation of the narrative that contains the most relevant (and most memorable) information of a respective text unit (Kintsch and van Dijk, 1978). Given this hierarchical organization, narrative shifts can overlap with unit boundaries on the micropropositional (i.e., predicate–argument structure) and surface level, but not every unit boundary on the lower level corresponds to a narrative shift. It also follows that narrative shifts can be identified at different levels of a macropropositional analysis, dependent on the iteration of the macro-rules. Therefore, narrative shifts depict observer-independent, time-locked events during text comprehension and do neither occur during the perception of coherent text passages nor at lower-level unit boundaries in general (e.g., at the end of a sentence). The cognitive processes operating at content shifts seem to overlap for different types of narrative units (i.e., micro- or macropropositions). Independent of whether local or global inconsistencies are detected, the reader/listener fails to integrate the input with previous information and needs to update the mental representation of the evolving cognitive model by adding new entries or replacing old references with current links to spatial, temporal or personal dimensions (Schulte and Vaisänen, 2006; Zwaan and Radvansky, 1998). Narrative shifts or updates, therefore, correspond to an enhanced integration effort (i.e., of current linguistic input) and a substitution of information in working memory. They are further characterized by attentional shifts between different dimensions of discourse representation (Zwaan and Radvansky, 1998) and increased processing costs in drawing inferences (for a review see Graesser et al., 1994; Virtue et al., 2006). The demand on working memory, inference and integration processes is likely to vary with the number of situational dimension that can be mapped onto previous stages as indicated by increased reading times and low coherence ratings scores (Zwaan and Radvansky, 1998). Text comprehension can thus be
361
characterized as a constant search for coherence, which becomes extremely difficult at narrative unit boundaries. Neuroimaging studies investigating coherence breaks both, on the sentence level (Ferstl and von Cramon, 2001, 2002; Kuperberg et al., 2006) or within larger narrative structures (Maguire et al., 1999; Martin-Loeches et al., 2008; Speer et al., 2007; Xu et al., 2005) have identified similar activation patterns. Enhanced BOLD responses were consistently reported in medial parietal structures including the precuneus and/or posterior cingulate gyrus (but cf. Sieboerger et al., 2007) and were accompanied by various study-specific signal increases in the medial frontal, temporal, lateral frontal or parietal cortices. It appears to be the medial parietal brain regions that define the coherence network and make it distinct from the neural correlate of text comprehension in general (for a meta-analysis see also Ferstl et al., 2008). Further, the close resemblance to the mental imagery network (for a review see Cavanna and Trimble, 2006) strengthens the role of the precuneus/posterior cingulate gyrus in situation model building. Unlike coherent text passage, narrative shifts provoke a partial renewal of the mental model, which increases the demand on processes related to mental imagery, such as attentive tracking and switching across dimensions (e.g., time, space). One could argue that the medial parietal BOLD enhancements in the study by Zacks et al. (2001) might have also been a result of similar higher-level semantic shifts that have triggered the segmentation process during video-clip viewing. Although the authors have associated unit boundaries with individual segments in the cognitive representation of the everyday activity, details about the event structure (e.g., on the micro- or macropropositional level) and its segmentation were missing to confirm this assumption. In sum, the neuroimaging findings imply that human brain activity in medial parietal structures is intrinsically time-locked to perceptive narrative event boundaries irrespective of the level of segmentation (micro- vs. macrostructure). Hence, narrative shifts are a suitable candidate to postulate critical time points of receptive speech for a continuous fMRI recording. The objective of the present study was to uncover the neural correlates of implicit processing of narrative shifts during passive, natural auditory story comprehension. So far it remains unclear whether the medial parietal network responds to narrative shifts that occur during natural language comprehension of larger narrative structures and in how far previous findings generalize to the auditory modality. The task required listening to a 23-minute story without any prior knowledge of story content or its event structure and the purpose of the study. Narrative shifts were defined on the basis of macropropositional analysis beforehand (Huber, 1990; van Dijk and Kintsch, 1983) and entered into an event-related fMRI analysis. We hypothesized that these theory-based, reliably defined transitions between linguistic units would elicit activation in a medial posterior network that has previously been related to coherence breaks at different levels of text representation. In contrast, BOLD responses at randomly chosen lower level boundaries (i.e., at the end of a sentence) occurring within coherent text passages should fail to activate these functionally specific structures. Materials and methods Participants Initially, 19 male subjects took part in the fMRI study. Due to head movement, data of three subjects had to be discarded from further analysis. All remaining 16 participants (mean age = 27.00 years, SD = 6.65; mean years of education = 14.50 years, SD = 1.67) were native German speakers, right-handed according to the Edinburgh Inventory of Handedness (Oldfield, 1971) and showed average or above average verbal IQ as assessed by the German MWT-B multiplechoice vocabulary test (Lehrl et al., 1995) (mean estimated verbal IQ = 120.06, SD = 17.16). Subjects with recent substance use or
362
C. Whitney et al. / NeuroImage 47 (2009) 360–366
general MRI incompatibility (e.g., metal implants) were excluded. All subjects gave informed consent and were paid 20 Euros for participation in the study. The study was approved by the local ethics committee.
recall was required. Answers were classified as “correct” or “incorrect”, accordingly. No time restrictions were given.
Stimuli
Narrative shifts were defined on the basis of a macropropositional analysis (Huber 1990; van Dijk and Kintsch 1983). In theories about discourse processing, macropropositions are specified as basic, narrative building blocks or events “constrained by referential continuity and general plausibility” (p. 156, Huber, 1990). The key features describing a narrative event are the character(s), time and location (Rich and Taylor, 2000) as well as the character's goals and their interaction with objects (Speer et al., 2007). Any change in the continuity of these features (i.e., the introduction of a new, relevant character; a shift of time, location, goal or interaction) indicates a transfer from one macroproposition to the next. Psychologically, such changes in continuity result in an update of the mental situation model. The following definitions of the four types of narrative shifts were applied to our text (Rich and Taylor, 2000; Schulte and Väisänen, 2006): (a) a shift in person is apparent whenever a new, relevant character is introduced or the focus switches from one to another, already known, character, (b) a shift in time is a jump forward in time, usually indicated by specific linguistic markers (e.g. the next day, 5 years later), ranging from hours to years in our story, (c) a shift in location is defined as a relocation of the narrative action from one place to another, (d) a shift in action is present whenever a new action becomes the central part of an event despite continuity in person, time or location. According to the criteria defined for person, time, location and action shifts, a group of four experts (WH, JK, TK, CW) marked narrative shifts on a written version of the novella during listening to its recording. The story was familiar to all experts beforehand. Only those narrative shifts were included into the analysis which were identified by the macropropositional analysis and marked by all four experts. Table 1 provides an overview of the novella's complete macropropositional structure and its narrative shifts. All but one narrative shift (M8) were identified at sentence boundaries, i.e. at the beginning of the sentence that introduced a change in one or more of the four dimensions. Three of the theoretically defined narrative shifts (M1, M11, M17) were not modelled in the SPM analysis: M1 and M11 were situated at the beginning of each run, i.e. part 1 and part 2 of the story. M17 was only identified by three out of four experts. Mean distance between two successive narrative shifts was 85.33 s (SD = 51.41 s). To verify whether brain activation related to narrative shifts was indeed specific to such narrative shifts and could not be attributed to sentence boundaries in general, we selected an equal number of events at sentence boundaries that were distinct from the narrative shifts. 15 sentence boundaries were chosen such that they lay approximately between two shifts or between the beginning of a session and its first corresponding shift.
A slightly modified version of the German novella “Der Kuli Kimgun” by Max Dauthendey (1909) was chosen for this study. The modifications concerned low-frequent or foreign words which were substituted by more familiar or high-frequent words. Furthermore, emotionally highly arousing text passages were replaced by less exciting material. The final version included a total of 3581 words. In general, novellas are short, well-structured narratives restricted to a few protagonists and basic narrative events which, together, are ascribed to a single, central conflict. They are written from an omniscient perspective, leaving out elaborate descriptions of the character's emotional or mental states. In the chosen novella, the sequence of events occurs chronologically, which allows the listener to build up a temporally continuous, mental representation of the story. Further details of the narrative structure are presented in the section on the narrative shift analysis. For auditory presentation during fMRI, the story was professionally recorded and spoken in a natural way by a trained, male speech therapist. The duration of the story was 23:32 min. Procedure The story was presented via MRI compatible headphones in two successive runs lasting 14:32 and 9:00 min, respectively. Participants were instructed to close their eyes and listen to the story carefully. Behavioural analysis To make sure that subjects attended to the content, they were informed at the beginning of the experiment about a short interview after the MRI session about the content of the story. Each of the ten questions, asked for in chronological order, required the recall of a critical event and referred to a specific episode/macroproposition as identified by the macropropositional analysis and listed in Table 1 (M1, 2, 4, 6, 9, 10, 12, 13, 15 and 17). Subjects succeeded whenever the essence of the respective event was captured, i.e. no overly detailed Table 1 Linguistic parameters of the stimulus novella (Dauthendey, 1909). Onset time (scans) M1a
3
Episode/macroproposition
Narrative shift
Ali living in his home town
Beginning of part 1 Location Action Location Location Action Action Character Location Action Beginning of part 2 Character Location Action Character Action Character Character
M2 M3 M4 M5 M6 M7 M8 M9 M10 M11a
55 75 97 196 218 257 327 349 400 3
News from the sunken bell Ali on his way to Rangoon Ali in the temple Ali in Rangoon Cleaning of the bell Procession Girl Festival Buying of bananas Ali and his wife living near the beach
M12 M13 M14 M15 M16 M17a M18
36 72 158 169 195 216 246
Birth of Ali's son and death of his wife Baby in the temple Ali on his way to the temple Tiger in the temple Ali's hopelessly attempted rescue Tiger's attack of the baby Shooting of the tiger
M = macroproposition. a Not modelled in the SPM analysis.
Narrative shift analysis
fMRI data acquisition All scanning was performed on an 1.5 T scanner (Gyroscan Intera, Philips Medical, Eindhoven, The Netherlands) using standard gradients and a circular polarized phase array head coil. For each subject, we acquired two series of functional volumes of T2⁎-weighted axial EPI-scans parallel to the AC/PC line with the following parameters: number of slices (NS), 22; slice thickness (ST), 5.0 mm; interslice gap (IG), 0.55 mm; matrix size (MS), 64 × 64; field of view (FOV), 240 × 240 mm; echo time (TE), 50 ms; repetition time (TR), 2.0 s. 436 functional volumes were acquired for the first part of the story and 270 functional volumes for the second part, adding up to 706 volumes in total.
C. Whitney et al. / NeuroImage 47 (2009) 360–366
fMRI data analysis MR images were analyzed using Statistical Parametric Mapping software (SPM2; www.fil.ion.ucl.ac.uk) implemented in MATLAB 7.0 (Mathworks Inc., Sherborn, MA). After discarding the first three volumes, all images were realigned to the first image to correct for head movement. Unwarping was used to correct for the interaction of susceptibility artifacts and head movement. After realignment and unwarping, the signal measured in each slice was shifted relative to the acquisition time of the middle slice using a sinc interpolation in time to correct for their different acquisition times. Volumes were then normalized into standard stereotaxic anatomical MNI-space by using the transformation matrix calculated from the first EPI-scan of each subject and the EPI-template. Afterwards, the normalized data with a resliced voxel size of 4 × 4 × 4 mm were smoothed with an 8 mm FWHM isotropic Gaussian kernel to accommodate intersubject variation in brain anatomy. The time series data were filtered with a high-pass cut-off of 1/128 Hz. The autocorrelation of the data was estimated and corrected for. The expected hemodynamic response at stimulus onset for each event-type, narrative shifts and random sentence boundaries, was modeled by two response functions, a canonical hemodynamic response function (HRF; Friston et al., 1998) and its temporal derivative. The latter was included in the model to account for the residual variance resulting from small temporal differences in the onset of the hemodynamic response, which is not explained by the canonical HRF alone. The functions were convolved with the eventtrain of stimulus onsets to create covariates in a general linear model. Subsequently, parameter estimates of the HRF regressor for each of the two different conditions were calculated from the least mean squares fit of the model to the time series. Parameter estimates for the temporal derivative were not further considered in any contrast. Using SPM5, first level contrasts were computed for narrative shifts and random sentence boundaries. These contrasts were entered into a flexible factorial ANOVA on the group level in order to compute the individual contrast for each condition (narrative shifts, random sentence boundaries) and the respective differential contrasts. Since we did not include a resting baseline (null events) in our experimental design (which would have been unnatural), the term baseline refers to all text passages that were not explicitly modeled (i.e., excluding the 15 narrative shifts and 15 random sentence boundaries). Brain areas reported during listening to narrative shifts vs. baseline or random sentence boundaries vs. baseline thus show an increase of brain activity at these specific time points as compared to the baseline
363
activation of listening to the narration. Differential contrasts were inclusively masked by the minuend at p b .001 (uncorrected). The results were computed on a voxel-wise threshold of p b 0.001. Subsequently, a. Monte Carlo simulation of the brain volume of the current study was conducted to establish an appropriate voxel contiguity threshold (Slotnick et al., 2003). The basic principle of the Monte Carlo approach involves the simulation of brain activity whereby each individual voxel of the functional image matrix becomes activated with a set type I error probability (i.e., 0.001). The resulting activation map is smoothed with a FWHM Gaussian kernel (i.e., 8 mm) and the size of each contiguous voxel cluster is determined. After multiple, consecutive simulations (i.e., 10,000) the extent threshold k is chosen such that the relative frequency of clusters of size k or greater in the sample is smaller than the desired corrected p-value of p b 0.5. Assuming an individual voxel type I error of p b 0.001 in our study, a cluster extent of 13 contiguous resampled voxels was indicated as necessary to correct for multiple voxel comparisons at p b 0.05. Anatomical labels for activated brain areas were computed using the SPM anatomy toolbox, which is based on anatomical probability maps of the brain reflecting the intersubject variability of selected brain areas in terms of location and extend (Eickhoff et al., 2007). A detailed anatomical map was available for the superior parietal cortex (BA 5; Scheperjans et al., 2005). The remaining areas were labeled on the macroanatomical level. Results Behavioural results During the post-scan interview about critical story episodes, all participants were able to recall the desired macropropositional detail in response to each of the ten questions. Answers to all questions were provided quickly and effortlessly. fMRI results Listening to narrative shifts compared to baseline (i.e., listening to text passages that neither contained narrative shifts nor the 15 randomly selected sentence boundaries) revealed stronger brain activation in the right middle and superior temporal gyrus, the right precuneus and a voxel cluster spanning the middle cingulate cortex, adjacent posterior cingulate cortex and precuneus bilaterally (see Table 2, Fig. 1). In contrast, no brain region was significantly stronger
Table 2 Brain activation related to narrative shifts and to random sentence boundaries during auditory story comprehension in a group of 16 healthy participants. Cl Narrative shifts N baseline
1 2 3
Random sentence boundaries N baseline Narrative shifts N random sentence boundaries
Random sentence boundaries N narrative shifts
H R R R R L L/R R L No region of
Anatomical region Middle temporal gyrus Superior temporal gyrus Precuneus Posterior cingulate cortex Precuneus Middle cingulate cortex Precuneus (BA 5) Posterior cingulate cortex statistical significance
4
R
Precuneus
5 6
L Middle cingulate cortex R Posterior cingulate cortex No region of statistical significance
Z
No. voxels/mm3
0 12 44 28 40 40 52 20
5.20 4.88 5.08 4.38 4.09 3.83 3.59 3.49
50/3200
− 72
44
4.76
30/1920
− 44 − 32
36 28
4.63 4.00
15/960 16/1024
Coordinates x
y
z
52 60 12 4 − 12 0 4 −4
− 44 − 44 − 72 − 32 − 48 − 40 − 44 − 44
8 −8 4
53/3392 71/4544
Cl = Cluster, H = hemisphere, L = left, R = right. If a voxel cluster comprises two or more distinct brain areas, all are listed under the same cluster. The significance level is given in Zvalues and cluster size in number of voxels and volume (mm3) for p b .05 (corrected) with a cluster extent of 13 voxel.
364
C. Whitney et al. / NeuroImage 47 (2009) 360–366
Fig. 1. Stronger brain activation during the comprehension of narrative shifts compared to baseline or pseudo-randomly chosen sentence boundaries projected onto selected brain sections of the standard SPM template. Note: Activation is corrected for multiple comparisons at p b .05, extent threshold = 13 voxels.
activated during the comprehension of pseudo-randomly chosen sentence boundaries compared to baseline. Direct comparison between narrative shifts and random sentence boundaries resulted in stronger BOLD responses for narrative shifts in three distinct voxel clusters comprising the right precuneus, the left middle cingulate cortex and the right posterior cingulate cortex (see Table 2, Fig. 1). The reverse contrast did not reveal any significant results. Discussion In the present study we demonstrated BOLD enhancements in the precuneus, right temporal gyrus, posterior and middle cingulate cortex for narrative shifts while subjects listened passively to a continuous story. Because most of these shifts occurred at the beginning/end of a sentence, we additionally compared the narrative shift-specific brain activity to BOLD signal changes during the comprehension of sentence boundaries that were distinct from narrative shifts. Again, stronger activation was observed in the right precuneus and posterior/middle cingulate structures bilaterally for narrative shifts compared to the control events, which strengthens the relevance of extra-perisylvian, midline structures for text-level processing. We assume that the underlying cognitive mechanisms during the narrative segmentation process equal an online update of the situation model during continuous story comprehension. Besides increased semantic integration effort, memory-based processes seem to be of particular importance. These include the encoding and maintenance of current knowledge and re-access of previous contents of the situation model. On the neural level we suggest that this complex, highly specific behaviour is reflected in BOLD enhancements in a posterior midline network whereby each neural component (posterior cingulate cortex, precuneus) subserves different aspects of narrative update.
Narrative update: the role of the posterior cingulate–precuneus network Neuroimaging studies of story and sentence comprehension have associated precuneus and posterior cingulate activity with memory retrieval, narrative integration effort and the establishment of coherence across sentences, all being key components in the construction and update of situation models (Almor et al., 2007; Ferstl and von Cramon, 2001, 2002, 2007; Fletcher et al., 1995; Kircher et al., 2001; Obleser et al., 2007; Speer et al., 2007; Tracy et al., 2003; Xu et al., 2005). For example, Ferstl et al. (2005) related bilateral activations of the intraparietal sulcus (IPS) and the medial parietal cortex (precuneus) to memory related processes of attention shifts during the recognition of narrative coherence breaks. Switching from local inputs to global contextual aspects seems to be central when previous information has to be recalled during the updating of situation models. Likewise, Maguire et al. (1999) and Martin-Loeches et al. (2008) observed distinct activations of the posterior cingulate cortex and the precuneus when the global theme had to be extracted from unusual stories by means of prior knowledge. Similar to Ferstl et al. (2005), the observed parietal activation was related to updating old with new information in the mental model, which was guiding story comprehension. The majority of the research reporting posterior cingulate and precuneus activation, however, is based on visuo-spatial experiments investigating mental imagery. Mental imagery, again, has been regarded as an essential process during language comprehension (Zwaan and Rapp, 2006; see also dual-coding model, Pavio, 1986) involving functions which can easily be transferred to the construction of situation models (e.g., orientating within space and time, attentive tracking and shifts). Accordingly, processes ascribed to the precuneus subsume attentive tracking and shifting, sequencing of actions and episodic memory retrieval (for a review see Cavanna and Trimble, 2006). It has also been suggested that the precuneus acts as
C. Whitney et al. / NeuroImage 47 (2009) 360–366
a part of a larger working memory buffer that contains information readily available for current decision-making processes (for a review see Wagner et al., 2005). Likewise, the posterior cingulate cortex has been implicated in this parietal working memory network (Wagner et al., 2005) but seems to be further important for evaluating the environment being perceived, i.e. in orientating within the environment and interpreting it (for a review see Vogt et al., 1992). Relating this idea to story comprehension, the posterior cingulate cortex might be involved in evaluating the linguistic input in order to accept or reject its integration with prior knowledge (e.g., Maguire et al., 1999; Mar, 2004; Martin-Loeches et al., 2008; Xu et al., 2005). This knowledge is supposed to be maintained in working memory as reflected by precuneus activity. Thus, the precuneus is likely to reflect attentional and memory components during narrative updates, which intrinsically complement the listener/reader's attempt to evaluate and integrate ongoing linguistic information, a process which is presumably subserved by the posterior cingulate cortex. Similar to previous investigations (Speer et al., 2007, Zacks et al., 2001), we have identified the neural basis of content shifts without further differentiations between, for example, spatial, temporal or personal changes. Collapsing across the different types of narrative shifts, however, might underestimate the number of brain regions involved in the update process. Different types of narrative shifts are most likely to evoke activation in different, content-specific brain structures (see Ferstl et al., 2005). The current activation pattern is most similar to the activation observed for sentence pairs that focussed on spatial information in Ferstl and von Cramon (2007) and for short stories that involved chronological changes in Ferstl et al. (2005) (i.e., precuneus and/or posterior cingulate cortex). Since we neither identified any shifts in time nor a dominance of spatial shifts in the current study (cf. Table 1: character (n = 5), space (n = 5) and action shifts (n = 6)), it seems unlikely that the medial parietal activation can be attributed to a specific type of shift processing. However, we do not dismiss the alternative interpretation that spatial and temporal aspects might also play a role in the action shifts detected in our study and that we might have detected a narrative shift network that predominantly resembles content changes along these dimensions. Identifying the full extent of such shift-specific networks during natural language comprehension needs to be investigated in future studies. Sentence boundaries In line with our predictions, brain activation in the precuneus and posterior cingulate cortex was significantly stronger during the comprehension of narrative shifts than sentence boundaries. Sentence boundaries were chosen such that they lay within a coherent text episode, so that no update process was required and, hence, no medial parietal brain activation detected. Further, failing to detect any significant BOLD increases during such control events was not unexpected. The implicit baseline condition, to which random sentence boundaries were compared to, also contained numerous events that described sentence boundaries. Further implications and limitations The current investigation was designed to tap specific intrinsic speech comprehension processes, i.e. the automatic segmentation of ongoing speech input into smaller, meaningful units under naturalistic conditions. The novella, which remained unchanged on its macropropositional level, revealed 15 reliable events that described boundaries between episodes in the narration. Although former studies were able to obtain larger stimulus sets by focussing on smaller content units (e.g., Ferstl and von Cramon, 2002; Speer et al., 2007; Zacks et al., 2001), brain activation overlapped in medial
365
parietal structures. Further, the strength of the neural response in the precuneus and posterior cingulate cortex was comparable to text or sentence comprehension studies with more events (Ferstl and von Cramon, 2001, 2002, 2007; Ferstl et al., 2005; Obleser et al., 2007; Siebörger et al., 2007; Speer et al., 2007; Virtue et al., 2006), and blocked designs (Fletcher et al., 1995; Lindenberg and Scheef, 2007; Maguire et al., 1999; Martin-Loeches et al., 2008; Mazoyer et al., 1993; Ozawa et al., 2000; Xu et al., 2005). Both, the strength of activation and the overlap in location suggest that situation model update as it occurs during natural text comprehension in the present investigation can be attributed to a distinct medial parietal network. It appears to mediate core functions of higher-level comprehension associated with changes in situational dimensions (e.g., character, location) irrespective of the degree of segmentation process being investigated (fine vs. coarse), stimulus manipulation/presentation (natural vs. artificial) or modality (cf., for text reading see Speer et al., 2007; for passive viewing of movie clips see Zacks et al., 2001). As outlined above, this network might not be exhaustive. This is primarily due to the limited control over confounding variables (e.g., the diversity of shifts being included), which is inevitable to more naturalistic designs, but should be less attributed to the restricted number of events used in the current study. It remains arguable though, in how far the identified network generalizes to other novellas. Although the event boundaries were derived from a single story (see also Speer et al., 2007), the set of narrative shifts under investigation was not homogeneous. Not only did the events describe inconsistencies in character, action or place, but they were also likely to differ with respect to, for example, the visuo-spatial mental image being produced during situation model building (cf. Zwaan and Rapp, 2006) or the resources demanded to detect different kinds of narrative shifts (cf. Ferstl and von Cramon, 2007). Further, previous observations indicate that activation in the precuneus/posterior cortex is very robust and independent of both, intra-stimulus (e.g., type of shift) and inter-stimulus/study (e.g., modality) manipulations (cf. Speer et al., 2007; Zacks et al., 2001). These aspects, again, argue in favour of a key role of the medial parietal network in narrative shift comprehension across different linguistic stimuli including different kinds of novellas. Finally, unlike previous studies, we particularly refrained from online tasks controlling for the subject's attention towards narrative shifts such as plausibility judgements or the explicit marking of unit boundaries during comprehension to maintain a high ecological validity (see also Speer et al., 2007). Since all listeners were able to recall the relevant details of the macropropositions given in the novella during the post-scan interview, we can be sure that each participant built up a mental representation of the story. Since Zacks et al. (2001) and Speer et al. (2007) have shown that the parsing of continuous input into meaningful units occurs automatically, we can also be confident that all of the subjects comprehended narrative shifts. Ceiling performance (with individuals and groups scoring up to 100% correct), as defined over the number of questions answered correctly vs. incorrectly, is commonly observed during post-scan interviews (Fletcher et al., 1995; Virtue et al., 2006; Xu et al., 2005; Yarkoni et al., 2008). We chose open questions and, hence, reduced the probability of ‘false positives’, i.e. participants that did not listen to the story but have not been identified as such by the test. If a participant was indeed not paying attention, our design would have successfully recognized this behaviour, while tests on the basis of true/false questions (e.g., Virtue et al., 2006) or multiple-choice approaches (e.g., Schmithorst et al., 2006; Tracy et al., 2003; Xu et al., 2005; Yarkoni et al., 2008) might have failed. Concluding comments In this study, continuous passive speech comprehension has been investigated under highly naturalistic conditions. The results
366
C. Whitney et al. / NeuroImage 47 (2009) 360–366
strengthen the relevance of medial parietal brain regions for narrative comprehension. They further demonstrate that the precuneus and posterior/middle cingulate cortex were involved in the segmentation of continuous speech input into smaller, meaningful events. This key component of story comprehension appeared to be implicit (i.e., listeners perform this process spontaneously and automatically without explicit instructions to do so) and independent of presentation modality (cf. visual stimulation: Speer et al., 2007; Zacks et al., 2001). Overall, the cingulate–precuneus network should be considered as part of a more complex neural system subserving narrative comprehension. Acknowledgments The study was supported by a grant from the Interdisciplinary Centre for Clinical Research “BIOMAT”. within the Faculty of Medicine at the RWTH Aachen University (IZKF VV N68) and the International Research Training Group 1328 supported by the German Research Foundation. References Almor, A., Smith, D.V., Bonilha, L., Fridriksson, J., Rorden, C., 2007. What is in a name? Spatial brain circuits are used to track discourse references. NeuroReport 18, 1215–1219. Bartels, A., Zeki, S., 2004. Functional brain mapping during free viewing of natural scenes. Hum. Brain Mapp. 21, 75–85. Cavanna, A.E., Trimble, M.R., 2006. The precuneus: a review of its functional anatomy and behavioural correlates. Brain 129, 564–583. Dauthendey, M., 1909. Lingam – Zwölf asiatische Novellen. Albert Langen Verlag, München. Eickhoff, S.B., Paus, T., Caspers, S., Grosbras, M-H., Evans, A.C., Zilles, K., Amunts, K., 2007. Assignment of functional activations to probabilistic cytoarchitectonic areas revisited. NeuroImage 36, 511–521. Ferstl, E.C., von Cramon, D.Y., 2001. The role of coherence and cohesion in text comprehension: an event-related fMRI study. Cogn. Brain Res. 11, 325–340. Ferstl, E.C., von Cramon, D.Y., 2002. What does the frontomedian cortex contribute to language processing: coherence or theory of mind? NeuroImage 17, 1599–1612. Ferstl, E., Rinck, M., von Cramon, D.Y., 2005. Emotional and temporal aspects of situation model processing during text comprehension: an event-related fMRI study. J. Cogn. Neurosci. 17, 724–739. Ferstl, E.C., von Cramon, D.Y., 2007. Time, space and emotion: fMRI reveals contentspecific activation during text comprehension. Neurosci. Lett. 427, 159–164. Ferstl, E.C., Neumann, J., Bogler, C., von Cramon, D.Y., 2008. The extended language network: a meta-analysis of neuroimaging studies on text comprehension. Hum. Brain Mapp. 29, 581–593. Fletcher, P.C., Happé, F., Frith, U., Baker, S.C., Dolan, R.J., Frackowiak, R.S.J., et al., 1995. Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension. Cognition 57, 109–128. Friston, K.J., Fletcher, P., Josephs, O., Holmes, A., Rugg, M.D., Turner, R., 1998. Eventrelated fMRI: characterizing differential responses. Neuroimage 7, 30–40. Gernsbacher, M.A., 1997. Two decades of structure building. Discourse Processes 23, 265–304. Golland, Y., Bentin, S., Gelbard, H., Benjamini, Y., Hella, R., Nir, Y., et al., 2007. Extrinsic and intrinsic systems in the posterior cortex of the human brain revealed during natural sensory simulation. Cereb. Cortex 17, 177–1766. Graesser, A.C., Singer, M., Trabasso, T., 1994. Constructing inferences during narrative text comprehension. Psychol. Rev. 101, 371–395. Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., Malach, R., 2004. Intersubject synchronization of cortical activity during natural vision. Science 303, 1634–1640. Huber, W., 1990. Text comprehension and production in aphasia: analysis in terms of micro- and macromapping. In: Joanette, Y., Brownell, H.H. (Eds.), Discourse Ability and Brain Damage. Springer Verlag, pp. 154–179. Johnson-Laird, P.N., 1983. Mental Models. Harvard University Press, Cambridge, MA. Karunananyaka, P.R., Holland, S.K., Schmithorst, J., Solodkin, A., Chen, E.E., Szaflarski, J.P., et al., 2007. Age-related connectivity changes in fMRI data from children listening to stories. NeuroImage 34, 349–360. Kintsch, W., van Dijk, T.A., 1978. Towards a model of text comprehension and production. Psychol. Rev. 85, 363–394. Kircher, T.T.J., Brammer, M.J., Levelt, W., Bartels, M., McGuire, P., 2004. Pausing for thought: activation of left temporal cortex during pauses in speech. NeuroImage 21, 84–90. Kircher, T.T., Brammer, M., Tous Andreu, N., Williams, S.C., McGuire, P.K., 2001. Engagement of right temporal cortex during processing of linguistic context. Neuropsychologia 39, 798–809.
Kircher, T.T.J., Brammer, M.J., Williams, S.C.R., McGuire, P.K., 2000. Lexical retrieval during fluent speech production: an fMRI study. NeuroReport 11, 4093–4096. Kuperberg, G.R., Lakshmanan, B.M., Caplan, D., Holcomb, P.J., 2006. Making sense of discourse: an fMRI study of causal inferencing across sentences. NeuroImage 33, 343–361. Lehrl, S., Triebig, G., Fischer, B., 1995. Multiple choice vocabulary test MWT as a valid and short test to estimate premorbid intelligence. Acta Neurol. Scand. 91, 335–345. Lindenberg, R., Scheef, L., 2007. Supramodal language comprehension: role of the left temporal lobe for listening and reading. Neuropsychologia 45, 2407–2415. Maguire, E.A., Frith, C.D., Morris, R.G.M., 1999. The functional neuroanatomy of comprehension and memory: the importance of prior knowledge. Brain 122, 1839–1850. Mar, R.A., 2004. The neuropsychology of narrative: story comprehension, story production and their interrelation. Neuropsychologia 42, 1414–1434. Martin-Loeches, M., Casado, P., Hernandez-Tamames, J.A., Alvarez-Linera, J., 2008. Brain activation in discourse comprehension: a 3 T fMRI study. NeuroImage 41, 614–622. Mazoyer, B.M., Dehaene, S., Tzourio, N., Murayama, N., Cohen, L., Levrier, O., Salamon, G., Syrota, A., Mehler, J., 1993. The cortical representation of speech. J. Cogn. Neurosci. 5, 467–479. Obleser, J., Wise, R.J.S., Dresner, M.A., Scott, S.K., 2007. Functional integration across brain regions improves speech perception under adverse listening conditions. J. Neurosci. 27, 2283–2289. Oldfield, R.C., 1971. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9, 97–113. Ozawa, F., Matsuo, K., Kato, C., Nakai, T., Isoda, H., Takehara, Y., Moriya, T., Sakahara, H., 2000. The effects of listening comprehension of various genres of literature on response in the linguistic area: an fMRI study. NeuroReport 11, 1141–1143. Pavio, A., 1986. Mental Representations: A Dual Coding Approach. Oxford University Press, Oxford, UK. Rich, S.S., Taylor, H.A., 2000. Not all narrative shifts function equally. Mem. Cognit. 28, 1257–1266. Schank, R.C., Abelson, R., 1977. Scripts, Plans, Goals and Understanding: An Inquiry Into Human Knowledge Structures. Erlbaum, Hillsdale. Schmithorst, J., Holland, S.K., Plante, E., 2006. Cognitive modules utilized for narrative comprehension in children: a functional magnetic resonance imaging study. NeuroImage 29, 254–266. Schulte, S., Väisänen, A.J., 2006. Narrative shifts: Die Wahrnehmung von Erzählwechseln bei Sprachgesunden – eine Validierungsstudie. Unpublished Bachelor Thesis, Hoogeschool Zuid, NL. Scheperjans, F., Grefkes, C., Palomero-Gallagher, N., Schleicher, A., Zilles, K., 2005. Subdivisions of human parietal area 5 revealed by quantitative receptor autoradiography: a parietal region between motor, somatosensory, and cingulate cortical areas. NeuroImage 25, 975–992. Siebörger, F.Th., Ferstl, E.C., von Cramon, D.Y., 2007. Making sense of nonsense: an fMRI study of task induced inference processes during discourse comprehension. Brain Res. 1166, 77–91. Slotnick, S.D., Moo, L.R., Segal, J.B., Hart Jr, J., 2003. Distinct prefrontal cortex activity associated with item memory and source memory for visual shapes. Cogn. Brain Res. 17, 75–82. Speer, N.K., Zacks, J.M., Reynolds, J.R., 2007. Human brain activity time-locked to narrative event boundaries. Psychol. Sci. 18, 449–455. Tracy, F., Flanders, A., Saaussan, M., Natale, P., Del vecchio, N., Pyrros, A., et al., 2003. The brain's response to incidental intruded words during focal text processing. NeuroImage 18, 117–126. van Dijk, T.A., Kintsch, W., 1983. Strategies of Discourse Comprehension. Academic Press, New York. Virtue, S., Heberman, J., Clancy, Z., Parrish, T., Jung Beeman, M., 2006. Neural activity of inferences during story comprehension. Brain Res. 1084, 104–114. Vogt, B.A., Finch, D.M., Olson, C.R., 1992. Functional heterogeneity in cingulate cortex: the anterior executive and posterior evaluative regions. Cereb. Cortex 2, 435–443. Wagner, A.D., Shannon, B.J., Kahn, I., Buckner, R.L., 2005. Parietal lobe contributions to episodic memory retrieval. Trends Cogn. Sci. 9, 445–451. Xu, J., Kemeny, S., Park, G., Frattali, C., Braun, A., 2005. Language in context: emergent features of word, sentence, and narrative comprehension. NeuroImage 25, 1002–1015. Yarkoni, T., Speer, N.K., Balota, D.A., McAvoy, M.P., Zacks, J.M., 2008. Pictures of a thousand words: investigating the neural mechanisms of reading with extremely rapid event-related fMRI. NeuroImage 42, 973–987. Zacks, J.M., Braver, T.S., Sheridan, M.A., Donaldson, D.I., Snyder, A.Z., Ollinger, J.M., et al., 2001. Human brain activity time-locked to perceptual event boundaries. Nat. Neurosci. 4, 651–655. Zwaan, R.A., Radvansky, G.A., 1998. Situation models in language comprehension and memory. Psychol. Bull. 123, 162–185. Zwaan, R.A., Rapp, D.N., 2006. Discourse comprehension, In: Traxler, M.J., Gernsbacher, A.M. (Eds.), Handbook of Psycholinguistics, 2nd ed. Academic Press, San Diego, pp. 725–764. Zwaan, R.A., Langston, M.C., Graesser, A.C., 1995. The construction of situation models in narrative comprehension: an event-indexing model. Psychol. Sci. 6, 292–297.
Contents lists available at ScienceDirect
NeuroImage j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / y n i m g
Neural correlates of narrative shifts during auditory story comprehension Carin Whitney a,⁎, Walter Huber b, Juliane Klann b,c, Susanne Weis d, Sören Krach e, Tilo Kircher e a
Department of Psychology, University of York, York, YO10 5DD, UK Section Neurolinguistics, Department of Neurology, RWTH Aachen University, Germany c Central Service Facility “Functional Imaging” at the ICCR-BIOMAT, RWTH Aachen University, Germany d Section Clinical Neuropsychology, Department of Neurology, RWTH Aachen University, Germany e Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany b
a r t i c l e
i n f o
Article history: Received 29 October 2008 Revised 31 March 2009 Accepted 8 April 2009 Available online 17 April 2009
a b s t r a c t The ability to segment continuous linguistic information online into larger, meaningful units is a key element in narrative comprehension. Narrative shifts, i.e. transitions between individual units, are postulated to continuously update the mental situation model. Their cerebral correlates, however, have hardly been investigated. Under highly naturalistic conditions this study seeks to identify the neural correlates of implicit processing of narrative shifts during continuous speech comprehension. 16 male native German speakers listened passively to a German novella for 23 min while BOLD signal was recorded with fMRI. Text comprehension was tested in a short post-scan interview asking for critical episodes of the story. Narrative shifts were defined on the basis of a macropropositional analysis. Compared to listening to text passages of the narrative that neither contained narrative shifts nor structurally similar linguistic control events (i.e., sentence boundaries), narrative shifts evoked increased BOLD signal changes in the right temporal gyrus, precuneus and posterior/middle cingulate cortex bilaterally. When narrative shifts were contrasted with sentence boundaries, activation in the right precuneus and cingulate cortex remained significant. The results strengthen the relevance of medial parietal structures for natural language comprehension. More precisely, the precuneus and posterior cingulate appear to be the neural substrate for updating mental story representations and can be regarded as critical parts of a more complex, distributed neural network underlying story comprehension. © 2009 Elsevier Inc. All rights reserved.
Introduction The majority of psycholinguistic neuroimaging studies are designed to investigate a single, highly specific component of language functioning that is isolated from other confounding processes, which might co-occur in more naturalistic conditions. Such paradigms reduce the ecological validity of the experiment, and it remains debatable to which degree the investigation of isolated processes contributes to the understanding of the same process in a more complex environment. It is also questionable in how far this approach is suitable for the investigation of implicit variables underlying complex linguistic performance such as text comprehension, which requires a more naturalistic setting as can be offered with traditional fMRI designs. In the last decade, a different approach has been introduced by presenting or recording continuous, more natural linguistic data during fMRI. In the first of these studies, recurrent expressive linguistic events within the continuous speech output during spontaneous picture description (e.g., hesitation pauses or segments
⁎ Corresponding author. Fax: +44 1904 433181. E-mail address: [email protected] (C. Whitney). 1053-8119/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.neuroimage.2009.04.037
of graded complexity of linguistic encoding) were determined posthoc for individual speakers and taken as input for subsequent eventrelated analyses (Kircher et al., 2000, 2004). A similar approach was used to study continuous movie and story perception, e.g. during video-clip viewing of everyday activities (Zacks et al., 2001), watching of commercial movies (Bartels and Zeki, 2004; Golland et al., 2007; Hasson et al., 2004) or story comprehension in children (Karunananayaka et al., 2007; Schmithorst et al., 2006). However, most studies that utilized continuous fMRI recording concentrated on a corroboration of standard fMRI findings with the application of this novel technique or with newly developed fMRI data analysis tools such as independent component analysis (ICA) (e.g., Karunananayaka et al., 2007; Schmithorst et al., 2006). For example, Zacks et al. (2001) applied post-hoc event-related analyses in order to explore event boundaries during active and passive video-clip viewing in a relatively naturalistic task. Participants were asked to segment video clips of everyday activities into natural and meaningful units by pressing a handheld button during fMRI scanning (active viewing condition). The resulting temporal event structure of each participant was taken to analyze the continuous fMRI signal under the active viewing condition and a passive version, during which no unit boundaries were marked. Remarkably, similar neural activations were found for both conditions in a bilateral
C. Whitney et al. / NeuroImage 47 (2009) 360–366
posterior network (precuneus, cuneus, inferior and superior temporal sulci, fusiform gyrus) signalling an implicit parsing capacity during which a temporally evolving environment was automatically junked into meaningful units. It should be noted, however, that the assumed capacity of parsing was purely operationally derived in this study. It remained unclear to what extent parsing relied on external visual cues or on underlying processes of semantic interpretation. Furthermore, one might object that these attempts of studying naturalistic comprehension lack theoretically based predictions on the cognitive and/or linguistic nature of the cues that trigger such common, timelocked human brain activity. Semantic theories of online story comprehension may provide a framework in which boundaries between natural and meaningful units can be specified. In particular, the notion of “narrative shift” appears to be promising. Most text comprehension models agree that narratives are represented as mental models of the situations being described in the text (e.g., ‘event-indexing model’ Zwaan and Radvansky, 1998; ‘mental model’ Johnson-Laird, 1983; ‘situation model’ van Dijk and Kintsch, 1983; ‘structure building framework’ Gernsbacher, 1997; for a review see Mar, 2004). Individuals constantly attempt to integrate the incoming linguistic information into a propositional text base which is then integrated with their world knowledge into a situation model. Development and changes in such a model occur when narrative units or ‘events’ fail to overlap in one or more of their constituent features such as character, time, action or location (e.g., Johnson-Laird, 1983; Rich and Taylor, 2000; Zwaan et al., 1995). Macropropositions form hereby the basis for such narrative events, which abstract from the text base and represent the ‘gist of the text’ (Kintsch and van Dijk, 1978) or ‘fact units’ (Huber, 1990). They are confined by knowledge or schemata (Schank and Abelson, 1977) of the described situation, on the basis of which irrelevant or redundant textual information is deleted, a sequence of individual propositions is generalized, or new propositions are constructed by means of inference. The same rules can be iteratively applied to the resulting macrostructure, creating a mental representation of the narrative that contains the most relevant (and most memorable) information of a respective text unit (Kintsch and van Dijk, 1978). Given this hierarchical organization, narrative shifts can overlap with unit boundaries on the micropropositional (i.e., predicate–argument structure) and surface level, but not every unit boundary on the lower level corresponds to a narrative shift. It also follows that narrative shifts can be identified at different levels of a macropropositional analysis, dependent on the iteration of the macro-rules. Therefore, narrative shifts depict observer-independent, time-locked events during text comprehension and do neither occur during the perception of coherent text passages nor at lower-level unit boundaries in general (e.g., at the end of a sentence). The cognitive processes operating at content shifts seem to overlap for different types of narrative units (i.e., micro- or macropropositions). Independent of whether local or global inconsistencies are detected, the reader/listener fails to integrate the input with previous information and needs to update the mental representation of the evolving cognitive model by adding new entries or replacing old references with current links to spatial, temporal or personal dimensions (Schulte and Vaisänen, 2006; Zwaan and Radvansky, 1998). Narrative shifts or updates, therefore, correspond to an enhanced integration effort (i.e., of current linguistic input) and a substitution of information in working memory. They are further characterized by attentional shifts between different dimensions of discourse representation (Zwaan and Radvansky, 1998) and increased processing costs in drawing inferences (for a review see Graesser et al., 1994; Virtue et al., 2006). The demand on working memory, inference and integration processes is likely to vary with the number of situational dimension that can be mapped onto previous stages as indicated by increased reading times and low coherence ratings scores (Zwaan and Radvansky, 1998). Text comprehension can thus be
361
characterized as a constant search for coherence, which becomes extremely difficult at narrative unit boundaries. Neuroimaging studies investigating coherence breaks both, on the sentence level (Ferstl and von Cramon, 2001, 2002; Kuperberg et al., 2006) or within larger narrative structures (Maguire et al., 1999; Martin-Loeches et al., 2008; Speer et al., 2007; Xu et al., 2005) have identified similar activation patterns. Enhanced BOLD responses were consistently reported in medial parietal structures including the precuneus and/or posterior cingulate gyrus (but cf. Sieboerger et al., 2007) and were accompanied by various study-specific signal increases in the medial frontal, temporal, lateral frontal or parietal cortices. It appears to be the medial parietal brain regions that define the coherence network and make it distinct from the neural correlate of text comprehension in general (for a meta-analysis see also Ferstl et al., 2008). Further, the close resemblance to the mental imagery network (for a review see Cavanna and Trimble, 2006) strengthens the role of the precuneus/posterior cingulate gyrus in situation model building. Unlike coherent text passage, narrative shifts provoke a partial renewal of the mental model, which increases the demand on processes related to mental imagery, such as attentive tracking and switching across dimensions (e.g., time, space). One could argue that the medial parietal BOLD enhancements in the study by Zacks et al. (2001) might have also been a result of similar higher-level semantic shifts that have triggered the segmentation process during video-clip viewing. Although the authors have associated unit boundaries with individual segments in the cognitive representation of the everyday activity, details about the event structure (e.g., on the micro- or macropropositional level) and its segmentation were missing to confirm this assumption. In sum, the neuroimaging findings imply that human brain activity in medial parietal structures is intrinsically time-locked to perceptive narrative event boundaries irrespective of the level of segmentation (micro- vs. macrostructure). Hence, narrative shifts are a suitable candidate to postulate critical time points of receptive speech for a continuous fMRI recording. The objective of the present study was to uncover the neural correlates of implicit processing of narrative shifts during passive, natural auditory story comprehension. So far it remains unclear whether the medial parietal network responds to narrative shifts that occur during natural language comprehension of larger narrative structures and in how far previous findings generalize to the auditory modality. The task required listening to a 23-minute story without any prior knowledge of story content or its event structure and the purpose of the study. Narrative shifts were defined on the basis of macropropositional analysis beforehand (Huber, 1990; van Dijk and Kintsch, 1983) and entered into an event-related fMRI analysis. We hypothesized that these theory-based, reliably defined transitions between linguistic units would elicit activation in a medial posterior network that has previously been related to coherence breaks at different levels of text representation. In contrast, BOLD responses at randomly chosen lower level boundaries (i.e., at the end of a sentence) occurring within coherent text passages should fail to activate these functionally specific structures. Materials and methods Participants Initially, 19 male subjects took part in the fMRI study. Due to head movement, data of three subjects had to be discarded from further analysis. All remaining 16 participants (mean age = 27.00 years, SD = 6.65; mean years of education = 14.50 years, SD = 1.67) were native German speakers, right-handed according to the Edinburgh Inventory of Handedness (Oldfield, 1971) and showed average or above average verbal IQ as assessed by the German MWT-B multiplechoice vocabulary test (Lehrl et al., 1995) (mean estimated verbal IQ = 120.06, SD = 17.16). Subjects with recent substance use or
362
C. Whitney et al. / NeuroImage 47 (2009) 360–366
general MRI incompatibility (e.g., metal implants) were excluded. All subjects gave informed consent and were paid 20 Euros for participation in the study. The study was approved by the local ethics committee.
recall was required. Answers were classified as “correct” or “incorrect”, accordingly. No time restrictions were given.
Stimuli
Narrative shifts were defined on the basis of a macropropositional analysis (Huber 1990; van Dijk and Kintsch 1983). In theories about discourse processing, macropropositions are specified as basic, narrative building blocks or events “constrained by referential continuity and general plausibility” (p. 156, Huber, 1990). The key features describing a narrative event are the character(s), time and location (Rich and Taylor, 2000) as well as the character's goals and their interaction with objects (Speer et al., 2007). Any change in the continuity of these features (i.e., the introduction of a new, relevant character; a shift of time, location, goal or interaction) indicates a transfer from one macroproposition to the next. Psychologically, such changes in continuity result in an update of the mental situation model. The following definitions of the four types of narrative shifts were applied to our text (Rich and Taylor, 2000; Schulte and Väisänen, 2006): (a) a shift in person is apparent whenever a new, relevant character is introduced or the focus switches from one to another, already known, character, (b) a shift in time is a jump forward in time, usually indicated by specific linguistic markers (e.g. the next day, 5 years later), ranging from hours to years in our story, (c) a shift in location is defined as a relocation of the narrative action from one place to another, (d) a shift in action is present whenever a new action becomes the central part of an event despite continuity in person, time or location. According to the criteria defined for person, time, location and action shifts, a group of four experts (WH, JK, TK, CW) marked narrative shifts on a written version of the novella during listening to its recording. The story was familiar to all experts beforehand. Only those narrative shifts were included into the analysis which were identified by the macropropositional analysis and marked by all four experts. Table 1 provides an overview of the novella's complete macropropositional structure and its narrative shifts. All but one narrative shift (M8) were identified at sentence boundaries, i.e. at the beginning of the sentence that introduced a change in one or more of the four dimensions. Three of the theoretically defined narrative shifts (M1, M11, M17) were not modelled in the SPM analysis: M1 and M11 were situated at the beginning of each run, i.e. part 1 and part 2 of the story. M17 was only identified by three out of four experts. Mean distance between two successive narrative shifts was 85.33 s (SD = 51.41 s). To verify whether brain activation related to narrative shifts was indeed specific to such narrative shifts and could not be attributed to sentence boundaries in general, we selected an equal number of events at sentence boundaries that were distinct from the narrative shifts. 15 sentence boundaries were chosen such that they lay approximately between two shifts or between the beginning of a session and its first corresponding shift.
A slightly modified version of the German novella “Der Kuli Kimgun” by Max Dauthendey (1909) was chosen for this study. The modifications concerned low-frequent or foreign words which were substituted by more familiar or high-frequent words. Furthermore, emotionally highly arousing text passages were replaced by less exciting material. The final version included a total of 3581 words. In general, novellas are short, well-structured narratives restricted to a few protagonists and basic narrative events which, together, are ascribed to a single, central conflict. They are written from an omniscient perspective, leaving out elaborate descriptions of the character's emotional or mental states. In the chosen novella, the sequence of events occurs chronologically, which allows the listener to build up a temporally continuous, mental representation of the story. Further details of the narrative structure are presented in the section on the narrative shift analysis. For auditory presentation during fMRI, the story was professionally recorded and spoken in a natural way by a trained, male speech therapist. The duration of the story was 23:32 min. Procedure The story was presented via MRI compatible headphones in two successive runs lasting 14:32 and 9:00 min, respectively. Participants were instructed to close their eyes and listen to the story carefully. Behavioural analysis To make sure that subjects attended to the content, they were informed at the beginning of the experiment about a short interview after the MRI session about the content of the story. Each of the ten questions, asked for in chronological order, required the recall of a critical event and referred to a specific episode/macroproposition as identified by the macropropositional analysis and listed in Table 1 (M1, 2, 4, 6, 9, 10, 12, 13, 15 and 17). Subjects succeeded whenever the essence of the respective event was captured, i.e. no overly detailed Table 1 Linguistic parameters of the stimulus novella (Dauthendey, 1909). Onset time (scans) M1a
3
Episode/macroproposition
Narrative shift
Ali living in his home town
Beginning of part 1 Location Action Location Location Action Action Character Location Action Beginning of part 2 Character Location Action Character Action Character Character
M2 M3 M4 M5 M6 M7 M8 M9 M10 M11a
55 75 97 196 218 257 327 349 400 3
News from the sunken bell Ali on his way to Rangoon Ali in the temple Ali in Rangoon Cleaning of the bell Procession Girl Festival Buying of bananas Ali and his wife living near the beach
M12 M13 M14 M15 M16 M17a M18
36 72 158 169 195 216 246
Birth of Ali's son and death of his wife Baby in the temple Ali on his way to the temple Tiger in the temple Ali's hopelessly attempted rescue Tiger's attack of the baby Shooting of the tiger
M = macroproposition. a Not modelled in the SPM analysis.
Narrative shift analysis
fMRI data acquisition All scanning was performed on an 1.5 T scanner (Gyroscan Intera, Philips Medical, Eindhoven, The Netherlands) using standard gradients and a circular polarized phase array head coil. For each subject, we acquired two series of functional volumes of T2⁎-weighted axial EPI-scans parallel to the AC/PC line with the following parameters: number of slices (NS), 22; slice thickness (ST), 5.0 mm; interslice gap (IG), 0.55 mm; matrix size (MS), 64 × 64; field of view (FOV), 240 × 240 mm; echo time (TE), 50 ms; repetition time (TR), 2.0 s. 436 functional volumes were acquired for the first part of the story and 270 functional volumes for the second part, adding up to 706 volumes in total.
C. Whitney et al. / NeuroImage 47 (2009) 360–366
fMRI data analysis MR images were analyzed using Statistical Parametric Mapping software (SPM2; www.fil.ion.ucl.ac.uk) implemented in MATLAB 7.0 (Mathworks Inc., Sherborn, MA). After discarding the first three volumes, all images were realigned to the first image to correct for head movement. Unwarping was used to correct for the interaction of susceptibility artifacts and head movement. After realignment and unwarping, the signal measured in each slice was shifted relative to the acquisition time of the middle slice using a sinc interpolation in time to correct for their different acquisition times. Volumes were then normalized into standard stereotaxic anatomical MNI-space by using the transformation matrix calculated from the first EPI-scan of each subject and the EPI-template. Afterwards, the normalized data with a resliced voxel size of 4 × 4 × 4 mm were smoothed with an 8 mm FWHM isotropic Gaussian kernel to accommodate intersubject variation in brain anatomy. The time series data were filtered with a high-pass cut-off of 1/128 Hz. The autocorrelation of the data was estimated and corrected for. The expected hemodynamic response at stimulus onset for each event-type, narrative shifts and random sentence boundaries, was modeled by two response functions, a canonical hemodynamic response function (HRF; Friston et al., 1998) and its temporal derivative. The latter was included in the model to account for the residual variance resulting from small temporal differences in the onset of the hemodynamic response, which is not explained by the canonical HRF alone. The functions were convolved with the eventtrain of stimulus onsets to create covariates in a general linear model. Subsequently, parameter estimates of the HRF regressor for each of the two different conditions were calculated from the least mean squares fit of the model to the time series. Parameter estimates for the temporal derivative were not further considered in any contrast. Using SPM5, first level contrasts were computed for narrative shifts and random sentence boundaries. These contrasts were entered into a flexible factorial ANOVA on the group level in order to compute the individual contrast for each condition (narrative shifts, random sentence boundaries) and the respective differential contrasts. Since we did not include a resting baseline (null events) in our experimental design (which would have been unnatural), the term baseline refers to all text passages that were not explicitly modeled (i.e., excluding the 15 narrative shifts and 15 random sentence boundaries). Brain areas reported during listening to narrative shifts vs. baseline or random sentence boundaries vs. baseline thus show an increase of brain activity at these specific time points as compared to the baseline
363
activation of listening to the narration. Differential contrasts were inclusively masked by the minuend at p b .001 (uncorrected). The results were computed on a voxel-wise threshold of p b 0.001. Subsequently, a. Monte Carlo simulation of the brain volume of the current study was conducted to establish an appropriate voxel contiguity threshold (Slotnick et al., 2003). The basic principle of the Monte Carlo approach involves the simulation of brain activity whereby each individual voxel of the functional image matrix becomes activated with a set type I error probability (i.e., 0.001). The resulting activation map is smoothed with a FWHM Gaussian kernel (i.e., 8 mm) and the size of each contiguous voxel cluster is determined. After multiple, consecutive simulations (i.e., 10,000) the extent threshold k is chosen such that the relative frequency of clusters of size k or greater in the sample is smaller than the desired corrected p-value of p b 0.5. Assuming an individual voxel type I error of p b 0.001 in our study, a cluster extent of 13 contiguous resampled voxels was indicated as necessary to correct for multiple voxel comparisons at p b 0.05. Anatomical labels for activated brain areas were computed using the SPM anatomy toolbox, which is based on anatomical probability maps of the brain reflecting the intersubject variability of selected brain areas in terms of location and extend (Eickhoff et al., 2007). A detailed anatomical map was available for the superior parietal cortex (BA 5; Scheperjans et al., 2005). The remaining areas were labeled on the macroanatomical level. Results Behavioural results During the post-scan interview about critical story episodes, all participants were able to recall the desired macropropositional detail in response to each of the ten questions. Answers to all questions were provided quickly and effortlessly. fMRI results Listening to narrative shifts compared to baseline (i.e., listening to text passages that neither contained narrative shifts nor the 15 randomly selected sentence boundaries) revealed stronger brain activation in the right middle and superior temporal gyrus, the right precuneus and a voxel cluster spanning the middle cingulate cortex, adjacent posterior cingulate cortex and precuneus bilaterally (see Table 2, Fig. 1). In contrast, no brain region was significantly stronger
Table 2 Brain activation related to narrative shifts and to random sentence boundaries during auditory story comprehension in a group of 16 healthy participants. Cl Narrative shifts N baseline
1 2 3
Random sentence boundaries N baseline Narrative shifts N random sentence boundaries
Random sentence boundaries N narrative shifts
H R R R R L L/R R L No region of
Anatomical region Middle temporal gyrus Superior temporal gyrus Precuneus Posterior cingulate cortex Precuneus Middle cingulate cortex Precuneus (BA 5) Posterior cingulate cortex statistical significance
4
R
Precuneus
5 6
L Middle cingulate cortex R Posterior cingulate cortex No region of statistical significance
Z
No. voxels/mm3
0 12 44 28 40 40 52 20
5.20 4.88 5.08 4.38 4.09 3.83 3.59 3.49
50/3200
− 72
44
4.76
30/1920
− 44 − 32
36 28
4.63 4.00
15/960 16/1024
Coordinates x
y
z
52 60 12 4 − 12 0 4 −4
− 44 − 44 − 72 − 32 − 48 − 40 − 44 − 44
8 −8 4
53/3392 71/4544
Cl = Cluster, H = hemisphere, L = left, R = right. If a voxel cluster comprises two or more distinct brain areas, all are listed under the same cluster. The significance level is given in Zvalues and cluster size in number of voxels and volume (mm3) for p b .05 (corrected) with a cluster extent of 13 voxel.
364
C. Whitney et al. / NeuroImage 47 (2009) 360–366
Fig. 1. Stronger brain activation during the comprehension of narrative shifts compared to baseline or pseudo-randomly chosen sentence boundaries projected onto selected brain sections of the standard SPM template. Note: Activation is corrected for multiple comparisons at p b .05, extent threshold = 13 voxels.
activated during the comprehension of pseudo-randomly chosen sentence boundaries compared to baseline. Direct comparison between narrative shifts and random sentence boundaries resulted in stronger BOLD responses for narrative shifts in three distinct voxel clusters comprising the right precuneus, the left middle cingulate cortex and the right posterior cingulate cortex (see Table 2, Fig. 1). The reverse contrast did not reveal any significant results. Discussion In the present study we demonstrated BOLD enhancements in the precuneus, right temporal gyrus, posterior and middle cingulate cortex for narrative shifts while subjects listened passively to a continuous story. Because most of these shifts occurred at the beginning/end of a sentence, we additionally compared the narrative shift-specific brain activity to BOLD signal changes during the comprehension of sentence boundaries that were distinct from narrative shifts. Again, stronger activation was observed in the right precuneus and posterior/middle cingulate structures bilaterally for narrative shifts compared to the control events, which strengthens the relevance of extra-perisylvian, midline structures for text-level processing. We assume that the underlying cognitive mechanisms during the narrative segmentation process equal an online update of the situation model during continuous story comprehension. Besides increased semantic integration effort, memory-based processes seem to be of particular importance. These include the encoding and maintenance of current knowledge and re-access of previous contents of the situation model. On the neural level we suggest that this complex, highly specific behaviour is reflected in BOLD enhancements in a posterior midline network whereby each neural component (posterior cingulate cortex, precuneus) subserves different aspects of narrative update.
Narrative update: the role of the posterior cingulate–precuneus network Neuroimaging studies of story and sentence comprehension have associated precuneus and posterior cingulate activity with memory retrieval, narrative integration effort and the establishment of coherence across sentences, all being key components in the construction and update of situation models (Almor et al., 2007; Ferstl and von Cramon, 2001, 2002, 2007; Fletcher et al., 1995; Kircher et al., 2001; Obleser et al., 2007; Speer et al., 2007; Tracy et al., 2003; Xu et al., 2005). For example, Ferstl et al. (2005) related bilateral activations of the intraparietal sulcus (IPS) and the medial parietal cortex (precuneus) to memory related processes of attention shifts during the recognition of narrative coherence breaks. Switching from local inputs to global contextual aspects seems to be central when previous information has to be recalled during the updating of situation models. Likewise, Maguire et al. (1999) and Martin-Loeches et al. (2008) observed distinct activations of the posterior cingulate cortex and the precuneus when the global theme had to be extracted from unusual stories by means of prior knowledge. Similar to Ferstl et al. (2005), the observed parietal activation was related to updating old with new information in the mental model, which was guiding story comprehension. The majority of the research reporting posterior cingulate and precuneus activation, however, is based on visuo-spatial experiments investigating mental imagery. Mental imagery, again, has been regarded as an essential process during language comprehension (Zwaan and Rapp, 2006; see also dual-coding model, Pavio, 1986) involving functions which can easily be transferred to the construction of situation models (e.g., orientating within space and time, attentive tracking and shifts). Accordingly, processes ascribed to the precuneus subsume attentive tracking and shifting, sequencing of actions and episodic memory retrieval (for a review see Cavanna and Trimble, 2006). It has also been suggested that the precuneus acts as
C. Whitney et al. / NeuroImage 47 (2009) 360–366
a part of a larger working memory buffer that contains information readily available for current decision-making processes (for a review see Wagner et al., 2005). Likewise, the posterior cingulate cortex has been implicated in this parietal working memory network (Wagner et al., 2005) but seems to be further important for evaluating the environment being perceived, i.e. in orientating within the environment and interpreting it (for a review see Vogt et al., 1992). Relating this idea to story comprehension, the posterior cingulate cortex might be involved in evaluating the linguistic input in order to accept or reject its integration with prior knowledge (e.g., Maguire et al., 1999; Mar, 2004; Martin-Loeches et al., 2008; Xu et al., 2005). This knowledge is supposed to be maintained in working memory as reflected by precuneus activity. Thus, the precuneus is likely to reflect attentional and memory components during narrative updates, which intrinsically complement the listener/reader's attempt to evaluate and integrate ongoing linguistic information, a process which is presumably subserved by the posterior cingulate cortex. Similar to previous investigations (Speer et al., 2007, Zacks et al., 2001), we have identified the neural basis of content shifts without further differentiations between, for example, spatial, temporal or personal changes. Collapsing across the different types of narrative shifts, however, might underestimate the number of brain regions involved in the update process. Different types of narrative shifts are most likely to evoke activation in different, content-specific brain structures (see Ferstl et al., 2005). The current activation pattern is most similar to the activation observed for sentence pairs that focussed on spatial information in Ferstl and von Cramon (2007) and for short stories that involved chronological changes in Ferstl et al. (2005) (i.e., precuneus and/or posterior cingulate cortex). Since we neither identified any shifts in time nor a dominance of spatial shifts in the current study (cf. Table 1: character (n = 5), space (n = 5) and action shifts (n = 6)), it seems unlikely that the medial parietal activation can be attributed to a specific type of shift processing. However, we do not dismiss the alternative interpretation that spatial and temporal aspects might also play a role in the action shifts detected in our study and that we might have detected a narrative shift network that predominantly resembles content changes along these dimensions. Identifying the full extent of such shift-specific networks during natural language comprehension needs to be investigated in future studies. Sentence boundaries In line with our predictions, brain activation in the precuneus and posterior cingulate cortex was significantly stronger during the comprehension of narrative shifts than sentence boundaries. Sentence boundaries were chosen such that they lay within a coherent text episode, so that no update process was required and, hence, no medial parietal brain activation detected. Further, failing to detect any significant BOLD increases during such control events was not unexpected. The implicit baseline condition, to which random sentence boundaries were compared to, also contained numerous events that described sentence boundaries. Further implications and limitations The current investigation was designed to tap specific intrinsic speech comprehension processes, i.e. the automatic segmentation of ongoing speech input into smaller, meaningful units under naturalistic conditions. The novella, which remained unchanged on its macropropositional level, revealed 15 reliable events that described boundaries between episodes in the narration. Although former studies were able to obtain larger stimulus sets by focussing on smaller content units (e.g., Ferstl and von Cramon, 2002; Speer et al., 2007; Zacks et al., 2001), brain activation overlapped in medial
365
parietal structures. Further, the strength of the neural response in the precuneus and posterior cingulate cortex was comparable to text or sentence comprehension studies with more events (Ferstl and von Cramon, 2001, 2002, 2007; Ferstl et al., 2005; Obleser et al., 2007; Siebörger et al., 2007; Speer et al., 2007; Virtue et al., 2006), and blocked designs (Fletcher et al., 1995; Lindenberg and Scheef, 2007; Maguire et al., 1999; Martin-Loeches et al., 2008; Mazoyer et al., 1993; Ozawa et al., 2000; Xu et al., 2005). Both, the strength of activation and the overlap in location suggest that situation model update as it occurs during natural text comprehension in the present investigation can be attributed to a distinct medial parietal network. It appears to mediate core functions of higher-level comprehension associated with changes in situational dimensions (e.g., character, location) irrespective of the degree of segmentation process being investigated (fine vs. coarse), stimulus manipulation/presentation (natural vs. artificial) or modality (cf., for text reading see Speer et al., 2007; for passive viewing of movie clips see Zacks et al., 2001). As outlined above, this network might not be exhaustive. This is primarily due to the limited control over confounding variables (e.g., the diversity of shifts being included), which is inevitable to more naturalistic designs, but should be less attributed to the restricted number of events used in the current study. It remains arguable though, in how far the identified network generalizes to other novellas. Although the event boundaries were derived from a single story (see also Speer et al., 2007), the set of narrative shifts under investigation was not homogeneous. Not only did the events describe inconsistencies in character, action or place, but they were also likely to differ with respect to, for example, the visuo-spatial mental image being produced during situation model building (cf. Zwaan and Rapp, 2006) or the resources demanded to detect different kinds of narrative shifts (cf. Ferstl and von Cramon, 2007). Further, previous observations indicate that activation in the precuneus/posterior cortex is very robust and independent of both, intra-stimulus (e.g., type of shift) and inter-stimulus/study (e.g., modality) manipulations (cf. Speer et al., 2007; Zacks et al., 2001). These aspects, again, argue in favour of a key role of the medial parietal network in narrative shift comprehension across different linguistic stimuli including different kinds of novellas. Finally, unlike previous studies, we particularly refrained from online tasks controlling for the subject's attention towards narrative shifts such as plausibility judgements or the explicit marking of unit boundaries during comprehension to maintain a high ecological validity (see also Speer et al., 2007). Since all listeners were able to recall the relevant details of the macropropositions given in the novella during the post-scan interview, we can be sure that each participant built up a mental representation of the story. Since Zacks et al. (2001) and Speer et al. (2007) have shown that the parsing of continuous input into meaningful units occurs automatically, we can also be confident that all of the subjects comprehended narrative shifts. Ceiling performance (with individuals and groups scoring up to 100% correct), as defined over the number of questions answered correctly vs. incorrectly, is commonly observed during post-scan interviews (Fletcher et al., 1995; Virtue et al., 2006; Xu et al., 2005; Yarkoni et al., 2008). We chose open questions and, hence, reduced the probability of ‘false positives’, i.e. participants that did not listen to the story but have not been identified as such by the test. If a participant was indeed not paying attention, our design would have successfully recognized this behaviour, while tests on the basis of true/false questions (e.g., Virtue et al., 2006) or multiple-choice approaches (e.g., Schmithorst et al., 2006; Tracy et al., 2003; Xu et al., 2005; Yarkoni et al., 2008) might have failed. Concluding comments In this study, continuous passive speech comprehension has been investigated under highly naturalistic conditions. The results
366
C. Whitney et al. / NeuroImage 47 (2009) 360–366
strengthen the relevance of medial parietal brain regions for narrative comprehension. They further demonstrate that the precuneus and posterior/middle cingulate cortex were involved in the segmentation of continuous speech input into smaller, meaningful events. This key component of story comprehension appeared to be implicit (i.e., listeners perform this process spontaneously and automatically without explicit instructions to do so) and independent of presentation modality (cf. visual stimulation: Speer et al., 2007; Zacks et al., 2001). Overall, the cingulate–precuneus network should be considered as part of a more complex neural system subserving narrative comprehension. Acknowledgments The study was supported by a grant from the Interdisciplinary Centre for Clinical Research “BIOMAT”. within the Faculty of Medicine at the RWTH Aachen University (IZKF VV N68) and the International Research Training Group 1328 supported by the German Research Foundation. References Almor, A., Smith, D.V., Bonilha, L., Fridriksson, J., Rorden, C., 2007. What is in a name? Spatial brain circuits are used to track discourse references. NeuroReport 18, 1215–1219. Bartels, A., Zeki, S., 2004. Functional brain mapping during free viewing of natural scenes. Hum. Brain Mapp. 21, 75–85. Cavanna, A.E., Trimble, M.R., 2006. The precuneus: a review of its functional anatomy and behavioural correlates. Brain 129, 564–583. Dauthendey, M., 1909. Lingam – Zwölf asiatische Novellen. Albert Langen Verlag, München. Eickhoff, S.B., Paus, T., Caspers, S., Grosbras, M-H., Evans, A.C., Zilles, K., Amunts, K., 2007. Assignment of functional activations to probabilistic cytoarchitectonic areas revisited. NeuroImage 36, 511–521. Ferstl, E.C., von Cramon, D.Y., 2001. The role of coherence and cohesion in text comprehension: an event-related fMRI study. Cogn. Brain Res. 11, 325–340. Ferstl, E.C., von Cramon, D.Y., 2002. What does the frontomedian cortex contribute to language processing: coherence or theory of mind? NeuroImage 17, 1599–1612. Ferstl, E., Rinck, M., von Cramon, D.Y., 2005. Emotional and temporal aspects of situation model processing during text comprehension: an event-related fMRI study. J. Cogn. Neurosci. 17, 724–739. Ferstl, E.C., von Cramon, D.Y., 2007. Time, space and emotion: fMRI reveals contentspecific activation during text comprehension. Neurosci. Lett. 427, 159–164. Ferstl, E.C., Neumann, J., Bogler, C., von Cramon, D.Y., 2008. The extended language network: a meta-analysis of neuroimaging studies on text comprehension. Hum. Brain Mapp. 29, 581–593. Fletcher, P.C., Happé, F., Frith, U., Baker, S.C., Dolan, R.J., Frackowiak, R.S.J., et al., 1995. Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension. Cognition 57, 109–128. Friston, K.J., Fletcher, P., Josephs, O., Holmes, A., Rugg, M.D., Turner, R., 1998. Eventrelated fMRI: characterizing differential responses. Neuroimage 7, 30–40. Gernsbacher, M.A., 1997. Two decades of structure building. Discourse Processes 23, 265–304. Golland, Y., Bentin, S., Gelbard, H., Benjamini, Y., Hella, R., Nir, Y., et al., 2007. Extrinsic and intrinsic systems in the posterior cortex of the human brain revealed during natural sensory simulation. Cereb. Cortex 17, 177–1766. Graesser, A.C., Singer, M., Trabasso, T., 1994. Constructing inferences during narrative text comprehension. Psychol. Rev. 101, 371–395. Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., Malach, R., 2004. Intersubject synchronization of cortical activity during natural vision. Science 303, 1634–1640. Huber, W., 1990. Text comprehension and production in aphasia: analysis in terms of micro- and macromapping. In: Joanette, Y., Brownell, H.H. (Eds.), Discourse Ability and Brain Damage. Springer Verlag, pp. 154–179. Johnson-Laird, P.N., 1983. Mental Models. Harvard University Press, Cambridge, MA. Karunananyaka, P.R., Holland, S.K., Schmithorst, J., Solodkin, A., Chen, E.E., Szaflarski, J.P., et al., 2007. Age-related connectivity changes in fMRI data from children listening to stories. NeuroImage 34, 349–360. Kintsch, W., van Dijk, T.A., 1978. Towards a model of text comprehension and production. Psychol. Rev. 85, 363–394. Kircher, T.T.J., Brammer, M.J., Levelt, W., Bartels, M., McGuire, P., 2004. Pausing for thought: activation of left temporal cortex during pauses in speech. NeuroImage 21, 84–90. Kircher, T.T., Brammer, M., Tous Andreu, N., Williams, S.C., McGuire, P.K., 2001. Engagement of right temporal cortex during processing of linguistic context. Neuropsychologia 39, 798–809.
Kircher, T.T.J., Brammer, M.J., Williams, S.C.R., McGuire, P.K., 2000. Lexical retrieval during fluent speech production: an fMRI study. NeuroReport 11, 4093–4096. Kuperberg, G.R., Lakshmanan, B.M., Caplan, D., Holcomb, P.J., 2006. Making sense of discourse: an fMRI study of causal inferencing across sentences. NeuroImage 33, 343–361. Lehrl, S., Triebig, G., Fischer, B., 1995. Multiple choice vocabulary test MWT as a valid and short test to estimate premorbid intelligence. Acta Neurol. Scand. 91, 335–345. Lindenberg, R., Scheef, L., 2007. Supramodal language comprehension: role of the left temporal lobe for listening and reading. Neuropsychologia 45, 2407–2415. Maguire, E.A., Frith, C.D., Morris, R.G.M., 1999. The functional neuroanatomy of comprehension and memory: the importance of prior knowledge. Brain 122, 1839–1850. Mar, R.A., 2004. The neuropsychology of narrative: story comprehension, story production and their interrelation. Neuropsychologia 42, 1414–1434. Martin-Loeches, M., Casado, P., Hernandez-Tamames, J.A., Alvarez-Linera, J., 2008. Brain activation in discourse comprehension: a 3 T fMRI study. NeuroImage 41, 614–622. Mazoyer, B.M., Dehaene, S., Tzourio, N., Murayama, N., Cohen, L., Levrier, O., Salamon, G., Syrota, A., Mehler, J., 1993. The cortical representation of speech. J. Cogn. Neurosci. 5, 467–479. Obleser, J., Wise, R.J.S., Dresner, M.A., Scott, S.K., 2007. Functional integration across brain regions improves speech perception under adverse listening conditions. J. Neurosci. 27, 2283–2289. Oldfield, R.C., 1971. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9, 97–113. Ozawa, F., Matsuo, K., Kato, C., Nakai, T., Isoda, H., Takehara, Y., Moriya, T., Sakahara, H., 2000. The effects of listening comprehension of various genres of literature on response in the linguistic area: an fMRI study. NeuroReport 11, 1141–1143. Pavio, A., 1986. Mental Representations: A Dual Coding Approach. Oxford University Press, Oxford, UK. Rich, S.S., Taylor, H.A., 2000. Not all narrative shifts function equally. Mem. Cognit. 28, 1257–1266. Schank, R.C., Abelson, R., 1977. Scripts, Plans, Goals and Understanding: An Inquiry Into Human Knowledge Structures. Erlbaum, Hillsdale. Schmithorst, J., Holland, S.K., Plante, E., 2006. Cognitive modules utilized for narrative comprehension in children: a functional magnetic resonance imaging study. NeuroImage 29, 254–266. Schulte, S., Väisänen, A.J., 2006. Narrative shifts: Die Wahrnehmung von Erzählwechseln bei Sprachgesunden – eine Validierungsstudie. Unpublished Bachelor Thesis, Hoogeschool Zuid, NL. Scheperjans, F., Grefkes, C., Palomero-Gallagher, N., Schleicher, A., Zilles, K., 2005. Subdivisions of human parietal area 5 revealed by quantitative receptor autoradiography: a parietal region between motor, somatosensory, and cingulate cortical areas. NeuroImage 25, 975–992. Siebörger, F.Th., Ferstl, E.C., von Cramon, D.Y., 2007. Making sense of nonsense: an fMRI study of task induced inference processes during discourse comprehension. Brain Res. 1166, 77–91. Slotnick, S.D., Moo, L.R., Segal, J.B., Hart Jr, J., 2003. Distinct prefrontal cortex activity associated with item memory and source memory for visual shapes. Cogn. Brain Res. 17, 75–82. Speer, N.K., Zacks, J.M., Reynolds, J.R., 2007. Human brain activity time-locked to narrative event boundaries. Psychol. Sci. 18, 449–455. Tracy, F., Flanders, A., Saaussan, M., Natale, P., Del vecchio, N., Pyrros, A., et al., 2003. The brain's response to incidental intruded words during focal text processing. NeuroImage 18, 117–126. van Dijk, T.A., Kintsch, W., 1983. Strategies of Discourse Comprehension. Academic Press, New York. Virtue, S., Heberman, J., Clancy, Z., Parrish, T., Jung Beeman, M., 2006. Neural activity of inferences during story comprehension. Brain Res. 1084, 104–114. Vogt, B.A., Finch, D.M., Olson, C.R., 1992. Functional heterogeneity in cingulate cortex: the anterior executive and posterior evaluative regions. Cereb. Cortex 2, 435–443. Wagner, A.D., Shannon, B.J., Kahn, I., Buckner, R.L., 2005. Parietal lobe contributions to episodic memory retrieval. Trends Cogn. Sci. 9, 445–451. Xu, J., Kemeny, S., Park, G., Frattali, C., Braun, A., 2005. Language in context: emergent features of word, sentence, and narrative comprehension. NeuroImage 25, 1002–1015. Yarkoni, T., Speer, N.K., Balota, D.A., McAvoy, M.P., Zacks, J.M., 2008. Pictures of a thousand words: investigating the neural mechanisms of reading with extremely rapid event-related fMRI. NeuroImage 42, 973–987. Zacks, J.M., Braver, T.S., Sheridan, M.A., Donaldson, D.I., Snyder, A.Z., Ollinger, J.M., et al., 2001. Human brain activity time-locked to perceptual event boundaries. Nat. Neurosci. 4, 651–655. Zwaan, R.A., Radvansky, G.A., 1998. Situation models in language comprehension and memory. Psychol. Bull. 123, 162–185. Zwaan, R.A., Rapp, D.N., 2006. Discourse comprehension, In: Traxler, M.J., Gernsbacher, A.M. (Eds.), Handbook of Psycholinguistics, 2nd ed. Academic Press, San Diego, pp. 725–764. Zwaan, R.A., Langston, M.C., Graesser, A.C., 1995. The construction of situation models in narrative comprehension: an event-indexing model. Psychol. Sci. 6, 292–297.