The neural predictors of choice preference in intertemporal choice

BR A I N R ES E A RC H 1 4 36 ( 20 1 2 ) 9 2 –10 0

Available online at www.sciencedirect.com

www.elsevier.com/locate/brainres

Research Report

The neural predictors of choice preference in intertemporal choice Lei Liua, 1 , Tingyong Fenga, b,⁎, 1 a

School of Psychology, Southwest University, Chongqing, China Key Laboratory of Cognition and Personality, Ministry of Education, China

b

A R T I C LE I N FO

AB S T R A C T

Article history:

Intertemporal choice may involve two processing stages: a valuation stage and a choice stage.

Accepted 9 December 2011

Decision makers must integrate the various dimensions of an option (e.g., money, time) into a

Available online 16 December 2011

single measure of its subjective value (the valuation stage) and then choose the option that is the most valuable (the choice stage). Although previous studies have implicated that subjective

Keywords:

values are represented by a diverse set of brain regions (e.g., vmPFC, VStr, and PCC) in

Intertemporal choice

intertemporal choice, it is not yet known which of these regions contain information that

Temporal discounting

directly predicts subsequent choice. To address this question, we measured brain activity

Valuation stage

using functional MRI while participants performed a simple intertemporal choice task. The

Choice stage

results found that participants' decision could be encoded by three specific brain areas (vmPFC,

fMRI

ACC, and PCC) even before they were required to make a choice, while the left posterior insula showed positively active in the choice stage when individuals selected the delayed rewards compared to the immediate rewards. These findings suggest that activation patterns in the vmPFC, ACC, and PCC were able to predict the subsequent choice preference; however, left posterior insula may play an important role for choice preference in the choice stage. © 2011 Elsevier B.V. All rights reserved.

1.

Introduction

Intertemporal choices which involve tradeoffs between costs and benefits occurring at different points in time are important and ubiquitous (Frederick et al., 2002). For example, decision makers must often decide whether to save retirement or to consume food. Behavioral research suggests that as the delay of delayed rewards increases, an individual's valuation of a delayed reward declines as a function of time, a phenomenon known as temporal discounting (Ainslie, 1975; Kirby and Santiesteban, 2003; Mazur, 1987). Increasing evidence suggests that intertemporal choice may involve two processing stages: a valuation stage and a choice stage (Hampton and

O'Doherty, 2007; Kable and Glimcher, 2009). Choice responses are often performed to obtain time-related reward, and the obligation of the valuation mechanism is to ensure that appropriate responses are selected. However, the neural processes of intertemporal choice remain unclear up to now. With the use of functional magnetic resonance imaging, recent research has approached the neural correlate of intertemporal choice. In a fMRI study, McClure et al. (2004) found that mesolimbic dopamine projection regions (including ventral striatum (VStr), ventral medial prefrontal cortex (vmPFC), and posterior cingulate (PCC)) were strongly active when choices involve immediate rewards; in contrast, lateral cortical regions (including dorsolateral prefrontal cortex (DLPFC) and posterior parietal

⁎ Corresponding author at: Department of Psychology, Southwest University, Chongqing 400715, China. Fax: +86 23 68367572. E-mail address: [email protected] (T. Feng). 1 Contributed equally to this work. 0006-8993/$ – see front matter © 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.brainres.2011.12.018

BR A I N R ES E A RC H 1 4 3 6 ( 20 1 2 ) 9 2 –10 0

cortex (PPC)) were more generally activated regardless of delay (McClure et al., 2004, 2007). Thus, McClure et al. proposed that intertemporal choice involved the integration of two independent valuation systems (the impatient β-process and the more patient δ-process systems) that competed with each other for behavioral control, which was considered as the dual-valuation account (McClure et al., 2004). However, a more recent study challenged the dual-valuation account. Kable and Glimcher (2007) found that activation in mesolimbic projection regions (including VStr, vmPFC, and PCC) correlated with a combination of the magnitude and delay of delayed rewards and they have identified that brain activation patterns decline to follow a hyperbolic discounting function in these areas (Kable and Glimcher, 2007, 2010), which was referred to the single-valuation account. Recently, neural signals have been identified related to choice preferences in intertemporal choice. Tanaka et al. (2004) found that the insular cortex and the striatum coded the choice of options for immediate versus delayed gratification. For example, ventroanterior regions of these two brain regions were activated for the immediate choices, whereas dorsoposterior regions were activated when subjects selected delayed rewards (Tanaka et al., 2004). Wittmann et al. found that when individuals chose the delay relative to the immediate reward, a strong activation was found in bilateral posterior insular cortex (Wittmann et al., 2007). In another similar study, they found that the ventral striatum was involved in the processing of choosing delay options as well as the consequences of choices with delays in the seconds' range (Wittmann et al., 2010). It has been known that decision making is typically viewed as a two-stage mechanism in which value is first assigned to each option and then compared to yield choice. Converging evidence suggested that these values were stored in the striatum and medial prefrontal cortex (mPFC), and were subsequently used to guide choice (Kable and Glimcher, 2009; Levy et al., 2011). In addition, other studies showed that in the reward-based decision-making task, anterior cingulate cortex, medial prefrontal cortex, and ventral striatum could encode information relevant to subsequent behavioral choice (Hampton

93

and O'Doherty, 2007). However, to our knowledge, studies have yet to demonstrate whether specific neural circuits can encode the subsequent choice preference in the valuation stage in intertemporal choice. The primary goal of this study was to further explore whether the activity of some neural circuits could be used to predict later choice preference. Using functional magnetic resonance imaging (fMRI), we developed a paradigm based on intertemporal choice task (Peters and Büchel, 2009). Subjects first saw the visual presentation of the immediate and delayed reward options for 2 s (the valuation stage), and then in order to prevent them from comparing values of options before the choice stage, there were two types of choices in the choice stage: forced-choice asking subjects to follow the computer's choices and free-choice indicating subjects to choose freely in which subjects indicated the strength of their decisions by using a four-point scale (see Fig. 1). Thus, in the valuation stage, the subjects did not compare values until the choice stage. To compute choice offers for the fMRI session, subjects first completed a behavioral pretest. Subsequently, they participated in the delayed discounting task during scanning. In line with previous studies (Hampton and O'Doherty, 2007; Kable and Glimcher, 2007; Wittmann et al., 2007), we expected that during the valuation stage, preference would activate neural circuits associated with the subjective valuation, e.g., vmPFC, ACC, VStr and so on; and during the choice stage, however, VStr, and insula would be greater activation when subjects chose the delayed rewards.

2.

Results

2.1.

Behavioral results

All subjects selected either immediate or delayed reward options during scanning, and the percentage of choices for the immediate ￥20 options ranged from 28% to 80% of the trials. We estimated a discount function for each subject consistent

Fig. 1 – Intertemporal choice task. Each trial started with a white fixation for 2–6 s. Next, the immediate and delayed options were presented for 2 s. After another white fixation for 2–6 s, they made the choice: forced-choice indicated by a red dot and free-choice suggested by a yellow dot.)

94

BR A I N R ES E A RC H 1 4 36 ( 20 1 2 ) 9 2 –10 0

with previous findings, and the discount curves for all subjects were characterized by a hyperbolic function (mean R2 from scanning sessions = 0.940, range = 0.578–0.996). Indifference amounts across subjects were calculated by averaging the amounts of the delayed options that included the point of preference reversal, corresponding to the amount at which subjects were indifferent between the immediate options and the delayed options. Indifference amounts were then converted into proportions of the fixed reward. Then, a hyperbolic function of the form was fit to these data to obtain discount rates k using Matlab. The mean discount function during fMRI (based on the median indifference points across subjects) was depicted in Fig. 2. On average, subjects chose strongly the immediate rewards on 27.8%±12.9% of all trials (range=9%–52%) and chose weakly the immediate rewards on 16.7%±5.0% of all trials (range=8%– 23%). Subjects chose strongly the delayed rewards on 27.2%± 13.9% of all trials (range=8%–41%) and chose weakly the delayed rewards on 28.3%±5.9% of all trials (range=11%–48%). The average decision times to the choice alternatives were 1.005 s± 0.303 s (immediate rewards—strongly accept) 1.1156 s±0.377 s (immediate rewards—weakly accept), 1.065 s±0.310 s (delayed rewards—strongly accept), and 0.935 s ± 0.199 s (delayed rewards—weakly accept). Using two-way repeated-measures ANOVAs on choice preference (immediate rewards, delayed rewards) and choice strength (strongly accept, weakly accept), we analyzed RTs to test effects of the value of the choice alternatives. There were significant main effects of choice preference (F(1, 17)= 8.653, p=0.009) and choice strength (F(1, 17)=23.132, p<0.001). However, the interaction effect between choice preference and choice strength was not significant (F(1, 17)=0.132, p=0.721).

2.2.

fMRI results

2.2.1. Immediate and delayed options versus baseline during the valuation stage The whole brain analysis (p < 0.05, corrected) revealed that several brain areas were activated during the valuation stage

(see Table 1). Specifically, there were significant activations in the anterior cingulate cortex (ACC), vmPFC, and PCC in the valuation condition relative to baseline. In addition, bilateral anterior insula, striatum, cuneus, inferior frontal/middle/medial frontal gyri, inferior/superior parietal lobule, precentral gyri, precuneus, thalamus and cuneus were positively active when individuals evaluated between the immediate and delayed options relative to the baseline.

2.2.2. Immediate and delayed options versus baseline during the choice stage Relative to baseline, bilateral posterior insula, inferior frontal/ middle frontal gyri, inferior/superior parietal lobule, precuneus, cingulate, thalamus, fusiform gyrus, and lingual gyrus were positively active when individuals decided between the immediate and delayed options relative to the baseline during the choice stage (whole brain analysis: p < 0.05, corrected; see Table 2).

2.2.3. VOI time courses for the immediate and delayed options during the valuation stage Time courses of activation were extracted from each VOI during the evaluation stage to verify the timing of neural responses to the strength of participants' choice preferences indicated by the statistical maps. Based on previous studies (Hampton and O'Doherty, 2007; Kable and Glimcher, 2007; Wittmann et al., 2007), the results showed that activations in VOIs in group maps (vmPFC, ACC, and PCC) significantly diverged between the immediate options and the delayed options, that was, the activations in these regions were greater when individuals chose immediate rewards. Fig. 3 showed brain regions significantly correlating with the subsequent choice of participants and Fig. 4 showed time courses of activations in these regions. Two-way repeated-measures ANOVAs on choice preference (immediate rewards, delayed rewards) and choice strength (strongly accept, weakly accept) demonstrated that activations in VOI regions including the ACC (F(1, 17) = 6.017, p = 0.025), vmPFC (F(1, 17) = 5.488, p = 0.032), and PCC (F(1, 17) = 5.466,

Fig. 2 – Behavioral data from fMRI experiment. A, Indifference points, plotted as a function of the imposed delay to the delayed reward. Indifference amounts across subjects were calculated by averaging the amounts of the delayed options at which the subject would choose the immediate and delayed rewards with equal frequency. The increase in indifference amounts with delay was fit by a line with a fixed intercept at $20. The standard bars were shown. B, Median discount functions from the fMRI sessions. Indifference points divided into $20 to obtain a discount function. The decrease in subjective value with delay was fit with a single-parameter hyperbolic function.

95

BR A I N R ES E A RC H 1 4 3 6 ( 20 1 2 ) 9 2 –10 0

Table 1 – Activation foci for immediate and delayed options versus baseline during valuation. Region

Volume

Max Z score

X

Y

Z

L ACC BA32 R vmPFC BA10 L PCC BA30 L inferior parietal lobule BA40 L inferior frontal gyrus BA9 L precentral gyrus BA6 R insula BA13 L insula BA13 R striatum R thalamus (medial dorsal nucleus) L middle frontal gyrus BA9 L medial frontal gyrus BA6 R medial frontal gyrus BA32 L precuneus BA7 R superior parietal lobule BA7 L superior parietal lobule BA7 R cuneus BA18

917 668 554 745 452 813 879 873 399 373 561 912 779 780 710 853 324

4.917 4.685 4.282 5.196 4.334 4.686 5.078 4.979 4.210 4.098 4.334 5.769 4.869 5.195 4.330 4.714 −4.167

−6 2 −6 −49 −45 −46 32 −30 15 8 −41 −7 5 −9 27 −32 23

42 54 − 53 − 31 5 3 17 18 3 − 18 18 2 9 − 70 − 68 − 59 − 84

−1 3 16 45 24 38 10 10 7 7 31 52 45 37 42 46 18

p = 0.032) showed significant main effects of choice preference. There were no significant main effects of choice strength in these regions (ACC: F(1, 17) = 3.480, p = 0.079; vmPFC: F(1, 17)= 2.869, p = 0.109; PCC: F(1, 17) = 1.979, p = 0.178). The interaction effects between choice preference and choice strength were also not significant in these regions (ACC: F(1, 17) = 0.166, p = 0.689; vmPFC: F(1, 17)= 0.241, p = 0.630; PCC: F(1, 17) = 0.833, p = 0.374). Using the same two-way repeated-measures ANOVAs on choice preference (immediate rewards, delayed rewards) and choice strength (strongly accept, weakly accept) in striatum region, there were no significant main effects of choice preference (F(1, 17)= 0.368, p = 0.552) and choice strength (F(1, 17) = 0.243, p = 0.629). The interaction effect between choice preference and choice strength was also not significant (F(1, 17) = 2.869, p = 0.109). Thus, the results suggested that the neural dissociation of choice preference could be presented by the active strength of vmPFC, ACC, and PCC, but the strength of their choice could not be presented by these regions. Together, these findings suggested that neural circuits associated with the subjective valuation could predict subsequent choice in

intertemporal choice in which the circuits were activated strongly when individuals chose immediate rewards.

2.2.4. VOI time courses for the immediate and delayed options during the choice stage Based on the results of intertemporal choices, activations for immediate rewards chosen and delayed rewards chosen were compared. The finding showed that left posterior insula was more active when delayed rewards were chosen. Fig. 5 showed brain regions significantly correlating with the choice of participants and time courses of activation in left posterior insula. Similarly, two-way repeated-measures ANOVAs on choice preference (immediate rewards, delayed rewards) and choice strength (strongly accept, weakly accept) demonstrated activation in left posterior insula showing significant main effect of choice preference (F(1, 17) = 7.760, p = 0.013). However, there was no significant main effect of choice strength (F(1, 17) = 1.256, p = 0.278). And the interaction between choice preference and choice strength was not significant (F(1, 17) = 2.199, p = 0.156). In addition, there were no significant main effect of

Table 2 – Activation foci for immediate and delayed options versus baseline during choice. Region

Volume

Max Z score

X

Y

Z

L posterior insula BA13 R posterior insula BA13 L cingulate gyrus BA23 L middle frontal gyrus BA10 R middle frontal gyrus BA10 R inferior parietal lobule BA40 L inferior parietal lobule BA40 R inferior frontal gyrus BA9 R thalamus L superior parietal lobule BA7 L precuneus BA7 R precuneus BA7 L fusiform gyrus BA19 R lingual gyrus BA18

354 587 574 626 688 738 519 808 733 357 984 915 636 579

5.262 5.609 4.986 5.307 5.855 5.753 5.556 5.040 5.720 4.931 6.396 8.170 8.683 9.375

− 44 46 0 − 40 34 47 − 42 50 25 − 26 − 21 19 − 29 30

−8 15 − 31 48 47 − 36 − 54 10 − 21 − 58 − 72 − 71 − 76 − 71

18 4 27 10 9 49 40 24 11 39 39 45 − 12 − 11

96

BR A I N R ES E A RC H 1 4 36 ( 20 1 2 ) 9 2 –10 0

Fig. 3 – Brain regions significantly correlating with the subsequent choice in the valuation stage (display threshold, p < 0.05, corrected; voxels ≥ 10).

choice preference in cingulate gyrus BA23 (F(1, 17) = 2.205, p = 0.154) and right posterior insula (F(1, 17) = 1.445, p = 0.246). And there were no significant main effects of choice strength (cingulate gyrus: F(1, 17) = 2.145, p = 0.161; right posterior insula: F(1, 17) = 0.876, p = 0.362). The interaction effect between choice preference and choice strength was also not significant (cingulate gyrus: F(1, 17) = 1.412, p = 0.251; right posterior insula: F(1, 17) = 2.331, p = 0.145). Thus, left posterior insula might play a role when participants selected the delayed rewards.

3.

Discussion

Using event-related fMRI, we investigated the specific neural circuits responsible for choice preferences during the valuation stage and the choice response stage in intertemporal choice. Behaviorally, the delayed options were discounted in a manner that was consistent with a hyperbolic discounting function (Bickel et al., 1999; Kirby and Santiesteban, 2003; Lane et al., 2003). When individuals selected the immediate rewards relative to the delayed rewards, the vmPFC, ACC and PCC showed strong activations in the valuation stage; however, when individuals selected the delayed rewards relative to the immediate rewards, the left posterior insula showed positive activity in the choice stage. These findings suggested that the activity of three brain areas associated with subjective value representations, the vmPFC, ACC and PCC, could be used to predict later choice preference, and posterior insula might play an important role for choice preference in the choice stage.

This study found that a subset of three regions seemed to contain information that encoded subsequent choice. When individuals selected the immediate reward compared to the delayed reward, strong activations were found in vmPFC, ACC and PCC during the valuation stage. Activation in the vmPFC has been reported to reflect product-related preference and attractiveness judgments (Lebreton et al., 2009; Paulus and Frank, 2003; Plassmann et al., 2008). Furthermore, the vmPFC has been proposed to impact consumer preferences and choices (Lee et al., 2006), and predict subsequent purchase decisions (Knutson et al., 2007). Similar findings have been reported for the ACC, which has been implicated in action–outcome learning (Rushworth et al., 2007), in effort-based decision making (Floresco and Ghods-Sharifi, 2007; Hauber and Sommer, 2009). Moreover, several electrophysiological studies have reported that ACC appeared to be vital for learning about the consequences of internally generated decisions (Procyk et al., 2000; Shidara and Richmond, 2002), assessing the value of chosen responses and guiding subsequent choices accordingly (Hampton and O'Doherty, 2007). Posterior cingulate cortex might serve as a link between selecting an action and monitoring its consequences to improve delayed decisions. PCC neurons responded to rewards (McCoy et al., 2003), signal preferences in a gambling task with matched reward rates (McCoy and Platt, 2005), and signal omission of predicted rewards (McCoy et al., 2003). Hayden et al. (2008) findings showed that PCC directly contributed to the neural processes that evaluated reward outcomes in subjective terms and used this information to influence subsequent

Fig. 4 – Time courses of activation during the valuation stage: time courses of % signal change binned by pre-Im strong, pre-Im weak, pre-De strong, and pre-De weak in vmPFC, ACC, and PCC.

BR A I N R ES E A RC H 1 4 3 6 ( 20 1 2 ) 9 2 –10 0

97

Fig. 5 – A, Brain region significantly correlating with the subsequent choice in the choice stage (display threshold, p <0.05, corrected; voxels ≥10). B, Time courses of activation during the choice stage: time courses of % signal change binned by Im strong, Im weak, De strong, and De weak in left posterior insula.

decisions. Thus, these observations suggest the hypothesis that PCC contributes to the integration of actions and their outcomes and thereby influences subsequent changes in behavior. Collectively, the three brain areas may play a critical role in preference valuation, and can be used to predict subsequent choices in intertemporal choice. Insula has been found in a large number of decisionmaking studies, e.g., anticipation of rewards (Critchley et al., 2001), and risk-taking (Ernst et al., 2002). Previous studies suggested that posterior insula might be involved in delaying gratification, that was, the ability to postpone immediate rewards for sake of delayed greater benefits (Wittmann et al., 2007). This is in line with our results, in which left posterior insula was more active, when participants selected the delayed rewards. Thus, posterior insular may integrate interoceptive information as a basis for judging reward values and then be involved in action choice, which suggests that posterior insula may play an important role for choice preference in the choice stage.

3.1. choice

Presentation of choice preferences in intertemporal

In two earlier studies subjects were scanned while choosing either the immediate reward or the delayed reward, in which the valuation stage and the choice stage were integrated into one decision making stage. By searching for brain areas, the first study showed that there were more activities for immediate rewards than for delayed rewards in ventroanterior regions of the insular cortex and the striatum, whereas dorsoposterior in the two regions were more active when choosing the delayed rewards (Tanaka et al., 2004); however, the second study only found that bilateral posterior insular cortex showed more activities when choosing the delayed rewards (Wittmann et al., 2007). Other studies have shown that ACC, mPFC, and VStr could encode information to guide subsequent choice in reward-based decision making task (Hampton and O'Doherty, 2007). Antecedent neural activation in ventral striatum and

insula could be used to predict upcoming financial decisions (Knutson et al., 2007). In a recent study, whether the choice was required or not, activation in the striatum and mPFC predicted subsequent choices (Levy et al., 2011). These studies suggested that activation in these circuits did not simply correlate with expectations, but also preceded and might promote decisions, possibly in opposing directions. This study was the first to directly link activation in the valuation stage to guide subsequent choices in intertemporal choice. Participants first integrated the various dimensions of an option (e.g., money, time) into a single measure of its subjective value, and then chose the option that was the most valuable. We found that activity in a specific subset of our interesting regions could decode the subsequent decisions, namely, the ACC, vmPFC, and PCC, whereas posterior insula was more activated for delayed than immediate reward in the choice stage. Our findings were also consistent with the proposal that decision making was best thought of as an emergent property of interactions between a distributed network of brain areas rather than being computed in any one single brain region (Hampton and O'Doherty, 2007). It could be assumed that value valuation was a major source for subsequent choices.

3.2. Strengths, limitations, and implications for future research The present study features a number of novel strengths. This study is the first to separate the processing stages in intertemporal choice, especially, examining specific neural circuits which can encode the subsequent choice preference in the valuation stage. The novel task was created that had several merits: (1) to appropriately differentiate valuation from choice, the valuation stage was designed such that participants cannot choose until the choice stage. For example, although both options were revealed in the valuation, the participants did know whether they would need to choice until the choice stage. Thus, two processing stages are thought to be discerned: the valuation and the choice stage. (2) Participants rated the subjective strength

98

BR A I N R ES E A RC H 1 4 36 ( 20 1 2 ) 9 2 –10 0

of their preference (weak or strong preference). Due to the methodological separation of processing stages, the study results could be related to different subprocesses. The benefits conferred by the present design also necessitated a few tradeoffs. In the study, the distinction relies on a 2–6 s time window imposed between the moment that participants see their options and the moment that participants make a response. Consequently, “valuation stage” activity could be related to assigning values or to comparing them. Choice preferences in the valuation stage might be activated and valuation in the choice phase might still be present. Thus, it might be difficult to disentangle the two processes so clearly. Every methodological approach has weaknesses and strengths. The present task cannot be seen as the “ideal” approach but as first approximation to a goal. To separate the valuation and choice stages clearly, increasing the unexpected possibility by adding the probability of forced-choice trials represents an interesting target for further research. However, adding the probability of forced-choice trials would require an additional study, since it would lengthen the length of the present experimental design. In addition, motor demands were not counterbalanced in our experiment. However, we did not think that this would be a confound between motor response and choice options. On the one hand, reward valuation areas should distinguish from motor areas. Previous studies showed that vmPFC, PCC, and VStr played an important role in reward-related process (Kable and Glimcher, 2007; Peters and Büchel, 2009). And these areas did not affect individuals' motor responses. On the other hand, the results were in line with previous studies which found that ACC, vmPFC, PCC, and posterior insula could decode the subsequent decisions (Hampton and O'Doherty, 2007; Wittmann et al., 2007). However, motor preparation and motor response may have the potential to influence the results. The best method is that motor demands are counterbalanced in the future study. In summary, in the context of the intertemporal choice task, it is possible for a decision to be computed before the implementation of the behavioral choice. By using fMRI techniques, we found that a subset of three regions seemed to contain information that was sufficient to decode subsequent behavioral decision making: ACC, vmPFC, and PCC. In addition, we found that posterior insula might play an important role for choice preference in intertemporal choice.

4.

Experimental procedures

4.1.

Participants

In total, data from 18 participants (10 male; all right-handed, mean age 22± 3 years) were included in the present data analysis. Subjects provided informed written consent, and the study was approved by the ethical committee of Southwest University (China). One additional subject participated but was excluded from analysis, because the subject did not choose the strong acceptance of immediate rewards by at least 15 trials. 4.2.

Stimulation procedure

Before the fMRI session, all participants completed a computerbased delay discounting procedure to estimate their discount

rates for a reward of ￥20 (roughly 3 Dollars) (1–3) (Peters and Büchel, 2009). Then, indifference amounts were calculated by averaging the amounts of the delayed options that included the point of preference reversal, corresponding to the amount at which subjects were indifferent between the immediate and the delayed options (Peters and Büchel, 2009). The magnitude of the delayed options varied over five amounts which were different for each subject, and the delay of the delayed options varied over five durations (1, 7, 30, 60, 90 days). Thus, in our experiment, the five delays were factorially combined with the five amounts to yield 25 unique choice pairs for each participant. Each trial started with a white fixation (jitter) duration between 2 and 6 s. Next, the immediate and delayed offers were shown for 2 s, followed by another white fixation (jitter) that was shown for random duration 2 and 6 s. Then, during the choice, the color of the central cue instructed subjects how to choose. A yellow dot indicated that subjects could choose either the smaller, immediate reward or the larger, delayed reward in which subjects indicated the strength of their choice preference by using a four-point scale (immediate reward—strongly accept, weakly accept; delayed reward—strongly accept, weakly accept) (Hare, et al., 2009) and a red dot suggested that the subjects should follow the computer's choices. In our experiment, for example, if participants wanted to choose the immediate rewards, they could press 1 or 2 key (1—strongly accept, 2—weakly accept); in contrast, if participants wanted to choose the delayed rewards, they could press 3 or 4 key (4—strongly accept, 3— weakly accept). After response confirmation, another 2–6 s jitter preceded the start of the next trial (see Fig. 1). Forced-choice and free-choice trials were randomly intermixed. Subjects completed four sessions, each of which lasted for 12 min and comprised 25 forced-choice trials and 25 free-choice trials, yielding a total of 100 trials per-condition for each subject. Prior to scanning, participants were told that one of their choices would be randomly selected following scanning. If the immediate reward was chosen, the amount was given immediately after the scanning session; if the delayed reward was chosen, the amount was transferred to the participants' bank account with the specified date. 4.3.

fMRI acquisition and analysis

We used a Siemens TRIO 3.0 T full-body MRI scanner to measure changes in blood oxygenation level-dependent (BOLD) activity. During each fMRI scan, a time series of volumes was acquired using a T2*-weighted echo planar imaging pulse sequence (TR, 2000 ms; TE, 30 ms; flip angle, 90°; field of view, 220; in-plane resolution, 64 × 64; 0.99 mm gap). In addition, anatomical images (256 × 256 × 176) with 1 × 1 × 1 mm3 resolution were obtained by a T1-weighted three-dimensional magnetization prepared rapid gradient echo (MPRAGE) sequence (inversion time, 900 ms; repetition time (TR), 1900 ms; echo time (TE), 2.52 ms; flip angle, 9°). fMRI data were analyzed using BrainVoyager QX (2.0). Preprocessing of functional scans included discarding the first 3 volumes, slice scan time correction, inter and intrasession three-dimensional motion correction and removal of low frequencies up to 5 cycles per scan (linear trend removal and high pass filtering). The images were then coregistered with each subject's high resolution anatomical scan, rotated into

BR A I N R ES E A RC H 1 4 3 6 ( 20 1 2 ) 9 2 –10 0

the anterior commissure–posterior commissure plane, and normalized into Talairach space (Talairach and Tournoux, 1988). 4.3.1. Statistical maps The variance in BOLD signal was decomposed in a general linear model (Friston et al., 1995), separately for each run. The time course of activity of each voxel was modeled as a sustained response during each trial, convolved with a standard estimate of the hemodynamic impulse response function (Boynton et al., 1996). During the valuation stage, there are five events modeled, based on the trial outcome: strongly accept for immediate rewards (pre-Im strong, free-choice trials), weakly accept for immediate rewards (pre-Im weak, free-choice trials), strongly accept for delayed rewards (pre-De strong, free-choice trials), weakly accept for delayed rewards (pre-De weak, free-choice trials), and forced-choice trials. The choice stage was also modeled using the same five events: strongly accept for immediate rewards (Im strong, free-choice trials), weakly accept for immediate rewards (Im weak, free-choice trials), strongly accept for delayed rewards (De strong, free-choice trials), weakly accept for delayed rewards (De weak, free-choice trials), and forcedchoice trials. The threshold for the random-effects maps was set at p < 0.05 corrected for false discovery rate (FDR) (Genovese et al., 2002). 4.3.2. Region of interest analysis Regions of interest (ROIs) were defined in regions which were positively active when individuals evaluated between the immediate and delayed options relative to the baseline within each individual subject. Previous studies showed that some regions could predict subsequent choice, for example, posterior insula (right: 41, −13, 12; left: −40, −6, 7) (Wittmann et al., 2007), vmPFC(−9, 50, 4) (Knutson et al., 2007; Paulus and Frank, 2003), striatum (right: 10, 12, −1; left: −10, 11, 0) (Knutson et al., 2007), ACC (12, 39 33) (Pine et al., 2009; Hampton and O'Doherty, 2007), and PCC (−8, −28, 32) (McClure et al., 2004; McCoy and Platt, 2005). In our study, we found that some regions which were in line with previous studies were significant activations by the task vs. baseline contrast. For example, there were significant activations in the ACC (−6, 42, −1), vmPFC (2, 54, 3), PCC (−6, −53, 3), and striatum (15, 3, 7) in the valuation condition relative to baseline. And there were significant activations in posterior insula (right: −44, −8, 18; left: 46, 15, 4) and cingulate (BA23; 0, −31, 27) in the choice condition relative to baseline. Thus, we used these regions (the valuation stage: ACC, vmPFC, PCC, and striatum; the choice stage: posterior insula and cingulate BA 32) as ROIs for analyzing. The location and size of each ROI were as follows: the valuation stage: ACC (−6, 42, −1; volume=917 mm3), vmPFC (2, 54, 3; volume=668 mm3), PCC (−6, −53, 3; volume=554 mm3), striatum (15, 3, 7; volume=399 mm3); the choice stage: posterior insula (right: −44, −8, 18; volume=587 mm3. left: 46, 15, 4; volume=354 mm3), cingulate (BA23: 0, −31, 27; volume= 574 mm3). Spatially averaged activation time courses were extracted from each volume of interest (VOI) and then divided by the average activation for each VOI over the course of the entire experiment to derive measures of percent signal change. That was, the calculations of “peak activation” from VOI time courses were based on an event-locked average. The events

99

were the valuation stage (signal at lag=6 s) and the choice stage (signal at lag=4 s). VOI peak activations during the valuation and choice response stages were submitted to two-way repeated-measures ANOVAs on their choice preference and their choice strength as the within-subjects factors and subjects as random effects.

Acknowledgments This study was supported by the National Natural Science Foundation of China (30800292), the National Key Discipline of Basic Psychology in Southwest University of China (NSKD08007) and the Fundamental Research Funds for the Central Universities (SWU1109009). The authors thank the anonymous reviewers for helpful comments.

REFERENCES

Ainslie, G., 1975. Specious reward: a behavioral theory of impulsiveness and impulse control. Psychol. Bull. 82, 463–496. Bickel, W.K., Odum, A.L., Madden, G.J., 1999. Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers. Psychopharmacology (Berl) 146, 447–454. Boynton, G.M., Engel, S.A., Glover, G.H., Heeger, D.J., 1996. Linear systems analysis of functional magnetic resonance imaging in human V1. J. Neurosci. 16, 4207–4221. Critchley, H.D., Mathias, C.J., Dolan, R.J., 2001. Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron 29, 537–545. Ernst, M., Bolla, K., Mouratidis, M., Contoreggi, C., Matochik, J.A., Kurian, V., et al., 2002. Decision-making in a risk-taking task: a PET study. Neuropsychopharmacology 26, 682–691. Floresco, S.B., Ghods-Sharifi, S., 2007. Amygdala-prefrontal cortical circuitry regulates effort-based decision making. Cereb. Cortex 17, 251–260. Frederick, S., Loewenstein, G., O'Donoghue, T., 2002. Time discounting and time preference: a critical review. J. Econ. Lit. 40, 351–401. Friston, J., Homes, A., Worsley, K., Poline, J., Frith, C., Frackwowiak, R., 1995. Statistical parametric maps in functional imaging: a general linear approach. Hum. Brain Mapp. 2, 189–210. Genovese, C.R., Lazar, N.A., Nichols, T., 2002. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15, 870–878. Hampton, A.N., O'Doherty, J.P., 2007. Decoding the neural substrates of reward-related decision making with functional MRI. PANS 104, 1377–1382. Hare, T.A., Camerer, C.F., Rangel, A., 2009. Self-control in decision-making involves modulation of the vmPFC valuation system. Science 324, 646–648. Hauber, W., Sommer, S., 2009. Prefrontostriatal circuitry regulates effort-related decision making. Cereb. Cortex 19, 2240–2247. Hayden, B.Y., Nair, A.C., McCoy, A.N., Platt, M.L., 2008. Posterior cingulate cortex mediates outcome-contingent allocation of behavior. Neuron 60, 19–25. Kable, J.W., Glimcher, P.W., 2007. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625–1633. Kable, J.W., Glimcher, P.W., 2009. The neurobiology of decision: consensus and controversy. Neuron 63, 733–745. Kable, J.W., Glimcher, P.W., 2010. An “as soon as possible” effect in human intertemporal decision making: behavioral evidence and neural mechanisms. J. Neurophysiol. 103, 2513–2531.

100

BR A I N R ES E A RC H 1 4 36 ( 20 1 2 ) 9 2 –10 0

Kirby, K.N., Santiesteban, M., 2003. Concave utility, transaction costs, and risk in measuring discounting of delayed rewards. J. Exp. Psychol. Learn. Mem. Cogn. 29, 66–79. Knutson, B., Rick, S., Elliott Wimmer, G.E., Prelec, D., Loewenstein, G., 2007. Neural predictors of purchases. Neuron 53, 147–156. Lane, S., Cherek, D., Pietras, C., Tcheremissine, O., 2003. Measurement of delay discounting using trial-by-trial consequences. Behav. Processes 64, 287–303. Lebreton, M., Jorge, S., Michel, V., Thirion, B., Pessiglione, M., 2009. An automatic valuation system in the human brain: evidence from functional neuroimaging. Neuron 64, 431–439. Lee, L., Frederick, S., Ariely, D., 2006. Try it, you'll like it: the influence of expectation, consumption, and revelation on preferences for beer. Psychol. Sci. 17, 1054–1058. Levy, I., Stephanie, C., Lazzaro, S.C., Rutledge, R.B., Glimcher, P.W., 2011. Choice from non-choice: predicting consumer preferences from blood oxygenation level-dependent signals obtained during passive viewing. J. Neurosci. 31, 118–125. Mazur, J., 1987. An Adjusting Procedure for Studying Delayed Reinforcement. Erlbaum, Hillsdale, NJ. McClure, S.M., Laibson, D.I., Loewenstein, G., Cohen, J.D., 2004. Separate neural systems value immediate and delayed monetary rewards. Science 306, 503–507. McClure, S.M., Ericson, K.M., Laibson, D.I., Loewenstein, G., Cohen, J.D., 2007. Time discounting for primary rewards. J. Neurosci. 27, 5796–5804. McCoy, A.N., Platt, M.L., 2005. Risk-sensitive neurons in macaque posterior cingulate cortex. Nat. Neurosci. 8, 1220–1227. McCoy, A.N., Crowley, J.C., Haghighian, G., Dean, H.L., Platt, M.L., 2003. Saccade reward signals in posterior cingulate cortex. Neuron 40, 1031–1040. Paulus, M.P., Frank, L.R., 2003. Ventromedial prefrontal cortex activation is critical for preference judgments. Neuroreport 14, 1311–1315.

Peters, J., Büchel, C., 2009. Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. J. Neurosci. 29, 15727–15734. Pine, A., Seymour, B., Roiser, J.P., Bossaerts, P., Friston, K.J., Valerie Curran, H., Dolan, R.J., 2009. Encoding of Marginal Utility across Time in the Human Brain. J. Neurosci. 29, 9575–9581. Plassmann, H., O'Doherty, J., Shiv, B., Rangel, A., 2008. Marketing actions can modulate neural representations of experienced pleasantness. PNAS 105, 1050–1054. Procyk, E., Tanaka, Y.L., Joseph, J.P., 2000. Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508. Rushworth, M.F., Behrens, T.E., Rudebeck, P.H., Walton, M.E., 2007. Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour. Trends Cogn. Sci. 11, 168–176. Shidara, M., Richmond, B.J., 2002. Anterior cingulate: single neuronal signals related to degree of reward expectancy. Science 296, 1709–1711. Talairach, J., Tournoux, P., 1988. Co-planar Stereotaxic Atlas of the Human Brain. Thieme, New York. Tanaka, S.C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., Yamawaki, S., 2004. Prediction of immediate and delayed rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893. Wittmann, M., Leland, D.S., Paulus, M.P., 2007. Time and decision making: differential contribution of the posterior insular cortex and the striatum during a delay discounting task. Exp. Brain Res. 179, 643–653. Wittmann, M., Lovero, K.L., Lane, S.D., Paulus, M.P., 2010. Now or later? Striatum and insula activation to immediate versus delayed rewards. J. Neurosci. Psychol. Econ. 3, 15–26.