Chemistry of the Adaptive Mind: Lessons from Dopamine

Neuron

Review Chemistry of the Adaptive Mind: Lessons from Dopamine Roshan Cools1,2,3,* 1Radboud University Medical Center, Department of Psychiatry, Donders Centre for Cognitive Neuroimaging, Kapittelweg 29, 6500 HB Nijmegen, the Netherlands 2Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, Donders Centre for Cognitive Neuroimaging, Kapittelweg 29, 6500 HB Nijmegen, the Netherlands 3Twitter: @CoolsControl *Correspondence: [email protected] https://doi.org/10.1016/j.neuron.2019.09.035

The brain faces various computational tradeoffs, such as the stability-flexibility dilemma. The major ascending neuromodulatory systems are well suited to dynamically regulate these tradeoffs depending on changing task demands. This follows from various general principles of chemical neuromodulation, which are illustrated with evidence from pharmacological neuroimaging studies on striatal dopamine’s role in output gating and cost-benefit choice of cognitive tasks. The work raises open questions, including those regarding the top-down cortical control of the midbrain dopamine system, and begins to elucidate the mechanisms underlying the variability in catecholaminergic drug effects. Such drug effects depend on the baseline state of distinct target brain regions, reflecting, in part, the systems’ self-regulatory capacity to maintain equilibrium. It is hypothesized that the basal tone of different dopaminergic projection systems reflects the perceived statistics of the environment computed in frontal cortex. By normalizing dopamine levels, dopaminergic drugs might counteract the bias elicited by the perceived environment. Introduction We live in the era of the connectome, where the brain is conceptualized as a distributed computing system, consisting of multiple structurally densely interconnected neural networks. However, these structural connections are relatively fixed. Therefore, one major question is: which mechanisms allow our brain to adapt flexibly to our constantly changing environment? Here, I review evidence suggesting that the major ascending neuromodulatory systems originating from the midbrain, such as dopamine, noradrenaline, and serotonin, implement this functionality (Figure 1). It is the chemical neuromodulators that adapt the output of our structurally fixed neural networks to our changing environment. Indeed, the human brain constantly faces a variety of computational dilemmas. Depending on task demands, our neural networks need to exhibit stability or flexibility, controlled or automatic processing, speed or accuracy, and generalization or multitasking (Cohen, 2017). In this review, I will focus on the first of these dilemmas: the stability-flexibility dilemma. These dilemmas are at the core of cognitive control, which is an ill-defined construct but generally refers to those mental processes that allow us to obtain our goals, by focusing on currently relevant representations and resisting distractions, temptations, and impulses. It is considered a hallmark of the human brain. Humans are particularly good at focusing on a current goal, and the prefrontal cortex, the brain region most strongly associated with cognitive control, is particularly well developed in the human species. Nevertheless, we fail to exert cognitive control all the time. Only a few of you will resist shifting attention to checking email or other novel input at various points during reading this paper, despite the clear goal of reading straight through until

the end. After all, focus is opportunity costly (Kurzban et al., 2013), and some novel, current task-irrelevant input might well turn out to be important and require a form of flexibility rather than focus. What we need is a meta-level ability to decide when to exert cognitive focus and when rather to let go of focus and flexibly respond to new input. How do we do this? Answering this question requires that the problem of cognitive control is reconceptualized as a problem of reinforcement learning and decision making instead of simply only a problem of implementation (Kool et al., 2010). The observation that the chemical neuromodulators adapt the output of our structurally fixed neural networks to our changing environment is grounded in part in the hypothesis that they contribute to arbitrating, or deciding, between different, often opposing computational strategies. In the first part of this paper, I will define chemical neuromodulation and highlight three key general principles of chemical neuromodulation, which provide the mechanistic basis of their role in adaptive cognition. This list of principles is not exhaustive, and for a more extensive review of key neurochemical principles, one might consider reading a previous review by Dayan (2012). Next, I will zoom in on the dopamine system and highlight one of the major problems of psychiatry and neurology: there is huge variability in dopaminergic drug effects on cognitive function, both across different and within the same individuals. I will then illustrate how the general principles of chemical neuromodulation can account for these paradoxical effects of dopaminergic drugs by reviewing a set of pharmacological and chemical neuroimaging studies with human volunteers that represent two relatively segregated lines of work, one focusing on functions associated most commonly with striatal dopamine, such as Neuron 104, October 9, 2019 ª 2019 Elsevier Inc. 113

Neuron

Review L-Tyrosine

Tryptophan

L-DOPA

Adrenaline

5-Hyrdoxtryptophan

Dopamine

Noradrenaline

Serotonin

value-based learning and choice, and the other on functions associated commonly with prefrontal dopamine, such as working memory and cognitive control. Subsequently, I will present current work that integrates these separate lines of work on striatal and prefrontal dopamine. Finally, a number of key open questions will be highlighted that arise from this work, such as: how are these midbrain systems controlled? While the highlighted principles likely generalize to other major ascending neuromodulatory systems (e.g., of serotonin, noradrenaline, and acetylcholine), the dopamine system represents a particularly useful case, in which disproportionately great advances have been made, not least because of the availability of sensitive techniques for its measurement and manipulation. At the same time, the dopamine system is also a special case, given its unique relatively selective abundance in the striatum and prefrontal cortex. This implies that we cannot necessarily generalize the specific implementation of these principles, outlined here for dopamine, to other systems. The reader is referred to other reviews for such implementational specifics of, for example, the hugely complex serotonin system, with its bewildering number of receptor types and distributions or the detailed specificity of cholinergic neuromodulation in cortical and subcortical regions. General Principles of Chemical Neuromodulation What is chemical neuromodulation, and what are its general principles? Chemical neuromodulation often refers to that set of neurochemical processes that curtail or prolong, augment or diminish, fast signaling in neuronal networks, often conditional on the baseline firing state of the postsynaptic cell (Iversen et al., 2009). This modulatory mode is distinguished from fast, direct signaling in the classical neurotransmitter mode via glutamate and GABA and refers to the control exerted on such classic neurotransmission. The first key general principle of chemical neuromodulation highlighted here is regional specialization. While neuromodulatory signals are often conceptualized as reflecting broadcast signals due to their widespread innervation of large parts of the brain (Schultz, 2007), their release and manipulation have radically different functional consequences depending on where they act in the brain. This regional specialization stems in part from regional differences in the density of different receptors (Zilles and Palomero-Gallagher, 2017), in the density of different projection neurons (Cragg, 2003), and in the speed of re-uptake

114 Neuron 104, October 9, 2019

Figure 1. Synthesis Pathways and Projections of Three Major Ascending Monoamines Adapted from Frobo¨se and Cools (2018).

(Sesack et al., 1998). For example, in the case of dopamine, which innervates multiple relatively segregated frontostriatal circuits, its motor, cognitive, and motivational consequences are now well established to reflect modulation of motor, dorsolateral, and ventromedial frontostriatal circuits, respectively (Alexander et al., 1986). Interestingly, a gradient in the speed of reuptake has been reported from ventral to dorsal components of the striatum, with the dorsal striatum being more sensitive to the temporally precise phasic signals than the ventral striatum (Cragg et al., 2000). This is not to say that the midbrain dopamine signal itself is not homogeneous, corresponding to, say, a reward prediction error (Montague et al., 1996), although evidence is accumulating that there is large heterogeneity in the neurophysiological responses of different clusters of midbrain dopamine cells (Engelhard et al., 2019). For dopamine, it is well possible that the functional consequences of a common value signal and a common selection computation, for example, at the level of the population of receiving striatal neurons, are diverse depending on the representational input to various target striatal subregions. A second principle is that these systems exhibit an exquisite capacity for self-regulation, with stimulation of presynaptic autoreceptors, present on the presynaptic elements of the midbrain neurons, being associated with inhibition of synthesis and release (Figure 2). In the case of dopamine, extensive evidence indicates biphasic inhibiting versus potentiating responses to low versus high doses of dopamine D2 receptor agonist administration, due to relatively greater sensitivity of pre- versus postsynaptic D2 receptors to low versus high doses, respectively (Frank and O’Reilly, 2006). The short-latency phasic responses have been argued to be controlled by tonic levels of activity, perhaps via negative feedback inhibition (Grace, 2000). Thus, although the proposal that separate phasic and tonic dopamine cell firing modes might subserve, respectively, reinforcement learning and motivation (Niv et al., 2007) is controversial (Hamid et al., 2016; Mohebi et al., 2019), there might well be a link between neurochemical self-regulation via autoreceptors and the functional opponency between reward prediction error-related dopamine firing and striatal dopamine levels reflecting average reward rate, which, in reinforcement learning models, serves as a reference against which prediction errors are compared (Cools et al., 2011). Whether this latter average reward rate or value (Mohebi et al., 2019) is a direct function of phasic dopamine cell firing (Lohani et al., 2018; Floresco et al., 2003) or is regulated independently from the phasic dopamine signal, for example, via local striatal control (Kosillo et al., 2016) is a matter of ongoing debate. Possibly related to this second principle of self-regulation is a third principle, baseline dependency, which refers to the now well-established, inverted

Neuron

Review pacity, measured with positron emission tomography with the tracer 6-[18F]fluoro-L-m-tyrosine (FMT). The same drug improved learning in low-dopamine subjects but impaired reward learning in high-dopamine subjects (Cools et al., 2009). There are various accounts of such detrimental ‘‘overdose’’ effects (Arnsten et al., 2012). According to one of these, the beneficial effect in low-dopamine subjects reflects predominant action at postsynaptic D2 receptors, the sensitivity of which would be increased in those subjects in order to increase the system’s sensitivity to dopamine increases. Conversely, the detrimental overdose effect in high-dopamine subjects might reflect a paradoxical net reduction of dopamine due to predominant action at presynaptic D2 receptors, the sensitivity of which would be increased as a result of the endogenous system’s tendency to regulate itself. Consistent with a unified view of dopamine’s roles in reward learning and motivation (Collins and Frank, 2014; Hamid et al., 2016), according to which the same dynamically fluctuating dopamine signal both alters the willingness to work and reinforces preceding action choices by encoding temporal difference reward prediction errors, similar baseline tone-dependent effects were observed on a task measuring the incentive motivation of cognitive control (Aarts et al., 2014): the promise of reward enhanced performance on a Stroop task in subjects with low baseline levels of dopamine synthesis capacity while impairing it in subjects with high baseline levels of dopamine. Together, these data provided strong evidence for the hypothesis that the human dopamine system expresses baseline dependency, as is the case for the nonhuman dopamine system (Williams and Goldman-Rakic, 1995). The implication of this finding is that isolation of dopaminergic drug effects on human cognition requires that the baseline level of dopamine is taken into account. Figure 2. Mechanisms of Dopamine Autoregulation The presynaptic nigrostriatal terminal releases dopamine (blue circles) and regulates extracellular dopamine levels through several mechanisms: dopamine reuptake from the extracellular fluid (via the DAT), dopamine transport into synaptic vesicles (via VMAT-2), dopamine synthesis (which is subjected to autoregulatory control via presynaptic D2 receptors), and dopamine metabolism (via MAO-B and COMT). The postsynaptic neuron responds to dopamine via two main types of receptors. AADC, aromatic L-amino acid decarboxylase; AC, adenylate cyclase; COMT, catechol-O-methyl-transferase; DAT, dopamine transporter; MAO-B, monoamine oxidase B; TH, tyrosine hydroxylase; VMAT-2, vesicular monoamine transporter 2. Figure reproduced from Cenci (2014), available via license: Creative Commons Attribution 4.0 International.

U-shaped dose-response curve: drugs that increase receptor stimulation have positive effects in systems with low baseline levels of activity but negative effects in systems with high baseline levels of activity. While such baseline dependency has been described most often for dopamine (Cools and D’Esposito, 2011), it is likely a characteristic of the other neuromodulatory systems, including noradrenaline (Arnsten et al., 2012), serotonin (Cano-Colino et al., 2014), and acetylcholine (Bentley et al., 2011), and in fact of any system (including body temperature and home central heating) characterized by self-regulation or homeostasis that ‘‘strives’’ to achieve its optimal level. In the case of dopamine, effects of the dopamine receptor agonist bromocriptine on reward learning were shown to depend critically on the baseline levels of striatal dopamine synthesis ca-

Dopaminergic Drug Effects on Human Cognition The cognitive role of dopamine is evidenced by its implication in not only mental disorders, like attention deficit hyperactivity disorder (ADHD) and Parkinson’s disease (PD), but also healthy states of stress and fatigue, which are all accompanied by cognitive (and impulse) control deficits. Critically, these states are often treated with cognitive-enhancing drugs that increase dopamine, such as levodopa or the dopamine and noradrenaline transporter blocker methylphenidate. Unfortunately, there is huge variability in treatment efficacy both across and within different individuals. Consider, for example, ADHD, where symptoms of inattention and impulsivity are often treated with dopamine-enhancing drugs like methylphenidate. Conversely, consider PD, where dopamine-enhancing medication can actually, in a considerable proportion of patients, contribute to cognitive deficits, leading, in some cases, to severe psychiatric abnormalities, including gambling addiction, hypersexuality, compulsive hobbying, and/ or addiction to medication intake (Weintraub, 2019). More pertinent, perhaps, are the diametrically opposite effects seen with the same drug administered to the same individuals across different tasks. For example, in one recent study, we observed, within the same group of healthy volunteers, anti-impulsive, distractibility-decreasing effects on a delayed response task of working memory (Fallon et al., 2017) but pro-impulsive, distractibility-increasing effects on a flanker task of selective attention

Neuron 104, October 9, 2019 115

Neuron

Review (ter Huurne et al., 2015), all as a function of the same dose of methylphenidate. How can the same drug have such different effects both across different and within the same individuals? I will illustrate how we have begun addressing these questions, noting that, as is the case for much of neuroscience, almost all of the advances that we make in our understanding of human chemical neuromodulation have leveraged insights from preclinical work with experimental animals (Floresco, 2013; Chudasama and Robbins, 2004; Schultz, 2007; Goldman-Rakic, 1995). Indeed, these systems are well conserved across species. Here, we focus exclusively on work with human volunteers, where the consequences of manipulating the midbrain systems are more complex, in part due to the increase in sophistication of both the brain regions that are targeted by chemical neuromodulation as well as the brain regions that exert its top-down control. Distinct Prefrontal and Striatal Systems for Dopaminergic Control of Cognition One reason why the dopamine system might be particularly well suited to help adapt the organism to its environment is because of its strong implication in reinforcement learning. In the late ’90s, major advances in our understanding of the neurophysiological signature of dopamine neuron firing (Schultz et al., 1997) converged remarkably with advances in computer science on the theory of reinforcement learning (Montague et al., 1996) and subsequently with evidence from interventional studies with dopaminergic drugs in human patients (Pessiglione et al., 2006). Together, these lines of work strongly suggested that phasic dopamine signals encode reward prediction errors that are paramount for learning from reward. Since then, much additional advancement has been made in understanding how dopamine activity computes action value and motivation (Hamid et al., 2016). By contrast, dopamine’s effects on functions classically associated with cognitive control and working memory have received much less attention and are more poorly understood. This relative lack of attention to prefrontal dopamine is despite seminal, pioneering work from Patricia Goldman-Rakic and colleagues (Arnsten, 2013), showing that 6-hydroxydopamine lesions of the prefrontal cortex impaired performance on a classic task of working memory almost to the same extent as did complete ablations of the prefrontal cortex (Brozoski et al., 1979) and that prefrontal neuron firing during the delay and probe phases of classic working memory tasks depends on dopamine D1 and D2 receptor activation, respectively (Wang et al., 2004). A select set of research groups subsequently pursued prefrontal dopamine’s role in working memory and cognitive control (Chudasama and Robbins, 2004; Floresco, 2013; Arnsten et al., 2012; Clark and Noudoost, 2014; Ott and Nieder, 2019), resulting not only in a growing body of empirical evidence, but also in an advance in theoretical understanding of prefrontal dopamine’s role (Braver and Cohen, 1999; Durstewitz and Seamans, 2008; ServanSchreiber et al., 1990; Seamans and Yang, 2004). For example, according to the dual-state theory of prefrontal dopamine function (Durstewitz and Seamans, 2008), which is grounded in biophysically realistic computational models and in vitro measurements of dopamine’s effects on prefrontal neurons and synaptic currents, two discrete dynamical regimes exist: a D1dominated state is characterized by a high-energy barrier among

116 Neuron 104, October 9, 2019

different attractor network patterns that favors robust online maintenance of representational states, whereas a D2-dominated state is characterized by a low-energy barrier that is beneficial for flexible and fast switching among representational attractor states. In this attractor network model, intermediate levels of dopamine promote the stabilization of working memory representations by predominantly stimulating prefrontal D1 receptors, while both low and high levels of dopamine promote the flexible updating of working memory representations by predominantly stimulation prefrontal D2 receptors. Strikingly, the development of this prefrontal class of theoretical models of dopamine’s control of cognition occurred in relative isolation from the (theoretical and empirical) advance made in parallel in the domain of striatal function. Particularly influential was work by O’Reilly, Frank, Hazy, et al. (Frank et al., 2001; Hazy et al., 2007), who had formulated a neural network account of dopamine’s contribution to working memory, according to which striatal dopamine regulates the opening and closing of a gate to working memory, akin to its role in selecting motor actions (Gurney et al., 2001). As such, the hypothesis that dopamine’s effects on cognition reflect modulation of, exclusively, the prefrontal cortex had already then long been revised. In fact, the current body of empirical evidence favors a multiple systems account of dopamine’s cognitive effects, highlighting the importance of the modulation of multiple distinct neural systems for the dopaminergic control of cognition. For example, 6-hydroxydopamine lesion work with nonhuman primates revealed contrasting effects of prefrontal and striatal dopamine depletion, which elicited increases in distractibility and greater inflexibility on an attentional set-shifting paradigm, respectively (Collins et al., 2000; Crofts et al., 2001; Roberts et al., 1994). Based on such empirical evidence, as well as the observation that there is neurochemical reciprocity between frontal and striatal dopamine (Roberts et al., 1994; Pycock et al., 1980; Meyer-Lindenberg et al., 2002), we have put forward the hypothesis that dopamine might modulate these distinct stabilizing and flexible aspects of cognition by acting on the prefrontal cortex and striatum, respectively (Figure 3). To test this hypothesis, we developed a delayed response paradigm to test the distinct stable and flexible aspects of control in terms of distractor resistance and task switching, respectively (Figure 3A). In keeping with the observed functional and neurochemical reciprocity, a series of studies followed showing the opposite effects of dopamine on these two forms of cognition. So, for example, patients with PD, characterized by severe striatal dopamine depletion, were found to exhibit increased switch costs (Cools et al., 2001a; Pollux, 2004; Rogers et al., 1998; Cameron et al., 2010; Fales et al., 2006) but also paradoxically reduced distractor costs (Cools et al., 2010; Moustafa et al., 2008) (Figure 3B): tfhey exhibited impaired cognitive flexibility but also enhanced cognitive stability. Dopaminergic medication reversed this pattern, enhancing cognitive flexibility but restoring to normal cognitive instability (Cools et al., 2010; Moustafa et al., 2008). Similar contrasting effects were seen with administration of dopamine D2 receptor agonists, like bromocriptine and cabergoline, in healthy volunteers, with drug-related increases in distractor costs (Bloemendaal et al., 2015; Broadway et al., 2018) and drug-related decreases in switch

Neuron

Review

Figure 3. Dopamine and the Stability/Flexibility Tradeoff (A) Event sequence of a trial in the adapted delayed response task that allowed the separate quantification of (1) cognitive flexibility in terms of switching attention between faces and scenes, depending on the color of the fixation cross, and (2) cognitive stability in terms of the resistance of the memorized face or scene representations to distraction (from a novel face or scene compared with a scrambled image). Figure reproduced from Cools and D’Esposito (2011). (B) Bromocriptine modulated the striatum and prefrontal cortex during, respectively, cognitive flexibility and stability but only in subjects with high trait impulsivity. Left: effects of bromocriptine on striatal activity during switching (the trait impulsivity 3 group interaction effect) are overlaid on 3 coronal slices (slice numbers displayed on top) from the Montreal Neurological Institute high-resolution single subject MR image. Abbreviations: L, left; R, right). Right: effects of bromocriptine on frontal activity during the distractor as a function of group (the trait impulsivity 3 drug interaction effect). The bar graphs reflect effects of bromocriptine on switch- and distractor-related activity in the striatum and left PFC in high-impulsive subjects only. (C) This same paradigm was used to show differential effects of dopaminergic medication in patients with PD on attention switching and distractor resistance (Cools et al., 2010). Left: as in prior studies (e.g., Cools et al., 2001b), PD patients exhibited increased switch costs when they were OFF, but not ON, medication, but this was not significant in this study. Right: distractor costs (reaction times after a novel face or scene compared with a scrambled image) were reduced in patients OFF, but not ON, medication. (D) Left: data from van Holstein et al. (2011) showing that bromocriptine reduced costs on a task-switching paradigm in subjects with genetically determined low baseline dopamine levels (i.e., DAT polymorphism 10R allele carriers). Right: data from Bloemendaal et al. (2015) showing that bromocriptine increased distractor costs on a delayed response paradigm similar to that in (A).

costs, albeit only in subjects with putative low baseline levels of dopamine (van Holstein et al., 2011; Cools et al., 2007) (Figure 3C). Conversely, the dopamine D2 receptor antagonist sulpiride enhanced distractor resistance of working memory while impairing set shifting (Mehta et al., 2004). Subsequent pharmacological fMRI work revealed a double dissociation, where bromocriptine modulated switch-related blood-oxygen-level-dependent (BOLD) signal in the striatum, but not the prefrontal cortex, while it modulated distractorrelated BOLD signal in the prefrontal cortex, but not the striatum (Cools et al., 2007) (Figure 3D). These findings led us to conclude that dopaminergic drugs have contrasting effects on the flexible

and stable aspects of cognitive control by modulating the striatum and prefrontal cortex, respectively (Cools and D’Esposito, 2011). This proposal reconciles a fairly large body of results from both previous and more recent pharmacological neuroimaging work with healthy volunteers or PD patients, demonstrating dopaminergic drug effects on prefrontal BOLD signal during working memory task performance (e.g., Bloemendaal et al., 2015; Mattay et al., 2003; Fallon et al., 2017), but effects on striatal BOLD signal and/or frontostriatal functional connectivity during switching (Nagano-Saito et al., 2008; Samanez-Larkin et al., 2013; Dodds et al., 2008). Thus, dopaminergic drugs might regulate the stability-flexibility tradeoff by altering the balance

Neuron 104, October 9, 2019 117

Neuron

Review

Figure 4. Mechanisms of Gating the Output of the Prefrontal Cortex (A) Schematic illustration of the hypothesis that the basal ganglia (BG) control attention switching by regulating top-down projections from prefrontal cortex (PFC) to sensory-specific regions in posterior cortex. (B) The spatial attention-switching paradigm required subjects to covertly attend to the left or right visual hemifield. On repeat trials, they had to discriminate the direction of a moving dot pattern on one side while ignoring the other side (random noise). On switch trials, the introduction of coherent movement of the dot pattern at the other side triggered a switch in attention to that side. Subjects then continued to perform the task on the opposite hemifield. (legend continued on next page)

118 Neuron 104, October 9, 2019

Neuron

Review between prefrontal and striatal dopamine. In PD, in line with the dopamine overdose hypothesis (Gotham et al., 1986), dopaminergic medication might have enhanced cognitive flexibility by restoring dopamine levels in the striatum while impairing distractor resistance by detrimentally overdosing dopamine levels in the less affected prefrontal cortex (Agid et al., 1993). More recent evidence has substantiated this general hypothesis that dopaminergic drugs might have contrasting effects on distinct prefrontal and striatal brain regions in PD (Kim et al., 2018). In parallel, neural network model-inspired fMRI work revealed that attractor-like working memory representations of task rules was associated with switching-specific thalamocorticostriatal activation—that is, with a system associated with flexible working memory updating and dopaminergic modulation of cognitive flexibility (Ueltzho¨ffer et al., 2015). Together, these results converge to suggest that the optimally adaptive mind requires a dynamic dopamine-dependent adjustment of a balance between distinct prefrontal and striatal brain regions depending on the need for cognitive focus and flexibility, respectively. Dopaminergic Gating of Prefrontal Output By which mechanisms might striatal dopamine shift the balance toward greater cognitive flexibility? According to classic gating models of motor actions (Gurney et al., 2001), striatal dopamine regulates the flexible gating of actions by, just at the right time, increasing activity in the direct Go pathway but decreasing activity of the indirect Nogo pathway of the basal ganglia in proportion to a ‘‘behavioral relevance signal.’’ Critically, this role of striatal dopamine has been proposed to extend to the flexible gating of cognitive actions, including task rules, maintained in working memory (Frank et al., 2001; Hazy et al., 2007), with the striatum contributing to both the ‘‘input gating’’ of working memory representations for maintenance as well as the ‘‘output gating’’ or prioritization of one out of multiple maintained representations for subsequent action selection (Frank and Badre; 2012; Chatham and Badre, 2015). In essence, these latter output gating models represent an integration of classic topdown biasing models of cognitive control, selective attention, and working memory (Miller and Cohen, 2001), with classic models of the basal ganglia, according to which striatal dopamine lowers the threshold for action selection (Gurney et al., 2001). Inspired by this theoretical work, we set up a series of studies to test the hypothesis that the biasing by the prefrontal cortex of task-relevant processing in posterior sensory cortex is controlled by the striatum so that task-relevant posterior processing is flexibly updated only when the striatum opens the gate for attentional switching (Figure 4). In these experiments, we presented subjects with bidimensional stimuli consisting of an overlapping face and scene (van Schouwenburg et al., 2014, 2013, 2010). On each trial, subjects had to discriminate between two of these stimuli based on either one of the dimensions: the faces or the scenes. Critically, they were instructed

to switch attention to the other dimension as soon as they detected a change of exemplars in this other stimulus dimension. These switch trials elicited highly significant BOLD signal in the striatum, thalamus, and frontal cortex. Moreover, there were strong attentional gain effects in stimulus-specific posterior sensory cortex, with greatest signal in the fusiform face area when people switched to faces and greatest signal in the parahippocampal place area when people switched to scenes. The overall pattern of BOLD signal was best captured by a dynamic causal model in which striatal switch-related activity modulated effective connectivity from the prefrontal cortex to the taskrelevant visual association cortex (van Schouwenburg et al., 2010, 2015). In a follow-up study, we sought evidence for a stronger version of the selective gating hypothesis, derived from the specific anatomical arrangement of the direct and indirect pathways, that the basal ganglia might control attention by inhibiting taskirrelevant processing while also amplifying task-relevant processing (van Schouwenburg et al., 2015). We tested this hypothesis by using a spatial attention-switching paradigm requiring subjects to switch attention between stimuli in the left and right visual hemifield. Simultaneous fMRI recordings allowed us to compare BOLD responses associated with the current and the alternative visual hemifields. In line with previous work (van Schouwenburg et al., 2010), the basal ganglia, prefrontal cortex, and visual cortex showed significantly increased signal during attentional switches. The visual cortex responded in a spatially selective manner: signal was increased when attention was switched toward versus away from a particular visual hemifield in the contralateral visual hemisphere. Non-linear dynamic causal modeling was adopted to statistically select between a variety of model sets in which the basal ganglia control visual processing through modulation of prefrontal top-down connections with the visual cortex. The first model set included basal ganglia modulation of frontal connections to task-relevant visual cortex to assess excitatory gating of fronto-posterior connectivity to the visual hemisphere processing the newly relevant visual hemifield. In contrast, the second model set included basal ganglia modulation of frontal connections to the newly task-irrelevant visual hemisphere. This model effectively assesses inhibitory gating of fronto-posterior connections to the visual hemisphere processing the now irrelevant visual hemifield. The third model set included modulatory influences of the basal ganglia on prefrontal cortex projections to both task-relevant and task-irrelevant hemifields. Based on Bayesian model averaging across the three model sets, we showed that modulatory influences of the basal ganglia on fronto-posterior connections exhibited a significant interaction between switch direction and hemisphere: basal ganglia activity enhanced prefrontal influence on the newly task-relevant visual cortex, while it suppressed prefrontal influence on the task-irrelevant visual cortex during attention switching. These findings

(C) Univariate analyses revealed strong switch-related BOLD signal, also in the basal ganglia and the inferior frontal gyrus (top), as well as the attentional gain signals, with greatest BOLD signal in the left visual cortex when subjects switched to the right hemifield (and vice versa for the right visual cortex). (D) Non-linear dynamic causal modeling showed that BOLD data, acquired during task performance, was accounted for by a model, in which the basal ganglia facilitates attention switching via modulating top-down control of stimulus-specific regions in posterior cortex. Adapted from van Schouwenburg et al. (2015).

Neuron 104, October 9, 2019 119

Neuron

Review Figure 5. Striatal Dopamine and Output Gating (A) A face/scene attention-switching paradigm with superimposed faces and scenes required subjects to discriminate between two compound stimuli based on attention to either the faces or the scenes. Subjects discovered the relevant dimension based on feedback that was provided every second trial. The face and scene exemplars were paired orthogonally across these two consecutive trials and subjects were instructed to consistently make the same choice on these two trials, enabling inference of the attended dimension. Subjects were also instructed to switch attention to the other dimension as soon as they detected the introduction of novel exemplars in this other dimension. (B) This task was employed in a pharmacological fMRI study (van Schouwenburg et al., 2013) to demonstrate that administration of the dopamine D2 receptor agonist bromocriptine modulates switchrelated BOLD signal in the striatum (C) The white matter region that exhibited a correlation between the behavioral index of switching and fractional anisotropy (van Schouwenburg et al., 2014). (D) Probabilistic tractography from this region revealed structural tracts (left) culminating in a region of the inferior frontal gyrus (right) that also exhibited a correlation with bromocriptine’s effect on functional connectivity with the striatum (in blue). Figure is adapted from van Schouwenburg et al. (2013, 2014).

suggest that the basal ganglia selectively gate cortical representations through a combination of enhanced task-relevant processing and suppressed task-irrelevant processing. In a subsequent drug study, administration of bromocriptine was shown to increase switch-related BOLD signal in the striatum during face/scene attention switching (van Schouwenburg et al., 2013). Intriguingly, this region is located immediately adjacent to a region that exhibited, in a separate diffusion tensor imaging study, a link between the behavioral index of attentional switching and white matter integrity, indexed by fractional anisotropy (van Schouwenburg et al., 2014). Moreover, probabilistic diffusion tractography showed that white matter fibers run

120 Neuron 104, October 9, 2019

from this region around the basal ganglia to the frontal cortex. In fact, this tract culminates in a region of inferior frontal cortex, which also exhibited a correlation with dopamine drug effects on functional connectivity with the striatal BOLD region (van Schouwenburg et al., 2013). Thus, dopamine receptor D2 stimulation with bromocriptine modulated switch-related BOLD signal in a region of the striatum that connects structurally and communicates functionally with the prefrontal cortex. Together, these data suggest that striatal dopamine potentiates cognitive flexibility by gating the output of the prefrontal cortex. In other words, striatal dopamine gated attention to a cognitive task rule via modulating prefrontal top-down control of task-relevant processing in stimulus-specific posterior cortex (Figure 5). The finding that the striatum controls the top-down effective connection from the prefrontal cortex to the stimulus-specific posterior cortex is remarkably reminiscent of the qualitative prediction made by the computational ‘‘prefrontal cortex basal ganglia working memory’’ (PBWM) model put forward by Frank and O’Reilly. Specifically, the results from these empirical studies demonstrate that the striatum regulates the output gating of cognitive task representations. Thus, accumulating empirical and theoretical evidence indicates that, whereas the prefrontal cortex maintains cognitive task representations,

Neuron

Review protecting them from interference (D’Esposito and Postle, 2015), the striatum contributes to gating the input but, critically, also the output of that working memory buffer so that maintained information can be subject to further selection to determine whether or not it should influence downstream processing (e.g., attention or motor response selection) (Chatham and Badre, 2015). In the PBWM model, the output units of the prefrontal cortex are activated only when their corresponding striatothalamic Go pathway unit fires. One implication of this is that information can be maintained in an active but somewhat ‘‘offline’’ form before being actively output to drive behavior (O’Reilly et al., 2012). This, in turn, is reminiscent of a growing literature on working memory showing that when attention is directed toward a subset of working memory representations, such prioritized information is often found to be represented in stimulus-specific posterior regions, while low-priority representations is not (Mallett and Lewis-Peacock, 2018). One possibility is that low-priority information is retained in prefrontal (and parietal) regions (Christophel et al., 2018) but requires a corresponding striatal Go signal for it to be output gated. The recent observation that distinct layers of the prefrontal cortex might encode different maintenance and output-related component processes of working memory (Finn et al., 2018) brings to the surface an intriguing prediction, which follows from the PBWM model that distinguishes between superficial input and deep output layers of the prefrontal cortex (O’Reilly et al., 2012): the striatum might gate inter-laminar connectivity within prefrontal cortex, so that output layer activity is updated to reflect the current input layer activity only upon a basal ganglia Go signal. An open question for future work is whether the observation that the dopamine D1 and D2 receptors are relatively abundant in, respectively, the more superficial input and deep output layers of the prefrontal cortex (Santana and Artigas, 2017) can help reconcile the hypothesis that dopamine modulates flexible and stable control by acting on the striatum and prefrontal cortex (Cools and D’Esposito, 2011) with the alternative dual-state hypothesis of prefrontal dopamine, according to which modulation of flexible versus stable control reflects stimulation of D1 and D2 receptors in the PFC, respectively (Durstewitz and Seamans, 2008). Integrating Dopamine’s Dual Roles in Value-Based Choice and Cognitive Control How might striatal dopamine set the threshold for gating a cognitive action or task? According to the gating models described above, as well as currently popular resource allocation models of control (Shenhav et al., 2013), we allocate resources to a cognitive task based on its expected value—that is, its benefits versus its costs. Cognitive demand carries effort, time, and opportunity costs, evidenced by studies showing that, on average, healthy participants prefer to perform a cognitively less demanding task (Kool et al., 2010) and choose to give up money to avoid a more demanding task (Westbrook et al., 2013). According to the expected value of the control model, it is the anterior cingulate cortex that regulates the intensity and direction of control depending on its value (Shenhav et al., 2013). As such, this model reconciled the long-known dual roles of the anterior cingulate cortex in, on the one hand, conflict monitoring and cognitive control (Botvinick et al., 2004) and, on the

other hand, value-based decision making (Rushworth et al., 2007). By analogy, these insights can be leveraged to integrate the dual roles of dopamine in, on the one hand, value-based learning and choice and, on the other hand, working memory and cognitive control (Westbrook and Braver, 2016). Specifically, striatal dopamine might affect cognitive control indirectly by modulating the value of cognitive control (Cools, 2016). This hypothesis generally concurs with the growing body of evidence for striatal dopamine’s role in signaling the value of ‘‘work’’ (or effortful action) (Hamid et al., 2016; cf. Gan et al., 2010) and is grounded, in part, on the Opponent Actor Learning (OpAL) model by Anne Collins and Michael Frank (Collins and Frank, 2014), according to which striatal dopamine increases the weight on the benefits versus the costs of cognitive actions and/or tasks by having opposite effects on the D1 (Go) and D2 (No-Go) pathways of the basal ganglia. The general implication of the proposal that striatal dopamine alters the costs and benefit of cognitive work would be that the cognitive effects of striatal dopamine-related disorders and their treatment reflect modulation of the willingness rather than the ability to exert cognitive control. Two separate lines of evidence provide tantalizing evidence for the proposal that cognitive task valuation depends on striatal dopamine. First, there is ample evidence for altered effects of incentives on cognitive control task performance, measured with rewarded task-switching or Stroop tasks, as a function of (striatal dopamine cell loss in) PD patients (Aarts et al., 2012; Timmer et al., 2018), dopaminergic medication in PD (Manohar et al., 2015), individual differences in striatal dopamine synthesis capacity (Aarts et al., 2014), and individual variation in a striatal dopamine transporter gene (Aarts et al., 2010). Moreover, striatal dopamine genetic variation is associated with altered (effects of methylphenidate on) striatal BOLD signal during incentivized cognitive control task performance (Aarts et al., 2010). Second, cognitive task avoidance is now established to be sensitive to manipulation of catecholamine transmission in experimental animals (Cocker et al., 2012), healthy human volunteers (Frobo¨se et al., 2018), and PD patients (McGuigan et al., 2019) (Figure 6). Evidence from work with rodents demonstrates that prolonging catecholamine transmission with amphetamine administration motivated them to choose a cognitively more demanding option for a higher reward, although this was true only for a subset of rodents (Cocker et al., 2012; but see Hosking et al., 2015). In healthy volunteers (n = 100), methylphenidate (20 mg, oral) altered cognitive demand avoidance learning (to avoid task switching), but the effect of methylphenidate depended on trait impulsivity: demand avoidance learning was reduced in low-impulsive participants but enhanced in highimpulsive participants (Frobo¨se et al., 2018). The hypothesis that catecholaminergic drugs enhance demand avoidance learning in high-impulsive individuals by reducing the value of cognitive effort is supported by data from a recent pharmacological study (Frobo¨se et al., 2019) using a cognitive effort discounting paradigm (Westbrook et al., 2013). In this study, we assessed the effects of the catecholamine precursor tyrosine on the subjective value assigned to difficult cognitive task performance. The experiment consisted of two phases. In a first phase, which took place before the tyrosine manipulation could have taken effect, subjects were exposed to a 1-back, 2-back, 3-back,

Neuron 104, October 9, 2019 121

Neuron

Review

Figure 6. Striatal Dopamine in the Value of Cognitive Work (A) Left: the demand avoidance task (Kool et al., 2010) requiring subjects to choose between two abstract visual patterns. Each choice is followed by a yellow or blue digit, requiring subjects to perform an odd/even or a high/low discrimination. One pattern is associated with 90% task switches, the other with 10%. Right: data shown from Frobo¨se et al. (2018). Across blocks of 80 trials, subjects learn to avoid the high-effort cognitive task option. Shown here is the effect of methylphenidate versus placebo on the proportion of low-demand choices across the 80 trials. Methylphenidate increased or decreased high-effort task avoidance depending on trait impulsivity. (B) Data from McGuigan et al. (2019). Left: subjects made choices between high- and low-cognitive-effort tasks (requiring attention to more or less rapid serial visual presentation streams) based on varying reward magnitude. The probability of accepting the higher-effort option decreased with increasing effort and with decreasing reward magnitude. PD patients accepted fewer higher-effort options when they were OFF their medication but not when they were ON their medication.

and 4-back working memory task. In this first learning phase, subjects learned to associate these tasks with different shapes. In the second phase, administered after tyrosine took effect, they chose between the different shapes, in order to indicate their preference for repeating one of the two tasks, in return for more or less monetary payoff. Critically, the monetary payoffs for the easy task were titrated during choice until subjective equivalence was reached. The subjective value of an offer to repeat a difficult task corresponds to the amount offered for the easiest task at indifference. Supplementation of the precursor tyrosine increased the value of cognitive control in low-impulsive participants but reduced the value of cognitive control in high-impulsive participants. We speculate that, in high-impulsive individuals, methylphenidate and tyrosine might have attenuated the subjective value of cognitive effort by paradoxically decreasing dopamine synthesis and dynamic dopamine response to offer presentation via D2 autoreceptors, eliciting a shift toward more cost and less benefit sensitivity (Box 1). Indeed, trait impulsivity has been associated with reduced availability (yet possibly more sensitive) dopamine (auto)receptors (Buckholtz et al., 2010). This evidence for catecholaminergic

122 Neuron 104, October 9, 2019

modulation of the value of cognitive work in healthy volunteers was recently complemented by results from a controlled dopaminergic medication withdrawal study in PD patients showing that dopaminergic drugs increase the subjective value of—that is, preferences for—high-demand, high-reward cognitive tasks over low-demand, low-reward tasks (McGuigan et al., 2019). To provide direct evidence for a key role of striatal dopamine in cognitive work, we recently employed an adapted version of the cognitive effort discounting paradigm in a large multisession pharmacological PET study with 50 healthy students to demonstrate that methylphenidate, the selective dopamine D2 receptor agent sulpiride, and individual variation in striatal dopamine synthesis capacity all increase willingness to expend cognitive effort for reward (Westbrook et al., 2019). All participants underwent a dopamine PET scan with the radiotracer [18F]DOPA, the uptake of which indexes the degree to which dopamine is synthesized in the striatal terminals of midbrain dopamine neurons. The subjective value of cognitive effort was significantly greater in subjects with high baseline levels of striatal dopamine than in low-dopamine subjects. Critically, this effect was baseline dependent. While the drugs enhanced the value of control in

Neuron

Review Box 1. Paradoxical Effects of Catecholamine Challenges in High-Impulsive Participants

The hypothesis that methylphenidate and tyrosine elicit a net reduction in dopamine levels in high-impulsive participants is based on the observation that methylphenidate and tyrosine administration might increase presynaptic (autoreceptor) rather than postsynaptic dopamine receptor binding in the striatum and that impulsive individuals differ in their presynaptic signaling sensitivity. Supporting the first assumption, the administration of phenylalanine, the precursor of tyrosine, to rats increased striatal dopamine release at lower doses but attenuated dopamine release at higher doses (During et al., 1988). By analogy, methylphenidate has been proposed to have its pro-cognitive, anti-impulsive effects, not just by enhancing catecholamine transmission in the prefrontal cortex, but also by paradoxically attenuating reward-related phasic dopamine activity in the striatum. Increasing the overall level of extracellular dopamine with a stimulant might trigger autoregulatory negative feedback inhibition of dopamine synthesis or release (Grace, 2000; Seeman and Madras, 2002) or simply provide a relatively high appetitive ‘‘aspiration level’’ against which the shortterm dopamine responses to rewards are compared, thereby making them look relatively less good, thus reducing their behavioral influence (Cools et al., 2011). Supporting the second assumption, trait impulsivity has been shown to be accompanied by high baseline levels of striatal dopamine release and low (but perhaps more sensitive) presynaptic dopamine D2 receptor availability in the midbrain (Buckholtz et al., 2010).

low-dopamine subjects, it left the value of work in high-dopamine subjects unaffected. To assess whether striatal dopamine increased motivation by altering the subjective weighting of costs and benefits, we next made, in a third phase of the task, a series of offers tailored to participants’ own indifference points, systematically biasing either high-cost, high-benefit or low-cost, low-benefit choices while also monitoring gaze at cost or benefit information. The results support that striatal dopamine promotes selection of high-cost, high-benefit alternatives across all participants and, moreover, that dopamine synthesis capacity and methylphenidate amplify the effect of benefits on choice. Gaze dynamics further support that attention to benefits versus costs increases motivation, and this effect is larger for individuals with higher dopamine synthesis capacity. Moreover, drift diffusion modeling supported that methylphenidate and higher synthesis capacity both amplified the effect of benefit information on the accumulation of evidence toward high-effort choices. These findings demonstrate that striatal dopamine contributes to the arbitration between difficult and easy cognitive actions by increasing the weight on the benefits versus the costs of available options. An open question is whether striatal dopamine also contributes to arbitrating between different higher-order (e.g., flexible versus stable) cognitive strategies, depending on the characteristics of the current task environment. According to recent neural network modeling of task-switching data, the cost of control constrains the gain of an activation function and thus the stabilization of currently active representations (Musslick et al., 2018). As such, there is certainly good reason to believe that modulation of the value (or cost) of cognitive effort might contribute to the arbitration between stability and flexibility. Given growing evidence that the prefrontal cortex is organized hierarchically, with more anterior regions encoding increasingly abstract representations, striatal dopamine is likely to also gate increasingly abstract representations in and out of increasingly anterior regions of the prefrontal cortex (Frank and Badre, 2012). Key Open Questions Neurochemical Specificity: The Case of Noradrenaline While we have focused on the dopamine system in this review, the general observation that the neuromodulatory activity con-

tributes to resolving key computational dilemmas, such as the stability-flexibility tradeoff, likely generalizes to other major ascending neuromodulators. The effects of methylphenidate, described above, might well reflect modulation of noradrenaline, given that methylphenidate is known to increase synaptic noradrenaline levels and to reduce spontaneous activity of locus coeruleus neurons via noradrenaline action at a-adrenergic receptors on locus coeruleus cells. Indeed, the locus-coeruleusnoradrenaline system has been associated with processes closely related to cognitive effort, such as mental fatigue (Berridge and Waterhouse, 2003) and task engagement (i.e., exploitation) versus disengagement (i.e., exploration) (Aston-Jones and Cohen, 2005). Evidence with nonhuman primates indicates distinct signatures of dopaminergic versus noradrenergic neuron activity during effort-based decisions with dopamine neurons encoding cost-discounted values of rewards and noradrenergic neurons encoding amount of effort required to obtain them (Varazzani et al., 2015). In keeping with the suggestion that noradrenaline also contributes to meta-level optimization of cognition is evidenced from recent neural network analyses of pharmacology resting-state (van den Brink et al., 2016) and pharmacological task-based fMRI data (Hernaus et al., 2017). This work demonstrated that atomoxetine, a noradrenaline reuptake inhibitor (that might also affect dopamine transmission but not in the striatum), leads to a reorganization of the functional connectome in a manner that is sensitive to ongoing cognitive demands via altering the balance between network-level segregation and integration (Shine et al., 2018). Specifically, atomoxetine potentiated network segregation during rest, but, conversely, the same drug potentiated network integration during a classic n-back working memory task. These diametrically opposite effects were argued to concur with work with experimental animals showing atomoxetine-induced decreases in tonic levels of noradrenaline but increases of phasic firing patterns in the locus coeruleus (Bari and Aston-Jones, 2013). Critically, the segregation effects during rest and integration effects during task were observed in the same neural regions and inversely correlated, supporting a task-dependent reorganization of the system (Shine et al., 2018). Furthermore, atomoxetine was also shown to increase the coupling between pupil diameter, often considered an

Neuron 104, October 9, 2019 123

Neuron

Review Box 2. Other Computational Tradeoffs

There is reason to believe that meta-level arbitration effects of dopaminergic drugs are not restricted to the adjustment of stabilityflexibility tradeoff but extend to other computational tradeoffs such the one between instrumental control, often associated with the dorsal striatum, and Pavlovian control, often associated with the ventral striatum (O’Doherty et al., 2004). In recent work (Swart et al., 2017), we have seen that methylphenidate shifts the balance between selective instrumental and nonselective Pavlovian control. Participants learned to make Go or Nogo responses in order to obtain reward or avoid punishment. Instrumental response-outcome contingencies were set up so that cues required either a Go response to obtain reward, a Nogo response to obtain reward, a Go response to avoid punishment, or a Nogo response to avoid punishment (Guitart-Masip et al., 2014). A Pavlovian autopilot bias in such tasks surfaces in terms of a greater proportion of Go responses for reward and greater proportion of Nogo responses to avoid punishment, irrespective of the instrumental contingencies. Methylphenidate was found to bias participants away from the putative ventral striatal autopilot strategy, reducing the tendency to exhibit behavioral activation for reward and behavioral inhibition for punishment but only for people with low working memory capacity. In people with high working memory capacity, methylphenidate actually biased the system toward this putative ventral striatal autopilot strategy. Thus, in people with low working memory capacity, increasing catecholamine transmission might have restored suboptimal levels in dorsal frontostriatal circuity, associated with selective instrumental behavioral control, but overdosed optimal levels in ventral frontostriatal circuitry, associated with nonselective autopiloting. Conversely, in people with high working memory capacity, increasing catecholamine transmission might have overdosed optimal levels in dorsal frontostriatal circuitry but restored suboptimal levels in ventral frontostriatal circuitry. This account parallels the dopamine overdose hypothesis of the contrasting effects of dopaminergic medication in Parkinson’s disease, put forward many decades ago (Gotham et al., 1986; Cools et al., 2001b; Swainson et al., 2000). According to this hypothesis, dopaminergic medication doses that are necessary to remediate severely depleted dopamine levels in the dorsal striatum overdose relatively intact dopamine levels in the relatively unaffected ventral striatum. Future neuroimaging and/or stimulation work will be needed to test the hypothesis that the effect of working memory capacity on the balance between Pavlovian versus instrumental control (and its modulation by catecholaminergic drugs) is accompanied by a shift in the balance between activity in ventral versus dorsal frontostriatal circuitry. Another intriguing question for future work is whether the capacity dependency, in healthy volunteers, of both the Pavlovian autopilot effects under baseline and the catecholaminergic drug effects reflects individual differences in the efficiency of medial frontal cortex, perhaps reflecting trait variation in the perceived controllability of the outcomes. This latter hypothesis is raised by recent evidence that reliance on a Pavlovian versus instrumental strategy in a rewarded Go/Nogo task varies with fluctuations in the controllability of the task environment (Dorfman and Gershman, 2019). Combining techniques for manipulating frontal cortex, via transcranial magnetic or ultrasound stimulation, with techniques for measuring and manipulating the midbrain systems, like dopamine PET and pharmacology, holds promise for elucidating the mechanisms of such environment-specific control of the midbrain.

indirect index of locus coeruleus (the main noradrenergic nucleus of the brain) activity (Joshi et al., 2016), and network integration, possibly indicative of enhanced performance in taskrelevant regions through arousal-mediated alterations in neural gain (Aston-Jones and Cohen, 2005). Such findings hold general relevance, for example in the context of acute stress, where stress-induced increases in noradrenergic activity have been argued to prompt large-scale neural network reconfiguration (Hermans et al. 2011). They are also reminiscent of earlier theoretical and modeling work, suggesting that noradrenaline acts as a ‘‘neural interrupt signal’’ that alerts the learner to an unpredicted change in the learning environment (Dayan and Yu, 2006) and performs a ‘‘network reset’’ (Bouret and Sara, 2005). In line with this, pharmacological manipulations and lesions of the noradrenergic system affect performance, in nonhuman animals and human volunteers, on tasks such as reversal learning and attentional set shifting, which require detection of and adaptation to environmental changes (Lapiz and Morilak, 2006; Middleton et al., 1999). Recent work provides empirical evidence (Muller et al., 2019) for a long-standing computational account of such increases in cognitive flexibility in terms of changes in the uncertainty of an internal model of the environment (Aston-Jones and Cohen, 2005; Yu and Dayan, 2005). This account is grounded in the observation that

124 Neuron 104, October 9, 2019

environments characterized by high volatility benefit from higher levels of perceived uncertainty (and cognitive flexibility) than do stable environments. So far, this review highlights that there is evidence for key roles of at least both dopamine and noradrenaline in the optimization of meta-level arbitration between distinct cognitive strategies (Box 2). While it is clear that there is considerable specificity when it comes to these different neuromodulators, the precise nature of this specificity remains to be elucidated. The use of more selective receptor agents, for example, in pretreatment designs (van der Schaaf et al., 2014), in future work will help to disentangle the likely complementary roles of dopamine and noradrenaline. Top-Down Control of the Midbrain An obvious question raised by the observation that the ascending neuromodulatory systems that originate from the midbrain contribute to arbitrating between different options or strategies is: what then controls the midbrain? In the case of dopamine, this question might correspond to asking how we set the value of the various available actions or strategies. This is not trivial, given that valuation is context dependent (e.g., Juechems and Summerfield, 2019). For example, cognitive work is not so valuable when the environment is volatile, uncontrollable, or opportunity costly. The value of task disengagement might be higher than the value of task engagement in an environment

Neuron

Review

Figure 7. The Dopaminergic Midbrain Switch Hypothesis This hypothesis states that the medial frontal cortex controls the baseline tone of dopamine in distinct target regions of the midbrain, as a function of the perceived statistical meta-parameters of the environment, such as its controllability, volatility, and opportunity cost. The middle panels illustrate the hypothesis that dopaminergic drugs have contrasting effects depending on the baseline tone of these distinct target regions.

where there is lots of opportunity for obtaining reward from alternative tasks. One intriguing possibility is that the dopaminergic midbrain is controlled by top-down signals from medial frontal regions that are thought to compute key statistical meta-parameters of the environment, like its volatility, controllability, and/or opportunity cost (Boureau et al., 2015; Behrens et al., 2007; Amat et al., 2005; Kolling et al., 2010) (Figure 7). This hypothesis that (medial) frontal cortex computes various environmental statistics and that the output of this computation is sent to the midbrain, thus controlling the release of certain neuromodulators, has been articulated previously for noradrenaline and acetylcholine (AstonJones and Cohen, 2005; Yu and Dayan, 2005; Nassar et al., 2012). Indeed, neuromodulatory systems are well known to receive input from brain regions that represent environmental structure and internal state (Ogawa et al., 2014). While there are various alternative computational accounts of noradrenaline-related meta-level arbitration, invoking uncertainty-based adjustments of learning rate (Nassar et al., 2010; Yu and Dayan, 2005), a mixture of learning rules with different time constants (Wilson et al., 2013), predictive coding (Mathys et al., 2011), or dynamic state inference (Muller et al., 2019), converging evidence strengthens the proposal that its neural implementation

involves top-down control of the midbrain. For example, recent findings revealed that activity of neurons in the anterior cingulate cortex predicts pupil dilation and temporally leads pupil-predicting locus coeruleus activity (Joshi et al., 2016). More specifically, Muller et al. (2019) showed that activity in the anterior cingulate cortex predicted an increase in the perceived uncertainty of an internal model of the environment. Moreover, this cortical activity also predicted the associated increase in pupil dilation strength in circumstances when model uncertainty should increase, for example, upon observations likely to indicate a change in environmental state. While such pupil results cannot be linked directly to noradrenaline, they are compatible with the hypothesis that the noradrenaline system provides a mechanism by which perceived uncertainty of the environment, computed by the anterior cingulate cortex, can be communicated to a wide variety of brain regions. In the case of dopamine, it might be the medial frontal cortex that prioritizes the distinct (frontal versus striatal) dopaminergic midbrain projections for maintaining or updating (effortful) actions, depending on the current statistics of the environment, by setting the baseline tone of these different systems (Figure 7). This hypothesis is in line with recent work showing that the rodent anterior cingulate cortex modulates the dopaminergic

Neuron 104, October 9, 2019 125

Neuron

Review midbrain during effort-based decision making (Elston and Bilkey, 2017). Alternatively, one might speculate that such an effect of medial frontal cortex on the dopaminergic midbrain is indirect and mediated by the locus coeruleus, stimulation of which is known to alter midbrain dopamine cell firing (Grenhoff et al., 1993; Mejias-Aponte, 2016). The proposal that the dopaminergic midbrain is under top-down (context-dependent) control generalizes a recent proposal, derived from fiber photometric and optogenetic evidence for environment-specific activity of another midbrain system, namely the dorsal raphe serotonin projection system, to the catecholamine systems. This system was argued to switch operational mode depending on characteristics of the task to promote environment-specific adaptive behaviors (Seo et al., 2019). This inference was made based on the finding that optogenetic stimulation of dorsal raphe serotonin neurons suppressed movement in a low-to-moderate-threat environment while promoting (escape) movement in a high-threat environment. This observation is reminiscent of but qualifies previous evidence from work with human volunteers, indicating that serotonin mediates the coupling between aversive Pavlovian predictions and behavioral inhibition (Dayan and Huys, 2008) such that lowering central serotonin levels by dietary acute tryptophan depletion releases behavioral inhibition elicited by punishment (Crockett et al., 2009; Geurts et al., 2013). Such evidence was interpreted to reflect modulation of an average punishment (or negative value) signal, corresponding to an opportunity cost of speed (Cools et al., 2011; Boureau and Dayan, 2011). As such, serotonin tone provides a potential opponent to dopamine tone’s association with an opportunity cost of sloth via the encoding of an average reward signal (Niv et al., 2007). The findings by Seo et al. (2019) suggest that the degree to which serotonin elicits behavioral inhibition upon aversive predictions depends on the specific nature of the aversive environment. The previous observation that dorsal raphe serotonin activity varies with the controllability of aversive outcomes (stressors), in a manner that depends on the integrity of the medial frontal cortex (Amat et al., 2005), raises the question of whether this threat dependence reflects differences in the perceived level of threat controllability, computed by medial frontal cortex. A reduction in the tendency to inhibit aversive thoughts, putatively mediated by frontal suppression of serotonin release, is indeed adaptive in a controllable context, whereas aversive thoughts are less useful and might thus well be inhibited in an uncontrollable context. More generally, these data support the working hypothesis that it is this medial frontal cortex that prioritizes computational strategy (gating [cognitive] work in the case of dopamine and inhibiting [cognitive] work in the case of serotonin) depending on the current statistics of the environment (Figure 7). One way the frontal cortex might do that is via altering the baseline tone of different projections systems. Evidence from work by Antonio Strafella combining transcranial magnetic stimulation with dopamine PET indicates that transcranial stimulation of human frontal cortex can indeed alter dopamine release in regions of the striatum that are strongly connected with the stimulated region (Strafella et al., 2001). Intriguingly, the medial frontal cortex (with other cortical regions) is now recognized not to represent a uniform region but rather to hold multiple representations of choice value based on different timescales of experi-

126 Neuron 104, October 9, 2019

ence organized in terms of systematic gradients across the cortex (Meder et al., 2017). Specifically, some parts of this area represent value estimates based on recent reward experience while others represent value estimates based on experience over the longer term, with aspects of these representations changing dynamically as the environment changes. This observation provides a tentative mechanistic basis for environmentspecific dynamic prioritization of distinct midbrain projection systems, particularly given the established topographic specificity of cortico-nigro-striato-cortical connections (Alexander et al., 1986; Haber et al., 2000) and the regional selectivity of dopamine release in striatal subregions corresponding to their cortically connected region (Strafella et al., 2001). One open issue concerns the mechanisms by which the cortex can control dopamine activity in distinct target brain regions. Does this happen via direct modulation of midbrain cell firing (Sesack and Carr, 2002), through indirect effects via the locus coeruleus (Grenhoff et al., 1993), or via local control of the midbrain targets, such as the striatum? Indeed, recent work with experimental rodents combining optogenetic tagging, microdialysis, voltammetry, and optimal sensor dLight suggests that dopamine fluctuations in the nucleus accumbens associated with motivation (the value of work) might arise independently from midbrain (VTA) dopamine cell firing, perhaps reflecting local non-spiking control (Mohebi et al., 2019). One possibility, particularly given their feedback inputs to dopamine cells, is that it is GABA neurons in either the dopaminergic midbrain or the striatum that are the target of top-down cortical control, as both striatal and midbrain GABA neurons have been shown to exhibit ramps to reward (Cohen et al., 2012). Further, cortical and thalamic glutamate inputs have been shown to modulate dopamine transmission by regulating striatal cholinergic interneurons as gatekeepers (Kosillo et al., 2016). This observation might turn out to hold particular relevance for meta-level arbitration between stability and flexibility, given the implication of the cholinergic system in the top-down control of distractor resistance in PD (Kim et al., 2019) as well as cognitive effort-based decision making (Hosking et al., 2014). Another obvious question concerns the timescale at which such a midbrain switch can act. Can the switch be flipped according to fluctuations in state rather than according to trait differences? This would require that the temporal resolution of changes in dopamine tone is as high as that of relevant strategy changes, consistent with recently observed task-specific striatal dopamine ramps (Hamid et al., 2019). If so, then this would provide a mechanism by which midbrain activity can contribute to optimizing the selection of computational strategies to suit the constantly changing environment. Of course, also to be addressed is the origin of the large heterogeneity of environmental statistics that have an effect on behavior and of the diversity of the behavioral effects of different modulators. Different neurons or frontal regions might compute distinct statistics, and these could target distinct neuromodulatory systems (Ogawa et al., 2014). Might Dopaminergic Drugs Counteract Biases in Cognitive Tradeoffs? The final key open question is whether and how this orchestration of meta-level arbitration can account for the large

Neuron

Review heterogeneity in dopaminergic drug effects. Together with the observation that dopaminergic drug effects depend on baseline levels of dopamine, these insights raise the hypothesis that dopaminergic drugs restore the balance between distinct control strategies by having paradoxical effects depending on context-specific baseline tone of distinct brain systems. In this framework, the contrasting effects of therapeutic dopaminergic drugs across different individuals reflect a normalization of biased tradeoffs. In PD, which is characterized by a biased stability-flexibility tradeoff, dopaminergic drugs might push the system toward more flexibility but less stability by restoring dopamine levels in the striatum but detrimentally ‘‘overdosing’’ dopamine levels in the relatively intact prefrontal cortex. In ADHD, by contrast, dopaminergic drugs might have the exact opposite effect, pushing the system toward greater stability by restoring prefrontal dopamine levels but away from flexibility by overdosing striatal dopamine levels (Figure 7). Future work will be required to assess whether dopaminergic drug effects also vary with putative state-related fluctuations in baseline tone in the same way as they depend on trait-related differences in baseline tone. If so, then dopaminergic drugs might disrupt rather than enhance contextdependent arbitration of cognitive strategies in healthy brains despite having a global enhancing effect in systems that are otherwise maladaptively biased to one state at the expense of another. In Conclusion The reviewed work illustrates three key general principles of chemical neuromodulation: regional specialization, self-regulation, and baseline dependency. While these principles likely generalize to the different ascending neuromodulatory systems, their specific implementation almost certainly varies between them. Here, I have focused on the ascending dopamine projections, which were shown to implement key computational tradeoffs, such as the stability-flexibility dilemma in the context of working memory and learning. Dopaminergic drugs have different effects by acting on distinct prefrontal and striatal systems and both the direction and extent of these effects depend on the baseline state of that system, probably reflecting in part the systems’ selfregulatory capacity to maintain equilibrium. We hypothesize that the basal tone of these different frontal and striatal projection systems varies as a function of the perceived statistics of the environment, such as its volatility, controllability, and opportunity cost, computed in medial frontal cortex. What follows is that therapeutic dopaminergic drugs might counteract the bias elicited by the perceived environment by enhancing activity in systems with low baseline tone and paradoxically reducing systems with high baseline tone. This proposal raises a wide variety of open issues, including those concerning the specific mechanisms by which this top-down control of the dopaminergic midbrain is implemented, taking into account the functional heterogeneity of the frontal cortex and the diversity of cortical inputs to the midbrain, as well as the degree to which dopaminergic drug truly act as cognitive enhancers in healthy brains. It is these questions that will be core in the future of cognitive neurochemistry.

ACKNOWLEDGMENTS R.C. is supported by the James McDonnell scholar Award and the Netherlands Organisation for Scientific Research (NWO): Vici Award (453.14.005). Thanks are due to my mentors Trevor Robbins and Mark D’Esposito as well as my collaborators and lab members. REFERENCES Aarts, E., Roelofs, A., Franke, B., Rijpkema, M., Ferna´ndez, G., Helmich, R.C., and Cools, R. (2010). Striatal dopamine mediates the interface between motivational and cognitive control in humans: evidence from genetic imaging. Neuropsychopharmacology 35, 1943–1951. Aarts, E., Helmich, R.C., Janssen, M.J.R., Oyen, W.J.G., Bloem, B.R., and Cools, R. (2012). Aberrant reward processing in Parkinson’s disease is associated with dopamine cell loss. Neuroimage 59, 3339–3346. Aarts, E., Wallace, D.L., Dang, L.C., Jagust, W.J., Cools, R., and D’Esposito, M. (2014). Dopamine and the cognitive downside of a promised bonus. Psychol. Sci. 25, 1003–1009. Agid, Y., Ruberg, M., Javoy-Agid, F., Hirsch, E., Raisman-Vozari, R., Vyas, S., Faucheux, B., Michel, P., Kastner, A., Blanchard, V., et al. (1993). Are dopaminergic neurons selectively vulnerable to Parkinson’s disease? Adv. Neurol. 60, 148–164. Alexander, G.E., DeLong, M.R., and Strick, P.L. (1986). Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 9, 357–381. Amat, J., Baratta, M.V., Paul, E., Bland, S.T., Watkins, L.R., and Maier, S.F. (2005). Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus. Nat. Neurosci. 8, 365–371. Arnsten, A.F.T. (2013). The neurobiology of thought: the groundbreaking discoveries of Patricia Goldman-Rakic 1937-2003. Cereb. Cortex 23, 2269–2281. Arnsten, A.F.T., Wang, M.J., and Paspalas, C.D. (2012). Neuromodulation of thought: flexibilities and vulnerabilities in prefrontal cortical network synapses. Neuron 76, 223–239. Aston-Jones, G., and Cohen, J.D. (2005). An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450. Bari, A., and Aston-Jones, G. (2013). Atomoxetine modulates spontaneous and sensory-evoked discharge of locus coeruleus noradrenergic neurons. Neuropharmacology 64, 53–64. Behrens, T.E., Woolrich, M.W., Walton, M.E., and Rushworth, M.F. (2007). Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221. Bentley, P., Driver, J., and Dolan, R.J. (2011). Cholinergic modulation of cognition: insights from human pharmacological functional neuroimaging. Prog. Neurobiol. 94, 360–388. Berridge, C.W., and Waterhouse, B.D. (2003). The locus coeruleus-noradrenergic system: modulation of behavioral state and state-dependent cognitive processes. Brain Res. Brain Res. Rev. 42, 33–84. Bloemendaal, M., van Schouwenburg, M.R., Miyakawa, A., Aarts, E., D’Esposito, M., and Cools, R. (2015). Dopaminergic modulation of distracter-resistance and prefrontal delay period signal. Psychopharmacology (Berl.) 232, 1061–1070. Botvinick, M.M., Cohen, J.D., and Carter, C.S. (2004). Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8, 539–546. Boureau, Y.-L., and Dayan, P. (2011). Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 36, 74–97. Boureau, Y.-L., Sokol-Hessner, P., and Daw, N.D. (2015). Deciding How To Decide: Self-Control and Meta-Decision Making. Trends Cogn. Sci. 19, 700–710. Bouret, S., and Sara, S.J. (2005). Network reset: a simplified overarching theory of locus coeruleus noradrenaline function. Trends Neurosci. 28, 574–582.

Neuron 104, October 9, 2019 127

Neuron

Review Braver, T.S., and Cohen, J.D. (1999). Dopamine, cognitive control, and schizophrenia: the gating model. Prog. Brain Res. 121, 327–349. Broadway, J.M., Frank, M.J., and Cavanagh, J.F. (2018). Dopamine D2 agonist affects visuospatial working memory distractor interference depending on individual differences in baseline working memory span. Cogn. Affect. Behav. Neurosci. 18, 509–520.

Cools, R., Frank, M.J., Gibbs, S.E., Miyakawa, A., Jagust, W., and D’Esposito, M. (2009). Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration. J. Neurosci. 29, 1538–1543. Cools, R., Miyakawa, A., Sheridan, M., and D’Esposito, M. (2010). Enhanced frontal function in Parkinson’s disease. Brain 133, 225–233.

Brozoski, T.J., Brown, R.M., Rosvold, H.E., and Goldman, P.S. (1979). Cognitive deficit caused by regional depletion of dopamine in prefrontal cortex of rhesus monkey. Science 205, 929–932.

Cools, R., Nakamura, K., and Daw, N.D. (2011). Serotonin and dopamine: unifying affective, activational, and decision functions. Neuropsychopharmacology 36, 98–113.

Buckholtz, J.W., Treadway, M.T., Cowan, R.L., Woodward, N.D., Li, R., Ansari, M.S., Baldwin, R.M., Schwartzman, A.N., Shelby, E.S., Smith, C.E., et al. (2010). Dopaminergic network differences in human impulsivity. Science 329, 532.

Cragg, S.J. (2003). Variable dopamine release probability and short-term plasticity between functional domains of the primate striatum. J. Neurosci. 23, 4378–4385.

Cameron, I.G.M., Watanabe, M., Pari, G., and Munoz, D.P. (2010). Executive impairment in Parkinson’s disease: response automaticity and task switching. Neuropsychologia 48, 1948–1957. Cano-Colino, M., Almeida, R., Gomez-Cabrero, D., Artigas, F., and Compte, A. (2014). Serotonin regulates performance nonmonotonically in a spatial working memory network. Cereb. Cortex 24, 2449–2463. Cenci, M.A. (2014). Presynaptic Mechanisms of l-DOPA-Induced Dyskinesia: The Findings, the Debate, and the Therapeutic Implications. Front. Neurol. 5, 242. Chatham, C.H., and Badre, D. (2015). Multiple gates on working memory. Curr. Opin. Behav. Sci. 1, 23–31. Christophel, T.B., Iamshchinina, P., Yan, C., Allefeld, C., and Haynes, J.-D. (2018). Cortical specialization for attended versus unattended working memory. Nat. Neurosci. 21, 494–496. Chudasama, Y., and Robbins, T.W. (2004). Dopaminergic modulation of visual attention and working memory in the rodent prefrontal cortex. Neuropsychopharmacology 29, 1628–1636. Clark, K.L., and Noudoost, B. (2014). The role of prefrontal catecholamines in attention and working memory. Front. Neural Circuits 8, 33.

Cragg, S.J., Hille, C.J., and Greenfield, S.A. (2000). Dopamine release and uptake dynamics within nonhuman primate striatum in vitro. J. Neurosci. 20, 8209–8217. Crockett, M.J., Clark, L., and Robbins, T.W. (2009). Reconciling the role of serotonin in behavioral inhibition and aversion: acute tryptophan depletion abolishes punishment-induced inhibition in humans. J. Neurosci. 29, 11993–11999. Crofts, H.S., Dalley, J.W., Collins, P., Van Denderen, J.C., Everitt, B.J., Robbins, T.W., and Roberts, A.C. (2001). Differential effects of 6-OHDA lesions of the frontal cortex and caudate nucleus on the ability to acquire an attentional set. Cereb. Cortex 11, 1015–1026. D’Esposito, M., and Postle, B.R. (2015). The cognitive neuroscience of working memory. Annu. Rev. Psychol. 66, 115–142. Dayan, P. (2012). Twenty-five lessons from computational neuromodulation. Neuron 76, 240–256. Dayan, P., and Huys, Q.J.M. (2008). Serotonin, inhibition, and negative mood. PLoS Comput. Biol. 4, e4. Dayan, P., and Yu, A.J. (2006). Phasic norepinephrine: a neural interrupt signal for unexpected events. Network 17, 335–350.

Cocker, P.J., Hosking, J.G., Benoit, J., and Winstanley, C.A. (2012). Sensitivity to cognitive effort mediates psychostimulant effects on a novel rodent cost/ benefit decision-making task. Neuropsychopharmacology 37, 1825–1837.

€ller, U., Clark, L., van Loon, A., Cools, R., and Robbins, T.W. Dodds, C.M., Mu (2008). Methylphenidate has differential effects on blood oxygenation leveldependent signal related to cognitive subprocesses of reversal learning. J. Neurosci. 28, 5976–5982.

Cohen, J.D. (2017). Cognitive Control: Core Constructs and Current Considerations. In The Wiley Handbook of Cognitive Control, T. Egner, ed. (WileyBlackwell), pp. 1–28.

Dorfman, H.M., and Gershman, S.J. (2019). Controllability Governs the Balance Between Pavlovian and Instrumental Action Selection. bioRxiv 596577, https://doi.org/10.1101/596577.

Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B., and Uchida, N. (2012). Neurontype-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88.

During, M.J., Acworth, I.N., and Wurtman, R.J. (1988). Phenylalanine administration influences dopamine release in the rat’s corpus striatum. Neurosci. Lett. 93, 91–95.

Collins, A.G.E., and Frank, M.J. (2014). Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol. Rev. 121, 337–366.

Durstewitz, D., and Seamans, J.K. (2008). The dual-state theory of prefrontal cortex dopamine function with relevance to catechol-o-methyltransferase genotypes and schizophrenia. Biol. Psychiatry 64, 739–749.

Collins, P., Wilkinson, L.S., Everitt, B.J., Robbins, T.W., and Roberts, A.C. (2000). The effect of dopamine depletion from the caudate nucleus of the common marmoset (Callithrix jacchus) on tests of prefrontal cognitive function. Behav. Neurosci. 114, 3–17.

Elston, T.W., and Bilkey, D.K. (2017). Anterior Cingulate Cortex Modulation of the Ventral Tegmental Area in an Effort Task. Cell Rep. 19, 2220–2230.

Cools, R. (2016). The costs and benefits of brain dopamine for cognitive control. Wiley Interdiscip. Rev. Cogn. Sci. 7, 317–329.

Engelhard, B., Finkelstein, J., Cox, J., Fleming, W., Jang, H.J., Ornelas, S., Koay, S.A., Thiberge, S.Y., Daw, N.D., Tank, D.W., and Witten, I.B. (2019). Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons. Nature 570, 509–513.

Cools, R., and D’Esposito, M. (2011). Inverted-U-shaped dopamine actions on human working memory and cognitive control. Biol. Psychiatry 69, e113–e125.

Fales, C.L., Vanek, Z.F., and Knowlton, B.J. (2006). Backward inhibition in Parkinson’s disease. Neuropsychologia 44, 1041–1049.

Cools, R., Barker, R.A., Sahakian, B.J., and Robbins, T.W. (2001a). Mechanisms of cognitive set flexibility in Parkinson’s disease. Brain 124, 2503–2512.

Fallon, S.J., van der Schaaf, M.E., Ter Huurne, N., and Cools, R. (2017). The Neurocognitive Cost of Enhancing Cognition with Methylphenidate: Improved Distractor Resistance but Impaired Updating. J. Cogn. Neurosci. 29, 652–663.

Cools, R., Barker, R.A., Sahakian, B.J., and Robbins, T.W. (2001b). Enhanced or impaired cognitive function in Parkinson’s disease as a function of dopaminergic medication and task demands. Cereb. Cortex 11, 1136–1143. Cools, R., Sheridan, M., Jacobs, E., and D’Esposito, M. (2007). Impulsive personality predicts dopamine-dependent changes in frontostriatal activity during component processes of working memory. J. Neurosci. 27, 5506–5514.

128 Neuron 104, October 9, 2019

Finn, E.S., Huber, L., Jangraw, D.C., and Bandettini, P.A. (2018). Layer-dependent activity in human prefrontal cortex during working memory. bioRxiv. https://doi.org/10.1101/425249. Floresco, S.B. (2013). Prefrontal dopamine and behavioral flexibility: shifting from an ‘‘inverted-U’’ toward a family of functions. Front. Neurosci. 7, 62.

Neuron

Review Floresco, S.B., West, A.R., Ash, B., Moore, H., and Grace, A.A. (2003). Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nat. Neurosci. 6, 968–973. Frank, M.J., and Badre, D. (2012). Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. Cereb. Cortex 22, 509–526. Frank, M.J., Loughry, B., and O’Reilly, R.C. (2001). Interactions between the frontal cortex and basal ganglia in working memory: A computational model. Cogn. Affect. Behav. Neurosci. 1, 137–160. Frank, M.J., and O’Reilly, R.C. (2006). A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav. Neurosci. 120, 497–517. Frobo¨se, M.I., and Cools, R. (2018). Chemical neuromodulation of cognitive control avoidance. Curr. Opin. Behav. Sci. 22, 121–127. Frobo¨se, M.I., Swart, J.C., Cook, J.L., Geurts, D.E.M., den Ouden, H.E.M., and Cools, R. (2018). Catecholaminergic modulation of the avoidance of cognitive control. J. Exp. Psychol. Gen. 147, 1763–1781. Frobo¨se, M.I., Westbrook, A., Bloemendaal, M., Aarts, E., and Cools, R. (2019). Catecholaminergic modulation of the cost of cognitive control in healthy older adults. PsyArXiv. https://doi.org/10.31234/osf.io/kypz3. Gan, J.O., Walton, M.E., and Phillips, P.E.M. (2010). Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat. Neurosci. 13, 25–27. Geurts, D.E.M., Huys, Q.J.M., den Ouden, H.E.M., and Cools, R. (2013). Serotonin and aversive Pavlovian control of instrumental behavior in humans. J. Neurosci. 33, 18932–18939. Goldman-Rakic, P.S. (1995). Cellular basis of working memory. Neuron 14, 477–485. Gotham, A.M., Brown, R.G., and Marsden, C.D. (1986). Levodopa treatment may benefit or impair ‘‘frontal’’ function in Parkinson’s disease. Lancet 2, 970–971. Grace, A.A. (2000). The tonic/phasic model of dopamine system regulation and its implications for understanding alcohol and psychostimulant craving. Addiction 95 (Suppl 2 ), S119–S128. Grenhoff, J., Nisell, M., Ferre´, S., Aston-Jones, G., and Svensson, T.H. (1993). Noradrenergic modulation of midbrain dopamine cell firing elicited by stimulation of the locus coeruleus in the rat. J. Neural Transm. (Vienna) 93, 11–25. Guitart-Masip, M., Duzel, E., Dolan, R., and Dayan, P. (2014). Action versus valence in decision making. Trends Cogn. Sci. 18, 194–202. Gurney, K., Prescott, T.J., and Redgrave, P. (2001). A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern. 84, 401–410. Haber, S.N., Fudge, J.L., and McFarland, N.R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci. 20, 2369–2382. Hamid, A.A., Pettibone, J.R., Mabrouk, O.S., Hetrick, V.L., Schmidt, R., Vander Weele, C.M., Kennedy, R.T., Aragona, B.J., and Berke, J.D. (2016). Mesolimbic dopamine signals the value of work. Nat. Neurosci. 19, 117–126. Hamid, A.A., Frank, M.J., and Moore, C.I. (2019). Dopamine waves as a mechanism for spatiotemporal credit assignment. bioRxiv. https://doi.org/10.1101/ 729640. Hazy, T.E., Frank, M.J., and O’Reilly, R.C. (2007). Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362, 1601–1613. Hermans, E.J., van Marle, H.J.F., Ossewaarde, L., Henckens, M.J.A.G., Qin, S., van Kesteren, M.T.R., Schoots, V.C., Cousijn, H., Rijpkema, M., Oostenveld, R., and Ferna´ndez, G. (2011). Stress-Related Noradrenergic Activity Prompts Large-Scale Neural Network Reconfiguration. Science 334, 1151–1153. Hernaus, D., Casales Santa, M.M., Offermann, J.S., and Van Amelsvoort, T. (2017). Noradrenaline transporter blockade increases fronto-parietal func-

tional connectivity relevant for working memory. Eur. Neuropsychopharmacol. 27, 399–410. Hosking, J.G., Lam, F.C.W., and Winstanley, C.A. (2014). Nicotine increases impulsivity and decreases willingness to exert cognitive effort despite improving attention in ‘‘slacker’’ rats: insights into cholinergic regulation of cost/benefit decision making. PLoS ONE 9, e111580. Hosking, J.G., Floresco, S.B., and Winstanley, C.A. (2015). Dopamine antagonism decreases willingness to expend physical, but not cognitive, effort: a comparison of two rodent cost/benefit decision-making tasks. Neuropsychopharmacology 40, 1005–1015. Iversen, L., Bloom, F.E., and Roth, R.H. (2009). Introduction to Neuropsychopharmacology (Oxford University Press). Joshi, S., Li, Y., Kalwani, R.M., and Gold, J.I. (2016). Relationships between pupil diameter and neuronal activity in the locus coeruleus, colliculi, and cingulate cortex. Neuron 89, 221–234. Juechems, K., and Summerfield, C. (2019). Where Does Value Come From? Trends Cogn. Sci. 23, 836–850. Kim, J., Zhang, K., Cai, W., YorkWilliams, S., Ua Cruadhlaoich, M.A.I., Llanes, S., Menon, V., and Poston, K.L. (2018). Dopamine-related dissociation of cortical and subcortical brain activations in cognitively unimpaired Parkinson’s disease patients OFF and ON medications. Neuropsychologia 119, 24–33. €ller, M.L.T.M., Bohnen, N.I., Sarter, M., and Lustig, C. (2019). The Kim, K., Mu cortical cholinergic system contributes to the top-down control of distraction: Evidence from patients with Parkinson’s disease. Neuroimage 190, 107–117. Kolling, N., Behrens, T.E., Mars, R.B., and Rushworth, M.F. (2012). Neural mechanisms of foraging. Science 336, 95–98. Kool, W., McGuire, J.T., Rosen, Z.B., and Botvinick, M.M. (2010). Decision making and the avoidance of cognitive demand. J. Exp. Psychol. Gen. 139, 665–682. Kosillo, P., Zhang, Y.-F., Threlfell, S., and Cragg, S.J. (2016). Cortical Control of Striatal Dopamine Transmission via Striatal Cholinergic Interneurons. Cereb. Cortex 26, 4160–4169. Kurzban, R., Duckworth, A., Kable, J.W., and Myers, J. (2013). An opportunity cost model of subjective effort and task performance. Behav. Brain Sci. 36, 661–679. Lapiz, M.D.S., and Morilak, D.A. (2006). Noradrenergic modulation of cognitive function in rat medial prefrontal cortex as measured by attentional set shifting capability. Neuroscience 137, 1039–1049. Lohani, S., Martig, A.K., Underhill, S.M., DeFrancesco, A., Roberts, M.J., Rinaman, L., Amara, S., and Moghaddam, B. (2018). Burst activation of dopamine neurons produces prolonged post-burst availability of actively released dopamine. Neuropsychopharmacology 43, 2083–2092. Mallett, R., and Lewis-Peacock, J.A. (2018). Behavioral decoding of working memory items inside and outside the focus of attention. Ann. N Y Acad. Sci. 1424, 256–267. Manohar, S.G., Chong, T.T.-J., Apps, M.A.J., Batla, A., Stamelou, M., Jarman, P.R., Bhatia, K.P., and Husain, M. (2015). Reward Pays the Cost of Noise Reduction in Motor and Cognitive Control. Curr. Biol. 25, 1707–1716. Mathys, C., Daunizeau, J., Friston, K.J., and Stephan, K.E. (2011). A bayesian foundation for individual learning under uncertainty. Front. Hum. Neurosci. 5, 39. Mattay, V.S., Goldberg, T.E., Fera, F., Hariri, A.R., Tessitore, A., Egan, M.F., Kolachana, B., Callicott, J.H., and Weinberger, D.R. (2003). Catechol O-methyltransferase val158-met genotype and individual variation in the brain response to amphetamine. Proc. Natl. Acad. Sci. USA 100, 6186–6191. McGuigan, S., Zhou, S.-H., Brosnan, M.B., Thyagarajan, D., Bellgrove, M.A., and Chong, T.T.-J. (2019). Dopamine restores cognitive motivation in Parkinson’s disease. Brain 142, 719–732. Meder, D., Kolling, N., Verhagen, L., Wittmann, M.K., Scholl, J., Madsen, K.H., Hulme, O.J., Behrens, T.E.J., and Rushworth, M.F.S. (2017). Simultaneous representation of a spectrum of dynamically changing value estimates during decision making. Nat. Commun. 8, 1942.

Neuron 104, October 9, 2019 129

Neuron

Review Mehta, M.A., Manes, F.F., Magnolfi, G., Sahakian, B.J., and Robbins, T.W. (2004). Impaired set-shifting and dissociable effects on tests of spatial working memory following the dopamine D2 receptor antagonist sulpiride in human volunteers. Psychopharmacology (Berl.) 176, 331–342. Mejias-Aponte, C.A. (2016). Specificity and impact of adrenergic projections to the midbrain dopamine system. Brain Res. 1641 (Pt B), 258–273. Meyer-Lindenberg, A., Miletich, R.S., Kohn, P.D., Esposito, G., Carson, R.E., Quarantelli, M., Weinberger, D.R., and Berman, K.F. (2002). Reduced prefrontal activity predicts exaggerated striatal dopaminergic function in schizophrenia. Nat. Neurosci. 5, 267–271. Middleton, H.C., Sharma, A., Agouzoul, D., Sahakian, B.J., and Robbins, T.W. (1999). Idazoxan potentiates rather than antagonizes some of the cognitive effects of clonidine. Psychopharmacology (Berl.) 145, 401–411. Miller, E.K., and Cohen, J.D. (2001). An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167–202. Mohebi, A., Pettibone, J.R., Hamid, A.A., Wong, J.T., Vinson, L.T., Patriarchi, T., Tian, L., Kennedy, R.T., and Berke, J.D. (2019). Dissociable dopamine dynamics for learning and motivation. Nature 570, 65–70. Montague, P.R., Dayan, P., and Sejnowski, T.J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947.

Roberts, A.C., De Salvia, M.A., Wilkinson, L.S., Collins, P., Muir, J.L., Everitt, B.J., and Robbins, T.W. (1994). 6-Hydroxydopamine lesions of the prefrontal cortex in monkeys enhance performance on an analog of the Wisconsin Card Sort Test: possible interactions with subcortical dopamine. J. Neurosci. 14, 2531–2544. Rogers, R.D., Sahakian, B.J., Hodges, J.R., Polkey, C.E., Kennard, C., and Robbins, T.W. (1998). Dissociating executive mechanisms of task control following frontal lobe damage and Parkinson’s disease. Brain 121, 815–842. Rushworth, M.F.S., Buckley, M.J., Behrens, T.E.J., Walton, M.E., and Bannerman, D.M. (2007). Functional organization of the medial frontal cortex. Curr. Opin. Neurobiol. 17, 220–227. Samanez-Larkin, G.R., Buckholtz, J.W., Cowan, R.L., Woodward, N.D., Li, R., Ansari, M.S., Arrington, C.M., Baldwin, R.M., Smith, C.E., Treadway, M.T., et al. (2013). A thalamocorticostriatal dopamine network for psychostimulant-enhanced human cognitive flexibility. Biol. Psychiatry 74, 99–105. Santana, N., and Artigas, F. (2017). Laminar and Cellular Distribution of Monoamine Receptors in Rat Medial Prefrontal Cortex. Front. Neuroanat. 11, 87. Schultz, W. (2007). Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288. Schultz, W., Dayan, P., and Montague, P.R. (1997). A neural substrate of prediction and reward. Science 275, 1593–1599.

Moustafa, A.A., Sherman, S.J., and Frank, M.J. (2008). A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism. Neuropsychologia 46, 3144–3156.

Seamans, J.K., and Yang, C.R. (2004). The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog. Neurobiol. 74, 1–58.

Muller, T.H., Mars, R.B., Behrens, T.E., and O’Reilly, J.X. (2019). Control of entropy in neural models of environmental state. eLife 8, e39404.

Seeman, P., and Madras, B. (2002). Methylphenidate elevates resting dopamine which lowers the impulse-triggered release of dopamine: a hypothesis. Behav. Brain Res. 130, 79–83.

Musslick, S., Jang, S.J., Shvartsman, M., Shenhav, A., and Cohen, J.D. (2018). Constraints associated with cognitive control and the stability-flexibility dilemma. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society (Cognitive Science Society), pp. 804–809. Nagano-Saito, A., Leyton, M., Monchi, O., Goldberg, Y.K., He, Y., and Dagher, A. (2008). Dopamine depletion impairs frontostriatal functional connectivity during a set-shifting task. J. Neurosci. 28, 3697–3706.

Seo, C., Guru, A., Jin, M., Ito, B., Sleezer, B.J., Ho, Y.-Y., Wang, E., Boada, C., Krupa, N.A., Kullakanda, D.S., et al. (2019). Intense threat switches dorsal raphe serotonin neurons to a paradoxical operational mode. Science 363, 538–542. Servan-Schreiber, D., Printz, H., and Cohen, J.D. (1990). A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 249, 892–895.

Nassar, M.R., Wilson, R.C., Heasly, B., and Gold, J.I. (2010). An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 30, 12366–12378.

Sesack, S.R., and Carr, D.B. (2002). Selective prefrontal cortex inputs to dopamine cells: implications for schizophrenia. Physiol. Behav. 77, 513–517.

Nassar, M.R., Rumsey, K.M., Wilson, R.C., Parikh, K., Heasly, B., and Gold, J.I. (2012). Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15, 1040–1046.

Sesack, S.R., Hawrylak, V.A., Guido, M.A., and Levey, A.I. (1998). Cellular and subcellular localization of the dopamine transporter in rat cortex. Adv. Pharmacol. 42, 171–174.

Niv, Y., Daw, N.D., Joel, D., and Dayan, P. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 191, 507–520.

Shenhav, A., Botvinick, M.M., and Cohen, J.D. (2013). The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 79, 217–240.

O’Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., and Dolan, R.J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454.

Shine, J.M., van den Brink, R.L., Hernaus, D., Nieuwenhuis, S., and Poldrack, R.A. (2018). Catecholaminergic manipulation alters dynamic network topology across cognitive states. Netw Neurosci 2, 381–396.

O’Reilly, R.C., Munakata, Y., Frank, M.J., and Hazy, T.E. (2012). CCNBook/Executive - Computational Cognitive Neuroscience Wiki. https://grey.colorado. edu/CompCogNeuro/index.php/CCNBook/Executive.

Strafella, A.P., Paus, T., Barrett, J., and Dagher, A. (2001). Repetitive transcranial magnetic stimulation of the human prefrontal cortex induces dopamine release in the caudate nucleus. J. Neurosci. 21, RC157.

Ogawa, S.K., Cohen, J.Y., Hwang, D., Uchida, N., and Watabe-Uchida, M. (2014). Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems. Cell Rep. 8, 1105–1118.

Swainson, R., Rogers, R.D., Sahakian, B.J., Summers, B.A., Polkey, C.E., and Robbins, T.W. (2000). Probabilistic learning and reversal deficits in patients with Parkinson’s disease or frontal or temporal lobe lesions: possible adverse effects of dopaminergic medication. Neuropsychologia 38, 596–612.

Ott, T., and Nieder, A. (2019). Dopamine and Cognitive Control in Prefrontal Cortex. Trends Cogn. Sci. 23, 213–234. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R.J., and Frith, C.D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442, 1042–1045.

Swart, J.C., Frobo¨se, M.I., Cook, J.L., Geurts, D.E., Frank, M.J., Cools, R., and den Ouden, H.E. (2017). Catecholaminergic challenge uncovers distinct Pavlovian and instrumental mechanisms of motivated (in)action. eLife 6, e22169.

Pollux, P.M.J. (2004). Advance preparation of set-switches in Parkinson’s disease. Neuropsychologia 42, 912–919.

ter Huurne, N., Fallon, S.J., van Schouwenburg, M., van der Schaaf, M., Buitelaar, J., Jensen, O., and Cools, R. (2015). Methylphenidate alters selective attention by amplifying salience. Psychopharmacology (Berl.) 232, 4317–4323.

Pycock, C.J., Kerwin, R.W., and Carter, C.J. (1980). Effect of lesion of cortical dopamine terminals on subcortical dopamine receptors in rats. Nature 286, 74–76.

Timmer, M.H.M., Aarts, E., Esselink, R.A.J., and Cools, R. (2018). Enhanced motivation of cognitive control in Parkinson’s disease. Eur. J. Neurosci. 48, 2374–2384.

130 Neuron 104, October 9, 2019

Neuron

Review Ueltzho¨ffer, K., Armbruster-Genc¸, D.J.N., and Fiebach, C.J. (2015). Stochastic Dynamics Underlying Cognitive Stability and Flexibility. PLoS Comput. Biol. 11, e1004331.

Varazzani, C., San-Galli, A., Gilardeau, S., and Bouret, S. (2015). Noradrenaline and dopamine neurons in the reward/effort trade-off: a direct electrophysiological comparison in behaving monkeys. J. Neurosci. 35, 7866–7877.

van den Brink, R.L., Pfeffer, T., Warren, C.M., Murphy, P.R., Tona, K.-D., van der Wee, N.J.A., Giltay, E., van Noorden, M.S., Rombouts, S.A.R.B., Donner, T.H., and Nieuwenhuis, S. (2016). Catecholaminergic Neuromodulation Shapes Intrinsic MRI Functional Connectivity in the Human Brain. J. Neurosci. 36, 7865–7876.

Wang, M., Vijayraghavan, S., and Goldman-Rakic, P.S. (2004). Selective D2 receptor actions on the functional circuitry of working memory. Science 303, 853–856.

van der Schaaf, M.E., van Schouwenburg, M.R., Geurts, D.E.M., Schellekens, A.F.A., Buitelaar, J.K., Verkes, R.J., and Cools, R. (2014). Establishing the dopamine dependency of human striatal signals during reward and punishment reversal learning. Cereb. Cortex 24, 633–642. van Holstein, M., Aarts, E., van der Schaaf, M.E., Geurts, D.E.M., Verkes, R.J., Franke, B., van Schouwenburg, M.R., and Cools, R. (2011). Human cognitive flexibility depends on dopamine D2 receptor signaling. Psychopharmacology (Berl.) 218, 567–578. van Schouwenburg, M.R., den Ouden, H.E.M., and Cools, R. (2010). The human basal ganglia modulate frontal-posterior connectivity during attention shifting. J. Neurosci. 30, 9910–9918. van Schouwenburg, M.R., Zwiers, M.P., van der Schaaf, M.E., Geurts, D.E.M., Schellekens, A.F.A., Buitelaar, J.K., Verkes, R.J., and Cools, R. (2013). Anatomical connection strength predicts dopaminergic drug effects on fronto-striatal function. Psychopharmacology (Berl.) 227, 521–531. van Schouwenburg, M.R., Onnink, A.M.H., ter Huurne, N., Kan, C.C., Zwiers, M.P., Hoogman, M., Franke, B., Buitelaar, J.K., and Cools, R. (2014). Cognitive flexibility depends on white matter microstructure of the basal ganglia. Neuropsychologia 53, 171–177. van Schouwenburg, M.R., den Ouden, H.E.M., and Cools, R. (2015). Selective attentional enhancement and inhibition of fronto-posterior connectivity by the basal ganglia during attention switching. Cereb. Cortex 25, 1527–1534.

Weintraub, D. (2019). Impulse control disorders in Parkinson’s disease: A 20-year odyssey. Mov. Disord. 34, 447–452. Westbrook, A., and Braver, T.S. (2016). Dopamine Does Double Duty in Motivating Cognitive Effort. Neuron 91, 708. Westbrook, A., Kester, D., and Braver, T.S. (2013). What is the subjective cost of cognitive effort? Load, trait, and aging effects revealed by economic preference. PLoS ONE 8, e68210. €a €tta €, J.I., Hofmans, L., Papadopetraki, Westbrook, A., van den Bosch, R., Ma D., Cools, R., and Frank, M.J. (2019). Dopamine Promotes Cognitive Effort by Biasing the Benefits versus Costs of Cognitive Work. bioRxiv. https://doi. org/10.1101/778134. Williams, G.V., and Goldman-Rakic, P.S. (1995). Modulation of memory fields by dopamine D1 receptors in prefrontal cortex. Nature 376, 572–575. Wilson, R.C., Nassar, M.R., and Gold, J.I. (2013). A mixture of delta-rules approximation to bayesian inference in change-point problems. PLoS Comput. Biol. 9, e1003150. Yu, A.J., and Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron 46, 681–692. Zilles, K., and Palomero-Gallagher, N. (2017). Multiple Transmitter Receptors in Regions and Layers of the Human Cerebral Cortex. Front. Neuroanat. 11, 78.

Neuron 104, October 9, 2019 131