- Email: [email protected]
Social norms, self-control, and the value of antisocial behavior
Available online at www.sciencedirect.com
ScienceDirect Social norms, self-control, and the value of antisocial behavior Joshua W Buckholtz1,2,3 Social norms facilitate large-scale cooperation by promoting prosocial interactions and constraining antisocial behavior. Dominant models of norm compliance emphasize the role of effortful, capacity-limited inhibitory control in prosocial cooperation. Similarly, clinical science has focused on inhibitory deficits as a key source of persistent norm-violating behavior. Support for an inhibition-based ‘braking success/ braking failure’ (BSBF) model is derived from evidence of dorsolateral prefrontal cortex (DLPFC) engagement during norm-guided behavior, and of DLPFC dysfunction in antisocial individuals. However, three challenges motivate an alternative explanation for links between self-control, DLPFC, and normbased behavior. Here, I propose a value-based alternative to the BSBF model, in which prosocial norm compliance and antisocial norm violations both arise from interactions between prefrontal model-based and striatal model-free decisionmaking systems. Addresses 1 Department of Psychology, United States 2 Center for Brain Science, Harvard University, United States 3 Department of Psychiatry, Massachusetts General Hospital, United States Corresponding author: Buckholtz, Joshua W. ([email protected])
(prosocial) behavior [3]. However, while norm compliance is extensive, it is far from universal, and our understanding of the mechanisms that drive norm violations (antisocial behavior) remains limited. Given the estimated societal costs of antisocial behavior — upwards of $1 trillion annually, by some accounts [4] — our relatively poor insight into its biological origins impedes treatment development and limits our ability to make informed policy decisions.
Social norms and self-control Norm compliance often involves engaging in other-regarding behavior that is counter to an agent’s immediate selfish interests. Thus, many have argued that norm-based behavior requires ‘self-control,’ [5,6] a term that is used somewhat interchangeably with ‘self-regulation,’ ‘impulse control,’ ‘cognitive control,’ and ‘executive function’. If prosocial behavior requires effortful self-control, it would thus naturally follow that persistent norm violating (antisocial) behavior arises from self-control failure, manifest as an inability to appropriately inhibit self-interested decision-making. Indeed, this is the conceptual foundation for a highly influential theory of crime (the ‘Self-Control Theory of Crime’) [7], and clinical science research has largely focused on identifying the cognitive and neural bases of self-control deficits in antisocial individuals [8].
Current Opinion in Behavioral Sciences 2015, 3:122–129 This review comes from a themed issue on Social behavior Edited by Molly J Crockett and Amy Cuddy For a complete overview see the Issue and the Editorial Available online 3rd April 2015 http://dx.doi.org/10.1016/j.cobeha.2015.03.004 2352-1546/# 2015 Elsevier Ltd. All rights reserved.
Our social landscape is shaped by social norms, a set of prescriptive and proscriptive rules that comprise the ‘grammar of social interaction’ [1] for Homo sapiens. Injunctions against dishonesty in exchange, physical harm, and theft promote social stability, community peace and economic prosperity. Thus, many have argued that norms were key for enabling the ultrasociality that is a signature of our species. In turn, their ability to facilitate cooperation depends on the widespread maintenance of norm compliance, accomplished by sanctioning norm violations [2] and by rewarding norm consistent Current Opinion in Behavioral Sciences 2015, 3:122–129
The ‘brakes’ metaphor of self-control The prevailing conceptualization of self-control in studies of norm-based cooperative behavior is that of an effortful, capacity-limited resource, used to inhibit automatic or prepotent responses. Social psychologists and behavioral economists have argued that cooperation arises from the capacity to actively override selfish impulses in order to promote selection of an alternative norm-consistent choice option [5,6]. Similarly, clinical scientists have proposed that antisocial behavior results from a deficit in the capacity to actively inhibit the execution of prepotent responses to threat and/or reward associated stimuli [8,9]. Together, these two perspectives comprise a dominant viewpoint wherein ‘braking success’ produces prosocial behavior and ‘braking failure’ leads to antisocial behavior. While this ‘braking success/braking failure’ (BSBF) model of norm-guided behavior has both intuitive appeal and experimental support, recent work suggests that the BSBF model is incomplete. In what follows, I will review brain imaging findings that are often cited to support the BSBF model, highlight conceptual and empirical challenges to the model, and articulate an alternative framework grounded in decision science. www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
Brakes, localized?: dorsolateral prefrontal cortex As noted above, many have suggested that norm-based prosocial behavior involves the deployment of self-control to actively inhibit automatic self-interested responses. Neurobiological studies of norm-based behavior have largely adopted this conceptualization, identifying the locus of the self-control ‘braking’ system in lateral prefrontal cortex. For example, dorsolateral prefrontal cortex (DLPFC) activation has been consistently noted in fMRI tasks that assess prosocial (‘altruistic’) punishment of norm violations, as well as in studies of cooperation, fairness, and social norm compliance [2,10]. A recent meta-analysis suggests that DLPFC activity during normbased decision-making reflects the need for ‘cognitive control from a reflective and deliberate System 2 to resolve conflict by . . . over-riding self-interest’ [11]. Of note, disruptive brain stimulation to DLPFC reduces both prosocial norm-enforcement [12] and norm compliance [13]. As other studies have shown that this region is important for response inhibition, some have inferred that the ability to actively override selfish or otherwise maladaptive responses is a cognitive sin qua non for prosocial behavior [5]. Similarly, it is widely assumed that antisocial behavior results from impaired inhibitory control [8,9]. This perspective is supported by behavioral and neurobiological evidence that antisocial individuals show deficient cognitive control in the context of heightened reactivity to threat and/or reward cues. Antisocial offenders exhibit reduced gray matter volume and cortical thickness within DLPFC [8,14], as well as compromised DLPFC activation during classic neuropsychological indices of inhibitory control [14,15]. By contrast, antisocial individuals appear to have relatively exaggerated responses to threat stimuli (within the amygdala) and reward cues (within the striatum) [16,17–19]. Interpreted through the lens of the BSBF model, such findings are taken as evidence that antisocial behavior occurs when bottom up ‘affective’ signals activate or generate a prepotent behavioral response that is inadequately inhibited by top down ‘cognitive’ resources due to poor prefrontal control. Taken together, a large body of work provides convergent support for the idea that DLPFC is causally involved in norm compliance, and that antisocial individuals exhibit deficits in DLPFC structure and function. While such findings appear to fit the BSBF model, three key challenges suggest that an alternative explanation for the link between DLPFC and pro-/anti-social behavior merits consideration.
Challenge 1: prosociality, no brakes required The first challenge comes from recent findings that prosocial behavior does not necessarily require the controlled inhibition of self-interested or maladaptive www.sciencedirect.com
123
automatic responses. Cooperation in both laboratory and real-life contexts is highest under conditions that promote fast, intuitive responding [20]. Developmental and cross-species data offer further support for the intuitive, automatic nature of prosociality: chimpanzees — even very young ones — engage in costly helping behavior, and prosocial responding in human children emerges prior to the development of inhibitory control [21]. Finally, neuroimaging work suggests that costly prosocial behavior may be facilitated by the heightened value assigned to prosocial choice options relative to available alternatives, rather than the active inhibition of automatic antisocial responses [22]. Together, these findings imply that norm-consistent behavior does not necessarily require effortful inhibition.
Challenge 2: there is no such (single) thing as ‘self-control’ The construct of self-control has a deceptively complex latent architecture. There is overwhelming evidence for multidimensionality, with several distinct cognitive components that can be grouped into at least two broad domains [23] (Figure 1). Capacities encompassed by the domain of ‘response inhibition’ enable agents to use internal representations or external cues to inhibit prepotent motor responses. A second domain contains processes that promote adaptive choice behavior by estimating the subjective value of different choice options and selecting actions based on optimal utility. Lesion and drug studies in animals, along with factor analytic work in humans, converge to suggest that response inhibition and value-based decision making are largely distinct cognitive capacities [24] with dissociable neurobiological substrates [23]. Despite the evident heterogeneity of ‘selfcontrol,’ it is often conflated with response inhibition or assessed by trait measures that map poorly to identifiable cognitive processes [25]. The net result of this construct neglect is two-fold. First, research on the relationship between self-control and norm-guided behavior has largely focused on response inhibition, while other facets of self-control remain relatively unexplored. Second, DLPFC engagement (or dysfunction) is often presumed to reflect inhibitory control, leaving alternative explanations for the role of DLPFC in pro-social and anti-social behavior relatively unexamined.
Challenge 3: DLPFC reconsidered Early work on the role of DLPFC in adaptive behavior was largely focused on inhibitory control: lesion, electrophysiology, and functional imaging studies demonstrated that DLPFC is crucial for the ability to use goals, task rules and external cues to inhibit the execution of contextually inappropriate motor responses [26–32]. However, recent brain imaging and electrophysiological data have broadened the view of DLPFC to include a crucial role in value-based decision-making. Lateral PFC has been shown to be important for integrating value-related Current Opinion in Behavioral Sciences 2015, 3:122–129
124 Social behavior
Figure 1
“Self-Control”
Value-Based
Action
Decision-Making
Control Response Inhibition
Model-Based
Model-Free
Cost-Benefit
Action Response Interference Response Cancellation Suppression Suppression Selection Continuous Performance
Probabilistic Response Reversal
Stop-Signal
n-Armed Bandit Tasks Delay Discounting Mixed Gambles Effort Discounting
Stroop
Go/NoGo
Flanker
Choice Reaction Time
Simon Task Anti-Saccade
2-Stage Markov Decision Task Reinforcer Devaluation Current Opinion in Behavioral Sciences
Self-Control is multidimensional. While commonly reduced to ‘response inhibition,’ the construct of self-control encompasses at least two broad domains of cognition, each containing several distinct cognitive processes. Here, I highlight the dissociation between two self-control domains: action control and value-based decision-making. Experimental paradigms are grouped according to the domain of cognition that they putatively access and the domain-specific processes that each measures.
decision variables at the time of choice, particularly when predicted action values must be calculated from multiple, simultaneously varying parameters (e.g. magnitude, probability, delay, uncertainty, and effort) [33–35]. This integrated signal may be incorporated into a process of episodic prospection that simulates outcomes for an array of action options, permitting a comparison of expected value for multiple potential actions [36–39]. Finally, DLPFC appears to be important for representing associations between abstract rules and expected outcome values [40]. By linking information about rules to reward outcomes, DLPFC may be able to generate a representation of the value of rules — a feature that may be particularly important for norm-guided decision making.
DLPFC modulates striatal value signals The challenges outlined above undermine the notion that effortful cognitive control is required for prosocial behavior, highlight the conflation of self-control with ‘response inhibition,’ and expand the role of DLPFC to encompass value-related computations. How, then, do we reconcile these challenges with clear evidence that, firstly, DLPFC is crucial for norm-guided prosocial Current Opinion in Behavioral Sciences 2015, 3:122–129
behavior and secondly, the structure and function of DLPFC is compromised in antisocial individuals? I propose that the ability of DLPFC representations to modulate choice-related value signals explains the prominent role for this region in both norm-based cooperation and antisocial behavior. As indicated above, DLPFC represents behavioral goals and abstract rule-value associations, integrates multiple streams of cost–benefit information, and prospectively maps action options to future outcome values. These higher order value-related processes may guide adaptive behavior by modulating action value representations in regions that are more directly involved in action selection, such as the striatum. DLPFC projections to the striatum are robust and wellcharacterized; these corticostriatal afferents comprise a crucial mechanism for modulating meso-striatal DA transmission and choice behavior [41]. Exposure to unpredicted rewards and reward-predicting cues induces burst firing in midbrain DA neurons, causing transient high-amplitude DA efflux from striatal terminals (the phasic DA signal). Phasic DA activity in striatum guides action selection based on stimulus-value associations [42,43]. While ascending midbrain efferents convey www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
stimulus value-dependent signals to induce phasic DA release in striatum, descending glutamatergic projections arising from LPFC maintain tonic DA levels at striatal axon terminals [44,45]. In turn, prefrontally regulated tonic DA acts on inhibitory presynaptic terminal autoreceptors to dampen phasic DA release [44] (Figure 2). Thus, higher-order value related information maintained in prefrontal cortex is able to bias choice
125
behavior by modulating the ability of reward-associated cues to drive action selection via phasic DA signaling. The functional-anatomical organization of mesocorticostriatal DA circuitry suggests that that diminishing prefrontal function could lead to maladaptive decisionmaking by disrupting prefrontal regulation of phasic DA release. Brain imaging and brain stimulation studies
Figure 2
Corticostriatal > Mesostriatal
LPFC
Striatum
GluR
D1 D2 GluR Medium Spiny Neuron
Motor Ctx
D2-AR
Midbrain
MB > MF Patient > Impulsive Prosocial > Antisocial
Mesostriatal > Corticostriatal
LPFC
Striatum
GluR
D1 D2 GluR Medium Spiny Neuron
Motor Ctx
D2-AR
Midbrain
MF > MB Impulsive > Patient Antisocial > Prosocial
Current Opinion in Behavioral Sciences
Circuit model for prefrontal modulation of value signals. Stronger ascending drive boosts striatal DA responses to learned reward cues, leading to impulsive, model-free choice behavior (bottom). Stronger corticostriatal inputs dampen phasic DA transmission in striatum, facilitating optimal action selection through the adaptive, context-appropriate re-weighting of model-free value signals (top). www.sciencedirect.com
Current Opinion in Behavioral Sciences 2015, 3:122–129
126 Social behavior
support this hypothesis. First, DLPFC activity is negatively associated with striatal DA transmission in humans [46,47], and disrupting DLPFC function induces striatal DA release [48], potentiates striatal fMRI responses to reward cues [49] and biases behavior toward impulsive reward-seeking [50]. Second, measures of functional and anatomical connectivity between DLPFC and striatum predict delay-based choice biases and striatal subjective value signals during cost–benefit decision making [51]. Such data are consistent with the notion that DLPFC guides behavior by modulating the value assigned to action options. Finally, changing the valuation of choice options (via cognitive reframing) is sufficient to induce far-sighted (patient) inter-temporal decision-making, even in the absence of effortful inhibitory control [52]. This finding accords well with data by Cooper and colleagues showing that impulsive decision-making can be decreased by cognitive manipulations that amplify striatal value signals for delayed (versus immediate) choice options [53]. On the whole, these studies suggest that DLPFC facilitates optimal decision-making by adaptively reweighting striatal action value signals, rather than by inhibiting action execution after valuation and selection have already occurred.
Model-free and model-based decisionmaking It may be useful to evaluate the putative value-modulating function of DLPFC described above in the context of an influential computational framework for action control. This approach distinguishes between two systems for behavioral control: a retrospective model-free system, which selects actions solely on the basis of reinforcement history, and a prospective model-based system that integrates reward experience with rules and context representations to create a map (i.e. model) of the external world [54]. From this perspective, choice behavior is driven either by learned stimulus-response ‘habits’ encoded in the model-free system, or by a process that computes optimal action values by prospectively mapping action options to future outcomes [55]. Model-free decision-making is supported by phasic DA transmission within the midbrain-striatum circuit described previously; by contrast, DLPFC (along with other cortical regions) appears crucial for model-based learning and choice [56,57]. Of note, the relationship between these two systems is likely interactive or cooperative, rather than directly competitive; experience can organize the structure of world models, and these models can in turn be used to modify learned action values [55]. On the whole, his account provides a useful framework for considering the role of value-based decisionmaking in norm-guided behavior.
A value-based framework for norm-guided choice and antisocial behavior Consider a scenario in which an agent is faced with choosing between self-interested (antisocial) gain and Current Opinion in Behavioral Sciences 2015, 3:122–129
prosocial action. Steve leaves a bank carrying an envelope filled with cash, which he quickly places it into his coat pocket and begins walking down the street. Unbeknownst to Steve, the envelope falls through a hole in his pocket and lands on the sidewalk, opening to reveal its contents. The street is deserted, but for a single bystander, John, who observes this turn of events. John is faced with a choice: pick up the envelope and enrich himself by walking off with it, or return it to its rightful owner. I argue that John’s ability to select the norm-consistent action depends, in part, on the interaction between prefrontal model-based representations and meso-striatal model-free action value signals. John will have acquired, via a lifetime of instrumental learning, a robust association between the possession of money and the consumption of primary rewards. All things being equal, actions linked to monetary gain (e.g. picking up an envelope filled with cash) will have, a priori, higher value compared to actions that are not (e.g. giving away an envelope filled with cash). This valuation is retrospective and contextindependent; there is nothing inherently prosocial or antisocial about selecting actions that result in the receipt of money per se. If John’s action values were driven solely by learned stimulus-response ‘habits,’ picking up the envelope and walking off with it should have the higher value. However, norms are a set of rules that can provide a model-based contextual modulation of that value vis-a´-vis other choice options. John’s world model incorporates cognitive information about social rules (e.g. proscriptive norms against taking other people’s property), rule-value associations based on first-hand and vicarious learning (e.g. people who take from others are ‘bad’, people who help others are virtuous), and information about the consequences of rule violations (e.g. people who steal go to jail). A model-based strategy will use these representations to make prospection-based cognitive predictions about the value of each action. This process may ultimately strengthen the value weight for norm-compliant action options, despite the learned value of selecting actions that result in self-interested gains. Findings of DLPFC dysfunction in antisocial offenders can be reinterpreted within this framework. I propose that functional and structural deficits in DLPFC predispose antisocial behavior by dysregulating striatal valuation processes, rather than (or perhaps in addition to) disrupting response inhibition. In such a model, weaker modelbased cortical modulation of model-free striatal value signals would impair adaptive behavior by limiting the ability of goal-based prospective representations to reweight actions values. Relatively weaker glutamatergic input into DLPFC would result in lower tonic DA, and thus greater control of action selection by learned reward signals encoded by phasic DA transmission (Figure 2). As noted, DLPFC is crucial for representing goals, delayed reward values, rule-values and prospection-based inferences about outcomes. Prefrontal impairment may lead to www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
the construction of a world model that is relatively impoverished, or to a failure to appropriately integrate modelbased inferences about outcomes into decision-making when faced with a learned reward cue. In either case, model-based computations would not be able to appropriately modify the value assigned to a norm-consistent action when model-free learning gives a strong value weight to an antisocial response. Consistent with the notion that frontostriatal projections accomplish this modulation by regulating striatal DA signaling, stronger model-based signals in lateral PFC are associated with higher levels of tonic DA and blunted model-free prediction errors in ventral striatum [58]. Furthermore, transient inactivation of DLPFC with transcranial magnetic stimulation — a manipulation known to impair norm-based decision-making [12] and amplify model-free striatal reward signals [49] — leads to a predominance of model-free over model-based choice behavior [59].
Conclusion Taken together, the findings described herein suggest a value-based alternative to the BSBF model. Cooperative, norm-consistent behavior putatively reflects an adaptive modulation of learned action values by model-based prospective computations that incorporate rule, context, and cost information. This modulation is effected via corticostriatal afferents that adjust the gain on phasic reward signals arising from midbrain DA neuron burst firing. By contrast, antisocial behavior may occur when DLPFC dysfunction impairs the development of a robust model of the world, or when it prevents the prospective calculations generated by this model from appropriately modulating ‘downstream’ action value signals. This model makes several testable predictions. First, antisocial individuals should show a combination of diminished model-based state prediction errors and amplified model-free reward prediction errors. Second, disrupting (or enhancing) DLPFC function should drive norm-guided behavior in a lawful manner, either by preventing (or facilitating) development of an enriched world model, or by attenuating (or potentiating) the influence of model-based signals on striatal action values, without having an intervening effect on response inhibition. Third, this model predicts that aspects of cognitive function linked to DLPFC should jointly track variation in norm compliance and model-based behavioral control. Of note, DLPFC-dependent working memory capacity has robust negative associations to impulsivity and antisocial behavior. Though speculative, this link may not reflect a direct causal relationship, but rather may arise epiphenomenally from the fact that DLPFC is important for both working memory capacity and model-based responding. In other words, deficits in working memory capacity and other aspects of executive function may be ‘third variable’ markers of compromised prefrontal value modulation. This hypothesis accords with recent data www.sciencedirect.com
127
showing that processing speed, cognitive control, and other DLPFC-dependent executive capacities are positively associated with model-based behavioral control [60,61]. The value-based model proposed here is necessarily speculative and is offered to generate tractable hypotheses that can be explored in future work. Importantly, in proposing this model it is not my intention to suggest that other important factors such as empathy or response inhibition play no role at all in norm-guided behavior. Rather, the impact of such factors in behavior may be the result of an interaction with valuation processes. For example, mental state representations can modulate action value during economic choice [62], and stimulusreward learning can both facilitate and disrupt response inhibition and other aspects of executive function [63]. Discovering how individual differences in brain circuit function drive individual differences in behavior by affecting such interactions [64] will be crucial for advancing our understanding of the cognitive and neurobiological roots of prosocial and antisocial behavior.
Conflict of interest statement Nothing declared.
Acknowledgments I wish to thank Joshua Greene, Sam Gershman, Justin Martin, and Michael Treadway for insightful discussion, and the National Institute on Drug Abuse (1R03DA034126-01), the Sloan Foundation, the Brain and Behavior Research Foundation, and the MGH Center for Law, Brain and Behavior for research support.
References and recommended reading Papers of particular interest, published within the period of review, have been highlighted as: of special interest of outstanding interest 1.
Bicchieri C: The Grammar of Society: the Nature and Dynamics of Social Norms. Cambridge University Press; 2006.
2.
Buckholtz JW, Marois R: The roots of modern justice: cognitive and neural foundations of social norms and their enforcement. Nat Neurosci 2012, 15:655-661.
3.
Rand DG, Dreber A, Ellingsen T, Fudenberg D, Nowak MA: Positive interactions promote public cooperation. Science 2009, 325:1272-1275.
4.
Anderson DA: The aggregate burden of crime. J Law Econ 1999, 42:611-642.
5.
Knoch D, Fehr E: Resisting the power of temptations: the right prefrontal cortex and self-control. Ann NY Acad Sci 2007, 1104:123-134.
6.
Baumeister RF: Self-regulation, ego depletion, and inhibition. Neuropsychologia 2014, 65:313-319.
7.
Hirschi T: Self-control and crime. Handbook of Self-Regulation. Guilford Press; 2004.
8.
Dolan M: The neuropsychology of prefrontal function in antisocial personality disordered offenders with varying degrees of psychopathy. Psychol Med 2012, 42:1715-1725. Current Opinion in Behavioral Sciences 2015, 3:122–129
128 Social behavior
9.
Patrick CJ, Durbin CE, Moser JS: Reconceptualizing antisocial deviance in neurobehavioral terms. Dev Psychopathol 2012, 24:1047-1071.
10. Treadway MT et al.: Corticolimbic gating of emotion-driven punishment. Nat Neurosci 2014, 17:1270-1275. 11. Feng C, Luo YJ, Krueger F: Neural signatures of fairness-related normative decision making in the ultimatum game: a coordinate-based meta-analysis. Hum Brain Mapp 2015, 36:591-602. 12. Baumgartner T, Knoch D, Hotz P, Eisenegger C, Fehr E: Dorsolateral and ventromedial prefrontal cortex orchestrate normative choice. Nat Neurosci 2011, 14:1468-1474. 13. Ruff CC, Ugazio G, Fehr E: Changing social norm compliance with noninvasive brain stimulation. Science 2013, 342:482-484. This study used transcranial direct current stimulation (tDCS) to investigate a causal role for DLPFC in voluntary and sanction-induced norm compliance during a monetary transfer task. Up-regulation and downregulation of right prefrontal function had opposing effects on compliance, without changing mental state or fairness assessments. 14. Yang Y, Raine A: Prefrontal structural and functional brain imaging findings in antisocial, violent, and psychopathic individuals: a meta-analysis. Psychiat Res: Neuroimag 2009, 174:81-88. 15. Moeller SJ et al.: Common and distinct neural correlates of inhibitory dysregulation: stroop fMRI study of cocaine addiction and intermittent explosive disorder. J Psychiat Res 2014, 58:55-62. 16. Hyde LW, Byrd AL, Votruba-Drzal E, Hariri AR, Manuck SB: Amygdala reactivity and negative emotionality: divergent correlates of antisocial personality and psychopathy traits in a community sample. J Abnorm Psychol 2014, 123:214-224. Amygdala activation was examined in a large sample of community volunteers who were assesed for psychopathic traits and antisocial behavior. After adjusting for overlapping variance, psychopathy and antisociality had opposing relationships to threat-related amygdala reactivity. Higher psychopathy scores were associated with lower amygdala response, while higher antisociality scores were related to greater amygdala reactivity, but only after it was adjusted for in the statistical model.
26. Oishi T, Mikami A, Kubota K: Local injection of bicuculline into area 8 and area 6 of the rhesus monkey induces deficits in performance of a visual discrimination GO/NO-GO task. Neurosci Res 1995, 22:163-177. 27. Simmonds DJ, Pekar JJ, Mostofsky SH: Meta-analysis of Go/Nogo tasks demonstrating that fMRI activation associated with response inhibition is task-dependent. Neuropsychologia 2008, 46:224-232. 28. Wallis JD, Anderson KC, Miller EK: Single neurons in prefrontal cortex encode abstract rules. Nature 2001, 411:953-956. 29. Funahashi S, Chafee MV, Goldman-Rakic PS: Prefrontal neuronal activity in rhesus monkeys performing a delayed anti-saccade task. Nature 1993, 365:753-756. 30. Wallis JD, Miller EK: From rule to response: neuronal processes in the premotor and prefrontal cortex. J Neurophysiol 2003, 90:1790-1806. 31. White IM, Wise SP: Rule-dependent neuronal activity in the prefrontal cortex. Exp Brain Res 1999, 126:315-335. 32. Kramer UM et al.: The role of the lateral prefrontal cortex in inhibitory motor control. Cortex (A journal devoted to the study of the nervous system and behavior) 2013, 49:837-849. 33. Basten U, Biele G, Heekeren HR, Fiebach CJ: How the brain integrates costs and benefits during decision making. Proc Natl Acad Sci 2010, 107:21767-21772. 34. Rushworth MFS, Noonan MP, Boorman ED, Walton ME, Behrens TE: Frontal cortex and reward-guided learning and decision-making. Neuron 2011, 70:1054-1069. 35. Kim S, Lee D: Prefrontal cortex and impulsive decision making. Biol Psychiat 2011, 69:1140-1146. 36. Tsujimoto S, Sawaguchi T: Neuronal activity representing temporal prediction of reward in the primate prefrontal cortex. J Neurophysiol 2005, 93:3687-3692. 37. Pan X et al.: Reward inference by primate prefrontal and striatal neurons. J Neurosci 2014, 34:1380-1396.
17. Coccaro EF, McCloskey MS, Fitzgerald DA, Phan KL: Amygdala and orbitofrontal reactivity to social threat in individuals with impulsive aggression. Biol Psychiat 2007, 62:168-178.
38. Boorman ED, Behrens TE, Rushworth MF: Counterfactual choice and learning in a neural network centered on human lateral frontopolar cortex. PLoS Biol 2011, 9:e1001093.
18. Carre´ JM, Hyde LW, Neumann CS, Viding E, Hariri AR: The neural signatures of distinct psychopathic traits. Soc Neurosci 2013, 8:122-135.
39. Badre D, Frank MJ: Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI. Cereb Cortex 2012, 22:527-536.
19. Buckholtz JW et al.: Mesolimbic dopamine reward system hypersensitivity in individuals with psychopathic traits. Nat Neurosci 2010, 13:419-421.
40. Dixon ML, Christoff K: The lateral prefrontal cortex and complex value-based learning and decision making. Neurosci Biobehav Rev 2014, 45:9-18.
20. Rand DG, Greene JD, Nowak MA: Spontaneous giving and calculated greed. Nature 2013, 489:427-430.
41. Haber SN, Kim K-S, Mailly P, Calzavara R: Reward-related cortical inputs define a large striatal region in primates that interface with associative cortical connections, providing a substrate for incentive-based learning. J Neurosci 2006, 26:8368-8376.
21. Warneken F, Tomasello M: Varieties of altruism in children and chimpanzees. Trends Cogn Sci 2009, 13:397-402. 22. Zaki J, Mitchell JP: Intuitive prosociality. Curr Dir Psychol Sci 2013, 22:466-470. 23. Dalley JW, Everitt BJ, Robbins TW: Impulsivity, compulsivity, and top-down cognitive control. Neuron 2011, 69:680-694. 24. Broos N et al.: The relationship between impulsive choice and impulsive action: a cross-species translational study. PLoS ONE 2012, 7:e36781. This paper combines human behavioral testing with preclinical neuroscience to dissociate two factors of self-control. In humans, performance on tasks of response inhibition did not correlate with discount rates on an intertemporal choice task. Factor analysis of a multi-task dataset confirmed that inhibitory control and cost–benefit decision-making represent distinct aspects of self-control. Parallel work in rats demonstrated that these two facets of self-control were differentially susceptible to pharmacological manipulations, suggesting that dissociable neurobiological mechanisms underpin performance on tasks of impulsive action and impulsive choice. 25. Nombela C, Rittman T, Robbins TW, Rowe JB: Multiple modes of impulsivity in Parkinson’s disease. PLoS ONE 2014, 9:e85747. Current Opinion in Behavioral Sciences 2015, 3:122–129
42. Roesch MR, Singh T, Brown PL, Mullins SE, Schoenbaum G: Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J Neurosci 2009, 29:13365-13376. 43. Tai L-H, Lee AM, Benavidez N, Bonci A, Wilbrecht L: Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat Neurosci 2012, 15:12811289. 44. Bilder RM, Volavka J, Lachman HM, Grace AA: The catechol-Omethyltransferase polymorphism: relations to the tonic– phasic dopamine hypothesis and neuropsychiatric phenotypes. Neuropsychopharmacol: Off Publ Am Coll Neuropsychopharmacol 2004, 29:1943-1961. 45. Antonelli T, Govoni BM, Bianchi C, Beani L: Glutamate regulation of dopamine release in guinea pig striatal slices. Neurochem Int 1997, 30:203-209. 46. Casey KF et al.: Individual differences in frontal cortical thickness correlate with the D-amphetamine-induced striatal www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
dopamine response in humans. J Neurosci 2013, 33: 15285-15294. By combining dopamine receptor imaging with structural MRI, this work provides strong empirical support for the idea that DLPFC negativelty regulates reward-related striatal DA response. Lower prefrontal thickness was associated with greater amphetamine-induced striatal DA release, confirming an inverse relationship between DLPFC integrity and striatal DA transmission.
129
55. Gershman SJ, Markman AB, Otto AR: Retrospective revaluation in sequential decision making: a tale of two systems. J Exp Psychol: Gen 2014, 143:182-194. 56. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ: Modelbased influences on humans’ choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
48. Strafella AP, Paus T, Barrett J, Dagher A: Repetitive transcranial magnetic stimulation of the human prefrontal cortex induces dopamine release in the caudate nucleus. J Neurosci 2001, 21:RC157.
57. Glascher J, Daw N, Dayan P, O’Doherty JP: States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595. This paper used model-based imaging to identify distinct neural circuits for model-free and model-based learning and choice. Consistent with prior work, ventral striatum was found to encode model-free signals; by contrast, model-based state prediction errors were identified in DLPFC and inferior parietal cortex.
49. Ott DV, Ullsperger M, Jocham G, Neumann J, Klein TA: Continuous theta-burst stimulation (cTBS) over the lateral prefrontal cortex alters reinforcement learning bias. NeuroImage 2011, 57:617-623.
58. Deserno L et al.: Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proc Natl Acad Sci USA 2015, 112:1595-1600.
50. Figner B et al.: Lateral prefrontal cortex and self-control in intertemporal choice. Nat Neurosci 2010, 13:538-539.
59. Smittenaar P, FitzGerald TH, Romei V, Wright ND, Dolan RJ: Disruption of dorsolateral prefrontal cortex decreases modelbased in favor of model-free control in humans. Neuron 2013, 80:914-919.
47. Meyer-Lindenberg A et al.: Reduced prefrontal activity predicts exaggerated striatal dopaminergic function in schizophrenia. Nat Neurosci 2002, 5:267-271.
51. van den Bos W, Rodriguez CA, Schweitzer JB, McClure SM: Connectivity strength of dissociable striatal tracts predict individual differences in temporal discounting. J Neurosci 2014, 34:10298-10310. 52. Magen E, Kim B, Dweck CS, Gross JJ, McClure SM: Behavioral and neural correlates of increased self-control in the absence of increased willpower. Proc Natl Acad Sci 2014, 111:9786-9791. This important study used a cognitive manipulation to dissociate impulsive reward decision-making from effortful cognitive control. A cognitive reframing intervention reduced striatal activation to immediate rewards without altering activity in frontoparietal control regions. These findings showed that impulsive choice behavior can result from differences in reward valuation rather than capacity-limited inhibition.
60. Otto AR, Skatova A, Madlon-Kay S, Daw ND: Cognitive control predicts use of model-based reinforcement learning. J Cogn Neurosci 2015, 27:319-333. 61. Schad DJ et al.: Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning. Front Psychol 2014, 5:1450. 62. Hare TA, Camerer CF, Knoepfle DT, O’Doherty JP, Rangel A: Value computations in ventral medial prefrontal cortex during charitable decision making incorporate input from regions involved in social cognition. J Neurosci 2010, 30:583-590.
53. Cooper N, Kable JW, Kim BK, Zauberman G: Brain activity in valuation regions while thinking about the future predicts individual discount rates. J Neurosci 2013, 33:13150-13156.
63. Krebs RM, Boehler CN, Egner T, Woldorff MG: The neural underpinnings of how reward associations can both guide and misguide attention. J Neurosci: Off J Soc Neurosci 2011, 31:9752-9759.
54. Daw ND, Niv Y, Dayan P: Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711.
64. Buckholtz JW, Meyer-Lindenberg A: Psychopathology and the human connectome: toward a transdiagnostic model of risk for mental illness. Neuron 2012, 74:990-1004.
www.sciencedirect.com
Current Opinion in Behavioral Sciences 2015, 3:122–129
ScienceDirect Social norms, self-control, and the value of antisocial behavior Joshua W Buckholtz1,2,3 Social norms facilitate large-scale cooperation by promoting prosocial interactions and constraining antisocial behavior. Dominant models of norm compliance emphasize the role of effortful, capacity-limited inhibitory control in prosocial cooperation. Similarly, clinical science has focused on inhibitory deficits as a key source of persistent norm-violating behavior. Support for an inhibition-based ‘braking success/ braking failure’ (BSBF) model is derived from evidence of dorsolateral prefrontal cortex (DLPFC) engagement during norm-guided behavior, and of DLPFC dysfunction in antisocial individuals. However, three challenges motivate an alternative explanation for links between self-control, DLPFC, and normbased behavior. Here, I propose a value-based alternative to the BSBF model, in which prosocial norm compliance and antisocial norm violations both arise from interactions between prefrontal model-based and striatal model-free decisionmaking systems. Addresses 1 Department of Psychology, United States 2 Center for Brain Science, Harvard University, United States 3 Department of Psychiatry, Massachusetts General Hospital, United States Corresponding author: Buckholtz, Joshua W. ([email protected])
(prosocial) behavior [3]. However, while norm compliance is extensive, it is far from universal, and our understanding of the mechanisms that drive norm violations (antisocial behavior) remains limited. Given the estimated societal costs of antisocial behavior — upwards of $1 trillion annually, by some accounts [4] — our relatively poor insight into its biological origins impedes treatment development and limits our ability to make informed policy decisions.
Social norms and self-control Norm compliance often involves engaging in other-regarding behavior that is counter to an agent’s immediate selfish interests. Thus, many have argued that norm-based behavior requires ‘self-control,’ [5,6] a term that is used somewhat interchangeably with ‘self-regulation,’ ‘impulse control,’ ‘cognitive control,’ and ‘executive function’. If prosocial behavior requires effortful self-control, it would thus naturally follow that persistent norm violating (antisocial) behavior arises from self-control failure, manifest as an inability to appropriately inhibit self-interested decision-making. Indeed, this is the conceptual foundation for a highly influential theory of crime (the ‘Self-Control Theory of Crime’) [7], and clinical science research has largely focused on identifying the cognitive and neural bases of self-control deficits in antisocial individuals [8].
Current Opinion in Behavioral Sciences 2015, 3:122–129 This review comes from a themed issue on Social behavior Edited by Molly J Crockett and Amy Cuddy For a complete overview see the Issue and the Editorial Available online 3rd April 2015 http://dx.doi.org/10.1016/j.cobeha.2015.03.004 2352-1546/# 2015 Elsevier Ltd. All rights reserved.
Our social landscape is shaped by social norms, a set of prescriptive and proscriptive rules that comprise the ‘grammar of social interaction’ [1] for Homo sapiens. Injunctions against dishonesty in exchange, physical harm, and theft promote social stability, community peace and economic prosperity. Thus, many have argued that norms were key for enabling the ultrasociality that is a signature of our species. In turn, their ability to facilitate cooperation depends on the widespread maintenance of norm compliance, accomplished by sanctioning norm violations [2] and by rewarding norm consistent Current Opinion in Behavioral Sciences 2015, 3:122–129
The ‘brakes’ metaphor of self-control The prevailing conceptualization of self-control in studies of norm-based cooperative behavior is that of an effortful, capacity-limited resource, used to inhibit automatic or prepotent responses. Social psychologists and behavioral economists have argued that cooperation arises from the capacity to actively override selfish impulses in order to promote selection of an alternative norm-consistent choice option [5,6]. Similarly, clinical scientists have proposed that antisocial behavior results from a deficit in the capacity to actively inhibit the execution of prepotent responses to threat and/or reward associated stimuli [8,9]. Together, these two perspectives comprise a dominant viewpoint wherein ‘braking success’ produces prosocial behavior and ‘braking failure’ leads to antisocial behavior. While this ‘braking success/braking failure’ (BSBF) model of norm-guided behavior has both intuitive appeal and experimental support, recent work suggests that the BSBF model is incomplete. In what follows, I will review brain imaging findings that are often cited to support the BSBF model, highlight conceptual and empirical challenges to the model, and articulate an alternative framework grounded in decision science. www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
Brakes, localized?: dorsolateral prefrontal cortex As noted above, many have suggested that norm-based prosocial behavior involves the deployment of self-control to actively inhibit automatic self-interested responses. Neurobiological studies of norm-based behavior have largely adopted this conceptualization, identifying the locus of the self-control ‘braking’ system in lateral prefrontal cortex. For example, dorsolateral prefrontal cortex (DLPFC) activation has been consistently noted in fMRI tasks that assess prosocial (‘altruistic’) punishment of norm violations, as well as in studies of cooperation, fairness, and social norm compliance [2,10]. A recent meta-analysis suggests that DLPFC activity during normbased decision-making reflects the need for ‘cognitive control from a reflective and deliberate System 2 to resolve conflict by . . . over-riding self-interest’ [11]. Of note, disruptive brain stimulation to DLPFC reduces both prosocial norm-enforcement [12] and norm compliance [13]. As other studies have shown that this region is important for response inhibition, some have inferred that the ability to actively override selfish or otherwise maladaptive responses is a cognitive sin qua non for prosocial behavior [5]. Similarly, it is widely assumed that antisocial behavior results from impaired inhibitory control [8,9]. This perspective is supported by behavioral and neurobiological evidence that antisocial individuals show deficient cognitive control in the context of heightened reactivity to threat and/or reward cues. Antisocial offenders exhibit reduced gray matter volume and cortical thickness within DLPFC [8,14], as well as compromised DLPFC activation during classic neuropsychological indices of inhibitory control [14,15]. By contrast, antisocial individuals appear to have relatively exaggerated responses to threat stimuli (within the amygdala) and reward cues (within the striatum) [16,17–19]. Interpreted through the lens of the BSBF model, such findings are taken as evidence that antisocial behavior occurs when bottom up ‘affective’ signals activate or generate a prepotent behavioral response that is inadequately inhibited by top down ‘cognitive’ resources due to poor prefrontal control. Taken together, a large body of work provides convergent support for the idea that DLPFC is causally involved in norm compliance, and that antisocial individuals exhibit deficits in DLPFC structure and function. While such findings appear to fit the BSBF model, three key challenges suggest that an alternative explanation for the link between DLPFC and pro-/anti-social behavior merits consideration.
Challenge 1: prosociality, no brakes required The first challenge comes from recent findings that prosocial behavior does not necessarily require the controlled inhibition of self-interested or maladaptive www.sciencedirect.com
123
automatic responses. Cooperation in both laboratory and real-life contexts is highest under conditions that promote fast, intuitive responding [20]. Developmental and cross-species data offer further support for the intuitive, automatic nature of prosociality: chimpanzees — even very young ones — engage in costly helping behavior, and prosocial responding in human children emerges prior to the development of inhibitory control [21]. Finally, neuroimaging work suggests that costly prosocial behavior may be facilitated by the heightened value assigned to prosocial choice options relative to available alternatives, rather than the active inhibition of automatic antisocial responses [22]. Together, these findings imply that norm-consistent behavior does not necessarily require effortful inhibition.
Challenge 2: there is no such (single) thing as ‘self-control’ The construct of self-control has a deceptively complex latent architecture. There is overwhelming evidence for multidimensionality, with several distinct cognitive components that can be grouped into at least two broad domains [23] (Figure 1). Capacities encompassed by the domain of ‘response inhibition’ enable agents to use internal representations or external cues to inhibit prepotent motor responses. A second domain contains processes that promote adaptive choice behavior by estimating the subjective value of different choice options and selecting actions based on optimal utility. Lesion and drug studies in animals, along with factor analytic work in humans, converge to suggest that response inhibition and value-based decision making are largely distinct cognitive capacities [24] with dissociable neurobiological substrates [23]. Despite the evident heterogeneity of ‘selfcontrol,’ it is often conflated with response inhibition or assessed by trait measures that map poorly to identifiable cognitive processes [25]. The net result of this construct neglect is two-fold. First, research on the relationship between self-control and norm-guided behavior has largely focused on response inhibition, while other facets of self-control remain relatively unexplored. Second, DLPFC engagement (or dysfunction) is often presumed to reflect inhibitory control, leaving alternative explanations for the role of DLPFC in pro-social and anti-social behavior relatively unexamined.
Challenge 3: DLPFC reconsidered Early work on the role of DLPFC in adaptive behavior was largely focused on inhibitory control: lesion, electrophysiology, and functional imaging studies demonstrated that DLPFC is crucial for the ability to use goals, task rules and external cues to inhibit the execution of contextually inappropriate motor responses [26–32]. However, recent brain imaging and electrophysiological data have broadened the view of DLPFC to include a crucial role in value-based decision-making. Lateral PFC has been shown to be important for integrating value-related Current Opinion in Behavioral Sciences 2015, 3:122–129
124 Social behavior
Figure 1
“Self-Control”
Value-Based
Action
Decision-Making
Control Response Inhibition
Model-Based
Model-Free
Cost-Benefit
Action Response Interference Response Cancellation Suppression Suppression Selection Continuous Performance
Probabilistic Response Reversal
Stop-Signal
n-Armed Bandit Tasks Delay Discounting Mixed Gambles Effort Discounting
Stroop
Go/NoGo
Flanker
Choice Reaction Time
Simon Task Anti-Saccade
2-Stage Markov Decision Task Reinforcer Devaluation Current Opinion in Behavioral Sciences
Self-Control is multidimensional. While commonly reduced to ‘response inhibition,’ the construct of self-control encompasses at least two broad domains of cognition, each containing several distinct cognitive processes. Here, I highlight the dissociation between two self-control domains: action control and value-based decision-making. Experimental paradigms are grouped according to the domain of cognition that they putatively access and the domain-specific processes that each measures.
decision variables at the time of choice, particularly when predicted action values must be calculated from multiple, simultaneously varying parameters (e.g. magnitude, probability, delay, uncertainty, and effort) [33–35]. This integrated signal may be incorporated into a process of episodic prospection that simulates outcomes for an array of action options, permitting a comparison of expected value for multiple potential actions [36–39]. Finally, DLPFC appears to be important for representing associations between abstract rules and expected outcome values [40]. By linking information about rules to reward outcomes, DLPFC may be able to generate a representation of the value of rules — a feature that may be particularly important for norm-guided decision making.
DLPFC modulates striatal value signals The challenges outlined above undermine the notion that effortful cognitive control is required for prosocial behavior, highlight the conflation of self-control with ‘response inhibition,’ and expand the role of DLPFC to encompass value-related computations. How, then, do we reconcile these challenges with clear evidence that, firstly, DLPFC is crucial for norm-guided prosocial Current Opinion in Behavioral Sciences 2015, 3:122–129
behavior and secondly, the structure and function of DLPFC is compromised in antisocial individuals? I propose that the ability of DLPFC representations to modulate choice-related value signals explains the prominent role for this region in both norm-based cooperation and antisocial behavior. As indicated above, DLPFC represents behavioral goals and abstract rule-value associations, integrates multiple streams of cost–benefit information, and prospectively maps action options to future outcome values. These higher order value-related processes may guide adaptive behavior by modulating action value representations in regions that are more directly involved in action selection, such as the striatum. DLPFC projections to the striatum are robust and wellcharacterized; these corticostriatal afferents comprise a crucial mechanism for modulating meso-striatal DA transmission and choice behavior [41]. Exposure to unpredicted rewards and reward-predicting cues induces burst firing in midbrain DA neurons, causing transient high-amplitude DA efflux from striatal terminals (the phasic DA signal). Phasic DA activity in striatum guides action selection based on stimulus-value associations [42,43]. While ascending midbrain efferents convey www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
stimulus value-dependent signals to induce phasic DA release in striatum, descending glutamatergic projections arising from LPFC maintain tonic DA levels at striatal axon terminals [44,45]. In turn, prefrontally regulated tonic DA acts on inhibitory presynaptic terminal autoreceptors to dampen phasic DA release [44] (Figure 2). Thus, higher-order value related information maintained in prefrontal cortex is able to bias choice
125
behavior by modulating the ability of reward-associated cues to drive action selection via phasic DA signaling. The functional-anatomical organization of mesocorticostriatal DA circuitry suggests that that diminishing prefrontal function could lead to maladaptive decisionmaking by disrupting prefrontal regulation of phasic DA release. Brain imaging and brain stimulation studies
Figure 2
Corticostriatal > Mesostriatal
LPFC
Striatum
GluR
D1 D2 GluR Medium Spiny Neuron
Motor Ctx
D2-AR
Midbrain
MB > MF Patient > Impulsive Prosocial > Antisocial
Mesostriatal > Corticostriatal
LPFC
Striatum
GluR
D1 D2 GluR Medium Spiny Neuron
Motor Ctx
D2-AR
Midbrain
MF > MB Impulsive > Patient Antisocial > Prosocial
Current Opinion in Behavioral Sciences
Circuit model for prefrontal modulation of value signals. Stronger ascending drive boosts striatal DA responses to learned reward cues, leading to impulsive, model-free choice behavior (bottom). Stronger corticostriatal inputs dampen phasic DA transmission in striatum, facilitating optimal action selection through the adaptive, context-appropriate re-weighting of model-free value signals (top). www.sciencedirect.com
Current Opinion in Behavioral Sciences 2015, 3:122–129
126 Social behavior
support this hypothesis. First, DLPFC activity is negatively associated with striatal DA transmission in humans [46,47], and disrupting DLPFC function induces striatal DA release [48], potentiates striatal fMRI responses to reward cues [49] and biases behavior toward impulsive reward-seeking [50]. Second, measures of functional and anatomical connectivity between DLPFC and striatum predict delay-based choice biases and striatal subjective value signals during cost–benefit decision making [51]. Such data are consistent with the notion that DLPFC guides behavior by modulating the value assigned to action options. Finally, changing the valuation of choice options (via cognitive reframing) is sufficient to induce far-sighted (patient) inter-temporal decision-making, even in the absence of effortful inhibitory control [52]. This finding accords well with data by Cooper and colleagues showing that impulsive decision-making can be decreased by cognitive manipulations that amplify striatal value signals for delayed (versus immediate) choice options [53]. On the whole, these studies suggest that DLPFC facilitates optimal decision-making by adaptively reweighting striatal action value signals, rather than by inhibiting action execution after valuation and selection have already occurred.
Model-free and model-based decisionmaking It may be useful to evaluate the putative value-modulating function of DLPFC described above in the context of an influential computational framework for action control. This approach distinguishes between two systems for behavioral control: a retrospective model-free system, which selects actions solely on the basis of reinforcement history, and a prospective model-based system that integrates reward experience with rules and context representations to create a map (i.e. model) of the external world [54]. From this perspective, choice behavior is driven either by learned stimulus-response ‘habits’ encoded in the model-free system, or by a process that computes optimal action values by prospectively mapping action options to future outcomes [55]. Model-free decision-making is supported by phasic DA transmission within the midbrain-striatum circuit described previously; by contrast, DLPFC (along with other cortical regions) appears crucial for model-based learning and choice [56,57]. Of note, the relationship between these two systems is likely interactive or cooperative, rather than directly competitive; experience can organize the structure of world models, and these models can in turn be used to modify learned action values [55]. On the whole, his account provides a useful framework for considering the role of value-based decisionmaking in norm-guided behavior.
A value-based framework for norm-guided choice and antisocial behavior Consider a scenario in which an agent is faced with choosing between self-interested (antisocial) gain and Current Opinion in Behavioral Sciences 2015, 3:122–129
prosocial action. Steve leaves a bank carrying an envelope filled with cash, which he quickly places it into his coat pocket and begins walking down the street. Unbeknownst to Steve, the envelope falls through a hole in his pocket and lands on the sidewalk, opening to reveal its contents. The street is deserted, but for a single bystander, John, who observes this turn of events. John is faced with a choice: pick up the envelope and enrich himself by walking off with it, or return it to its rightful owner. I argue that John’s ability to select the norm-consistent action depends, in part, on the interaction between prefrontal model-based representations and meso-striatal model-free action value signals. John will have acquired, via a lifetime of instrumental learning, a robust association between the possession of money and the consumption of primary rewards. All things being equal, actions linked to monetary gain (e.g. picking up an envelope filled with cash) will have, a priori, higher value compared to actions that are not (e.g. giving away an envelope filled with cash). This valuation is retrospective and contextindependent; there is nothing inherently prosocial or antisocial about selecting actions that result in the receipt of money per se. If John’s action values were driven solely by learned stimulus-response ‘habits,’ picking up the envelope and walking off with it should have the higher value. However, norms are a set of rules that can provide a model-based contextual modulation of that value vis-a´-vis other choice options. John’s world model incorporates cognitive information about social rules (e.g. proscriptive norms against taking other people’s property), rule-value associations based on first-hand and vicarious learning (e.g. people who take from others are ‘bad’, people who help others are virtuous), and information about the consequences of rule violations (e.g. people who steal go to jail). A model-based strategy will use these representations to make prospection-based cognitive predictions about the value of each action. This process may ultimately strengthen the value weight for norm-compliant action options, despite the learned value of selecting actions that result in self-interested gains. Findings of DLPFC dysfunction in antisocial offenders can be reinterpreted within this framework. I propose that functional and structural deficits in DLPFC predispose antisocial behavior by dysregulating striatal valuation processes, rather than (or perhaps in addition to) disrupting response inhibition. In such a model, weaker modelbased cortical modulation of model-free striatal value signals would impair adaptive behavior by limiting the ability of goal-based prospective representations to reweight actions values. Relatively weaker glutamatergic input into DLPFC would result in lower tonic DA, and thus greater control of action selection by learned reward signals encoded by phasic DA transmission (Figure 2). As noted, DLPFC is crucial for representing goals, delayed reward values, rule-values and prospection-based inferences about outcomes. Prefrontal impairment may lead to www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
the construction of a world model that is relatively impoverished, or to a failure to appropriately integrate modelbased inferences about outcomes into decision-making when faced with a learned reward cue. In either case, model-based computations would not be able to appropriately modify the value assigned to a norm-consistent action when model-free learning gives a strong value weight to an antisocial response. Consistent with the notion that frontostriatal projections accomplish this modulation by regulating striatal DA signaling, stronger model-based signals in lateral PFC are associated with higher levels of tonic DA and blunted model-free prediction errors in ventral striatum [58]. Furthermore, transient inactivation of DLPFC with transcranial magnetic stimulation — a manipulation known to impair norm-based decision-making [12] and amplify model-free striatal reward signals [49] — leads to a predominance of model-free over model-based choice behavior [59].
Conclusion Taken together, the findings described herein suggest a value-based alternative to the BSBF model. Cooperative, norm-consistent behavior putatively reflects an adaptive modulation of learned action values by model-based prospective computations that incorporate rule, context, and cost information. This modulation is effected via corticostriatal afferents that adjust the gain on phasic reward signals arising from midbrain DA neuron burst firing. By contrast, antisocial behavior may occur when DLPFC dysfunction impairs the development of a robust model of the world, or when it prevents the prospective calculations generated by this model from appropriately modulating ‘downstream’ action value signals. This model makes several testable predictions. First, antisocial individuals should show a combination of diminished model-based state prediction errors and amplified model-free reward prediction errors. Second, disrupting (or enhancing) DLPFC function should drive norm-guided behavior in a lawful manner, either by preventing (or facilitating) development of an enriched world model, or by attenuating (or potentiating) the influence of model-based signals on striatal action values, without having an intervening effect on response inhibition. Third, this model predicts that aspects of cognitive function linked to DLPFC should jointly track variation in norm compliance and model-based behavioral control. Of note, DLPFC-dependent working memory capacity has robust negative associations to impulsivity and antisocial behavior. Though speculative, this link may not reflect a direct causal relationship, but rather may arise epiphenomenally from the fact that DLPFC is important for both working memory capacity and model-based responding. In other words, deficits in working memory capacity and other aspects of executive function may be ‘third variable’ markers of compromised prefrontal value modulation. This hypothesis accords with recent data www.sciencedirect.com
127
showing that processing speed, cognitive control, and other DLPFC-dependent executive capacities are positively associated with model-based behavioral control [60,61]. The value-based model proposed here is necessarily speculative and is offered to generate tractable hypotheses that can be explored in future work. Importantly, in proposing this model it is not my intention to suggest that other important factors such as empathy or response inhibition play no role at all in norm-guided behavior. Rather, the impact of such factors in behavior may be the result of an interaction with valuation processes. For example, mental state representations can modulate action value during economic choice [62], and stimulusreward learning can both facilitate and disrupt response inhibition and other aspects of executive function [63]. Discovering how individual differences in brain circuit function drive individual differences in behavior by affecting such interactions [64] will be crucial for advancing our understanding of the cognitive and neurobiological roots of prosocial and antisocial behavior.
Conflict of interest statement Nothing declared.
Acknowledgments I wish to thank Joshua Greene, Sam Gershman, Justin Martin, and Michael Treadway for insightful discussion, and the National Institute on Drug Abuse (1R03DA034126-01), the Sloan Foundation, the Brain and Behavior Research Foundation, and the MGH Center for Law, Brain and Behavior for research support.
References and recommended reading Papers of particular interest, published within the period of review, have been highlighted as: of special interest of outstanding interest 1.
Bicchieri C: The Grammar of Society: the Nature and Dynamics of Social Norms. Cambridge University Press; 2006.
2.
Buckholtz JW, Marois R: The roots of modern justice: cognitive and neural foundations of social norms and their enforcement. Nat Neurosci 2012, 15:655-661.
3.
Rand DG, Dreber A, Ellingsen T, Fudenberg D, Nowak MA: Positive interactions promote public cooperation. Science 2009, 325:1272-1275.
4.
Anderson DA: The aggregate burden of crime. J Law Econ 1999, 42:611-642.
5.
Knoch D, Fehr E: Resisting the power of temptations: the right prefrontal cortex and self-control. Ann NY Acad Sci 2007, 1104:123-134.
6.
Baumeister RF: Self-regulation, ego depletion, and inhibition. Neuropsychologia 2014, 65:313-319.
7.
Hirschi T: Self-control and crime. Handbook of Self-Regulation. Guilford Press; 2004.
8.
Dolan M: The neuropsychology of prefrontal function in antisocial personality disordered offenders with varying degrees of psychopathy. Psychol Med 2012, 42:1715-1725. Current Opinion in Behavioral Sciences 2015, 3:122–129
128 Social behavior
9.
Patrick CJ, Durbin CE, Moser JS: Reconceptualizing antisocial deviance in neurobehavioral terms. Dev Psychopathol 2012, 24:1047-1071.
10. Treadway MT et al.: Corticolimbic gating of emotion-driven punishment. Nat Neurosci 2014, 17:1270-1275. 11. Feng C, Luo YJ, Krueger F: Neural signatures of fairness-related normative decision making in the ultimatum game: a coordinate-based meta-analysis. Hum Brain Mapp 2015, 36:591-602. 12. Baumgartner T, Knoch D, Hotz P, Eisenegger C, Fehr E: Dorsolateral and ventromedial prefrontal cortex orchestrate normative choice. Nat Neurosci 2011, 14:1468-1474. 13. Ruff CC, Ugazio G, Fehr E: Changing social norm compliance with noninvasive brain stimulation. Science 2013, 342:482-484. This study used transcranial direct current stimulation (tDCS) to investigate a causal role for DLPFC in voluntary and sanction-induced norm compliance during a monetary transfer task. Up-regulation and downregulation of right prefrontal function had opposing effects on compliance, without changing mental state or fairness assessments. 14. Yang Y, Raine A: Prefrontal structural and functional brain imaging findings in antisocial, violent, and psychopathic individuals: a meta-analysis. Psychiat Res: Neuroimag 2009, 174:81-88. 15. Moeller SJ et al.: Common and distinct neural correlates of inhibitory dysregulation: stroop fMRI study of cocaine addiction and intermittent explosive disorder. J Psychiat Res 2014, 58:55-62. 16. Hyde LW, Byrd AL, Votruba-Drzal E, Hariri AR, Manuck SB: Amygdala reactivity and negative emotionality: divergent correlates of antisocial personality and psychopathy traits in a community sample. J Abnorm Psychol 2014, 123:214-224. Amygdala activation was examined in a large sample of community volunteers who were assesed for psychopathic traits and antisocial behavior. After adjusting for overlapping variance, psychopathy and antisociality had opposing relationships to threat-related amygdala reactivity. Higher psychopathy scores were associated with lower amygdala response, while higher antisociality scores were related to greater amygdala reactivity, but only after it was adjusted for in the statistical model.
26. Oishi T, Mikami A, Kubota K: Local injection of bicuculline into area 8 and area 6 of the rhesus monkey induces deficits in performance of a visual discrimination GO/NO-GO task. Neurosci Res 1995, 22:163-177. 27. Simmonds DJ, Pekar JJ, Mostofsky SH: Meta-analysis of Go/Nogo tasks demonstrating that fMRI activation associated with response inhibition is task-dependent. Neuropsychologia 2008, 46:224-232. 28. Wallis JD, Anderson KC, Miller EK: Single neurons in prefrontal cortex encode abstract rules. Nature 2001, 411:953-956. 29. Funahashi S, Chafee MV, Goldman-Rakic PS: Prefrontal neuronal activity in rhesus monkeys performing a delayed anti-saccade task. Nature 1993, 365:753-756. 30. Wallis JD, Miller EK: From rule to response: neuronal processes in the premotor and prefrontal cortex. J Neurophysiol 2003, 90:1790-1806. 31. White IM, Wise SP: Rule-dependent neuronal activity in the prefrontal cortex. Exp Brain Res 1999, 126:315-335. 32. Kramer UM et al.: The role of the lateral prefrontal cortex in inhibitory motor control. Cortex (A journal devoted to the study of the nervous system and behavior) 2013, 49:837-849. 33. Basten U, Biele G, Heekeren HR, Fiebach CJ: How the brain integrates costs and benefits during decision making. Proc Natl Acad Sci 2010, 107:21767-21772. 34. Rushworth MFS, Noonan MP, Boorman ED, Walton ME, Behrens TE: Frontal cortex and reward-guided learning and decision-making. Neuron 2011, 70:1054-1069. 35. Kim S, Lee D: Prefrontal cortex and impulsive decision making. Biol Psychiat 2011, 69:1140-1146. 36. Tsujimoto S, Sawaguchi T: Neuronal activity representing temporal prediction of reward in the primate prefrontal cortex. J Neurophysiol 2005, 93:3687-3692. 37. Pan X et al.: Reward inference by primate prefrontal and striatal neurons. J Neurosci 2014, 34:1380-1396.
17. Coccaro EF, McCloskey MS, Fitzgerald DA, Phan KL: Amygdala and orbitofrontal reactivity to social threat in individuals with impulsive aggression. Biol Psychiat 2007, 62:168-178.
38. Boorman ED, Behrens TE, Rushworth MF: Counterfactual choice and learning in a neural network centered on human lateral frontopolar cortex. PLoS Biol 2011, 9:e1001093.
18. Carre´ JM, Hyde LW, Neumann CS, Viding E, Hariri AR: The neural signatures of distinct psychopathic traits. Soc Neurosci 2013, 8:122-135.
39. Badre D, Frank MJ: Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI. Cereb Cortex 2012, 22:527-536.
19. Buckholtz JW et al.: Mesolimbic dopamine reward system hypersensitivity in individuals with psychopathic traits. Nat Neurosci 2010, 13:419-421.
40. Dixon ML, Christoff K: The lateral prefrontal cortex and complex value-based learning and decision making. Neurosci Biobehav Rev 2014, 45:9-18.
20. Rand DG, Greene JD, Nowak MA: Spontaneous giving and calculated greed. Nature 2013, 489:427-430.
41. Haber SN, Kim K-S, Mailly P, Calzavara R: Reward-related cortical inputs define a large striatal region in primates that interface with associative cortical connections, providing a substrate for incentive-based learning. J Neurosci 2006, 26:8368-8376.
21. Warneken F, Tomasello M: Varieties of altruism in children and chimpanzees. Trends Cogn Sci 2009, 13:397-402. 22. Zaki J, Mitchell JP: Intuitive prosociality. Curr Dir Psychol Sci 2013, 22:466-470. 23. Dalley JW, Everitt BJ, Robbins TW: Impulsivity, compulsivity, and top-down cognitive control. Neuron 2011, 69:680-694. 24. Broos N et al.: The relationship between impulsive choice and impulsive action: a cross-species translational study. PLoS ONE 2012, 7:e36781. This paper combines human behavioral testing with preclinical neuroscience to dissociate two factors of self-control. In humans, performance on tasks of response inhibition did not correlate with discount rates on an intertemporal choice task. Factor analysis of a multi-task dataset confirmed that inhibitory control and cost–benefit decision-making represent distinct aspects of self-control. Parallel work in rats demonstrated that these two facets of self-control were differentially susceptible to pharmacological manipulations, suggesting that dissociable neurobiological mechanisms underpin performance on tasks of impulsive action and impulsive choice. 25. Nombela C, Rittman T, Robbins TW, Rowe JB: Multiple modes of impulsivity in Parkinson’s disease. PLoS ONE 2014, 9:e85747. Current Opinion in Behavioral Sciences 2015, 3:122–129
42. Roesch MR, Singh T, Brown PL, Mullins SE, Schoenbaum G: Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J Neurosci 2009, 29:13365-13376. 43. Tai L-H, Lee AM, Benavidez N, Bonci A, Wilbrecht L: Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat Neurosci 2012, 15:12811289. 44. Bilder RM, Volavka J, Lachman HM, Grace AA: The catechol-Omethyltransferase polymorphism: relations to the tonic– phasic dopamine hypothesis and neuropsychiatric phenotypes. Neuropsychopharmacol: Off Publ Am Coll Neuropsychopharmacol 2004, 29:1943-1961. 45. Antonelli T, Govoni BM, Bianchi C, Beani L: Glutamate regulation of dopamine release in guinea pig striatal slices. Neurochem Int 1997, 30:203-209. 46. Casey KF et al.: Individual differences in frontal cortical thickness correlate with the D-amphetamine-induced striatal www.sciencedirect.com
Social norms, self-control, and antisocial behavior Buckholtz
dopamine response in humans. J Neurosci 2013, 33: 15285-15294. By combining dopamine receptor imaging with structural MRI, this work provides strong empirical support for the idea that DLPFC negativelty regulates reward-related striatal DA response. Lower prefrontal thickness was associated with greater amphetamine-induced striatal DA release, confirming an inverse relationship between DLPFC integrity and striatal DA transmission.
129
55. Gershman SJ, Markman AB, Otto AR: Retrospective revaluation in sequential decision making: a tale of two systems. J Exp Psychol: Gen 2014, 143:182-194. 56. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ: Modelbased influences on humans’ choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
48. Strafella AP, Paus T, Barrett J, Dagher A: Repetitive transcranial magnetic stimulation of the human prefrontal cortex induces dopamine release in the caudate nucleus. J Neurosci 2001, 21:RC157.
57. Glascher J, Daw N, Dayan P, O’Doherty JP: States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595. This paper used model-based imaging to identify distinct neural circuits for model-free and model-based learning and choice. Consistent with prior work, ventral striatum was found to encode model-free signals; by contrast, model-based state prediction errors were identified in DLPFC and inferior parietal cortex.
49. Ott DV, Ullsperger M, Jocham G, Neumann J, Klein TA: Continuous theta-burst stimulation (cTBS) over the lateral prefrontal cortex alters reinforcement learning bias. NeuroImage 2011, 57:617-623.
58. Deserno L et al.: Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proc Natl Acad Sci USA 2015, 112:1595-1600.
50. Figner B et al.: Lateral prefrontal cortex and self-control in intertemporal choice. Nat Neurosci 2010, 13:538-539.
59. Smittenaar P, FitzGerald TH, Romei V, Wright ND, Dolan RJ: Disruption of dorsolateral prefrontal cortex decreases modelbased in favor of model-free control in humans. Neuron 2013, 80:914-919.
47. Meyer-Lindenberg A et al.: Reduced prefrontal activity predicts exaggerated striatal dopaminergic function in schizophrenia. Nat Neurosci 2002, 5:267-271.
51. van den Bos W, Rodriguez CA, Schweitzer JB, McClure SM: Connectivity strength of dissociable striatal tracts predict individual differences in temporal discounting. J Neurosci 2014, 34:10298-10310. 52. Magen E, Kim B, Dweck CS, Gross JJ, McClure SM: Behavioral and neural correlates of increased self-control in the absence of increased willpower. Proc Natl Acad Sci 2014, 111:9786-9791. This important study used a cognitive manipulation to dissociate impulsive reward decision-making from effortful cognitive control. A cognitive reframing intervention reduced striatal activation to immediate rewards without altering activity in frontoparietal control regions. These findings showed that impulsive choice behavior can result from differences in reward valuation rather than capacity-limited inhibition.
60. Otto AR, Skatova A, Madlon-Kay S, Daw ND: Cognitive control predicts use of model-based reinforcement learning. J Cogn Neurosci 2015, 27:319-333. 61. Schad DJ et al.: Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning. Front Psychol 2014, 5:1450. 62. Hare TA, Camerer CF, Knoepfle DT, O’Doherty JP, Rangel A: Value computations in ventral medial prefrontal cortex during charitable decision making incorporate input from regions involved in social cognition. J Neurosci 2010, 30:583-590.
53. Cooper N, Kable JW, Kim BK, Zauberman G: Brain activity in valuation regions while thinking about the future predicts individual discount rates. J Neurosci 2013, 33:13150-13156.
63. Krebs RM, Boehler CN, Egner T, Woldorff MG: The neural underpinnings of how reward associations can both guide and misguide attention. J Neurosci: Off J Soc Neurosci 2011, 31:9752-9759.
54. Daw ND, Niv Y, Dayan P: Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711.
64. Buckholtz JW, Meyer-Lindenberg A: Psychopathology and the human connectome: toward a transdiagnostic model of risk for mental illness. Neuron 2012, 74:990-1004.
www.sciencedirect.com
Current Opinion in Behavioral Sciences 2015, 3:122–129