The role of the dorsal striatum in extinction: A memory systems perspective

Accepted Manuscript The role of the dorsal striatum in extinction: A memory systems perspective Jarid Goodman, Mark G. Packard PII: DOI: Reference:

S1074-7427(18)30054-6 https://doi.org/10.1016/j.nlm.2018.02.028 YNLME 6820

To appear in:

Neurobiology of Learning and Memory

Received Date: Revised Date: Accepted Date:

21 November 2017 25 January 2018 28 February 2018

Please cite this article as: Goodman, J., Packard, M.G., The role of the dorsal striatum in extinction: A memory systems perspective, Neurobiology of Learning and Memory (2018), doi: https://doi.org/10.1016/j.nlm.2018.02.028

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

1 The role of the dorsal striatum in extinction: A memory systems perspective

Review

Jarid Goodman1 & Mark G. Packard2 1

Department of Psychiatry, Dell Medical School, University of Texas at Austin

2

Department of Psychological and Brain Sciences, Institute for Neuroscience, Texas A&M University

Correspondence:

Mark G. Packard Department of Psychology Texas A&M University College Station, TX 77843 e-mail: [email protected] Fax: 979-845-4727 Phone: 979-845-9504

Highlights: The DLS mediates extinction in maze learning and instrumental learning tasks. Consistent with its role in acquisition, the DLS selectively mediates extinction of habit memory and not cognitive memory. Findings may be relevant to understanding the neurobiology underlying extinction of maladaptive habits in human psychopathology.

2

ABSTRACT

The present review describes a role for the dorsal striatum in extinction. Evidence from brain lesion and pharmacological studies indicate that the dorsolateral region of the striatum (DLS) mediates extinction in various maze learning and instrumental learning tasks. Within the context of a multiple memory systems view, the role of the DLS in extinction appears to be selective. Specifically, the DLS mediates extinction of habit memory and is not required for extinction of cognitive memory. Thus, extinction mechanisms mediated by the DLS may involve response-produced inhibition (e.g. inhibition of existing stimulus-response associations or formation of new inhibitory stimulus-response associations), as opposed to cognitive mechanisms (e.g. changes in expectation). Evidence also suggests that NMDA-dependent forms of synaptic plasticity may be part of the mechanism through which the DLS mediates extinction of habit memory. In addition, in some learning situations, DLS inactivation enhances extinction, suggesting a competitive interaction between multiple memory systems during extinction training. Consistent with a multiple memory systems perspective, it is suggested that the DLS represents one of several distinct neural systems that specialize in extinction of different kinds of memory. The relevance of these findings to the development of behavioral and pharmacological therapies that target the maladaptive habit-like symptoms in human psychopathology is also briefly considered.

3 INTRODUCTION The dorsal striatum of the mammalian brain is part of a group of midbrain structures called the basal ganglia. The dorsal striatum, along with the ventral striatum, is the major input structure of the basal ganglia motor loop, and thus has been classically regarded as an important brain structure for controlling movement (Albin et al., 1989; Mink et al., 1996; Redgrave et al., 1999). However, in the latter half of the 20th century, evidence began to surface indicating that the dorsal striatum also mediates specific kinds of memory (for reviews, see Packard, 2001; Packard and Knowlton, 2002; White, 2009). In terms of its mnemonic function, the dorsal striatum is functionally heterogeneous (Yin and Knowlton, 2006; Devan et al., 2011; Goodman and Packard, 2017). The dorsolateral region of the striatum (DLS) mediates associations between stimuli and responses, i.e. stimulus-response (S-R) learning, as well as acquisition and expression of habitual behavior and egocentric navigation (Packard and Knowlton, 2002). In contrast, the dorsomedial striatum (DMS) mediates associations between actions and outcomes, i.e. action-outcome learning, as well as allocentric spatial navigation and cognitive flexibility (Devan et al., 2011). Differences in the mnemonic functions of these brain regions may be attributed to their distinct input/output pathways (Voorn et al., 2004). The DLS primarily receives input from sensorimotor areas of the cortex, whereas the DMS is innervated by cortical visual/associative areas, as well as subcortical limbic regions mediating emotional and cognitive memory processes, such as the amygdala and hippocampus (Groenewegen et al., 1987; Kelley and Domesick, 1982; Kelley et al., 1982; Lopez-Figuero et al., 1995; McGeorge and Faull, 1989).

4 Although studies on dorsal striatal memory functions have primarily focused on acquisition, consolidation, and retrieval of memory, there is increasing evidence that the dorsal striatum also mediates extinction behavior. Extinction occurs following initial acquisition of a memory, when the original reinforcement is withdrawn, resulting in a decrement of the previously reinforced response. Extinction does not reflect erasure of the original memory, but rather involves new extinction learning which interferes with the old memory (Bouton, 2004; Dunsmoor et al., 2016). The present paper reviews the role of the dorsal striatum in extinction across maze learning and instrumental learning tasks. Evidence specifically implicating the medial region of the dorsal striatum in extinction is limited, and therefore the present review focuses on experiments involving the DLS. Based on these studies, it is inferred that extinction mechanisms subserved by the DLS involve response-produced inhibition (i.e. a mechanism whereby performance of the response during extinction training is necessary to produce a response decrement). In addition, evidence is presented suggesting that the DLS selectively mediates extinction of habit memory. These findings are considered within a multiple memory systems view of extinction. Finally, the relevance of these findings to treating some human psychopathologies is considered, in particular those involving dysfunctional habitual behaviors mediated in part by the DLS.

THE DORSAL STRIATUM AND EXTINCTION: EARLY STUDIES Although the majority of studies examining the role of the dorsal striatum in extinction have been conducted over the past few decades, these recent studies were preceded by a handful of experiments suggesting a role for this brain region in extinction across a range of tasks. For

5 instance, early evidence in monkeys indicated that lesions of the dorsal striatum impaired extinction of instrumental bar-pressing behavior (Butters and Rosvold, 1968; Thompson, 1963). In addition, lesion or electrical stimulation of the dorsal striatum impaired extinction of classically conditioned alimentary responses in cats and dogs (Baranov, 1977; Denisova, 1972, 1981; for a more recent study, see Makarova, 2001), and electrolytic lesions of the dorsal striatum impaired extinction of spatial reversal learning in rats (Kolb, 1977). Electrical stimulation or lesions of the dorsal striatum similarly disrupted instrumental learning and conditioned avoidance responses in rats (Herz and Peeke, 1971; Sanberg, Pisa, and Fibiger, 1978; Schmaltz and Isaacson, 1972; Suvorov, Ermolenko, and Zhodzhaeva, 1974). These early studies provided evidence that manipulating dorsal striatal function impaired extinction learning in some tasks; however, they contained limitations. For instance, lesions were typically made before initial acquisition of the task. Thus, it remains uncertain whether impaired extinction may be attributed directly to an effect on extinction learning, or indirectly via the impairing effect on initial task acquisition (e.g. Herz and Peeke, 1971; Schmaltz and Isaacson, 1972; Kolb, 1977; Sanberg et al., 1978). In addition, manipulations of the dorsal striatum were extensive, incorporating both medial and lateral aspects of the brain region. Therefore, it is difficult to determine whether functional heterogeneity may exist when examining the role of the dorsal striatum in extinction. More recent studies conducted in maze learning, instrumental learning, and conditioned place preference experiments have typically employed updated methodologies to circumvent these issues, while taking into account potential regional differences in dorsal striatal function.

6 EXTINCTION OF MAZE LEARNING Several experiments indicate that the dorsal striatum mediates extinction in a variety of maze learning tasks. In a straight alley maze, rats are trained to traverse a straight path to obtain a food reinforcer at the opposite end of the maze. Over the course of training, animals display a decline in runway latencies to reach the goal end, indicating successful acquisition of the task. For extinction training, the food reinforcer is removed, and over time the runway latencies increase, indicating extinction. Permanent electrolytic lesions conducted before acquisition training and incorporating both medial and lateral aspects of the dorsal striatum impair extinction in the straight alley maze (Thullier et al., 1996). A similar impairment in extinction of straight alley maze performance was also observed using pre-acquisition kainic-acid induced lesions confined to the DLS, but no effect on extinction was observed with DMS lesions (Dunnett and Iversen, 1981). It should be noted that the DLS lesions also impaired acquisition in the straight alley maze (Dunnett and Iversen, 1981). Thus, based on these findings, it is difficult to determine whether the effect of DLS lesions on extinction in the straight alley maze can be attributed to prior differences in initial task acquisition. These findings suggest that the DLS may be involved in straight alley maze extinction, but do not identify the extinction mechanisms through which the DLS operates. According to the S-R view of extinction (Hull, 1942), upon removal of the reinforcer, stimuli in the learning environment may activate inhibition of the running approach response. Similar to other theories of extinction learning involving response-produced inhibition (Hulse, Egeth, and Deese, 1975), the subject must make the original running response during extinction training for the inhibitory S-R association to be acquired. On the other hand, cognitive mechanisms may be involved in extinction to the extent that the animal learns to associate the original goal location with the

7 absence of reinforcement, resulting in a change in expectation (Tolman, 1932). Contrary to the S-R view of extinction, this change in expectation can be acquired without performing the original response, a phenomenon termed “latent extinction” (Seward and Levy, 1949). Response- or cognitive-based extinction procedures may effectively produce a response decrement in the straight alley maze. For response extinction, animals are released from the original start position and have the opportunity to perform the original running response to the empty goal location. For latent extinction (a cognitive form of extinction training), animals are confined to the original goal location without food. Upon returning to the start position, animals that previously received latent extinction show greater latencies to approach the goal location, suggesting a change in expectation (i.e. expecting the absence of reinforcement; Seward and Levy, 1949). Reversibly inactivating the DLS with the sodium channel blocker bupivacaine prior to each session of extinction training attenuates the increase in runway latencies during response extinction training, but does not influence the effectiveness of latent extinction training (Goodman, Gabriele, and Packard, 2017). These results suggest that DLS function is required for extinction mechanisms involving response-produced inhibition (e.g. inhibitory S-R associations), but not for cognitive extinction mechanisms (e.g. changes in expectation regarding the absence of reinforcement). In addition, the use of reversible DLS inactivation immediately prior to extinction training rules out the possibility that impaired extinction may be attributed to DLS influencing initial task acquisition. Regarding the neurobiology of latent extinction, evidence indicates a role for hippocampal function (Gabriele and Packard, 2006; Goodman, Gabriele, and Packard, 2016). However, the hippocampus, as well as the DMS, are not required for response extinction in the

8 straight alley maze (Gabriele and Packard, 2006; Dunnett and Iversen, 1981). Taken together, these findings demonstrate a double dissociation, in which the DLS mediates response extinction but not latent extinction, and the hippocampus mediates latent extinction but not response extinction. Thus, it may be speculated that the DLS and hippocampus promote extinction via different learning mechanisms, the former through response-produced inhibition and the latter through changes in expectation. The precise nature of the individual extinction mechanisms mediated by the DLS and hippocampus in this task requires further investigation. In addition to the straight alley maze, the dorsal striatum may also mediate extinction in the T-maze. This task involves a maze with three arms arranged in a “T” formation. For each trial, the rat is placed into the start arm of the maze and may retrieve food reinforcement by making a consistent body turn (i.e. a right or left turn) at the intersection of the maze to enter a consistently reinforced goal arm. Pre-acquisition kainic acid-induced lesions of the DLS or DMS impaired both acquisition and extinction in the T-maze (Dunnett and Iversen, 1981). In a modified water-maze version of the T-maze, rats are trained to make a consistent body turn at the choice point to enter a goal arm that contains a submerged platform, which the animal can mount to escape the water. The DLS was inactivated immediately after extinction training in this task (Campus et al., 2015). Post-training drug administration targets memory consolidation (McGaugh, 2000), i.e. the process through which a labile short-term memory is transformed into a long-term memory trace. Thus, animals that had received post-training DLS inactivation with the GABA-A agonist muscimol demonstrated lower latencies to the previously reinforced goal location compared to control animals, indicating that consolidation of the extinction memory was impaired (Campus et al., 2015). In addition, since the DLS was temporarily inactivated after extinction training, the observed impairment may not be attributed to a potential effect of DLS

9 inactivation on non-mnemonic factors occurring during extinction training or the retrieval test (e.g. motor performance, sensory processing, motivation, etc.).

EXTINCTION OF HABIT MEMORY The straight alley maze and T-maze experiments discussed above have revealed a role for the DLS in the acquisition and consolidation of extinction memory. However, the kind of memory underlying initial acquisition in the straight alley maze and T-maze remains unknown. It is possible that during initial acquisition in these tasks, animals learn (1) the allocentric spatial location of the goal (i.e. a place strategy), (2) a habitual running/turning response (i.e. an S-R strategy), or (3) memory incorporating both spatial and S-R information. As the kind of memory undergoing extinction in these tasks remains unspecified, it is difficult to determine whether the role of the DLS in extinction is selective for a specific kind of memory. To examine the potential selectivity of the DLS in extinction, rats were trained in a pair of plus-maze tasks, i.e. a response learning task and a place learning task, each of which requires acquisition of a specific kind of memory (Goodman, Ressler, and Packard, 2016). In a response learning task, rats were released from opposite starting locations (i.e. North and South arms) throughout initial acquisition and were reinforced to make a consistent body-turn at the choice point (i.e. a left turn) to retrieve food reinforcement from one of the goal arms (East or West). Response learning involves acquisition of an egocentric turning response and requires DLS function, but not hippocampal function (Asem and Holland, 2015; Chang and Gold, 2003, 2004; Compton, 2004; Packard and McGaugh, 1996; Yin and Knowlton, 2004). A separate group of rats were trained in a place learning task, in which animals were similarly released from opposite starting positions, while food reinforcement remained in a consistent goal location. This task

10 requires memory for the spatial location of the food reinforcer in order to navigate from different starting positions to the goal. Previous evidence indicates that place learning in the plus-maze involves hippocampal function, but not DLS function (Chang and Gold, 2003; Compton, 2004; Packard and McGaugh, 1996; Schroeder, Wingard, and Packard, 2002). Following acquisition in the DLS-dependent response learning task or hippocampusdependent place learning task, rats underwent extinction training. Immediately after extinction training, the DLS was inactivated with bupivacaine and 24 hours later rats were subjected to a second day of extinction training to probe the effects of prior DLS inactivation. Post-training DLS inactivation blocked consolidation of extinction in the response learning task, but not in the place learning task (Goodman, Ressler, and Packard, 2016). These findings suggest that the role of the DLS in extinction is selective to DLS-dependent habit memory and is not required for extinction of hippocampus-dependent cognitive memory. The selective role of the DLS in extinction of DLS-dependent habit memory may be helpful for interpreting the findings from prior T-maze and straight alley maze experiments. Initial acquisition of the T-maze and straight alley maze may involve both DLS- and hippocampus-dependent memory processes (see Dunnett and Iversen, 1981; Kirkby, Polgar, and Coyle, 1981; Packard and McGaugh, 1996; Rawlins, Feldon, Ursin, and Gray, 1985). However, given the findings of the place and response learning experiments discussed above, it is possible that the impairing effect of DLS manipulations on extinction in the T-maze and straight alley maze tasks observed in earlier studies may be attributed to impaired extinction of DLSdependent habit memory, rather than hippocampus-dependent cognitive memory.

11 EXTINCTION OF DRUG-REINFORCED INSTRUMENTAL LEARNING AND CONDITIONED PLACE PREFERENCE BEHAVIOR In addition to maze learning tasks, early evidence indicates that the dorsal striatum is also involved in extinction of instrumental learning (Butters and Rosvold, 1968; Herz and Peeke, 1971; Sanberg, Pisa, and Fibiger, 1978; Schmaltz and Isaacson, 1972; Suvorov, Ermolenko, and Zhodzhaeva, 1974; Thompson, 1963). Although several early studies employed natural rewards as reinforcers, such as food and water, extensive recent evidence has indicated a role for the DLS in extinction of instrumental responding for addictive drugs. These experiments have employed self-administration paradigms, in which during initial acquisition animals learn to press a lever to self-administer an addictive drug. During subsequent extinction training, lever presses no longer result in drug infusions. Extinction of instrumental responding for cocaine or methamphetamine has been associated with multiple changes in dorsal striatal physiology. These changes include higher proenkephalin expression (Crespo et al., 2001), adaptations in D2 and A2A receptor binding (Frankowska et al., 2013), higher NMDA receptor expression (Ghasemzadeh et al., 2009a), increased expression of metabotropic glutamate receptors (Ghasemzadeh et al., 2009b; Schwendt et al., 2012), greater expression of PICK1 scaffolding protein (Ghasemzadeh et al., 2009a), and higher neurotensin levels (Hanson et al., 2013). In addition, suppression of the immediate early gene Arc in the DLS impairs extinction of cocaine self-administration (Hearing et al., 2011), suggesting a role for plasticity-related genes in the DLS. Disrupting function of metabotropic glutamate receptors in the DLS also impairs extinction of cocaine self-administration, which may be attributed to downstream disruption of DLS Arc expression (Knackstedt et al., 2014; Knackstedt and Schwendt, 2016). In possible contrast to studies suggesting involvement of the

12 DLS in extinction of drug-reinforced behavior, one study found that after prolonged drug abstinence, pre-training DLS inactivation was associated with a blockade of cocaine seeking during an initial drug free “relapse session” and blockade of “rebound” responding during subsequent extinction of cocaine-reinforced lever pressing (See, Elliott and Feltenstein, 2007). One limitation to these drug self-administration studies discussed above is that the precise kind of memory undergoing extinction remains unknown. As previously discussed for the straight alley maze and T-maze paradigms, drug self-administration may be acquired using different learning strategies (Dickinson, 1985). For instance, it is possible that (1) animals acquire a goal-directed strategy, in which they learn to press a lever to obtain the outcome (i.e. intravenous drug administration) or (2) animals may acquire a habit-based strategy, in which stimuli in the learning environment activate habitual lever pressing. Evidence indicates that habitual lever pressing depends on DLS function (Yin, Knowlton, and Balleine, 2004), whereas goal-directed lever pressing involves DMS or hippocampal function (Corbit and Balleine, 2000; Yin, Ostlund, Knowlton, and Balleine, 2005). Considering that the drug self-administration studies discussed above did not gauge whether initial acquisition was guided by a habitual or goal-directed strategy, it remains unknown whether the DLS was implicated in extinction of DLS-dependent habit memory or DMS/hippocampus-dependent cognitive memory. However, addictive drugs typically bias animals toward the use of a habit-based strategy over a goal-directed strategy in instrumental learning tasks (for review, see Goodman and Packard, 2016). Therefore, it is possible that the findings indicating a role for the DLS in extinction of drug self-administration might reflect the involvement of this brain region in extinction of DLS-dependent habit memory. In one study consistent with this idea, investigators trained rats to self-administer ethanol using either a goal-

13 directed or habit-based strategy (Klein, Fanelli, and Robinson, 2012). During extinction training, single-unit electrophysiological recordings indicated more phasic changes in neural firing in the DMS relative to the DLS in goal-directed rats, and more phasic changes in the DLS relative to the DMS in habitual rats. These findings suggest that the extent to which the DLS or DMS participates in extinction of drug-self administration depends on the memory system used during acquisition, similar to that observed in the plus-maze (Goodman, Ressler, and Packard, 2016). In addition to drug-self administration studies, a potential role for the dorsal striatum in extinction of drug-related behavior has also been demonstrated in conditioned place preference (CPP) tasks. In drug-induced CPP, the affective consequences of an addictive drug are associated with one environment, whereas a control treatment is associated with another “neutral” environment. During a subsequent drug-free preference test, in which animals have the opportunity to freely explore both environments, rats typically show a preference for the context previously paired with the drug, suggesting animals had associated the rewarding properties of the drug with the environmental cues of the drug-paired compartment. Following acquisition, extinction of CPP can be produced by re-confining the animal to the preferred compartment without drug administration (Hsu and Packard, 2008; Hsu, Schroeder, and Packard, 2002; Schroeder and Packard, 2003). Whereas acquisition of a morphine CPP is associated with increased BDNF levels in the dorsal striatum, subsequent extinction in this task is associated with a decrease in dorsal striatal BDNF levels (Meng et al., 2013). In addition, extinction of a cocaine-induced CPP involves function of midkine neurotrophic factor and tyrosine phosphorylation of the peroxiredoxin 6 enzyme in the dorsal striatum (Gramage et al., 2013). It should be noted that these CPP studies did not examine whether there were regional differences across the medial and lateral aspects of the dorsal striatum, and thus it is not clear whether the

14 DLS and DMS are differentially involved in extinction of drug-induced CPP. Moreover, these CPP studies employed correlational analyses, and therefore do not reveal a causal role for the dorsal striatum in extinction of drug-induced CPP.

MULTIPLE MEMORY SYSTEMS AND EXTINCTION The dorsal striatum, hippocampus, and amygdala have been regarded as the central structures of distinct memory systems, in which the dorsal striatum mediates S-R/habit memory, the hippocampus mediates cognitive spatial memory, and the amygdala mediates stimulusaffect/emotional memories (White and McDonald, 2002; Squire, 2004; White, Packard, and McDonald, 2013). This multiple memory systems hypothesis has been primarily supported by extensive research examining initial acquisition, consolidation, and retrieval of memory. However, there is evidence that extinction of memory also involves multiple memory systems. Indeed, experiments reviewed above suggest that the DLS selectively mediates extinction of habit memory, but not extinction of cognitive spatial memory (Goodman, Ressler, and Packard, 2016) or extinction of conditioned fear (Wendler et al., 2014). Extinction of cognitive spatial memory is mediated in part by the hippocampus (Gabriele and Packard, 2006; Goodman, Gabriele, and Packard, 2016), whereas extinction of conditioned fear is mediated by a circuit involving the medial prefrontal cortex and amygdala, among other brain regions (Maren, 2015). Therefore, the DLS, hippocampus, and amygdala may represent the nodes of distinct neural systems that each support extinction of different kinds of memory. In addition, there may also be regional differences across the dorsal striatum itself. Although data specifically pertaining to the DMS are limited, in some learning situations the DMS and DLS are differentially implicated in extinction. For instance, the DMS but not the

15 DLS is required for extinction of a spatial alternation in the T-maze (Moussa et al., 2011). In addition, in an instrumental learning task, the DLS but not the DMS has been implicated in extinction of S-R/habitual learning, whereas the DMS but not the DLS has been associated with extinction of goal-directed learning (Klein et al., 2012). Thus, it appears that the DLS selectively mediates extinction of habit memory, whereas the DMS mediates extinction of cognitive memory, similar to their roles in initial acquisition and retrieval (Yin and Knowlton, 2006; Devan et al., 2011).

COMPETITION BETWEEN MEMORY SYSTEMS DURING EXTINCTION In addition to the studies discussed above in which the DLS critically mediates extinction, there are a few studies suggesting that in some learning situations lesion or inactivation of the DLS actually enhances extinction. Inactivating the DLS immediately after extinction training with bupivacaine slightly enhances the consolidation of extinction in the place learning version of the plus-maze (Goodman, Ressler, and Packard, 2016). Likewise, excitotoxic lesions of the DLS enhance extinction of spatial alternation in a modified T-maze, whereas DMS lesions impair extinction in this task (Moussa et al., 2011). Lastly, selective lesions of either the DLS or DMS are associated with enhanced extinction of two-way active avoidance (Wendler et al., 2014). These findings are consistent with the view that anatomically distinct memory systems sometimes interact with each other in a competitive manner, such that disrupting one memory system may enhance function of another intact memory system (Poldrack and Packard, 2003). Consistent with a competitive interaction between memory systems, the DLS may mediate learning mechanisms that are not ideal for extinction in some tasks. Lesions of this brain region

16 may eliminate this sub-optimal kind of extinction learning and allow another brain region that is better suited for extinction in these tasks (e.g. the hippocampus or DMS) to seize control. Competitive interactions between memory systems have been demonstrated primarily during initial acquisition, consolidation, and retrieval of memory (for reviews, see Poldrack and Packard, 2003; Packard and Goodman, 2013). The observation that DLS inactivation enhances extinction in some tasks suggests that memory systems may continue to compete during extinction, although the neural mechanisms underlying competitive interactions among memory systems during extinction are unknown.

POTENTIAL DORSAL STRIATAL MECHANISMS OF EXTINCTION: FROM LEARNING THEORY TO SYNAPTIC PLASTICITY There are multiple potential learning mechanisms that the DLS could subserve in mediating extinction. As mentioned above, several investigators have proposed responseproduced inhibition theories of extinction learning, including the hypothesis that extinction may be achieved through acquisition of a novel inhibitory S-R association (Hull, 1943; Rescorla, 2001). According to this hypothesis, during extinction training, stimuli in the learning environment may develop the ability to inhibit the original response. Consistent with extensive evidence that the DLS mediates S-R learning (Packard and Knowlton, 2002), this brain region may also mediate inhibitory S-R associations, which can putatively promote extinction (Rescorla, 2001). Importantly, the DLS may not mediate changes in expectation during extinction training given that DLS inactivation does not influence latent extinction (Goodman, Gabriele, and Packard, 2017).

17 The DLS may also promote extinction via learned avoidance. Acquisition of passive avoidance and active avoidance is partially mediated by DLS function (Salado-Castillo et al., 1996; Wendler et al., 2014). Thus, one possibility is that during extinction training the DLS mediates avoidance behavior, in which the original response is suppressed. This avoidance behavior may be negatively reinforced by preventing frustration, which is hypothesized to occur during extinction training when the original response no longer leads to a positive outcome (Amsel, 1962). Another possibility is that as the animal learns to avoid the original response, a new response is acquired in parallel. For instance, during extinction training in the response learning plus-maze task, animals may learn to make the opposite body-turn response. To the extent that the DLS has been critically implicated in initial acquisition of response learning, this brain region may also be involved in acquiring novel responses during extinction training. Indeed, DLS function has been associated with reversal learning in a T-maze task (Rueda-Orozco et al., 2008), and the DLS has also been implicated in acquiring new strategies following the removal of reinforcement (Skelin et al., 2014). Regarding the neurobiological mechanisms through which the dorsal striatum mediates extinction, there is evidence for a role of glutamatergic neurotransmission. Intra-DLS administration of the NMDA receptor antagonist AP5 is sufficient to impair extinction of response learning in the plus-maze, whereas intra-DLS administration of the NMDA receptor agonist D-cycloserine enhances extinction in this task (Goodman, Ressler, and Packard, 2017). In addition, as mentioned above, extinction of drug self-administration is associated with increased expression of metabotropic glutamate receptors and NMDA receptors in the DLS (Ghasemzadeh et al., 2009a,b; Schwendt et al., 2012), whereas inhibiting DLS metabotropic

18 glutamate receptors impairs extinction of drug self-administration (Knackstedt et al., 2014). These findings are consistent with extensive prior evidence indicating that glutamatergic neurotransmission (in particular NMDA receptor activation) mediates extinction across a range of learning and memory tasks (Baker and Azorlosa, 1996; Falls et al., 1992; Gabriele and Packard, 2007; Goodman, Gabriele, and Packard, 2016; Kehoe et al., 1996; Thompson and Disterhoft, 1997). The observation that dorsal striatal extinction mechanisms partially depend on activation of NMDA receptors suggests a potential role for NMDA-dependent forms of synaptic plasticity. Indeed, many investigators have considered synaptic plasticity as a potential neurobiological substrate for learning and memory (Bliss & Collingridge, 1993; Hebb, 1949; Kandel, 2001; Martin, Grimwood, and Morris, 2000), including for extinction memory (e.g. Maren, 2015). Evidence indicates that NMDA receptor activity at corticostriatal synapses is required for longterm potentiation (LTP) and synaptic depotentiation (i.e. the reversal of LTP) in the DLS (Li, Li, and Han, 2009; Picconi et al., 2011). Therefore, it is tempting to speculate that long-term changes in corticostriatal glutamate release following LTP or synaptic depotentiation may underlie extinction in the maze learning and instrumental learning tasks discussed above. The suggestion that DLS mechanisms of extinction involve synaptic plasticity is consistent with evidence indicating that Arc (i.e. a plasticity-related immediate early gene) in the DLS is required for extinction of drug self-administration (Hearing et al., 2011; Knackstedt and Schwendt, 2016). Considering the well-known role of dopamine in learning and memory processes of the dorsal striatum (Packard and White, 1991; Packard and McGaugh, 1994; Faure, Haberland, Condé, and Massioui, 2005; for review, see Packard and Goodman, 2017), it is possible that

19 dopaminergic mechanisms in this brain region also contribute to extinction. D1 and D2 dopamine receptors are preferentially expressed on distinct subtypes of striatal medium spiny neurons (MSNs) which form two different pathways with putatively opposite roles in action selection. The D1 direct pathway is believed to promote desired action sequences, whereas the D2 indirect pathway suppresses alternative, undesired action sequences (Albin, Young, and Penney, 1989; Delong, 1990; Kravitz et al., 2010; Jin et al., 2014). Consistent with their roles in action selection, it may be speculated that D1 and D2 pathways modulate extinction in a competitive manner. D1 pathway activation may be associated with inhibition of extinction, whereas D2 pathway activation may promote extinction. Indeed, evidence indicates that intraventricular administration of a D1 antagonist enhances, and intra-ventricular administration of a D2 antagonist impairs, extinction in an appetitive T-maze task (André and Manahan-Vaughan, 2016). Whether these effects can be specifically attributed to inhibition of D1 and D2 pathways in the striatum has not been investigated.

EXTINCTION OF MALADAPTIVE HABITS IN PSYCHOPATHOLOGY The hypothesis that some human neuropsychiatric disorders reflect maladaptive dorsal striatal memory processes has garnered increasing attention. Habit-like behavioral symptoms, such as tics, repetitive/stereotyped behaviors, or drug taking, may reflect heightened engagement of DLS-dependent habit memory in a variety of human psychopathologies, including Tourette’s syndrome, obsessive-compulsive disorder, drug addiction/relapse, post-traumatic stress disorder, and others (Everitt and Robbins, 2005, 2013; Gillan and Robbins, 2014; Goh and Peterson, 2012; Goodman et al., 2012, 2014; Goodman and Packard, 2016; Graybiel and Rauch, 2000; Schwabe et al., 2011; White, 1996). In addition, the enhanced formation and expression of habit-like

20 symptoms in these disorders following stressful life events, such as stress-induced relapse to drug taking, may be attributed to stress and anxiety augmenting DLS-dependent memory function (for review, see Goodman et al., 2012; Goodman and Packard, 2016; Schwabe et al., 2011; Schwabe, 2013). Considering the putative role of DLS-dependent habit memory in some human psychopathologies, extinction of habit memory may be relevant to understanding the neurobehavioral mechanisms through which dysfunctional habits (e.g. drug addiction) may be suppressed. Indeed, relative to a cognitive extinction protocol (i.e. latent extinction), DLSdependent response extinction protocols are more effective at extinguishing habit memory in the plus-maze and cocaine-reinforced running in the straight alley maze (Gabriele, Setlow, and Packard, 2009; Goodman, Gabriele, and Packard, 2016; Goodman and Packard, 2015). Therefore, it may be speculated that response-based therapy would be more effective than cognitive-based therapy in alleviating the maladaptive habits present in some human psychopathologies. Research on extinction of habit memory may not only be relevant for selecting an appropriate behavioral therapy, but may also be instructive in combatting maladaptive habits through pharmacological methods. For instance, research implicating NMDA receptor activity in extinction of habit memory suggests that pharmacological treatments targeting the NMDA receptor may be useful for suppressing dysfunctional habit-like symptoms. As mentioned above, the NMDA receptor agonist D-cycloserine improves extinction of habit memory in the plusmaze (Goodman, Ressler, and Packard, 2017), and therefore D-cycloserine may be considered as a potential adjunct to behavioral extinction-based therapy for alleviating maladaptive habits. Indeed, D-cycloserine augments extinction in a variety of animal learning tasks, which serve as

21 models of maladaptive memory, including Pavlovian fear conditioning (Ledgerwood et al, 2003; Walker et al., 2002) and cocaine self-administration (Thanos et al., 2011). Consistent with the translational applications of this research, D-cycloserine has also been shown to improve extinction of maladaptive memory in humans. For instance, D-cycloserine boosts the efficacy of extinction-based therapy in post-traumatic stress disorder (Heresco-Levy et al., 2002), drug addiction (Oliveto et al., 2003), obsessive-compulsive disorder (Kushner et al., 2007), and phobia disorders (Ressler et al., 2004). Interestingly, each of these disorders may also be associated with habit-like symptoms mediated by DLS-dependent memory (Gillan and Robbins, 2014; Goodman and Packard, 2016; Graybiel and Rauch, 2000; Schwabe, Wolf, and Oitzl, 2010). Whether D-cycloserine specifically targets the habit-like behavioral symptoms in these disorders has not been thoroughly established. In addition, it will be useful to determine in future studies whether other pharmacological treatments that enhance extinction of conditioned fear and drug-reinforced behavior (e.g., drugs targeting dopamine, mGluR, and cannabinoid systems; Lafenêtre, Chaouloff, and Marsicano, 2007; Cleva, Gass, Widholm, & Olive, 2010; Abraham, Neve, and Lattal, 2014; Xu et al., 2013) may also facilitate extinction of habit memory.

CONCLUSION The present review describes a prominent role for the dorsal striatum in extinction (see Table 1). In particular, these findings indicate that the DLS mediates extinction across maze learning and instrumental learning tasks. The DLS may incite a kind of response-produced inhibition (e.g. inhibitory S-R associations) to promote extinction, rather than relying on cognitive forms of extinction learning, such as those involving changes in expectation. Although

22 the precise neurobiological mechanisms in the DLS that support extinction remain unknown, there is evidence indicating a role for NMDA receptors, potentially implicating the involvement of NMDA-dependent forms of synaptic plasticity (e.g. LTP and synaptic depotentiation) at corticostriatal synapses. Findings regarding the behavioral and neural mechanisms underlying extinction of DLS-dependent habit memory may be useful for developing treatments to combat the maladaptive habit-like behavioral features of some human psychopathologies.

23

REFERENCES Abraham, A. D., Neve, K. A., & Lattal, K. M. (2014). Dopamine and extinction: a convergence of theory with fear and reward circuitry. Neurobiology of Learning and Memory, 108, 65-77. Albin, R. L., Young, A. B., & Penney, J. B. (1989). The functional anatomy of basal ganglia disorders. Trends in neurosciences, 12(10), 366-375. Amsel, A. (1962). Frustrative nonreward in partial reinforcement and discrimination learning: Some recent history and a theoretical extension. Psychological Review, 69(4), 306. André, M. A. E., & Manahan-Vaughan, D. (2016). Involvement of dopamine D1/D5 and D2 receptors in context-dependent extinction learning and memory reinstatement. Frontiers in Behavioral Neuroscience, 9, 372. Asem, J. S. A., & Holland, P. C. (2015). Dorsolateral striatum implicated in the acquisition, but not expression, of immediate response learning in rodent submerged T-maze. Neurobiology of learning and memory, 123, 205-216. Baker, J. D., & Azorlosa, J. L. (1996). The NMDA antagonist MK-801 blocks the extinction of Pavlovian fear conditioning. Behavioral Neuroscience, 110(3), 618-620. Baranov, V. V. (1977). Extinction of inhibition following damage to the orbital cortex and ventral portion of the head of the caudate nucleus in dogs. Z imeni IP Pavlova, 27(1), 196-199. Bliss, T.V., & Collingridge, G.L. (1993). A synaptic model of memory: long-term potentiation in the hippocampus. Nature, 361, 31-39. Bouton, M. E. (2004). Context and behavioral processes in extinction. Learning & memory, 11(5), 485-494. Butters, N., & Rosvold, H. E. (1968). Effect of caudate and septal nuclei lesions on resistance to extinction and delayed-alternation. Journal of comparative and physiological psychology, 65(3p1), 397-403. Campus, P., Colelli, V., Orsini, C., Sarra, D., & Cabib, S. (2015). Evidence for the involvement of extinction-associated inhibitory learning in the forced swimming test. Behavioural Brain Research, 278, 348-355. Chang, Q., & Gold, P. E. (2003). Intra-hippocampal lidocaine injections impair acquisition of a place task and facilitate acquisition of a response task in rats. Behavioural brain research, 144(1), 19-24.

24 Chang, Q., & Gold, P. E. (2004). Inactivation of dorsolateral striatum impairs acquisition of response learning in cue-deficient, but not cue-available, conditions. Behavioral neuroscience, 118(2), 383-388. Cleva, R. M., Gass, J. T., Widholm, J. J., & Olive, M. F. (2010). Glutamatergic targets for enhancing extinction learning in drug addiction. Current Neuropharmacology, 8(4), 394-408. Compton, D. M. (2004). Behavior strategy learning in rat: effects of lesions of the dorsal striatum or dorsal hippocampus. Behavioural Processes, 67(3), 335-342. Corbit, L. H., & Balleine, B. W. (2000). The role of the hippocampus in instrumental conditioning. Journal of Neuroscience, 20(11), 4233-4239. Crespo, J. A., Manzanares, J., Oliva, J. M., Corchero, J., Palomo, T., & Ambrosio, E. (2001). Extinction of cocaine self-administration produces a differential time-related regulation of proenkephalin gene expression in rat brain. Neuropsychopharmacology, 25(2), 185-194. DeLong, M. R. (1990). Primate models of movement disorders of basal ganglia origin. Trends in Neurosciences, 13(7), 281-285. Denisova, A. S. (1972). Role of caudate nuclei in the mechanism of extinction of conditioned alimentary reactions. Zhurnal vysshei nervnoi deiatelnosti imeni IP Pavlova, 22(2), 260-265. Denisova, A. S. (1981). Effect of electrical stimulation of the somatosensory cortex and caudate nucleus on extinctive inhibition of a food conditioned reflex to sound. Neuroscience and behavioral physiology, 11(6), 576-581. Devan, B. D., Hong, N. S., & McDonald, R. J. (2011). Parallel associative processing in the dorsal striatum: Segregation of stimulus–response and cognitive control subregions. Neurobiology of learning and memory, 96(2), 95-120. Dickinson, A. (1985). Actions and habits: the development of behavioural autonomy. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 308(1135), 67-78. Dunnett, S. B., & Iversen, S. D. (1981). Learning impairments following selective kainic acidinduced lesions within the neostriatum of rats. Behavioural brain research, 2(2), 189-209. Dunsmoor, J. E., Niv, Y., Daw, N., & Phelps, E. A. (2015). Rethinking extinction. Neuron, 88(1), 47-63. Everitt, B. J., & Robbins, T. W. (2005). Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nature Neuroscience, 8(11), 1481-1489. Everitt, B. J., & Robbins, T. W. (2013). From the ventral to the dorsal striatum: devolving views of their roles in drug addiction. Neuroscience & Biobehavioral Reviews, 37(9), 1946-1954.

25

Falls, W. A., Miserendino, M. J., & Davis, M. (1992). Extinction of fear-potentiated startle: blockade by infusion of an NMDA antagonist into the amygdala. The Journal of Neuroscience, 12(3), 854-863. Faure, A., Haberland, U., Condé, F., & El Massioui, N. (2005). Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. Journal of Neuroscience, 25(11), 2771-2780. Frankowska, M., Marcellino, D., Adamczyk, P., Filip, M., & Fuxe, K. (2013). Effects of cocaine self‐administration and extinction on D2‐like and A2A receptor recognition and D2‐like/Gi protein coupling in rat striatum. Addiction Biology, 18(3), 455-466. Gabriele, A., & Packard, M. G. (2006). Evidence of a role for multiple memory systems in behavioral extinction. Neurobiology of Learning and Memory, 85(3), 289-299. Gabriele, A., & Packard, M. G. (2007). D-Cycloserine enhances memory consolidation of hippocampus-dependent latent extinction. Learning & memory, 14(7), 468-471. Gabriele, A., Setlow, B., & Packard, M. G. (2009). Cocaine self-administration alters the relative effectiveness of multiple memory systems during extinction. Learning & Memory, 16(5), 296299. Ghasemzadeh, M. B., Mueller, C., & Vasudevan, P. (2009a). Behavioral sensitization to cocaine is associated with increased glutamate receptor trafficking to the postsynaptic density after extended withdrawal period. Neuroscience, 159(1), 414-426. Ghasemzadeh, M. B., Vasudevan, P., Mueller, C. R., Seubert, C., & Mantsch, J. R. (2009b). Region-specific alterations in glutamate receptor expression and subcellular distribution following extinction of cocaine self-administration. Brain Research, 1267, 89-102. Gillan, C. M., & Robbins, T. W. (2014). Goal-directed learning and obsessive–compulsive disorder. Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1655), 20130475. Goh, S., & Peterson, B. S. (2012). Imaging evidence for disturbances in multiple learning and memory systems in persons with autism spectrum disorders. Developmental Medicine & Child Neurology, 54(3), 208-213. Goodman, J., Gabriele, A., & Packard, M. G. (2016). Hippocampus NMDA receptors selectively mediate latent extinction of place learning. Hippocampus, 26, 1115-1123. Goodman, J., Gabriele, A., & Packard, M. G. (2017). Differential effects of neural inactivation of the dorsolateral striatum on response and latent extinction. Behavioral Neuroscience, 131, 143148.

26 Goodman, J., Leong, K. C., & Packard, M. G. (2012). Emotional modulation of multiple memory systems: implications for the neurobiology of post-traumatic stress disorder. Goodman, J., Marsh, R., Peterson, B. S., & Packard, M. G. (2014). Annual research review: the neurobehavioral development of multiple memory systems–implications for childhood and adolescent psychiatric disorders. Journal of Child Psychology and Psychiatry, 55(6), 582-610. Goodman, J., & Packard, M. G. (2015). The memory system engaged during acquisition determines the effectiveness of different extinction protocols. Frontiers in Behavioral Neuroscience, 9, 314. Goodman, J., & Packard, M. G. (2016). Memory systems and the addicted brain. Frontiers in Psychiatry, 7, 24. Goodman, J., & Packard, M. G. (2017). Memory systems of the basal ganglia. In H. Steiner & K. Y. Tseng (Eds.), Handbook of basal ganglia structure and function 2nd ed. (pages 725-740). Academic Press/Elsevier. Goodman, J., Ressler, R. L., & Packard, M. G. (2016). The dorsolateral striatum selectively mediates extinction of habit memory. Neurobiology of Learning and Memory, 136, 54-62. Goodman, J., Ressler, R. L., & Packard, M. G. (2017). Enhancing and impairing extinction of habit memory through modulation of NMDA receptors in the dorsolateral striatum. Neuroscience, 352, 216-225. Gramage, E., Pérez-García, C., Vicente-Rodríguez, M., Bollen, S., Rojo, L., & Herradón, G. (2013). Regulation of extinction of cocaine-induced place preference by midkine is related to a differential phosphorylation of peroxiredoxin 6 in dorsal striatum. Behavioural brain research, 253, 223-231. Graybiel, A. M., & Rauch, S. L. (2000). Toward a neurobiology of obsessive-compulsive disorder. Neuron, 28(2), 343-347. Groenewegen HJ, Vermeulen-Van der Zee E, te Kortschot A, Witter MP (1987) Organization of the projections from the subiculum to the ventral striatum in the rat A study using anterograde transport of Phaseolus vulgaris leucoagglutinin. Neuroscience, 23, 103-120. Hanson, G. R., Hoonakker, A. J., Robson, C. M., McFadden, L. M., Frankel, P. S., & Alburges, M. E. (2013). Response of neurotensin basal ganglia systems during extinction of methamphetamine self-administration in rat. Journal of Pharmacology and Experimental Therapeutics, 346(2), 173-181. Hearing, M. C., Schwendt, M., & McGinty, J. F. (2011). Suppression of activity-regulated cytoskeleton-associated gene expression in the dorsal striatum attenuates extinction of cocaineseeking. International Journal of Neuropsychopharmacology, 14(6), 784-795.

27 Hebb, D.O. (1949). The organization of behavior. New York: Wiley. Heresco-Levy, U., Kremer, I., Javitt, D. C., Goichman, R., Reshef, A., Blanaru, M., & Cohen, T. (2002). Pilot-controlled trial of D-cycloserine for the treatment of post-traumatic stress disorder. The International Journal of Neuropsychopharmacology, 5(04), 301-307. Herz, M. J., & Peeke, H. V. (1971). Impairment of extinction with caudate nucleus stimulation. Brain research, 33(2), 519-522. Hull, C. L. (1943) Principles of behavior. New York: Appleton-Century-Crofts. Hulse, S.H., Egeth, H., & Deese, J. (1975). The psychology of learning. New York: McGrawHill. Hsu, E., & Packard, M. G. (2008). Medial prefrontal cortex infusions of bupivacaine or AP-5 block extinction of amphetamine conditioned place preference. Neurobiology of learning and memory, 89(4), 504-512. Hsu, E. H., Schroeder, J. P., & Packard, M. G. (2002). The amygdala mediates memory consolidation for an amphetamine conditioned place preference. Behavioural brain research, 129(1), 93-100. Kandel, E.R. (2001). Neuroscience – The molecular biology of memory storage: a dialogue between genes and synapses. Science, 294, 1030-1038. Jin, X., Tecuapetla, F., & Costa, R. M. (2014). Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences. Nature Neuroscience, 17(3), 423-430. Kehoe, E. J., Macrae, M., & Hutchinson, C. L. (1996). MK-801 protects conditioned responses from extinction in the rabbit nictitating membrane preparation. Psychobiology, 24(2), 127-135. Kelley AE, Domesick VB (1982) The distribution of the projection from the hippocampal formation to the nucleus accumbens in the rat – an anterograde-horseradish and retrogradehorseradish peroxidase study. Neuroscience, 7, 2321-2335. Kelley AE, Domesick VB, Nauta WJH (1982) The amygdalostriatal projection in the rat – an anatomical study by anterograde and retrograde tracing methods. Neuroscience 7:615-630.

Kirkby, R. J., Polgar, S., & Coyle, I. R. (1981). Caudate nucleus lesions impair the ability of rats to learn a simple straight-alley task. Perceptual and motor skills, 52(2), 499-502. Klein, J. T., Fanelli, R. R., & Robinson, D. L. (2012). Phasic neural activity in the dorsomedial and dorsolateral striatum during extinction and reinstatement in goal-directed and habit-trained rats. Alcoholism: Clinical & Experimental Research, 36, 27A.

28 Knackstedt, L. A., & Schwendt, M. (2016). mGlu5 receptors and relapse to cocaine-seeking: the role of receptor trafficking in postrelapse extinction learning deficits. Neural plasticity, 2016, Article ID 9312508. doi:10.1155/2016/9312508 Knackstedt, L. A., Trantham‐Davidson, H. L., & Schwendt, M. (2014). The role of ventral and dorsal striatum mGluR5 in relapse to cocaine‐seeking and extinction learning. Addiction biology, 19(1), 87-101. Kolb, B. (1977). Studies on the caudate-putamen and the dorsomedial thalamic nucleus of the rat: Implications for mammalian frontal-lobe functions. Physiology & Behavior, 18(2), 237-244. Kravitz, A. V., Freeze, B. S., Parker, P. R., Kay, K., Thwin, M. T., Deisseroth, K., & Kreitzer, A. C. (2010). Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature, 466(7306), 622-626. Kushner, M. G., Kim, S. W., Donahue, C., Thuras, P., Adson, D., Kotlyar, M., ... & Foa, E. B. (2007). D-cycloserine augmented exposure therapy for obsessive-compulsive disorder. Biological Psychiatry, 62(8), 835-838. Lafenêtre, P., Chaouloff, F., & Marsicano, G. (2007). The endocannabinoid system in the processing of anxiety and fear and how CB1 receptors may modulate fear extinction. Pharmacological Research, 56(5), 367-381. Ledgerwood, L., Richardson, R., & Cranney, J. (2003). Effects of D-cycloserine on extinction of conditioned freezing. Behavioral neuroscience, 117(2), 341-349. Li, P., Li, Y. H., & Han, T. Z. (2009). NR2A-containing NMDA receptors are required for LTP induction in rat dorsolateral striatum in vitro. Brain Research,1274, 40-46. Lopez-Figuero MO, Ramirez-Gonzalez JA, Divac I (1995) Projections from the visual areas to the neostriatum in rats – reexamination. Acta Neurobiol Exp (Wars) 55:165-175. Makarova, I. I. (2001). Early Afferent Reactions in Caudate Nucleus and Neocortex during Consolidation and Extinction of Conditioned. Bulletin of experimental biology and medicine, 131(2), 106-108. Maren, S. (2015). Out with the old and in with the new: Synaptic mechanisms of extinction in the amygdala. Brain research, 1621, 231-238. Martin, S.J., Grimwood, P.D., & Morris, R.G. (2000). Synaptic plasticity and memory: an evaludation and the hypothesis. Annual Review in Neuroscience, 23, 649-711. McGaugh, J. L. (2000). Memory--a century of consolidation. Science, 287(5451), 248-251. McGeorge AJ, Faull RLM (1989) The organization of the projection from the cerebral cortex to the striatum in the rat. Neuroscience, 29:503-537.

29

Meng, M., Zhao, X., Dang, Y., Ma, J., Li, L., & Gu, S. (2013). Region-specific expression of brain-derived neurotrophic factor splice variants in morphine conditioned place preference in mice. Brain research, 1519, 53-62. Mink, J. W. (1996). The basal ganglia: focused selection and inhibition of competing motor programs. Progress in neurobiology, 50(4), 381-425. Moussa, R., Poucet, B., Amalric, M., & Sargolini, F. (2011). Contributions of dorsal striatal subregions to spatial alternation behavior. Learning & Memory, 18(7), 444-451. Oliveto, A., Benios, T., Gonsai, K., Feingold, A., Poling, J., & Kosten, T. R. (2003). Dcycloserine--naloxone interactions in opioid-dependent humans under a novel-response naloxone discrimination procedure. Experimental and Clinical Psychopharmacology, 11(3), 237-246. Packard, M. G. (2001). On the neurobiology of multiple memory systems: Tolman versus Hull, system interactions and the emotion-memory link. Cognitive Processes, 2, 3-24. Packard, M. G., & Goodman, J. (2013). Factors that influence the relative use of multiple memory systems. Hippocampus, 23(11), 1044-1052. Packard, M. G., Goodman, J. (2017). Procedural Learning in Animals. In: Learning and Memory, Third Edition, Macmillan Publishers. Packard, M. G., & Knowlton, B. J. (2002). Learning and memory functions of the basal ganglia. Annual review of Neuroscience, 25(1), 563-593. Packard, M. G., & McGaugh, J. L. (1994). Quinpirole and d-amphetamine administration posttraining enhances memory on spatial and cued discriminations in a water maze. Psychobiology, 22(1), 54-60. Packard, M. G., & McGaugh, J. L. (1996). Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiology of learning and memory, 65(1), 65-72. Packard, M. G., & White, N. M. (1991). Dissociation of hippocampus and caudate nucleus memory systems by posttraining intracerebral injection of dopamine agonists. Behavioral Neuroscience, 105(2), 295-306. Pavlov, I.P. (1927). Conditioned Reflexes. London: Oxford University Press. Picconi, B., Centonze, D., Håkansson, K., Bernardi, G., Greengard, P., Fisone, G., ... & Calabresi, P. (2003). Loss of bidirectional striatal synaptic plasticity in L-DOPA–induced dyskinesia. Nature Neuroscience, 6(5), 501-506.

30 Poldrack, R. A., & Packard, M. G. (2003). Competition among multiple memory systems: converging evidence from animal and human brain studies. Neuropsychologia, 41(3), 245-251. Rawlins, J. N. P., Feldon, J., Ursin, H., & Gray, J. A. (1985). Resistance to extinction after schedules of partial delay or partial reinforcement in rats with hippocampal lesions. Experimental brain research, 59(2), 273-281. Redgrave, P., Prescott, T. J., & Gurney, K. (1999). The basal ganglia: a vertebrate solution to the selection problem? Neuroscience, 89, 1009-1023. Rescorla, R. A. (2001). Experimental extinction. In: R. R. Mowrer & S. B. Klein (Eds.), Handbook of Contemporary Learning Theories (pages 119-154). London: Lawrence Erlbaum Associates, Publishers. Ressler, K. J., Rothbaum, B. O., Tannenbaum, L., Anderson, P., Graap, K., Zimand, E., ... & Davis, M. (2004). Cognitive enhancers as adjuncts to psychotherapy: Use of D-cycloserine in phobic individuals to facilitate extinctionof fear. Archives of General Psychiatry, 61(11), 11361144. Rueda-Orozco, P. E., Montes-Rodriguez, C. J., Soria-Gomez, E., Méndez-Díaz, M., & ProspéroGarcía, O. (2008). Impairment of endocannabinoids activity in the dorsolateral striatum delays extinction of behavior in a procedural memory task in rats. Neuropharmacology, 55(1), 55-62. Salado-Castillo, R., Alvarado, R., Quirarte, G. L., & Prado-Alcalá, R. A. (1996). Effects of regional GABAergic blockade of the striatum on memory consolidation. Neurobiology of Learning and Memory, 66(2), 102-108. Sanberg, P. R., Pisa, M., & Fibiger, H. C. (1979). Avoidance, operant and locomotor behavior in rats with neostriatal injections of kainic acid. Pharmacology Biochemistry and Behavior, 10(1), 137-144. Schmaltz, L. W., & Isaacson, R. L. (1972). Effect of caudate and frontal lesions on acquisition and extinction of an operant response. Physiology & behavior, 9(2), 155-159. Schroeder, J. P., & Packard, M. G. (2003). Systemic or intra‐amygdala injections of glucose facilitate memory consolidation for extinction of drug‐induced conditioned reward. European Journal of Neuroscience, 17(7), 1482-1488. Schroeder, J. P., Wingard, J. C., & Packard, M. G. (2002). Post‐training reversible inactivation of hippocampus reveals interference between memory systems. Hippocampus, 12(2), 280-284. Schwabe, L. (2013). Stress and the engagement of multiple memory systems: integration of animal and human studies. Hippocampus, 23(11), 1035-1043.

31 Schwabe, L., Dickinson, A., & Wolf, O. T. (2011). Stress, habits, and drug addiction: a psychoneuroendocrinological perspective. Experimental and clinical psychopharmacology, 19(1), 53-63. Schwabe, L., Wolf, O. T., & Oitzl, M. S. (2010). Memory formation under stress: quantity and quality. Neuroscience & Biobehavioral Reviews, 34(4), 584-591. Schwendt, M., Reichel, C. M., & See, R. E. (2012). Extinction-dependent alterations in corticostriatal mGluR2/3 and mGluR7 receptors following chronic methamphetamine selfadministration in rats. PLoS One, 7(3), e34299. See, R. E., Elliott, J. C., & Feltenstein, M. W. (2007). The role of dorsal vs ventral striatal pathways in cocaine-seeking behavior after prolonged abstinence in rats. Psychopharmacology, 194(3), 321-331. Seward, J. P., & Levy, N. (1949). Sign learning as a factor in extinction. Journal of Experimental Psychology, 39(5), 660-668. Shugalev, N. P., Ol'shanskiĭ, A. S., & Hartmann, G. (2000). Effect of neurotensin injections into caudate nuclei on realization and extinction of conditioned motor reflex in rats. Zhurnal vysshei nervnoi deiatelnosti imeni IP Pavlova, 51(4), 473-476. Skelin, I., Hakstol, R., VanOyen, J., Mudiayi, D., Molina, L. A., Holec, V., ... & Gruber, A. J. (2014). Lesions of dorsal striatum eliminate lose‐switch responding but not mixed‐response strategies in rats. European Journal of Neuroscience, 39(10), 1655-1663. Squire, L. R. (2004). Memory systems of the brain: a brief history and current perspective. Neurobiology of learning and memory, 82(3), 171-177. Suvorov, V. V., Ermolenko, S. F., & Zhodzhaeva, N. U. (1974). Role of the caudate nucleus in the formation and extinction of conditioned avoidance reactions in rats of various ages. , 24(2), 272-278. Thanos, P. K., Bermeo, C., Wang, G. J., & Volkow, N. D. (2011). D‐cycloserine facilitates extinction of cocaine self‐administration in rats. Synapse, 65(9), 938-944. Thompson, R. (1963). Effects of caudate lesions on performance of monkeys in mixed fixed ratio-extinction schedules. American Psychologist, 18, 464. Thompson, L. T., & Disterhoft, J. F. (1997). Age-and dose-dependent facilitation of associative eyeblink conditioning by {d}-cycloserine in rabbits. Behavioral Neuroscience, 111(6), 13031312. Thullier, F., Lalonde, R., Mahler, P., Joyal, C. C., & Lestienne, F. (1996). Dorsal striatal lesions in rats. 2: Effects on spatial and non-spatial learning. Archives of physiology and biochemistry, 104(3), 307-312.

32

Tolman, E. C. (1932) Purposive behavior in animals and men. New York: Appleton-CenturyCrofts. Voorn, P., Vanderschuren, L. J., Groenewegen, H. J., Robbins, T. W., & Pennartz, C. M. (2004). Putting a spin on the dorsal–ventral divide of the striatum. Trends in Neurosciences, 27(8), 468474. Walker, D. L., Ressler, K. J., Lu, K. T., & Davis, M. (2002). Facilitation of conditioned fear extinction by systemic administration or intra-amygdala infusions of D-cycloserine as assessed with fear-potentiated startle in rats. The Journal of Neuroscience, 22(6), 2343-2351. Wendler, E., Gaspar, J. C., Ferreira, T. L., Barbiero, J. K., Andreatini, R., Vital, M. A., ... & Da Cunha, C. (2014). The roles of the nucleus accumbens core, dorsomedial striatum, and dorsolateral striatum in learning: performance and extinction of Pavlovian fear-conditioned responses and instrumental avoidance responses. Neurobiology of learning and memory, 109, 2736. White, N. M. (1996). Addictive drugs as reinforcers: multiple partial actions on memory systems. Addiction, 91(7), 921-950. White, N. M. (2009). Some highlights of research on the effects of caudate nucleus lesions over the past 200 years. Behavioural brain research, 199(1), 3-23. White, N. M., Packard, M. G., & McDonald, R. J. (2013). Dissociation of memory systems: The story unfolds. Behavioral neuroscience, 127(6), 813-834. Xu, J., Zhu, Y., Kraniotis, S., He, Q., Marshall, J. J., Nomura, T., Stauffer, S. R., Lindsley, C. W., Conn, P. J., & Contractor, A. (2013). Potentiating mGluR5 function with a positive allosteric modulator enhances adaptive learning. Learning & Memory, 20(8), 438-445. Yin, H. H., & Knowlton, B. J. (2004). Contributions of striatal subregions to place and response learning. Learning & Memory, 11(4), 459-463. Yin, H. H., & Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nature Reviews Neuroscience, 7(6), 464-476. Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European journal of neuroscience, 19(1), 181-189. Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005). The role of the dorsomedial striatum in instrumental conditioning. European Journal of Neuroscience, 22(2), 513-523.

33 Table 1. The effect of dorsal striatal manipulations on extinction in a variety of tasks Paradigm Subject Manipulation Effect on References Extinction Instrumental barMonkeys DS lesion Impaired Butters and Rosvold, 1968; pressing Thompson, 1963 Rats

DS lesion

Impaired

Sanberg et al., 1978; Schmaltz and Issacson, 1972

Electrical stimulation of DS

Impaired

Herz and Peeke, 1971

Pavlovian conditioning of alimentary responses

Dogs

DS lesion

Impaired

Baranov, 1977; Denisova, 1972

Impaired

Denisova, 1981

Spatial reversal learning in a Grice box

Rats

Electrical stimulation of DS DS lesion

Impaired

Kolb, 1977

Straight alley maze (response extinction)

Rats

DS lesion

Impaired

Thullier et al., 1996

Impaired No effect Impaired

Straight alley maze (latent extinction) T-maze

Rats

DLS lesion DMS lesion Temporary DLS inactivation Temporary DLS inactivation DLS lesion DMS lesion Temporary DLS inactivation Temporary DLS inactivation Intra-DLS AP5

Dunnett and Iversen, 1981 Dunnett and Iversen, 1981 Goodman, Gabriele, and Packard, 2017 Goodman, Gabriele, and Packard, 2017 Dunnett and Iversen, 1981 Dunnett and Iversen, 1981 Campus et al., 2015

Intra-DLS D-cycloserine Temporary DLS inactivation DLS lesion DMS lesion Inhibition of Arc in the DLS Inhibition of DLS metabotropic glutamate receptors

Enhanced

DLS lesion

Rats Mice

Response learning in the plus-maze

Rats

Place learning in the plus-maze Spatial alternation in a modified T-maze

Rats

Cocaine selfadministration

Rats

Rats

Two-way active avoidance

Rats

Fear conditioning

Rats

No effect Impaired Impaired Impaired Impaired

Enhanced Impaired Impaired

Goodman, Ressler, and Packard, 2016 Goodman, Ressler, and Packard, 2017 Goodman, Ressler, and Packard, 2017 Goodman, Ressler, and Packard, 2016 Moussa et al., 2011 Moussa et al., 2011 Hearing et al., 2011

Impaired

Knackstedt et al., 2014

Enhanced

Wendler et al., 2014

Impaired

Enhanced

DMS lesion Enhanced Wendler et al., 2014 DLS lesion No effect Wendler et al., 2014 DMS lesion No effect Wendler et al., 2014 DS: dorsal striatum, DLS: dorsolateral striatum, DMS: dorsomedial striatum.