Neural mechanisms for model-free and model-based reinforcement strategies in humans performing a multi-step navigation task

Abstracts / Neuroscience Research 68S (2010) e223–e334

P2-k07 Endogenous IL-1␤ and IL-1 receptor antagonist in the brain are involved in poly I:C-induced immunological fatigue Masanori Yamato 1 , Kaori Okuyama 1,3 , Guanghua Jin 1,3 , Asami Eguchi 1,3 , Yasuyoshi Watanabe 2,3 , Yosky Kataoka 1,3 1

Cellular Function Imaging Lab., RIKEN CMIS, Kobe, Japan 2 Molecular Probe Dynamics Lab., RIKEN CMIS, Kobe, Japan 3 Department Physiol, Osaka City University Grad. Sch. Med., Osaka, Japan Intraperitoneal (i.p.) injection of polyriboinosinic:polyribocytidylic acid (poly I:C), double-strand RNA, is known to mimic viral infection, and induces immunological fatigue-like behavior including transient fever and the suppression of locomotor activity in rats. To date, we have reported that interleukin (IL)-1␤ expression was up-regulated in various brain regions of rats administered with poly I:C, and that i.c.v. infusion of rat recombinant IL1␤ suppressed locomotor activity in normal rats. These observations suggest that brain IL-1␤ is a key mediator for the induction of immunological fatigue in viral infection. IL-1 family has an endogenous IL-1 receptor antagonist. Recently, we demonstrated that i.c.v infusion of IL-1 receptor antagonist significantly attenuated poly I:C-induced fatigue-like behavior. The balance of IL-1␤ and the antagonist is thought to regulate immunological fatigue-like behavior. In the present report, we will shed light on the role of IL-1 receptor antagonist which is endogenously produced in the brain of poly I:C-injected rats on the fatigue-like behavior. doi:10.1016/j.neures.2010.07.1265

P2-k08 Possible changes in neurotrophin signalings and levels of ER␣ are involved in anxiety– and depressive– like behaviors in postpartum rats after weaning Miyako Furuta 1,2 , Tadahiro Numakawa 2 , Midori Ninomiya 2,3 , Shuichi Chiba 2 , Yu Kajiyama 2,3 , Shigenobu Shibata 3 , Toshiya Funabashi 1 , Tatsuo Akema 1 , Hiroshi Kunugi 2,4 1

Department of Physiology, St. Marianna University School of Medicine Department of Mental Disorder Res., National Inst. of Neuroscience National Center of Neurology and Psychiatry, Tokyo, Japan 3 Waseda University, School of Science and Engineering, Tokyo, Japan 4 CREST, JST, Tokyo, Japan 2

A drastic reduction in estrogen levels at delivery and lactation is suggested to be implicated in anxiety, depression, and postpartum psychosis. However, which estrogen receptors, ER␣ or ER␤, mediates such psychological disorders, and how ER␣ and/or ER␤ contribute the underlying mechanism is still unclear. Here, we found that primiparous rats at 3 weeks–postpartum showed increased anxiety and depressive behaviors in elevated plus maze (EP) and forced swimming (FS) tests, compared with diestrus nulliparous females. ER␣–selective agonist, propyl–pyrazole–triol (PPT; 1.0 mg/kg) and 17␤–estradiol (0.25 mg/kg), displayed anxiolytic and antidepressant effects in EP and FS tests at 3 weeks–postpartum after subcutaneously injection once daily for 4 days. Interestingly, immunohistochemical studies revealed that a significant difference exists in the distribution of ER␣ between the 3 weekspostpartum and other groups. Furthermore, we found a marked alteration in the expression of TrkB and BDNF (brain–derived neurotrophic factor) receptors, and in resultant intracellular signalings of the 3 weeks–postpartums. This raises a possibility that changes of neurotrophin–signalings via ER␣ stimulation are associated with the anxiety and the depression-like behaviors. doi:10.1016/j.neures.2010.07.1266

P2-k09 Alarm pheromone suppresses male but female sexual behavior in rats Tatsuya Kobayashi Mori

, Yasushi Kiyokawa, Yukari Takeuchi, Yuji

Vet. Ethol., University of Tokyo, Tokyo We have reported that foot-shocked male rats release the “alarm pheromone” that aggravates the stress-induced hyperthermia (temporary rise of body temperature) in recipient rats. This pheromone is shown to be produced in a testosterone-independent manner and released from the perianal region of the donor rat. Recipient rats perceive this pheromone by the vomeronasal organ and show anxiety-related responses, which could, at leaset partly, be blocked by pretreatment of anxiolytics suggesting the involvement of regulatory system of anxiety. In the present study we examined if the alarm pheromone affects the sexual behavior in either male or female rats or both. When a pair of male and female rats were exposed to

e285

the alarm pheromone during the copulation, the number of mounts preceding ejaculation increased and the hit rate (number of intromissions/number of mounts and intromissions) decreased in the male rat. In contrast, female sexual behavior was not affected by the alarm pheromone. Then we further examined the pheromone effect by exposing only the male or female of the pair to the pheromone just before copulation. Obtained results were similar, namely the pheromone effects were evident only in males but not in females. We concluded that the alarm pheromone has suppressive effects on the sexual behavior in rats, but the effect is limited to the male.

Acknowledgements This study was supported by Grants-in-Aid for Scientific Research (21228006) and for JSPS Fellows (19-5683). doi:10.1016/j.neures.2010.07.1267

P2-k10 Response of neurons in the monkey prefrontal cortex during learning of new cue-target associations Makoto Kusunoki 1,2 Duncan 1,2

, Natasha Sigala 1,2 , David Gaffan 1 , John

1

University Oxford, Department Exp. Psych., Oxford, UK and Brain Sci. Unit, Cambridge, UK

2

MRC, Cognition

We have reported that many neurons in the lateral prefrontal cortex (LPFC) code target and nontarget objects in a cued target search task (Kusunoki et al., 2009). To investigate whether this result depends on extensive training with particular cue and target pairs, we compared LPFC activity in two conditions: one with only well trained pairs (learned pairs), and one with newly introduced pairs. In this task, one of three cue objects was presented at trial onset to the left or the right of the central fixation point, followed by 0-3 nontargets (targets associated with other cues, or a neutral object) and then the instructed target. Reward was given for a saccade to the correct target. In learned pair blocks, there were three familiar cue-target pairs. In new pair blocks, one learned pair was replaced with a new pair. The neutral nontarget was also replaced with a new object, so that animals could not find the new target simply by selecting an unfamiliar object. Both animals tested acquired new pairs quickly, following just a few error responses. Many neurons changed their response pattern in new pair blocks. Some showed strong responses to new targets and neutral nontargets with significant selectivity between them in the late phase of stimulus presentation (later than 300 ms after stimulus onset). This selectivity often developed within several correct responses to new targets. Other neurons did not respond at all to new objects while they were still responding to learned objects. There were also neurons which changed response to learned pair objects during new pair blocks. The results suggest that the representation of targets and nontargets in LPFC is dynamic. Activity is modulated by the whole set of pairs in a test block, and is not completely locked to particular pairs. The result that some LPFC cells responded strongly to new objects, while reducing responses to learned objects, suggests redirection of neural activity towards new pair associations. doi:10.1016/j.neures.2010.07.1268

P2-k12 Neural mechanisms for model-free and modelbased reinforcement strategies in humans performing a multi-step navigation task Alan Fermin 1 , Takehiko Yoshida 1,2 , Makoto Ito 1 , Junichiro Yoshimoto 1,2 , Kenji Doya 1,2,3 1

Neural Computation Unit, Okinawa Institute of Science and Technology Nara Institute of Science and Technlogy 3 ATR Computational Neuroscience Laboratories

2

Humans can learn actions from scratch, or by using knowledge from past experiences. Reinforcement Learning (RL), a computational theory of adaptive optimal control, suggests two methods that resemble real human behavior: Model-Free (MF) method uses action value functions to predict future rewards based on current states, and Model-Based (MB) method uses a forward model to predict the future states reached by hypothetical actions. We tested whether humans utilize MF and MB strategies by having subjects perform a grid-sailing task whose goal was to move a cursor from a start position to a target by sequentially pressing three keys, each moved the cursor in a different direction. After one day of training, subjects were tested inside the fMRI scanner under three task conditions: (1) learning of new key-map and start-goal positions, (2) use of learned key-map for new start-goal positions, (3) well-learned action sequences. In half of the trials

e286

Abstracts / Neuroscience Research 68S (2010) e223–e334

a response started immediately after the go signal, or after a delay period of 4–6 s. Analysis of reward acquisition revealed high performance in condition 3; condition 2 was significantly better in trials with a delay; condition 1 had the lowest performance. By calculating the posterior probability of whether MF or MB algorithms generate subjects’ action selection, we showed their actions in conditions 1 and 2 could be explained by MF and MB models, respectively. The fMRI analysis of this delay period using condition 3 as the control task showed that condition 2 activated the left DLPFC, ventral premotor cortex, anterior basal ganglia and right posterior cerebellum, whereas task condition 1 activated the left dorsal premotor cortex, parietal and visual areas bilaterally. Analysis of the time course and signal intensity of activation in these areas showed the strongest activation in condition 2 in this anterior prefrontal–basal–ganglia–cerebellum network, a candidate to implement MB strategy. doi:10.1016/j.neures.2010.07.1269

P2-k13 A dual information task for analyzing cell assembly dynamics underlying different internal cognitive process Tomoaki Nakazono 1 , Susumu Takahashi 2,3 , Yoshio Sakurai 1 1

Department Psychol., Kyoto University, Kyoto 2 Kyoto Sangyo University, Kyoto 3 PRESTO, JST, Kawaguchi

Synchronized firing of certain neurons (cell assembly) is thought to play an important role in internal cognitive processes. But we don’t know enough about how such cell assemblies encode internal information and how dynamically the cell assemblies change by types and levels of internal information. To investigate this dynamics, we should compare activity of same cell assemblies in different cognitive processes. Thus, we need to develop a behavioral paradigm which imposes subjects on different internal cognitive processes with same sensory input and motor output. In this study, we have developed a behavioral task for dual information processes to investigate the connection dynamics property of cell assembly. In the task, a cue has two information meanings (time and position). The rats are required to select and use just one information meaning (internal information) to get a reward effectively. We record hippocampal multineuronal activity with tetrodes when the rats are performing this dual information task. Then we analyze the neuronal data to show how hippocampal cell assemblies change when the internal information the rats employ change. We report preliminary behavioral and neuronal data. doi:10.1016/j.neures.2010.07.1270

P2-k14 Sequential information processing in rats in serial reaction time task Seiya Ishino 1 , Susumu Takahashi 2,3 , Yoshio Sakurai 1 1 3

Department Psychol., Kyoto Univ, Kyoto 2 Kyoto Sangyo University, Kyoto PRESTO, JST, Kawaguchi

All behaviors consist of sequences of action elements. When specific sequences of action elements have meaning, they are termed serial-order behaviors or sequential behaviors. Then, how do humans and other animals process sequential information and perform serial-order behavior? Though a lot of psychological experiments have executed about serial learning and serial-order behavior in humans and primates, neuronal mechanisms are unclear yet. The goal of this study is to train rats a serial reaction time (SRT) task and to elucidate the neuronal mechanisms of sequential information processing. In the SRT task we used a chamber with several nose-poke holes arranged horizontally on the wall and required each rat to poke its nose into these holes in specific sequences. First we conducted repeated sequence condition in which a specific sequence was repeated on each trial. Then we conducted random sequence condition in which a randomly selected sequence was presented on each trial.We compared behavioral data in the two conditions. Subsequently, we recorded multineuronal activity from the hippocampus and analyzed neuronal activity expressing sequence information in the rats. We will report preliminary behavioral and neuronal data. doi:10.1016/j.neures.2010.07.1271

P2-k15 Neural activity in macaque prefrontal cortex during learning through trial-and-error behaviors Atsushi Fujimoto , Satoshi Nishida, Tomohiro Tanaka, Tadashi Ogawa Integrative Brain Science, Graduate School of Medicine, Kyoto University, Kyoto When facing an unknown problem, we can get the knowledge to solve it through the repetition of trial-and-error behaviors. To elucidate the neural mechanisms underlying this type of learning, we trained monkeys to perform a learning task that required them to repeatedly make trial-and-error behaviors and recorded single unit activity from the lateral prefrontal cortex (LPF), which was implicated to represent abstract rules. The monkey had to search for a target from six different colored elements in a search array, without any information about the target color. The target color was kept in each of trial blocks and randomly changed into another color after a block switch.The monkey showed a step-like elevation in the success rate after once it made the correct response, suggesting knowledge-based learning. We defined the trials before the first success trial as ‘trial-and-error phase’ (TE phase) and those after the first success trial as ‘maintenance phase’ (M phase). Reaction times (reaction time to the fixation point and target stimulus) and choice tendency to select the pre-rewarded color after error trials were significantly different between TE and M phases, suggesting that the monkey utilized different task-solution strategies. Consistently, some LPF neurons exhibited a clear change in their activity at a switch of two phases, although the activity was relatively stable within each of the phases. This differential activity might not be explained simply by the behavioral performance, because the activity was not different between the success and error trials within M phase. The reward-size control task also confirmed that the differential activity could not be explained by the reward expectancy. Furthermore, in a part of these neurons the enhanced activity was observed after the outcome in the first success trial, suggesting that LPF neurons potentially play roles not only at maintaining but also at switching the different task-solution strategies. doi:10.1016/j.neures.2010.07.1272

P2-k16 Theta phase shift during alert immobility in rat hippocampal CA1 neurons Muneyoshi Takahashi 1 , Johan Lauwereyns 2 , Yoshio Sakurai 3,4 , Minoru Tsukada 1 1

Tamagawa University Brain Sci. Inst., Tokyo, Japan 2 Grad. Sch. Sys. Life Sci., Kyushu University, Fukuoka, Japan 3 Department Psychol., Grad. Sch. Letters, Kyoto University, Kyoto, Japan 4 CREST, Kawaguchi, Japan Extensive research has focused on hippocampal activity during active exploration (i.e., place cell activity) when Type 1 theta oscillation (i.e., translational movement-related theta) is dominant. During this period, hippocampal place cells exhibit a phenomenon called “theta phase precession,” firing spikes late in the ongoing theta cycle when a rat first enters the place field of the cell, but firing spikes earlier in the theta cycle as the rat progresses through the field. However, little is known about phase precession during alert immobility when Type 2 theta oscillation (i.e., attention/arousal-related theta) is dominant. Here we report the first evidence that the spikes of CA1 neurons shift approximately half of the theta cycle during a one-second period of immobile fixation, while the rat is fully alert. Four rats were trained on a delayed spatial alternation task using a sustained nose-poking paradigm. Two hundred forty eight putative CA1 pyramidal neurons were recorded using a 14-tetrode hyperdrive assembly during the task behavior. We concentrated on 49 neurons that showed a fixation-specific firing trend. Thirty-two of these neurons exhibited significant correlation, indicating a theta phase shift during the fixation period. Population analysis revealed that the firing trend shifted from the positive peak of the theta cycle, recorded around stratum pyramidale of CA1 at the beginning of fixation, to the negative trough at the end of fixation. It is known that the strongest input from entorhinal cortex occurs at the positive peak of the CA1 cell layer theta, whereas the strongest input from CA3 is associated with the trough of the theta. We hypothesize that CA1 pyramidal cells perform a function as an adaptive filter between entorhinal cortex and CA3, propagating relevant information depending on task requirements. doi:10.1016/j.neures.2010.07.1273