- Email: [email protected]
A dynamic model of learning in the septo-hippocampal system
Neurocomputing 32}33 (2000) 501 } 507
A dynamic model of learning in the septo-hippocampal system Bas Rokers *, Catherine E. Myers, Mark A. Gluck Department of Psychology, Rutgers University, Room 301, 101 Warren Street, Newark, NJ 07102, USA Department of Psychology, Rutgers University, 101 Warren Street, Newark, NJ 07102, USA Center for Molecular and Behavioral Neuroscience, Rutgers University, 197 University Avenue, Newark, NJ 07102, USA Accepted 13 January 2000
Abstract Gluck and Myers (Hippocampus 3 (1993) 491}516) modeled the hippocampus as an autoencoder; Myers et al. (Neurobiol. Learning Memory 66 (1996) 51}66) argued that the cholinergic input from medial septum modulates learning rate in this auto-encoder. Neurophysiological evidence suggests the hippocampus self-regulates septal acetylcholine release in response to novel stimuli (Hasselmo and Schnell, J. Neurosci. 14 (1994) 3898}3914). We have extended our earlier model of septohippocampal modulation to include such a feedback loop. The resulting dynamic model learns faster and better than the earlier version on phenomena such as blocking and shift reversal. It can also be applied to data regarding the e!ects of the anticholinergic drug scopolamine. 2000 Elsevier Science B.V. All rights reserved. Keywords: Hippocampus; Septum; Acetylcholine; Neuromodulation; Model
1. Introduction Myers et al. [8] proposed that cholinergic projections from medial septum to hippocampus could modulate hippocampal learning. Speci"cally, hippocampal learning rates were assumed to be proportional to this cholinergic input, and to be reduced by anticholinergic drugs such as scopolamine. This model accounted for a range of data regarding the e!ects of scopolamine on classical conditioning [9].
* Corresponding author. 0925-2312/00/$ - see front matter 2000 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 - 2 3 1 2 ( 0 0 ) 0 0 2 0 5 - 8
502
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
There is also a reciprocal pathway from hippocampus to medial septum. Hasselmo et al. (1995) proposed that the hippocampus uses this pathway to self-regulate cholinergic input. Speci"cally, high novelty in the input pattern should cause activity in the hippocampal}septal pathway, which should in turn activate cholinergic septo}hippocampal pathways and drive storage in the hippocampus. We propose to combine these two computational models. In the Myers et al. [8] model, the hippocampus is implemented as an auto-encoder, which learns to reconstruct its inputs. High output error thus re#ects stimulus novelty. This global error measure can be used to determine hippocampal learning rates in the model, just as hippocampal}septal activity may drive cholinergic septo-hippocampal inputs. It has already been shown that dynamic learning rates can improve learning [10,2]. Here, we incorporate a dynamic learning rate in the &static' model of Gluck and Myers [3] and test performance on blocking and cholinergic modi"cation.
2. The Gluck and Myers (1993) model Gluck and Myers "rst proposed a hippocampal auto-encoder that taps into a cerebellar feed-forward network. The hippocampal module re-encodes stimuli to enable storage in the cerebellar module through back-propagation of error [11]. Next to changes of weights between nodes based on current encoding error, the back-propagation algorithm makes use of a momentum term. The direction and rate of previous learning a!ects current weight modi"cation. This renders the delta rule for modi"cation of a connection weight from node i to node j *w "bd y #a(*w ), GH H G GH where b is a learning rate parameter, and a is the momentum parameter, which is set to 0.9. b typically falls in the 10.1, 0.012 range, and unless otherwise speci"ed, will be set to 0.05 for simulations to be discussed here. In the hippocampal module, the error signal for each output node j is calculated by d "y (1!y )(I !y ), H H H H H where I denotes the desired value of node j and y is the current activation of the H H output node. Obviously, desired output and current input denote the same value in auto-encoders. For nodes in the hidden layer the error is calculated as
d "y (1!y ) w d , H H H HI I I where w is the weight from hidden node j to output node k, and d is the error of HI I output node k. This error measure is used to determine weight modi"cation through the delta-rule [11]. Myers et al. [8] argued that the e!ects of scopolamine, which disrupt septohippocampal cholinergic projections, was to reduce the learning rate parameter b.
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
503
3. Dynamic learning To implement a dynamic learning rule and simulate cholinergic feedback, we extend the error signal calculation by adding a global measure of error. This takes the form of a Hamming distance metric H" "I !y " H H H for all output nodes j in the hippocampal module. This Hamming function computes an aggregate measure of error within the model. By de"nition the summation over all di!erences between current and desired outputs over all nodes j in the output layer has a range of 10, j2, assuming each node itself has range 10,12. Finally, we rede"ne the initial static learning rate parameter b in the original delta rule to b "bH. This dynamic learning rate parameter b incorporates the e!ect of cholinergic feedback on encoding error. E!ectively, the learning rate will be decreased in case H falls within the 10,12 range and increased otherwise, e.g. in case H is larger than unity. Initially, all stimuli generate high encoding errors and modify the error signal considerably. Once associations have been acquired, encoding errors and thus e!ects of cholinergic modulation decrease. E!ective cholinergic modulation can be reinduced again, only when encoding errors increase as a result of novel or highly salient stimuli.
4. Benchmarks We might expect that the introduction of H will increase the error signal to such an extent that the back-propagation procedure will not converge. However, on average, the inclusion of a measure of overall error (H) decreases the value of the error signal d . H This can easily be inferred from Fig. 1.
Fig. 1. Learning rate parameters (b for static, and b for dynamic) as a function of training.
504
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
Fig. 2. Response error as a function of training.
Fig. 3. Comparison of orthogonal distance between representations in the cortical module as a function of training.
A lower learning rate suggests lower performance. However, the inclusion of H in the error signal enhances performance. As shown in Figs. 2 and 3, the network converges quicker and spreads input patterns further apart in representational space.
5. Blocking When an animal is trained on a CS}US pair, the addition of another CS to the pair will not lead to an association between the second CS and the US. This is known as blocking. However, presentation of the second novel CS leads to a short period of &confusion', where attention is drawn to the new item and response error increases. This &confusion' period has been shown in one-trial blocking experiments. Fig. 4 shows that the dynamic model is in better accordance with empirical onetrial blocking data, showing an initial rise in response error at the onset of stage two. The static model does not respond directly at the onset of the second CS. Since the
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
505
Fig. 4. Error as a function of training under a blocking paradigm for both the static and the dynamic model.
Fig. 5. Error as a function of training for either the dynamic or the scopolamine modi"ed dynamic model.
second stimulus does not add to US prediction, response error decays quickly in the dynamic model, whereas the static model has some di$culty accommodating the new information. 6. Scopolamine The introduction of scopolamine, an acetylcholine antagonist, is supposed to retard but not abolish learning by impairing novelty detection. This is simulated by dividing values of H by a factor 10 and shown in Fig. 5. 7. Conclusion Though theoretically the inclusion of Hamming distance in the delta rule could lead to an impossibly high learning rate and correspondingly low performance,
506
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
in practice the dynamic nature of the learning rule has a self-regulating e!ect in conditioning. Since a dynamic model looks promising based on discussed blocking and scopolamine results, we aim to further evaluate performance on a range of conditioning data, such as latent inhibition and shift reversal. Furthermore, it has been shown that cholinergic agonists initially enhance learning, but impair it in higher doses. We believe that this behavior could be explained in a dynamical model as homeostasis. A future article will address this issue.
8. For further reading The following references are also of interest to the reader: [1,4}7,12].
References [1] E. Barkai, M. Hasselmo, Modulation of the input/output function of rat piriform cortex pyramidal cells, J. Neurophysiol. 72 (1994) 644}658. [2] P. den Dulk, B. Rokers, R.H. Phaf, Connectionist simulations with a dual route model of fear conditioning, in: B. Kokinov (Ed.), Perspective on Cognitive Science, Vol. 4, New Bulgarian University Press, So"a, 1996. [3] M.A. Gluck, C.E. Myers, Hippocampal mediation of stimulus representation: a computational theory, Hippocampus 3 (1993) 491}516. [4] M. Hasselmo, Neuromodulation and cortical function: modeling the physiological basis of behavior, Behav. Brain Res. 67 (1995) 1}27. [5] M. Hasselmo, E. Barkai, Cholinergic modulation of activity-dependent synaptic plasticity in the piriform cortex and associative memory function in a network biophysiological simulation, J. Neurosci. 15 (1995) 6592}6604. [6] P. Huerta, J. Lisman, Heightened synaptic plasticity of hippocampal CA1 neurons during a cholinergically induced rhytmic state, Nature 364 (1993) 723}725. [7] D. Madison, B. Lancaster, R. Nicoll, Voltage clamp analysis of cholinergic action in the hippocampus, J. Neurosci. 7 (1987) 733}741. [8] C.E. Myers, B.R. Ermita, K. Harris, M. Hasselmo, P. Solomon, M.A. Gluck, A computational model of the e!ects of septohippocampal disruption on classical eyeblink conditioning, Neurobiol. Learn. Memory 66 (1996) 51}66. [9] C.E. Myers, B.R. Ermita, M. Hasselmo, M.A. Gluck, Further implications of a computational model of septohippocampal modulation in eyeblink conditioning, Psychobiology 26 (1) (1998) 1}20. [10] R.H. Phaf, A.G. Tijsseling, E. Lebert, Self-organizing CALM maps, submitted for publication. [11] D. Rumelhart, G. Hinton, R. Williams, Learning internal representations by error propagation, in: D. Rumelhart, J. McClelland (Eds.), Parallel Distributed Processing: explorations in the Microstructure of Cognition, MIT Press, Cambridge, 1986, pp. 318}362. [12] B. Widrow, M.E. Ho!, Adaptive switching circuits, WESCON Convention Record, Vol. 4, 1960.
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
507
Catherine Myers is a Research Assistant Professor in the Department of Psychology at Rutgers University in Newark. She received her B.S. in Cognitive Science from the University of Delaware in 1987, and her Ph.D. in Neural Systems Engineering from the University of London in 1991. She studies the brain substrates of learning and memory with an emphasis on memory and memory dysfunction in humans. One area of interest is computational neuroscience, developing neural networks of how various brain regions interact during normal learning, and how lesions a!ect the ability to acquire new information. A second area of interest is experimental neuropsychology, particularly developing simple associative tasks, which may dissociate super"cially similar amnesic syndromes depending on which underlying brain systems have been damaged.
Bas Rokers is a Cognitive Science Ph.D. Student in the Department of Psychology at Rutgers University in Newark. He received his B.S. degree in Cognitive Science and Arti"cial Intelligence from Utrecht University, the Netherlands in 1998. He studies the brain substrates of learning and memory with an emphasis on neural development and neuro-modulation. One area of interest is computational neuroscience, developing neural networks of how various brain regions interact during normal learning, and how neurogenesis a!ects the ability to acquire new information. A second area of interest is computational neuropsychology, particularly the interaction between emotion and cognition, through the function and interaction of their respective neural structures, the hippocampus and the amygdala.
Mark Gluck works at the interface between cognition, animal learning, behavioral neuroscience, and computational modeling. He combines his training in human and animal psychology with neural-network analyses to understand the mechanisms of learning and memory. His earlier work in cognitive psychology demonstrated how behavioral properties of animal learning can be related to higher-order forms of learning in humans. Currently, his lab works to identify, model, and empirical evaluate fundamental components of both animal and human memory systems, with a special emphasis on the functional role of the hippocampus in learning.
A dynamic model of learning in the septo-hippocampal system Bas Rokers *, Catherine E. Myers, Mark A. Gluck Department of Psychology, Rutgers University, Room 301, 101 Warren Street, Newark, NJ 07102, USA Department of Psychology, Rutgers University, 101 Warren Street, Newark, NJ 07102, USA Center for Molecular and Behavioral Neuroscience, Rutgers University, 197 University Avenue, Newark, NJ 07102, USA Accepted 13 January 2000
Abstract Gluck and Myers (Hippocampus 3 (1993) 491}516) modeled the hippocampus as an autoencoder; Myers et al. (Neurobiol. Learning Memory 66 (1996) 51}66) argued that the cholinergic input from medial septum modulates learning rate in this auto-encoder. Neurophysiological evidence suggests the hippocampus self-regulates septal acetylcholine release in response to novel stimuli (Hasselmo and Schnell, J. Neurosci. 14 (1994) 3898}3914). We have extended our earlier model of septohippocampal modulation to include such a feedback loop. The resulting dynamic model learns faster and better than the earlier version on phenomena such as blocking and shift reversal. It can also be applied to data regarding the e!ects of the anticholinergic drug scopolamine. 2000 Elsevier Science B.V. All rights reserved. Keywords: Hippocampus; Septum; Acetylcholine; Neuromodulation; Model
1. Introduction Myers et al. [8] proposed that cholinergic projections from medial septum to hippocampus could modulate hippocampal learning. Speci"cally, hippocampal learning rates were assumed to be proportional to this cholinergic input, and to be reduced by anticholinergic drugs such as scopolamine. This model accounted for a range of data regarding the e!ects of scopolamine on classical conditioning [9].
* Corresponding author. 0925-2312/00/$ - see front matter 2000 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 - 2 3 1 2 ( 0 0 ) 0 0 2 0 5 - 8
502
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
There is also a reciprocal pathway from hippocampus to medial septum. Hasselmo et al. (1995) proposed that the hippocampus uses this pathway to self-regulate cholinergic input. Speci"cally, high novelty in the input pattern should cause activity in the hippocampal}septal pathway, which should in turn activate cholinergic septo}hippocampal pathways and drive storage in the hippocampus. We propose to combine these two computational models. In the Myers et al. [8] model, the hippocampus is implemented as an auto-encoder, which learns to reconstruct its inputs. High output error thus re#ects stimulus novelty. This global error measure can be used to determine hippocampal learning rates in the model, just as hippocampal}septal activity may drive cholinergic septo-hippocampal inputs. It has already been shown that dynamic learning rates can improve learning [10,2]. Here, we incorporate a dynamic learning rate in the &static' model of Gluck and Myers [3] and test performance on blocking and cholinergic modi"cation.
2. The Gluck and Myers (1993) model Gluck and Myers "rst proposed a hippocampal auto-encoder that taps into a cerebellar feed-forward network. The hippocampal module re-encodes stimuli to enable storage in the cerebellar module through back-propagation of error [11]. Next to changes of weights between nodes based on current encoding error, the back-propagation algorithm makes use of a momentum term. The direction and rate of previous learning a!ects current weight modi"cation. This renders the delta rule for modi"cation of a connection weight from node i to node j *w "bd y #a(*w ), GH H G GH where b is a learning rate parameter, and a is the momentum parameter, which is set to 0.9. b typically falls in the 10.1, 0.012 range, and unless otherwise speci"ed, will be set to 0.05 for simulations to be discussed here. In the hippocampal module, the error signal for each output node j is calculated by d "y (1!y )(I !y ), H H H H H where I denotes the desired value of node j and y is the current activation of the H H output node. Obviously, desired output and current input denote the same value in auto-encoders. For nodes in the hidden layer the error is calculated as
d "y (1!y ) w d , H H H HI I I where w is the weight from hidden node j to output node k, and d is the error of HI I output node k. This error measure is used to determine weight modi"cation through the delta-rule [11]. Myers et al. [8] argued that the e!ects of scopolamine, which disrupt septohippocampal cholinergic projections, was to reduce the learning rate parameter b.
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
503
3. Dynamic learning To implement a dynamic learning rule and simulate cholinergic feedback, we extend the error signal calculation by adding a global measure of error. This takes the form of a Hamming distance metric H" "I !y " H H H for all output nodes j in the hippocampal module. This Hamming function computes an aggregate measure of error within the model. By de"nition the summation over all di!erences between current and desired outputs over all nodes j in the output layer has a range of 10, j2, assuming each node itself has range 10,12. Finally, we rede"ne the initial static learning rate parameter b in the original delta rule to b "bH. This dynamic learning rate parameter b incorporates the e!ect of cholinergic feedback on encoding error. E!ectively, the learning rate will be decreased in case H falls within the 10,12 range and increased otherwise, e.g. in case H is larger than unity. Initially, all stimuli generate high encoding errors and modify the error signal considerably. Once associations have been acquired, encoding errors and thus e!ects of cholinergic modulation decrease. E!ective cholinergic modulation can be reinduced again, only when encoding errors increase as a result of novel or highly salient stimuli.
4. Benchmarks We might expect that the introduction of H will increase the error signal to such an extent that the back-propagation procedure will not converge. However, on average, the inclusion of a measure of overall error (H) decreases the value of the error signal d . H This can easily be inferred from Fig. 1.
Fig. 1. Learning rate parameters (b for static, and b for dynamic) as a function of training.
504
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
Fig. 2. Response error as a function of training.
Fig. 3. Comparison of orthogonal distance between representations in the cortical module as a function of training.
A lower learning rate suggests lower performance. However, the inclusion of H in the error signal enhances performance. As shown in Figs. 2 and 3, the network converges quicker and spreads input patterns further apart in representational space.
5. Blocking When an animal is trained on a CS}US pair, the addition of another CS to the pair will not lead to an association between the second CS and the US. This is known as blocking. However, presentation of the second novel CS leads to a short period of &confusion', where attention is drawn to the new item and response error increases. This &confusion' period has been shown in one-trial blocking experiments. Fig. 4 shows that the dynamic model is in better accordance with empirical onetrial blocking data, showing an initial rise in response error at the onset of stage two. The static model does not respond directly at the onset of the second CS. Since the
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
505
Fig. 4. Error as a function of training under a blocking paradigm for both the static and the dynamic model.
Fig. 5. Error as a function of training for either the dynamic or the scopolamine modi"ed dynamic model.
second stimulus does not add to US prediction, response error decays quickly in the dynamic model, whereas the static model has some di$culty accommodating the new information. 6. Scopolamine The introduction of scopolamine, an acetylcholine antagonist, is supposed to retard but not abolish learning by impairing novelty detection. This is simulated by dividing values of H by a factor 10 and shown in Fig. 5. 7. Conclusion Though theoretically the inclusion of Hamming distance in the delta rule could lead to an impossibly high learning rate and correspondingly low performance,
506
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
in practice the dynamic nature of the learning rule has a self-regulating e!ect in conditioning. Since a dynamic model looks promising based on discussed blocking and scopolamine results, we aim to further evaluate performance on a range of conditioning data, such as latent inhibition and shift reversal. Furthermore, it has been shown that cholinergic agonists initially enhance learning, but impair it in higher doses. We believe that this behavior could be explained in a dynamical model as homeostasis. A future article will address this issue.
8. For further reading The following references are also of interest to the reader: [1,4}7,12].
References [1] E. Barkai, M. Hasselmo, Modulation of the input/output function of rat piriform cortex pyramidal cells, J. Neurophysiol. 72 (1994) 644}658. [2] P. den Dulk, B. Rokers, R.H. Phaf, Connectionist simulations with a dual route model of fear conditioning, in: B. Kokinov (Ed.), Perspective on Cognitive Science, Vol. 4, New Bulgarian University Press, So"a, 1996. [3] M.A. Gluck, C.E. Myers, Hippocampal mediation of stimulus representation: a computational theory, Hippocampus 3 (1993) 491}516. [4] M. Hasselmo, Neuromodulation and cortical function: modeling the physiological basis of behavior, Behav. Brain Res. 67 (1995) 1}27. [5] M. Hasselmo, E. Barkai, Cholinergic modulation of activity-dependent synaptic plasticity in the piriform cortex and associative memory function in a network biophysiological simulation, J. Neurosci. 15 (1995) 6592}6604. [6] P. Huerta, J. Lisman, Heightened synaptic plasticity of hippocampal CA1 neurons during a cholinergically induced rhytmic state, Nature 364 (1993) 723}725. [7] D. Madison, B. Lancaster, R. Nicoll, Voltage clamp analysis of cholinergic action in the hippocampus, J. Neurosci. 7 (1987) 733}741. [8] C.E. Myers, B.R. Ermita, K. Harris, M. Hasselmo, P. Solomon, M.A. Gluck, A computational model of the e!ects of septohippocampal disruption on classical eyeblink conditioning, Neurobiol. Learn. Memory 66 (1996) 51}66. [9] C.E. Myers, B.R. Ermita, M. Hasselmo, M.A. Gluck, Further implications of a computational model of septohippocampal modulation in eyeblink conditioning, Psychobiology 26 (1) (1998) 1}20. [10] R.H. Phaf, A.G. Tijsseling, E. Lebert, Self-organizing CALM maps, submitted for publication. [11] D. Rumelhart, G. Hinton, R. Williams, Learning internal representations by error propagation, in: D. Rumelhart, J. McClelland (Eds.), Parallel Distributed Processing: explorations in the Microstructure of Cognition, MIT Press, Cambridge, 1986, pp. 318}362. [12] B. Widrow, M.E. Ho!, Adaptive switching circuits, WESCON Convention Record, Vol. 4, 1960.
B. Rokers et al. / Neurocomputing 32}33 (2000) 501}507
507
Catherine Myers is a Research Assistant Professor in the Department of Psychology at Rutgers University in Newark. She received her B.S. in Cognitive Science from the University of Delaware in 1987, and her Ph.D. in Neural Systems Engineering from the University of London in 1991. She studies the brain substrates of learning and memory with an emphasis on memory and memory dysfunction in humans. One area of interest is computational neuroscience, developing neural networks of how various brain regions interact during normal learning, and how lesions a!ect the ability to acquire new information. A second area of interest is experimental neuropsychology, particularly developing simple associative tasks, which may dissociate super"cially similar amnesic syndromes depending on which underlying brain systems have been damaged.
Bas Rokers is a Cognitive Science Ph.D. Student in the Department of Psychology at Rutgers University in Newark. He received his B.S. degree in Cognitive Science and Arti"cial Intelligence from Utrecht University, the Netherlands in 1998. He studies the brain substrates of learning and memory with an emphasis on neural development and neuro-modulation. One area of interest is computational neuroscience, developing neural networks of how various brain regions interact during normal learning, and how neurogenesis a!ects the ability to acquire new information. A second area of interest is computational neuropsychology, particularly the interaction between emotion and cognition, through the function and interaction of their respective neural structures, the hippocampus and the amygdala.
Mark Gluck works at the interface between cognition, animal learning, behavioral neuroscience, and computational modeling. He combines his training in human and animal psychology with neural-network analyses to understand the mechanisms of learning and memory. His earlier work in cognitive psychology demonstrated how behavioral properties of animal learning can be related to higher-order forms of learning in humans. Currently, his lab works to identify, model, and empirical evaluate fundamental components of both animal and human memory systems, with a special emphasis on the functional role of the hippocampus in learning.