- Email: [email protected]
Stochastic interaction in associative nets
ARTICLE IN PRESS
Neurocomputing 65–66 (2005) 387–392 www.elsevier.com/locate/neucom
Stochastic interaction in associative nets Thomas Wennekersa,,1, Nihat Ayb,1 a
Centre for Theoretical and Computational Neuroscience, Plymouth Institute of Neuroscience, University of Plymouth, Portland Square, Room 218, Drake Circus, Plymouth PL4 8AA, United Kingdom b Institute of Mathematics, University of Erlangen, 91054 Erlangen, Germany Available online 15 December 2004
Abstract Spatio-temporal correlations in spike trains of simultaneously recorded neurons characterize stochastic interactions in neural networks. Information theoretic measures for ‘‘spatial’’ and ‘‘temporal stochastic interaction’’ can measure the total amount of dependence in a set of cells. In the present work we calculate these interaction measures for associative networks, the most prominent models for cortical gamma-oscillations and precisely repetiting spike patterns (synfire chains). Stochastic interaction in these networks appears to be very high conflicting with the common belief that neurons are largely independent Poisson processes. r 2004 Elsevier B.V. All rights reserved. Keywords: Spike correlations; Stochastic interaction; Associative memory models
1. Spike-correlations and associative models Simultaneous recordings of multiple cortical units revealed various types of correlation patterns on the time-scale of single spikes [1,2,6,9]. Fig. 1F, for instance, displays the correlation function of two (artificially generated) spike trains and reveals a dominant peak at time zero indicative for frequent synchronous firings. The side-peaks at 20 ms further relate to a preference of the cells to fire periodically Corresponding author. Tel.: +44 1752 23 3593; fax: +44 1752 23 3349.
E-mail address: [email protected] (T. Wennekers). Part of this work has been performed when both authors were at the Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany. 1
0925-2312/$ - see front matter r 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.neucom.2004.10.033
ARTICLE IN PRESS T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
388
(F)
(B)
60
1
(A) 1 auto assoc. 1
36
#
# 40 20
36
(C) 1
0
-60 -40 -20 #
36
(D) 1 1
sfc (cyclic)
36
0
τ
20
40
60
(G) 36 1
(E)
#
#
36
36
time
τ1
τ2
Fig. 1. A: Auto-associative synaptic coupling matrix storing six non-overlapping patterns of six neurons each. B: Persistent synchronized and C: asynchronous activity as raster plots. D: Hetero-associative matrix of six patterns (cyclic). E: Ordered retrieval of the stored sequence. F: Correlation function showing synchronized noisy oscillation of 2 neurons. G: Synfire pattern comprising 3 units, i.e., 2 time-shifts.
around 50 Hz. ‘‘Synchronized gamma-oscillations’’ of this type in real data have been proposed as expressing the binding of feature coding neurons into ‘‘coherent’’ object representations [9]. Fig. 1G displays a correlation function for three units. The abscissa here counts spike-triplets, where the first unit fires at some time t, the second unit fires t1 ms later, and the third one t2 ms after the first. Most triplets occur at a low rate consistent with independent firing. The prominent peak in Fig. 1G, however, defines a certain pattern that appears significantly more often than expected by chance. Recordings from monkey prefrontal cortex [1] indicate that such ‘‘synfire patterns’’ occur behavior-dependent, but their functional role is unknown. The above types of spatio-temporal correlations have been related to tentative network architectures. By far the most prominent networks used for that purpose are instances of associative memory models with more or less realistic artificial neurons and a synaptic connectivity matrix that stores a set of binary patterns using a Hebbian learning rule [8,11]. Realistic simulations with compartmental neurons can, e.g., be found in [10]. In the present paper we solely focus on firing patterns and not on how they are generated. Fig. 1 displays some (non-exhaustive) prototypical types of firing patterns that can arise in associative networks. Fig. 1A, e.g., shows an auto-associative synaptic matrix storing six non-overlapping patterns in a network of 36 units. Figs. 1B and C reveal two modes of dynamic activity possible in an auto-associative net: In B all neurons of the first pattern fire periodically and synchronized. This example is the most simple one that explains the basic features of oscillatory synchronization as in Fig. 1F. In Fig. 1C the fifth pattern has been activated but now the neurons fire asynchronously though still periodically.
ARTICLE IN PRESS T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
389
Other types of behavior can be possible in concrete auto-associative networks. Thus, the observable spike patterns are not a unique function of connectivity. Fig. 1D shows a connectivity matrix, where six patterns have been stored in cyclic order. Correspondingly, a sequence of firings of the whole patterns as in Fig. 1E can result. This principle basically constitutes the ‘‘synfire chain model’’ as proposed by Abeles [1] in order to explain precise repetiting firing patterns as described in Fig. 1G. However, observe that also the cells in Fig. 1C fire in a fixed temporal order although they are homogeneously coupled. Thus, repetiting patterns can occur in different architectures. These simple examples show that the same architecture can lead to various types of firing correlations, and that different architectures can lead to spike patterns with comparable properties. Certainly, in real neural networks and complex neural simulations the relation between connectivity and correlation structure would be even more involved as in the simple example above (see, e.g. [2,6,10]). Therefore, one should carefully distinguish between the correlation structure in spike data, also called ‘‘functional couplings’’ [2], and the underlying physical connectivity.
2. Spatial and temporal stochastic interaction Attempts have been made to characterize spike correlations and more general stochastic interactions in spike data mathematically [3–7]. For a stationary probability distribution p on the joint state space O of N units, spatial (stochastic) interaction can be defined in terms of Shannon-entropies as IðpÞ:¼
N X
Hðpv Þ HðpÞ;
(1)
v¼1
where the pv are the marginals of p. IðpÞ can be shown to measure the Kullback–Leibler distance from independence, whence, the name ‘‘interaction’’ [4]. Spatial interaction has been extended to stochastic processes by Ay [4]: Given a probability distribution p on the states of N units and a Markov kernel K for the probabilities of state transitions, temporal (stochastic) interaction, Iðp; KÞ can be represented analogous to (1) as Iðp; KÞ ¼
N X
Hðpv ; K v Þ Hðp; KÞ;
(2)
v¼1
where the conditional entropy of a Markov transition ðp; KÞ is defined as Hðp; KÞ:¼ P 0 0 o;o0 2O pðoÞ Kðo j oÞ ln Kðo j oÞ; and the pv and K v are marginal probabilities and Markov kernels for unit v, respectively. Iðp; KÞ is the K–L distance from a product of independent (‘‘split’’) Markov chains [4]. Maximum values for IðpÞ and Iðp; KÞ are N ln 2: No system comprising N binary units can reveal stronger total interactions. Because associative memory models are the most prominent models proposed to explain stimulus-dependent synchronization and synfire patterns, we now study the stochastic interaction of firing patterns ideally observed in them. In order to map the
ARTICLE IN PRESS T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
390
1−ε
1 1
1
1
1−ε 1
0
2
1
1 0
2
1
1
4
(A)
1
5
3
(B)
1
4
1
1
1−ε
(C)
1−ε
ε
2
0
3
5
ε
ε
1
1
5
3
ε
ε
4
ε
1−ε
1−ε
Fig. 2. Transition diagrams for (A) the auto-associative and (B) the synfire chain model and six stored patterns. Labels at edges denote one-step transition probabilities. Graph C contains A and B as limiting cases for ¼ 0 and 1.
time-continuous dynamics in real systems to time-discrete Markov chains, we assume a bin-size of the length of the firing-period in Fig. 2B and C and of the switching time between patterns in Fig. 2E. This perfectly leads to state-transition diagrams between activated patterns as depicted in Figs. 2A and B. Obviously, our simplifications neglect spike jitter, overlaps between patterns, and the different firing modes in Figs. 1B and C, but they enable a comparison of the prototypical modes in auto-associative and hetero-associative nets, i.e., attractor networks and synfire chain models. Formally, we consider M patterns si ¼ ðsi1 ; . . . ; siN Þ 2 f0; 1gN ; i ¼ 0; 1; . . . ; M 1 such that for all units v 2 f1; . . . ; Ng: jfi: siv ¼ 1gjp1 (i.e., each unit is in at most one i i pattern). We define PM1the size of the patterns s as N i :¼jfv: sv ¼ 1gj; and their average 1 ¯ size as N:¼ M i¼0 N i : Corresponding with Fig. 2C we use a parametric family of transition kernels: For e 2 ½0; 1 we set K e ðsi j sj Þ:¼e if i ¼ ðj þ 1Þ mod M; K e ðsi j P sj Þ:¼1 e if i ¼ j; and K e ðsi j sj Þ:¼0 otherwise. The probability distribution e 1 p:¼ M M1 i¼0 dsi is stationary for all kernels K such that IðpÞ results from (1) as ! M1 X 1 1 M 1 M 1 ln ln IðpÞ ¼ Ni ln M M M M M i¼0 ¯ ¯ ¼ ðNM 1Þ ln M NðM 1Þ lnðM 1Þ; which obviously is independent of e: Now, we compute the temporal stochastic interaction. The marginal kernels K ev are for all v K ev ð0j0Þ ¼
M 1e ; M 1
K ev ð0j1Þ ¼ e;
K ev ð1j0Þ ¼
e ; M 1
K ev ð1j1Þ ¼ 1 e:
Using these kernels and (2) the temporal stochastic interaction gets ¯ ¯ Iðp; K e Þ ¼ NðM 1Þ lnðM 1Þ NðM 1 eÞ lnðM 1 eÞ ¯ 1Þe ln e ðN ¯ 1Þð1 eÞ lnð1 eÞ: ð2N ¯ ¼ 6; i ¼ 0; . . . ; 5 as in Fig. 3(left) displays IðpÞ and Iðp; KÞ for M ¼ 6 and N i ¼ N Fig. 1. Apparently, IðpÞ ¼ 14:4 is constant at roughly 58% of the maximally possible
ARTICLE IN PRESS
25
25
20
20
15 10
I(p), I(p,K)
I(p), I(p,K)
T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
391
15 10
5
5
0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 epsilon
0
2
4
6
8
10 12 14 16 18 Nbar
Fig. 3. Temporal (solid) and spatial interaction (dashed) in the Willshaw associative memory; dotted line: ¯ ¼ 6:; M ¼ 6: maximally possible value, N ln 2 24:95: Left: Interaction as function of e with N ¼ 36; N ¯ with N ¼ 36; M ¼ N=N; ¯ e ¼ 1: The thin dashed line Right: Interaction as a function of pattern size N represents the number of storable (non-overlapping, equal sized) patterns.
value of 36 ln 2 ¼ 24:95 for 36 independent binary units. So, the attractor and synfire chain dynamics have equal spatial interaction. Nonetheless, as Fig. 3 (left) shows, the attractor dynamics (e ¼ 0) has zero temporal interaction, whereas that of the synfire dynamics (e ¼ 1) is high. Fig. 3 (right) displays interaction values as a function of pattern size for ¼ 1; i.e., the synfire chain model. Increasing pattern size decreases the number of storable (non-overlapping) patterns (thin dashed line). For ¯ ¼ N=2 ¼ 18 we can store just two patterns which must therefore alternate in each N ¯ ¼ 1; step (e ¼ 1). Then, the temporal interaction becomes zero. Conversely, for N units are silent for N 1 steps after firing also corresponding with a low interaction. Only for intermediate pattern sizes the temporal interaction can be large. In contrast, ¯ ¼ spatial interaction increases monotonically and obtains its optimal value at N N=2 ¼ 18; where M ¼ 2; such that the single unit entropies are maximal (each unit is 0 in one pattern and 1 in the other). These entropies decrease for smaller patterns ¯ The picture remains basically the same for such that IðpÞ gets small for small N: larger networks, but note, that storage of a reasonable number of patterns requires small pattern sizes. Then the interaction values may fall considerably below the maximum value of N ln 2: Preliminary estimates show that overlapping patterns, e.g., random patterns, si ; in that regime again result in interactions that reach a high fraction of the maximally possible values. Correlations as in Figs. 1F and G derived from experiments are usually only small, synfire patterns are hardly observable at all (cf., [1]). This is often interpreted as suggesting that cortical neurons act largely as independent Poisson processes. The spatial and temporal stochastic interaction measures for N-independent Poisson processes, however, are zero (because these measures represent a K–L distance from independence). In contrast, we have shown here that the interaction measures for firing patterns expected in the most prominent models for synchronized gammaoscillations and synfire patterns, reach a moderate amount of the absolutely possible values for N binary units (N ln 2): Such systems are therefore quite strongly interacting! As long as it is assumed that the models in principle cover aspects of cortical function, this seeming contradiction leaves as the only possibility that pairwise and low-order correlations are not well suited measures to determine the dependence structure in multiple unit firing patterns. This would only show up in
ARTICLE IN PRESS 392
T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
measurements from a large number of cells. Finally, note that temporal interaction is non-vanishing only in networks comprising hetero-associative propagation of activity. In [4,5] we argued that neural systems may try to maximize temporal interaction. In that case activity propagation along neural pathways should be the rule rather than persistent attractor states. References [1] M. Abeles, Corticonics, Cambridge University Press, Cambridge, UK, 1991. [2] A.M.H.J. Aertsen, G.L. Gerstein, M.K. Habib, G. Palm, Dynamics of neuronal firing correlation: modulation of ‘‘Effective Connectivity’’, J. Neurophysiol. 61 (1989) 900–917. [3] N. Ay, Locality of global stochastic interaction in directed acyclic networks, Neural Comput. 14 (2002) 2959–2980. [4] N. Ay, Information geometry on complexity and stochastic interaction, IEEE Trans. Inform. Theory, (2003) under revision. [5] N. Ay, T. Wennekers, Dynamical properties of strongly interacting Markov chains, Neural Networks 16 (2003) 1483–1497. [6] L. Martignon, H. von Hasseln, S. Gru¨n, A. Aertsen, G. Palm, Detecting higher-order interactions among the spiking events in a group of neurons, Biol. Cybern. 73 (1995) 69–81. [7] H. Nakahara, S. Amari, Information geometric measure for neural spike trains, Neural Comput. 14 (2002) 2269–2316. [8] G. Palm, Neural Assemblies, Springer, Berlin, 1982. [9] W. Singer, C.M. Gray, Visual feature integration and the temporal correlation hypotheses, Ann. Rev. Neurosci. 18 (1995) 555–586. [10] F. Sommer, T. Wennekers, Synfire chains with conductance-based neurons: internal timing and coordination with timed input, Neurocomputing (2005) in press. [11] D.J. Willshaw, O.P. Buneman, H.C. Longuet-Higgins, Non-holographic associative memory, Nature 222 (1969) 960–962.
Neurocomputing 65–66 (2005) 387–392 www.elsevier.com/locate/neucom
Stochastic interaction in associative nets Thomas Wennekersa,,1, Nihat Ayb,1 a
Centre for Theoretical and Computational Neuroscience, Plymouth Institute of Neuroscience, University of Plymouth, Portland Square, Room 218, Drake Circus, Plymouth PL4 8AA, United Kingdom b Institute of Mathematics, University of Erlangen, 91054 Erlangen, Germany Available online 15 December 2004
Abstract Spatio-temporal correlations in spike trains of simultaneously recorded neurons characterize stochastic interactions in neural networks. Information theoretic measures for ‘‘spatial’’ and ‘‘temporal stochastic interaction’’ can measure the total amount of dependence in a set of cells. In the present work we calculate these interaction measures for associative networks, the most prominent models for cortical gamma-oscillations and precisely repetiting spike patterns (synfire chains). Stochastic interaction in these networks appears to be very high conflicting with the common belief that neurons are largely independent Poisson processes. r 2004 Elsevier B.V. All rights reserved. Keywords: Spike correlations; Stochastic interaction; Associative memory models
1. Spike-correlations and associative models Simultaneous recordings of multiple cortical units revealed various types of correlation patterns on the time-scale of single spikes [1,2,6,9]. Fig. 1F, for instance, displays the correlation function of two (artificially generated) spike trains and reveals a dominant peak at time zero indicative for frequent synchronous firings. The side-peaks at 20 ms further relate to a preference of the cells to fire periodically Corresponding author. Tel.: +44 1752 23 3593; fax: +44 1752 23 3349.
E-mail address: [email protected] (T. Wennekers). Part of this work has been performed when both authors were at the Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany. 1
0925-2312/$ - see front matter r 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.neucom.2004.10.033
ARTICLE IN PRESS T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
388
(F)
(B)
60
1
(A) 1 auto assoc. 1
36
#
# 40 20
36
(C) 1
0
-60 -40 -20 #
36
(D) 1 1
sfc (cyclic)
36
0
τ
20
40
60
(G) 36 1
(E)
#
#
36
36
time
τ1
τ2
Fig. 1. A: Auto-associative synaptic coupling matrix storing six non-overlapping patterns of six neurons each. B: Persistent synchronized and C: asynchronous activity as raster plots. D: Hetero-associative matrix of six patterns (cyclic). E: Ordered retrieval of the stored sequence. F: Correlation function showing synchronized noisy oscillation of 2 neurons. G: Synfire pattern comprising 3 units, i.e., 2 time-shifts.
around 50 Hz. ‘‘Synchronized gamma-oscillations’’ of this type in real data have been proposed as expressing the binding of feature coding neurons into ‘‘coherent’’ object representations [9]. Fig. 1G displays a correlation function for three units. The abscissa here counts spike-triplets, where the first unit fires at some time t, the second unit fires t1 ms later, and the third one t2 ms after the first. Most triplets occur at a low rate consistent with independent firing. The prominent peak in Fig. 1G, however, defines a certain pattern that appears significantly more often than expected by chance. Recordings from monkey prefrontal cortex [1] indicate that such ‘‘synfire patterns’’ occur behavior-dependent, but their functional role is unknown. The above types of spatio-temporal correlations have been related to tentative network architectures. By far the most prominent networks used for that purpose are instances of associative memory models with more or less realistic artificial neurons and a synaptic connectivity matrix that stores a set of binary patterns using a Hebbian learning rule [8,11]. Realistic simulations with compartmental neurons can, e.g., be found in [10]. In the present paper we solely focus on firing patterns and not on how they are generated. Fig. 1 displays some (non-exhaustive) prototypical types of firing patterns that can arise in associative networks. Fig. 1A, e.g., shows an auto-associative synaptic matrix storing six non-overlapping patterns in a network of 36 units. Figs. 1B and C reveal two modes of dynamic activity possible in an auto-associative net: In B all neurons of the first pattern fire periodically and synchronized. This example is the most simple one that explains the basic features of oscillatory synchronization as in Fig. 1F. In Fig. 1C the fifth pattern has been activated but now the neurons fire asynchronously though still periodically.
ARTICLE IN PRESS T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
389
Other types of behavior can be possible in concrete auto-associative networks. Thus, the observable spike patterns are not a unique function of connectivity. Fig. 1D shows a connectivity matrix, where six patterns have been stored in cyclic order. Correspondingly, a sequence of firings of the whole patterns as in Fig. 1E can result. This principle basically constitutes the ‘‘synfire chain model’’ as proposed by Abeles [1] in order to explain precise repetiting firing patterns as described in Fig. 1G. However, observe that also the cells in Fig. 1C fire in a fixed temporal order although they are homogeneously coupled. Thus, repetiting patterns can occur in different architectures. These simple examples show that the same architecture can lead to various types of firing correlations, and that different architectures can lead to spike patterns with comparable properties. Certainly, in real neural networks and complex neural simulations the relation between connectivity and correlation structure would be even more involved as in the simple example above (see, e.g. [2,6,10]). Therefore, one should carefully distinguish between the correlation structure in spike data, also called ‘‘functional couplings’’ [2], and the underlying physical connectivity.
2. Spatial and temporal stochastic interaction Attempts have been made to characterize spike correlations and more general stochastic interactions in spike data mathematically [3–7]. For a stationary probability distribution p on the joint state space O of N units, spatial (stochastic) interaction can be defined in terms of Shannon-entropies as IðpÞ:¼
N X
Hðpv Þ HðpÞ;
(1)
v¼1
where the pv are the marginals of p. IðpÞ can be shown to measure the Kullback–Leibler distance from independence, whence, the name ‘‘interaction’’ [4]. Spatial interaction has been extended to stochastic processes by Ay [4]: Given a probability distribution p on the states of N units and a Markov kernel K for the probabilities of state transitions, temporal (stochastic) interaction, Iðp; KÞ can be represented analogous to (1) as Iðp; KÞ ¼
N X
Hðpv ; K v Þ Hðp; KÞ;
(2)
v¼1
where the conditional entropy of a Markov transition ðp; KÞ is defined as Hðp; KÞ:¼ P 0 0 o;o0 2O pðoÞ Kðo j oÞ ln Kðo j oÞ; and the pv and K v are marginal probabilities and Markov kernels for unit v, respectively. Iðp; KÞ is the K–L distance from a product of independent (‘‘split’’) Markov chains [4]. Maximum values for IðpÞ and Iðp; KÞ are N ln 2: No system comprising N binary units can reveal stronger total interactions. Because associative memory models are the most prominent models proposed to explain stimulus-dependent synchronization and synfire patterns, we now study the stochastic interaction of firing patterns ideally observed in them. In order to map the
ARTICLE IN PRESS T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
390
1−ε
1 1
1
1
1−ε 1
0
2
1
1 0
2
1
1
4
(A)
1
5
3
(B)
1
4
1
1
1−ε
(C)
1−ε
ε
2
0
3
5
ε
ε
1
1
5
3
ε
ε
4
ε
1−ε
1−ε
Fig. 2. Transition diagrams for (A) the auto-associative and (B) the synfire chain model and six stored patterns. Labels at edges denote one-step transition probabilities. Graph C contains A and B as limiting cases for ¼ 0 and 1.
time-continuous dynamics in real systems to time-discrete Markov chains, we assume a bin-size of the length of the firing-period in Fig. 2B and C and of the switching time between patterns in Fig. 2E. This perfectly leads to state-transition diagrams between activated patterns as depicted in Figs. 2A and B. Obviously, our simplifications neglect spike jitter, overlaps between patterns, and the different firing modes in Figs. 1B and C, but they enable a comparison of the prototypical modes in auto-associative and hetero-associative nets, i.e., attractor networks and synfire chain models. Formally, we consider M patterns si ¼ ðsi1 ; . . . ; siN Þ 2 f0; 1gN ; i ¼ 0; 1; . . . ; M 1 such that for all units v 2 f1; . . . ; Ng: jfi: siv ¼ 1gjp1 (i.e., each unit is in at most one i i pattern). We define PM1the size of the patterns s as N i :¼jfv: sv ¼ 1gj; and their average 1 ¯ size as N:¼ M i¼0 N i : Corresponding with Fig. 2C we use a parametric family of transition kernels: For e 2 ½0; 1 we set K e ðsi j sj Þ:¼e if i ¼ ðj þ 1Þ mod M; K e ðsi j P sj Þ:¼1 e if i ¼ j; and K e ðsi j sj Þ:¼0 otherwise. The probability distribution e 1 p:¼ M M1 i¼0 dsi is stationary for all kernels K such that IðpÞ results from (1) as ! M1 X 1 1 M 1 M 1 ln ln IðpÞ ¼ Ni ln M M M M M i¼0 ¯ ¯ ¼ ðNM 1Þ ln M NðM 1Þ lnðM 1Þ; which obviously is independent of e: Now, we compute the temporal stochastic interaction. The marginal kernels K ev are for all v K ev ð0j0Þ ¼
M 1e ; M 1
K ev ð0j1Þ ¼ e;
K ev ð1j0Þ ¼
e ; M 1
K ev ð1j1Þ ¼ 1 e:
Using these kernels and (2) the temporal stochastic interaction gets ¯ ¯ Iðp; K e Þ ¼ NðM 1Þ lnðM 1Þ NðM 1 eÞ lnðM 1 eÞ ¯ 1Þe ln e ðN ¯ 1Þð1 eÞ lnð1 eÞ: ð2N ¯ ¼ 6; i ¼ 0; . . . ; 5 as in Fig. 3(left) displays IðpÞ and Iðp; KÞ for M ¼ 6 and N i ¼ N Fig. 1. Apparently, IðpÞ ¼ 14:4 is constant at roughly 58% of the maximally possible
ARTICLE IN PRESS
25
25
20
20
15 10
I(p), I(p,K)
I(p), I(p,K)
T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
391
15 10
5
5
0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 epsilon
0
2
4
6
8
10 12 14 16 18 Nbar
Fig. 3. Temporal (solid) and spatial interaction (dashed) in the Willshaw associative memory; dotted line: ¯ ¼ 6:; M ¼ 6: maximally possible value, N ln 2 24:95: Left: Interaction as function of e with N ¼ 36; N ¯ with N ¼ 36; M ¼ N=N; ¯ e ¼ 1: The thin dashed line Right: Interaction as a function of pattern size N represents the number of storable (non-overlapping, equal sized) patterns.
value of 36 ln 2 ¼ 24:95 for 36 independent binary units. So, the attractor and synfire chain dynamics have equal spatial interaction. Nonetheless, as Fig. 3 (left) shows, the attractor dynamics (e ¼ 0) has zero temporal interaction, whereas that of the synfire dynamics (e ¼ 1) is high. Fig. 3 (right) displays interaction values as a function of pattern size for ¼ 1; i.e., the synfire chain model. Increasing pattern size decreases the number of storable (non-overlapping) patterns (thin dashed line). For ¯ ¼ N=2 ¼ 18 we can store just two patterns which must therefore alternate in each N ¯ ¼ 1; step (e ¼ 1). Then, the temporal interaction becomes zero. Conversely, for N units are silent for N 1 steps after firing also corresponding with a low interaction. Only for intermediate pattern sizes the temporal interaction can be large. In contrast, ¯ ¼ spatial interaction increases monotonically and obtains its optimal value at N N=2 ¼ 18; where M ¼ 2; such that the single unit entropies are maximal (each unit is 0 in one pattern and 1 in the other). These entropies decrease for smaller patterns ¯ The picture remains basically the same for such that IðpÞ gets small for small N: larger networks, but note, that storage of a reasonable number of patterns requires small pattern sizes. Then the interaction values may fall considerably below the maximum value of N ln 2: Preliminary estimates show that overlapping patterns, e.g., random patterns, si ; in that regime again result in interactions that reach a high fraction of the maximally possible values. Correlations as in Figs. 1F and G derived from experiments are usually only small, synfire patterns are hardly observable at all (cf., [1]). This is often interpreted as suggesting that cortical neurons act largely as independent Poisson processes. The spatial and temporal stochastic interaction measures for N-independent Poisson processes, however, are zero (because these measures represent a K–L distance from independence). In contrast, we have shown here that the interaction measures for firing patterns expected in the most prominent models for synchronized gammaoscillations and synfire patterns, reach a moderate amount of the absolutely possible values for N binary units (N ln 2): Such systems are therefore quite strongly interacting! As long as it is assumed that the models in principle cover aspects of cortical function, this seeming contradiction leaves as the only possibility that pairwise and low-order correlations are not well suited measures to determine the dependence structure in multiple unit firing patterns. This would only show up in
ARTICLE IN PRESS 392
T. Wennekers, N. Ay / Neurocomputing 65–66 (2005) 387–392
measurements from a large number of cells. Finally, note that temporal interaction is non-vanishing only in networks comprising hetero-associative propagation of activity. In [4,5] we argued that neural systems may try to maximize temporal interaction. In that case activity propagation along neural pathways should be the rule rather than persistent attractor states. References [1] M. Abeles, Corticonics, Cambridge University Press, Cambridge, UK, 1991. [2] A.M.H.J. Aertsen, G.L. Gerstein, M.K. Habib, G. Palm, Dynamics of neuronal firing correlation: modulation of ‘‘Effective Connectivity’’, J. Neurophysiol. 61 (1989) 900–917. [3] N. Ay, Locality of global stochastic interaction in directed acyclic networks, Neural Comput. 14 (2002) 2959–2980. [4] N. Ay, Information geometry on complexity and stochastic interaction, IEEE Trans. Inform. Theory, (2003) under revision. [5] N. Ay, T. Wennekers, Dynamical properties of strongly interacting Markov chains, Neural Networks 16 (2003) 1483–1497. [6] L. Martignon, H. von Hasseln, S. Gru¨n, A. Aertsen, G. Palm, Detecting higher-order interactions among the spiking events in a group of neurons, Biol. Cybern. 73 (1995) 69–81. [7] H. Nakahara, S. Amari, Information geometric measure for neural spike trains, Neural Comput. 14 (2002) 2269–2316. [8] G. Palm, Neural Assemblies, Springer, Berlin, 1982. [9] W. Singer, C.M. Gray, Visual feature integration and the temporal correlation hypotheses, Ann. Rev. Neurosci. 18 (1995) 555–586. [10] F. Sommer, T. Wennekers, Synfire chains with conductance-based neurons: internal timing and coordination with timed input, Neurocomputing (2005) in press. [11] D.J. Willshaw, O.P. Buneman, H.C. Longuet-Higgins, Non-holographic associative memory, Nature 222 (1969) 960–962.