- Email: [email protected]
Neural networks as models of associative memories
Computer Physics Communications 55 (1989) 77—84 North-Holland, Amsterdam
77
NEURAL NETWORKS AS MODELS OF ASSOCL4T1VE MEMORIES
*
N. PARGA Centro Atbmico Bariloche and Instituto Balseiro, Comisibn Nacional de Energia Atômica, 8400 S.C. de Bariloche, Rio Negro, Argentina Received 27 December 1988
Neural networks have proved to be useful models of associative memories. After a brief review of the standard Hopfield model we discuss how to introduce some realistic features such as categorization of the stored information and asymmetric synapsis.
1. Introduction Although the application of statistical mechamcs tools to neural networks started very recently [1,2], these systems have received important contributions from many areas of science since W. McCulloch and W. Pitts showed that they are general computing devices [3]. Learning in neural networks was introduced later on by D. Hebb [4], who proposed a rule which has been the basis of many models of content addressable memories, McCulloch’s and Pitts’ formal neurons are simple processors which can stay in two possible states: firing and quiescent. At a given time the neuron chooses its state as a function of its activation field defined as a weighed sum of the states of other similar elements. The weights represent the synaptic strengths and they are determined by the task the network is prepared to perform. A task is given by a mapping from an initial state of the network to an output state. The ability of the system to perform it depends on its architecture, i.e. the way neurons are connected to each other. A simple machine, the perceptron [5], consists of two layers of neurons with feed-forward processing; those units in the input layer de*
Paper presented at the Adriatico Research Conference on: Computer Simulation Techniques for the Study of Microscopic Phenomena, Trieste, Italy, July 19—22, 1988.
termine their state from the outer world while each output neuron is a linear threshold unit. An interesting property of the perceptron is that, according to a convergence theorem, whenever it is able to solve a task a simple gradient algorithm can find the value of the couplings between the input and output neurons in a finite time. However, it was soon realized that many problems cannot be solved by a simple perceptron [6]. An example is given by the exclusive-OR of two input units; the perceptron is not able to make the right classification because the problem is not linearly separable. When the input and output layers are identified the system becomes a fully connected network. While the perceptron is a one time step machine the new system of linear threshold units evolves following a sequential or parallel dynamics until it reaches a fixed point or a limit cycle. It works as a content addressable memory. The space of neuron states is divided into several sectors each of them containing the attraction basin of one of the fixed points. These give the neural representations of information which has been stored in the network, while any point in its attraction basin represents a partial or distorted information. Given an initial state this will flow under the dynamics until the correct memorized state is retrieved. A model with these characteristics was studied by Hopfield [7] and it soon attracted the attention of physicists
OO1O-4655/89/$03.50 © Elsevier Science Publishers B.V. (North-Holland Physics Publishing Division)
78
N. Parga
/
Neural networks as models of associative memories
who could solve it by using techniques which had been developed previously to study models of spin glasses [2]. The Hopfield model has many unrealistic features. This led physicists to propose related systems which introduce dilution, asymmetric synapsis, correlation between the stored patterns, limit cycles and non-linearities. At present their effects on capacity and retrieval have been studied with some detail. Parallel to this effort multilayer systems with non-linear activation functions have received a great deal of attention [8]. A theorem due to Kolmogorov [9] states that a function of N variables taking values in [0, 1]N can be expressed in terms of a layered structure of functions of a single variable. Although the theorem says nothing about how to determine these functions it guarantees the existence of a non-linear mapping of N inputs to one output. This paper is devoted to reviewing some aspects of the recent contribution of physics to the field of neural networks. In the next section the Hopfield model is described emphasizing those features which make an analytical treatment possible. Its unrealistic properties are mentioned. Section 3 contains a discussion of some modifications done in that model to get rid of those problems or to adapt the neural network to different situations, Conclusions are given in section 4. 2. The Hopfield model of associative memories The participation of physicists in this field was motivated by a model proposed by J.J. Hopfield [7] who made just the right assumptions to allow an almost straightforward application of what had been learnt during the previous decade about mean field models of spin glasses [2]. The basic objects of the Hopfield model are still two-state neurons. There are two other hypotheses which are fundamental to make the application of equilibrium statistical mechanics possible: each neuron is connected to all the others through the synaptic connections J,1, the diagonal terms ~ are taken equal to zero, —
—
The off-diagonal elements are symmetrical, i.e. J,~= J~. Defining the activation potential h. at site i as the weighed sum of the activation of all the other neurons, we have at time t: N
h
t
~ ‘c’ J /
‘‘
S
=
“
~t
~
(2 1
~‘
neuron S, is updated according to S1 (t
+
1)
[h1(t)]
sign
=
(2.2)
.
and its new value is immediately used to correct h1(t) (the dynamics is sequential). These assumptions allow us to define an energy function H= ~ SS (2 3a) —
.~
‘~
‘
~
N —
—
~
~‘
h S
2
i
2 3b
i~
which decreases systematically under the dynamics. Because of its sequential character and the symmetry of the couplings it ends up in a fixed point. Information is stored by assigning a meaning to the minima of H. The system works as an associative or content addressable memory: a neuron configuration inside the attraction basin of one of those minima represents a partial knowledge of the information; under the dynamics the system flows to one of those minima, when a fixed point is reached the memory is completely retrieved. Information is stored in the synaptic connections according to the Hebb’s rule [4]. If the P neuron configurations to be memorized are denoted by (E~)(fi = 1,..., F), the J,., are given by =
~
~‘
~i!:. ~ N
=1
~(ThE(fl)
(2.4)
‘
and J~,= 0. Notice that J~= J~. The dynamics given in (2.2) is a zero temperature quenching; a straightforward generalization of the model is to replace it by a stochastic
N. Parga
/
Neural networks as models of associative memories
dynamics corresponding to the heat bath version of a Monte Carlo simulation [10]. Neuron states are updated according to the probability ,.
exp(—$h1S,)
\
P~S11~ exp(—$h.~’.)+ exp(/3h.fl’
~2 ~\ ~ ‘
where l~ is the inverse of the temperature 7’. The quantities of interest are thermal averages, The free energy per neuron is given by —
~ln N
Z
(2 6~ ‘
‘
‘
with Z = tr [exp( /3H)]. Since we are not interested in the detailed properties of, a set of memories but in those which are genenc to most of them we consider the quenched average of eq. (2.6) over the distribution used to select the PN variables Denoting it with a bar: —
et al. [10]. Above the line 7~= 1 + v~the system is in a paramagnetic phase; below it there is a spin glass phase, which is always stable. It is characterized by ergodicity breaking and the existence of an exponentially large number of metastable states [2]. Since the system may fall into those states, the retrieval properties of the model may be spoiled in this phase. However, below a line TM which goes from T = 1 for a = 0 to a = 0.145 at T = 0, another phase characterized by good retrieval appears and eventually it becomes the global minimum at a temperature 7~(a)
~
~
I
f=
—-~-ii~~.
(2.7)
The generalized Hopfield model is closely related to the mean-field approximation of spin glass models, proposed by Sherrington and Kirkpatrick (SK) little more than ten years ago [11]. The SK model was originally proposed with the hope that it was easily solvable; however, it took a decade to understand its properties and the nature of its low temperature phase [12,13,2]. The Hamiltonian of the SK model is also given by (2.3); the difference between the two models lies in the way the disorder is chosen. In the SK model the random variables are 2). the On couplings the other themselves, hand in the which Hopnumber is 0(N field model the .1,, are given in terms of the PN variables ~9), In this sense the Hopfield model is more general than the spin glass model; only when P becomes 0(N) the two models coincide. The rather sophisticated techniques developed to solve the SK model [12,14] can be used to obtain the free energy of the generalized Hopfield model [10] or to count its metastable states [15]. The interesting regime is the one where P = aN; in particular a = 0 means a finite number of memorized patterns in an infinite system. The phase diagram in the (T, a) plane was obtained by Arnit
79
=
-~—
~ (S1)~~,
(2.8)
,=1
which gives the overlap between the local magnetization KS,) and the pattern fi. m~ 1 means good retrieval. At T = TM(a) this quantity changes discontinuously. At T = 0 this occurs for a = a~,where it drops from m~ 0.97 to m~ 0.35. Above this value the memory collapses; it is not able to recognize any of the P patterns and the system is always attracted to the metastable states of the spin glass phase. It was soon realized that even when the Hopfield model had some attractive features and it could be solved and understood usingnot well-known techniques, in many aspects it was realistic. Among these unrealistic features we mention: At the critical value a~the memory collapses. The retrieval properties are lost completely. The system is not only unable to learn new information but is also forgets what it had learnt before. The Hopfield model is important because it makes an analytical treatment possible. Since each neuron is connected to all the others the mean field solution is exact. However, in the cortex neurons are not connected to the nearly 1011 other neurons but only to iO~—iO~. —
—
80
N. Parga
/ Neural networks as models of associative memories
In nature the synapsis are not symmetrical. In particular if neuron i is connected to neuron j the inverse may not be true. The neural representations ()} are not orthogonal. More than one half of the neurons are inactive, what introduces correlations among the stored patterns. A particular interesting case of correlations corresponds to a hierarchical organization of the memorized patterns. In this case they can be classified in categories according to their cornmon features. Not only fixed points may be relevant for different brain functions. A natural extension corresponds to limit cycles. Its existence might be responsible for rhythmic motor behaviour governed by the so called central pattern generators [16].
—
—
—
—
3. Beyond the Hopfield model In this section we discuss some of the attempts which have been made to introduce realistic features in the Hopfield model. The discussion includes the effect of non-linearities in the learning rule, hierarchically correlated patterns, limit cycles and the effect of asymmetric synapses. 3.1. Short term memories An effort to avoid the deterioration of the Hopfield model at a critical value of a led to the proposal of networks where the memory never collapses, instead that if new patterns are memorized the older ones are forgotten. A way to achieve this is to alterand the the linear relationstrengths. between the activation fields synaptic For the case of uncorrelated memories if new information is stored, the h, ‘s follow a random walk and the resulting Gaussian distribution has a variance ~ A natural modification of the Hopfield model is to introduce a non-linear behaviour when the activation field reaches a value A 0.4. The choice made in ref. [17] is to saturate the synaptic connections,
with f(x) = —A f(x) = x f(x) =A
for x < —A, for —AA.
(3.2)
The crucial question is how many of the last k patterns are retrieved correctly, for instance, with at least 90% of the bits reproduced correctly. Defining k = xN, a good retrieval occurs for x < 0.04. The value of A is chosen such that the capacity is maximum [17] (A 0.35, close to the value mentioned before). As in the standard Hopfield model the capacity is still linear in N. There is, however, an important difference between the two networks: while in the Hopfield model for a < a~all patterns are retrieyed correctly, in this one only the last xN patterns are memorized. This means that the network becomes a model for short-term memory. The discontinuous transition at a critical capacity disappears. Short-term memory models have also been proposed by other authors [18]. One of them belongs to a family of solvable models [19]. In this case synapses are modified according to J11(t+ 1) =X[J~1(t) +/N~t+1)e51+1)j, (3.3) where the parameter c measures the acquisition amplitude of each new pattern. Again synapses are not allowed to grow freely. A suitable condition is
i~—~1/N,
(3.4)
=
which determines to theeq.parameter A = (1 + ~2/ 1~’2.According (3.3) the oldest patterns N) are multiplied by a large power of A given by the time elapsed since the pattern was learnt. For large N, A exp( C 2/2 N), thus the most recent patterns have a uniform weight during a period of 0(N). As was already mentioned, this model belongs to a family of Hopfield like networks which can be solved analytically. The optimal value of e, defined as that for which the capacity is maximal, is ~ = 4.108; the corresponding value of a is —
J, 1(t +
1) =f(J,1(t)
+ 1/V~~t+1)E.~t+1)),
(3.1)
aopt
=
0.0489. The acquisition amplitude cannot
N.
Parga
/
Neural networks as models of associative memories
be smaller than e~= 2.465. These exact values confirm estimates obtained previously in ref. [18]. 3.2. Hierarchically correlated patterns We discuss models with hierarchically correlated patterns. Simpler correlations can be handled as particular cases of them. In this case the memorized patterns can be classified into categories such that members in the same class are closer to each other than members in different classes. Categories can of course be split into subcategories and these into smaller sets until the lowest level, where each pattern forms a class by itself, is reached. A hierarchical structure was first found in physics in ref. [20], wasorganized shown that the pure states of the SKwhere modelit are in this way. The proximity of two pure states a and /3 with local magnetizations (m~} and (m~) is measured in terms of their overlap, N =
-~
~ ~ =
(3.5)
1
The hierarchical organization is equivalent to a property known as ultrametricity: given three states a, /3, y, they form a triangle which is either equilateral or isoceles with the largest overlap as its base. Following an analysis based on the SK model a learning rule was proposed [21] that for the particular case of the hierarchy shown in fig. 1 reads N
NJ 1~=
1 q
~ ~a~a a= 1 p
+
1
—
q
~
~
—
~) (~7fl
—
~).
a=1 /3=1
(3.6)
Fig. 1. A tree describing a two level hierarchy. In this figure N~=4, r =3.
81
Here ~ denotes the overlap between any pair of states in the same category, Na is the number of these and r refers to the number of patterns in each class. The pattern {~$)has two indices: a denotes the category and /3 one of its members. The category itself is ~ The generalization of this expression to more complicated hierarchical trees or to simpler ones (a single category with patterns of mutual average overlap ~J is straightforward. The thermodynamics of this model can be solved [22] and it was shown that it has only one transition located at the same a~as the Hopfield model. This can be seen calculating the order parameters N
~
=
1
~
(r’1
—
~)(s,),
(3.7a)
N
ma
=
~
~
(3.7b)
Equation (3.7a) gives the overlap of the thermal average of the neuron state with the difference between the stored pattern (saP } and the prototype of its class. The other equation gives its overlap with the prototype. Feigelman and loffe found that both of them are zero above a~= 0.145; in other words, when the network cannot recognize the memory (ji’~ = 0) it cannot recognize the category either (ma = 0). However, the thermodynamic analysis is based on the saddle point technique which can detect only free energies 0(N). For a <0.145 the system is in a spin glass phase characterized by the existence of metastable states separated by barriers which are smaller than 0(N). Under the zero temperature dynamics given by eq. (2.2) it will fall into states which are stable at least under one spin flip. In the SK model this gives rise to remanence effects [23] and the same is true for the Hopfield model [10]. As a consequence the overlaps used as order parameters may not be zero in the spin glass phase. If these effects were such that the correct category is distinguished then the system could be able to recognize it, even when the details of the memorized patterns were already lost. This was shown to be so in ref. [24] by means of a numerical
82
N. Parga 0.75
I
I
I
I
/
Neural networks as models of associative memories .00
I
5, 0
IE
IE 0.60 I~
0
0
025
0 0
020
0.40
0.60
O~J
1.00
0.2
01
0
0.3
04
05
cx (I Fig. 2. (a) Average overlap 83 between the output state and the correct category for a tree with two categories. N = 500 (0) and N 1000 (•). (b) Upper points are the same as before but for y = a/2. Lower points are the average maximum overlap M between the output state and the other categories, also for y = a/2. Here N = 1000. The dashed line indicates the classification errors.
analysis. There, it was seen that for a finite number of categories (say 2), if the initial configuration is one of the patterns in category 1, the
plied more recently to the Hopfield model without [15] and with dilution [25]. N( m, m 1, m2) was evaluated in ref. [26] for the
output state is such that it has finite overlap with most of the members of that category but zero with most of the patterns in the other class, The result obtained in ref. [24] is shown in figs. 2a and 2b. For a < a~the average overlap between the output state and the category is constant, while above a~it decreases but remains finite. On the other hand its overlap with the other category is close to zero. Because of this effect the right category can be easily recognized if a list of the prototypes is provided. The fact that the number of categories is finite guarantees that the list can be checked in a finite time. In contrast to this result, fig. 2b shows what happens for a large number of categories, i.e. N = yN. Now both the overlaps between the output and the right category and with at least one of the other categories are finite. One could argue that category recognition is still possible because they become equal only for a cc, however, problems with numencal precision and the fact that now the hst of categones is 0(N) makes recognition impractical. Good category retrieval is possible thanks to a proper organization of the metastable states into classes. This can be checked by direct evaluation of the number N(m, m1, m2) of one spin flip stable states with overlap m with a pattern in category 1, and overlaps m1 and m2 with classes 1 and 2, respectively, The technique to count these states was first developed for the SK model [14] and it was ap-
model with couplings given by eq. (3.6) and a generalized version [22] where the coefficient of the second term is taken as a new~parameter. In this case the total number of metastable states can be expressed as
—~
f°J] N
N
=
0
dA, tr~6(h, AS.) —
i=i —
=
Jdm dm1 dm2N(m, m1, m2),
(3.8)
where the bar denotes the average over the branched stochastic process used to select the memorized patterns. The main result is that the distribution N(m, m1, m2) is dominated by a saddle point where m2 = 0 but m1 # 0, in agreement with the analysis of the dynamics [24]. 3.3. Limit cycles The simplest recipe to memorize a cycle of length L is [7] 1 ~
L
a±1 a
E~
=
~j,
(3.9)
a =1
with ~çL±1) = ~(1) However this does not work because it does not allow the pattern (~) to stabilize before the system goes to the next { ~a ±1), This problem can be solved by introducing two different processes, one acting in a short time scale which takes care of stabilizing the current pattern and a second one
N. Parga
/
Neural networks as models of associative memories
acting in a longer time scale which is responsible for the transition [27]. The corresponding synapses are, j(S)
— —
1 ~ t(a)t(a)
13 10
N a~i~’ ~ A
(L) =
N
V’
\
a1
(a±1) ~ ~
1ob~
13 ‘-
‘
‘
~‘
a—i
where S and L stand for the short and long time scales respectively. The corresponding internal fields are
83
have been observed. However, the authors of ref. [28] argue that the observed values can be taken as a diluted system and since neural networks are robust under dilution both the theoretical model and the real system exhibit similar behaviours. The reason why the theoretical model is able to describe the CPG of the Tritonia Diomedea probably is the simplicity of this system. There is no reason to expect that a more complicated CPG could be explained by this simple model. A detailed study of the Tritoma as well as other organisms would be necessary to elucidate this point.
N
h~(t)
=
h~(t)
=
~
(3.lla)
j(S)~(~)
3,4. Asymmetry ~ i=l
dt’w(t
J.(.L)[f
—
t’)~1(t’)].
-
(3.llb) The dynamics is still like in eq. (2.2) with h. h~+ h~.The function ~ is a monotonously decreasing one with characteristic time T. The relative strength of the two components is chosen such that the transition to the new pattern {~(a+i)) occurs after the system stayed at { Eta) for a time of O(T). An interesting application of this model was proposed by D.K. Kleinfeld and H. Sompolinsky [28] who used it to generate the rhythmic swimming movements of the Tritonia Diomedea. Biological studies [16] of this mollusk have determined the structure of the group of neurons which generate the neural activity necessary to produce the right movements. This set of neurons is the central pattern generator (CPG) of the Tritonia Diomedea. The firing patterns that are produced by the four premotor interneurons while swimming are known [16]. Using these in formulae (3.10) Kleinfeld and Sompolinsky could predict a synaptic matrix with short and long time scale components which are in good agreement with the experimental ones. Most of the fast synaptic components predicted by the theory are observed and have the right signs. For the slow components the theory predicts more synapses than those which =
The symmetry of the synaptic connections adopted by the Hopfield model is far from being realized in a biological system. In particular if neuron i sends its axon to the dendrites of neuron j, the inverse is not necessarily true; a case usually called asymmetric dilution. The number of metastable states in easily calculated using the techniques of ref. [14] for several types of asymmetry [25]. Surprisingly the number of non-retrieval states is not appreciably altered with respect to the symmetric model. This, however, does not say everything about the dynamics since the size and shape of the attraction basins is also important. It was found in ref. [29] that in spite of the large number of metastable states, the time necessary to reach them grows exponentially with the size of the system. The dynamic behaviour is chaotic. Apart from the description of models with asymmetric synaptic matrixes, a cognitive role of asymmetry was suggested by G. Parisi [30]. When the dynamics reaches a fixed point indicating that something has been recognized there is nothing in the Hopfield model which guarantees that it is one of the memorized patterns. Spin glass states are always present and they may confuse the system, since there is no way to distinguish them from the retrieval states, Then asymmetry helps: since those spurious states are replaced by chaotic trajectories the problem is automatically avoided.
84
N. Parga
/ Neural networks as models of associative memories
4. Conclusions
[4] D.O. Hebb, The Organization of Behaviour (Wiley, New York, 1949).
Some of the most recent contributions of physics to neural networks have been reviewed in this work. Starting from a fully connected model where a mean field analysis and the application of methods developed to solve disordered magnetic systems is straightforward, physicists could later introduce the effect of pattern correlations, non-linearities, dilution and asymmetry. At the same time different types of neural networks have been considered, usually in connection with their application to specific tasks [8]. These are multilayer models where neurons on a layer determine the activity of those on the next one through a non-linear (sigmoid) function of the activity field. These networks have been applied with some success to explain features of language acquisition [8], early vision [31], the contiguity problem [32], etc. Most of these works have been motivated by the ability of the multilayer non-linear networks to recognize the relevant features of a task after a set of examples have been presented to them during a learning session. Learning by examples allows the machine to find algorithms to solve the task. Until this moment, however, physics had little influence in the analysis of these networks. Even the questions. physics would try to answer about them should be different from those addressed up to now. Instead of bothering about how to construct a network useful for a specific task one should study properties which hold for most of them by considering ensembles of problems.
[5] F. Rosemblatt, Principles Books, New York, 1962). of Neurodynamics (Spartan [6] M. Minsky and S. Pappert, Perceptrons (MIT Press, Cambridge, MA, 1969). [7] J.J. Hopfield, Proc. Nat. Acad. Sci. USA 79 (1982) 2554. [8] D.E. Rumeihart and J.L. McCleiland, Parallel Distributed Processing, Vol. 1 and 2 (Bradford Books, Cambridge, ~ 1986). [9] A.N. Kolmogorov, Dokl. Akad. Nauk. SSSR 114 (1957) 953 (in Russian). AMS translating 2 (1957) 55. [10] D.J. Amit, H. Gutfreund and H. Sompolinsky, Phys. Rev. A32 (1985) 1007; Phys. Rev. Lett. 55 (1985) 1530; Ann. Phys. 173 (1987) 30. [11] D. Shemngton and S. Kirkpatrick, Phys. Rev. Lett. 35 (1975) 1792. [121G. Parisi, Phys. Rev. Lett. 43 (1979) 1754; J. Phys. A13 (1980) L-115, 1101, 1887. [13] G. Parisi, Phys. Rev. Lett. 50 (1983) 1946. [14] A.J. Bray and M.A. Moore, J. Phys. C13 (1980) L-469. [15] E. Gardner, J. Phys. A19 (1986) L-1047. [16] P.A. Getting and MS. Dekin, in: Model Neural Networks and Behavior, A.I. Selverston, ed. (Plenum Press, New York, 1985). [17] G. Parisi, J. Phys. A19 (1986) L-617. [18] J.P. Nadal, G. Toulouse, J.-P. Changeux and S. Dehaene, Europhys. Lett. 1 (1986) 535. [19] M. Mézard, J.P. Nadal and G. Toulouse, J. Physique 47 (1986) 1457. [201M. Mézard, G. Parisi, N. Sourlas, G. Toulouse and M. Virasoro, J. Physique 45 (1984) 843. [21] N. Parga and M.A. Virasoro, J. Physique 47 (1986) 1857. [22] M.V. Feigelman and L.B. loffe, mt. J. of Mod. Phys. Bi (1987) 51. [23] W. Kinzel, Z. Phys. B60 (1985) 205.
References
[28] print D. Kleinfeld and H. Sompolinsky, AlT Bell Labs. pre(1987), submitted to J. Neurosci. [29] A. Crisanti and H. Sompolinsky, Jerusalem preprint RI/87/89. [30] G. Parisi, J. Phys. A19 (1986) L675. [31] R. Linsker, Proc. Nat. Acad. Sci. USA 83 (1986) 7508, 8779, 8390. [32] 5. Solla, S. Ryckebush and D. Schwartz, to be published.
[1] D.J. Amit, in: Proc. Heidelberg Colloquium on Glassy Dynamics, J. van Hemmen and I. Morgenstern, eds. (Springer, Berlin, 1987). [2] M. Mézard, G. Parisi and M.A. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987). [3] W.S. McCulloch and W.A. Pitts, Bull. Math. Biophys. 5 (1943) 115.
[24] S. Bacci, J. Alfaro, C. Wiecko and N. Parga, J. Physique, to appear (April 1989). [25] A. Treves and D.J. Aniit, Jerusalem preprint RJ/88/119. [26] S. Bacci, G. Mato and N. Parga, Bariloche preprint. [27] D. Kleinfeld, Proc. Nat. Acad. Sci. USA 83 (1986) 9469; H. Sompolinsky and I. Kanter, Phys. Rev. Lett. 57 (1986) 2861.
77
NEURAL NETWORKS AS MODELS OF ASSOCL4T1VE MEMORIES
*
N. PARGA Centro Atbmico Bariloche and Instituto Balseiro, Comisibn Nacional de Energia Atômica, 8400 S.C. de Bariloche, Rio Negro, Argentina Received 27 December 1988
Neural networks have proved to be useful models of associative memories. After a brief review of the standard Hopfield model we discuss how to introduce some realistic features such as categorization of the stored information and asymmetric synapsis.
1. Introduction Although the application of statistical mechamcs tools to neural networks started very recently [1,2], these systems have received important contributions from many areas of science since W. McCulloch and W. Pitts showed that they are general computing devices [3]. Learning in neural networks was introduced later on by D. Hebb [4], who proposed a rule which has been the basis of many models of content addressable memories, McCulloch’s and Pitts’ formal neurons are simple processors which can stay in two possible states: firing and quiescent. At a given time the neuron chooses its state as a function of its activation field defined as a weighed sum of the states of other similar elements. The weights represent the synaptic strengths and they are determined by the task the network is prepared to perform. A task is given by a mapping from an initial state of the network to an output state. The ability of the system to perform it depends on its architecture, i.e. the way neurons are connected to each other. A simple machine, the perceptron [5], consists of two layers of neurons with feed-forward processing; those units in the input layer de*
Paper presented at the Adriatico Research Conference on: Computer Simulation Techniques for the Study of Microscopic Phenomena, Trieste, Italy, July 19—22, 1988.
termine their state from the outer world while each output neuron is a linear threshold unit. An interesting property of the perceptron is that, according to a convergence theorem, whenever it is able to solve a task a simple gradient algorithm can find the value of the couplings between the input and output neurons in a finite time. However, it was soon realized that many problems cannot be solved by a simple perceptron [6]. An example is given by the exclusive-OR of two input units; the perceptron is not able to make the right classification because the problem is not linearly separable. When the input and output layers are identified the system becomes a fully connected network. While the perceptron is a one time step machine the new system of linear threshold units evolves following a sequential or parallel dynamics until it reaches a fixed point or a limit cycle. It works as a content addressable memory. The space of neuron states is divided into several sectors each of them containing the attraction basin of one of the fixed points. These give the neural representations of information which has been stored in the network, while any point in its attraction basin represents a partial or distorted information. Given an initial state this will flow under the dynamics until the correct memorized state is retrieved. A model with these characteristics was studied by Hopfield [7] and it soon attracted the attention of physicists
OO1O-4655/89/$03.50 © Elsevier Science Publishers B.V. (North-Holland Physics Publishing Division)
78
N. Parga
/
Neural networks as models of associative memories
who could solve it by using techniques which had been developed previously to study models of spin glasses [2]. The Hopfield model has many unrealistic features. This led physicists to propose related systems which introduce dilution, asymmetric synapsis, correlation between the stored patterns, limit cycles and non-linearities. At present their effects on capacity and retrieval have been studied with some detail. Parallel to this effort multilayer systems with non-linear activation functions have received a great deal of attention [8]. A theorem due to Kolmogorov [9] states that a function of N variables taking values in [0, 1]N can be expressed in terms of a layered structure of functions of a single variable. Although the theorem says nothing about how to determine these functions it guarantees the existence of a non-linear mapping of N inputs to one output. This paper is devoted to reviewing some aspects of the recent contribution of physics to the field of neural networks. In the next section the Hopfield model is described emphasizing those features which make an analytical treatment possible. Its unrealistic properties are mentioned. Section 3 contains a discussion of some modifications done in that model to get rid of those problems or to adapt the neural network to different situations, Conclusions are given in section 4. 2. The Hopfield model of associative memories The participation of physicists in this field was motivated by a model proposed by J.J. Hopfield [7] who made just the right assumptions to allow an almost straightforward application of what had been learnt during the previous decade about mean field models of spin glasses [2]. The basic objects of the Hopfield model are still two-state neurons. There are two other hypotheses which are fundamental to make the application of equilibrium statistical mechanics possible: each neuron is connected to all the others through the synaptic connections J,1, the diagonal terms ~ are taken equal to zero, —
—
The off-diagonal elements are symmetrical, i.e. J,~= J~. Defining the activation potential h. at site i as the weighed sum of the activation of all the other neurons, we have at time t: N
h
t
~ ‘c’ J /
‘‘
S
=
“
~t
~
(2 1
~‘
neuron S, is updated according to S1 (t
+
1)
[h1(t)]
sign
=
(2.2)
.
and its new value is immediately used to correct h1(t) (the dynamics is sequential). These assumptions allow us to define an energy function H= ~ SS (2 3a) —
.~
‘~
‘
~
N —
—
~
~‘
h S
2
i
2 3b
i~
which decreases systematically under the dynamics. Because of its sequential character and the symmetry of the couplings it ends up in a fixed point. Information is stored by assigning a meaning to the minima of H. The system works as an associative or content addressable memory: a neuron configuration inside the attraction basin of one of those minima represents a partial knowledge of the information; under the dynamics the system flows to one of those minima, when a fixed point is reached the memory is completely retrieved. Information is stored in the synaptic connections according to the Hebb’s rule [4]. If the P neuron configurations to be memorized are denoted by (E~)(fi = 1,..., F), the J,., are given by =
~
~‘
~i!:. ~ N
=1
~(ThE(fl)
(2.4)
‘
and J~,= 0. Notice that J~= J~. The dynamics given in (2.2) is a zero temperature quenching; a straightforward generalization of the model is to replace it by a stochastic
N. Parga
/
Neural networks as models of associative memories
dynamics corresponding to the heat bath version of a Monte Carlo simulation [10]. Neuron states are updated according to the probability ,.
exp(—$h1S,)
\
P~S11~ exp(—$h.~’.)+ exp(/3h.fl’
~2 ~\ ~ ‘
where l~ is the inverse of the temperature 7’. The quantities of interest are thermal averages, The free energy per neuron is given by —
~ln N
Z
(2 6~ ‘
‘
‘
with Z = tr [exp( /3H)]. Since we are not interested in the detailed properties of, a set of memories but in those which are genenc to most of them we consider the quenched average of eq. (2.6) over the distribution used to select the PN variables Denoting it with a bar: —
et al. [10]. Above the line 7~= 1 + v~the system is in a paramagnetic phase; below it there is a spin glass phase, which is always stable. It is characterized by ergodicity breaking and the existence of an exponentially large number of metastable states [2]. Since the system may fall into those states, the retrieval properties of the model may be spoiled in this phase. However, below a line TM which goes from T = 1 for a = 0 to a = 0.145 at T = 0, another phase characterized by good retrieval appears and eventually it becomes the global minimum at a temperature 7~(a)
~
~
I
f=
—-~-ii~~.
(2.7)
The generalized Hopfield model is closely related to the mean-field approximation of spin glass models, proposed by Sherrington and Kirkpatrick (SK) little more than ten years ago [11]. The SK model was originally proposed with the hope that it was easily solvable; however, it took a decade to understand its properties and the nature of its low temperature phase [12,13,2]. The Hamiltonian of the SK model is also given by (2.3); the difference between the two models lies in the way the disorder is chosen. In the SK model the random variables are 2). the On couplings the other themselves, hand in the which Hopnumber is 0(N field model the .1,, are given in terms of the PN variables ~9), In this sense the Hopfield model is more general than the spin glass model; only when P becomes 0(N) the two models coincide. The rather sophisticated techniques developed to solve the SK model [12,14] can be used to obtain the free energy of the generalized Hopfield model [10] or to count its metastable states [15]. The interesting regime is the one where P = aN; in particular a = 0 means a finite number of memorized patterns in an infinite system. The phase diagram in the (T, a) plane was obtained by Arnit
79
=
-~—
~ (S1)~~,
(2.8)
,=1
which gives the overlap between the local magnetization KS,) and the pattern fi. m~ 1 means good retrieval. At T = TM(a) this quantity changes discontinuously. At T = 0 this occurs for a = a~,where it drops from m~ 0.97 to m~ 0.35. Above this value the memory collapses; it is not able to recognize any of the P patterns and the system is always attracted to the metastable states of the spin glass phase. It was soon realized that even when the Hopfield model had some attractive features and it could be solved and understood usingnot well-known techniques, in many aspects it was realistic. Among these unrealistic features we mention: At the critical value a~the memory collapses. The retrieval properties are lost completely. The system is not only unable to learn new information but is also forgets what it had learnt before. The Hopfield model is important because it makes an analytical treatment possible. Since each neuron is connected to all the others the mean field solution is exact. However, in the cortex neurons are not connected to the nearly 1011 other neurons but only to iO~—iO~. —
—
80
N. Parga
/ Neural networks as models of associative memories
In nature the synapsis are not symmetrical. In particular if neuron i is connected to neuron j the inverse may not be true. The neural representations ()} are not orthogonal. More than one half of the neurons are inactive, what introduces correlations among the stored patterns. A particular interesting case of correlations corresponds to a hierarchical organization of the memorized patterns. In this case they can be classified in categories according to their cornmon features. Not only fixed points may be relevant for different brain functions. A natural extension corresponds to limit cycles. Its existence might be responsible for rhythmic motor behaviour governed by the so called central pattern generators [16].
—
—
—
—
3. Beyond the Hopfield model In this section we discuss some of the attempts which have been made to introduce realistic features in the Hopfield model. The discussion includes the effect of non-linearities in the learning rule, hierarchically correlated patterns, limit cycles and the effect of asymmetric synapses. 3.1. Short term memories An effort to avoid the deterioration of the Hopfield model at a critical value of a led to the proposal of networks where the memory never collapses, instead that if new patterns are memorized the older ones are forgotten. A way to achieve this is to alterand the the linear relationstrengths. between the activation fields synaptic For the case of uncorrelated memories if new information is stored, the h, ‘s follow a random walk and the resulting Gaussian distribution has a variance ~ A natural modification of the Hopfield model is to introduce a non-linear behaviour when the activation field reaches a value A 0.4. The choice made in ref. [17] is to saturate the synaptic connections,
with f(x) = —A f(x) = x f(x) =A
for x < —A, for —A
(3.2)
The crucial question is how many of the last k patterns are retrieved correctly, for instance, with at least 90% of the bits reproduced correctly. Defining k = xN, a good retrieval occurs for x < 0.04. The value of A is chosen such that the capacity is maximum [17] (A 0.35, close to the value mentioned before). As in the standard Hopfield model the capacity is still linear in N. There is, however, an important difference between the two networks: while in the Hopfield model for a < a~all patterns are retrieyed correctly, in this one only the last xN patterns are memorized. This means that the network becomes a model for short-term memory. The discontinuous transition at a critical capacity disappears. Short-term memory models have also been proposed by other authors [18]. One of them belongs to a family of solvable models [19]. In this case synapses are modified according to J11(t+ 1) =X[J~1(t) +/N~t+1)e51+1)j, (3.3) where the parameter c measures the acquisition amplitude of each new pattern. Again synapses are not allowed to grow freely. A suitable condition is
i~—~1/N,
(3.4)
=
which determines to theeq.parameter A = (1 + ~2/ 1~’2.According (3.3) the oldest patterns N) are multiplied by a large power of A given by the time elapsed since the pattern was learnt. For large N, A exp( C 2/2 N), thus the most recent patterns have a uniform weight during a period of 0(N). As was already mentioned, this model belongs to a family of Hopfield like networks which can be solved analytically. The optimal value of e, defined as that for which the capacity is maximal, is ~ = 4.108; the corresponding value of a is —
J, 1(t +
1) =f(J,1(t)
+ 1/V~~t+1)E.~t+1)),
(3.1)
aopt
=
0.0489. The acquisition amplitude cannot
N.
Parga
/
Neural networks as models of associative memories
be smaller than e~= 2.465. These exact values confirm estimates obtained previously in ref. [18]. 3.2. Hierarchically correlated patterns We discuss models with hierarchically correlated patterns. Simpler correlations can be handled as particular cases of them. In this case the memorized patterns can be classified into categories such that members in the same class are closer to each other than members in different classes. Categories can of course be split into subcategories and these into smaller sets until the lowest level, where each pattern forms a class by itself, is reached. A hierarchical structure was first found in physics in ref. [20], wasorganized shown that the pure states of the SKwhere modelit are in this way. The proximity of two pure states a and /3 with local magnetizations (m~} and (m~) is measured in terms of their overlap, N =
-~
~ ~ =
(3.5)
1
The hierarchical organization is equivalent to a property known as ultrametricity: given three states a, /3, y, they form a triangle which is either equilateral or isoceles with the largest overlap as its base. Following an analysis based on the SK model a learning rule was proposed [21] that for the particular case of the hierarchy shown in fig. 1 reads N
NJ 1~=
1 q
~ ~a~a a= 1 p
+
1
—
q
~
~
—
~) (~7fl
—
~).
a=1 /3=1
(3.6)
Fig. 1. A tree describing a two level hierarchy. In this figure N~=4, r =3.
81
Here ~ denotes the overlap between any pair of states in the same category, Na is the number of these and r refers to the number of patterns in each class. The pattern {~$)has two indices: a denotes the category and /3 one of its members. The category itself is ~ The generalization of this expression to more complicated hierarchical trees or to simpler ones (a single category with patterns of mutual average overlap ~J is straightforward. The thermodynamics of this model can be solved [22] and it was shown that it has only one transition located at the same a~as the Hopfield model. This can be seen calculating the order parameters N
~
=
1
~
(r’1
—
~)(s,),
(3.7a)
N
ma
=
~
~
(3.7b)
Equation (3.7a) gives the overlap of the thermal average of the neuron state with the difference between the stored pattern (saP } and the prototype of its class. The other equation gives its overlap with the prototype. Feigelman and loffe found that both of them are zero above a~= 0.145; in other words, when the network cannot recognize the memory (ji’~ = 0) it cannot recognize the category either (ma = 0). However, the thermodynamic analysis is based on the saddle point technique which can detect only free energies 0(N). For a <0.145 the system is in a spin glass phase characterized by the existence of metastable states separated by barriers which are smaller than 0(N). Under the zero temperature dynamics given by eq. (2.2) it will fall into states which are stable at least under one spin flip. In the SK model this gives rise to remanence effects [23] and the same is true for the Hopfield model [10]. As a consequence the overlaps used as order parameters may not be zero in the spin glass phase. If these effects were such that the correct category is distinguished then the system could be able to recognize it, even when the details of the memorized patterns were already lost. This was shown to be so in ref. [24] by means of a numerical
82
N. Parga 0.75
I
I
I
I
/
Neural networks as models of associative memories .00
I
5, 0
IE
IE 0.60 I~
0
0
025
0 0
020
0.40
0.60
O~J
1.00
0.2
01
0
0.3
04
05
cx (I Fig. 2. (a) Average overlap 83 between the output state and the correct category for a tree with two categories. N = 500 (0) and N 1000 (•). (b) Upper points are the same as before but for y = a/2. Lower points are the average maximum overlap M between the output state and the other categories, also for y = a/2. Here N = 1000. The dashed line indicates the classification errors.
analysis. There, it was seen that for a finite number of categories (say 2), if the initial configuration is one of the patterns in category 1, the
plied more recently to the Hopfield model without [15] and with dilution [25]. N( m, m 1, m2) was evaluated in ref. [26] for the
output state is such that it has finite overlap with most of the members of that category but zero with most of the patterns in the other class, The result obtained in ref. [24] is shown in figs. 2a and 2b. For a < a~the average overlap between the output state and the category is constant, while above a~it decreases but remains finite. On the other hand its overlap with the other category is close to zero. Because of this effect the right category can be easily recognized if a list of the prototypes is provided. The fact that the number of categories is finite guarantees that the list can be checked in a finite time. In contrast to this result, fig. 2b shows what happens for a large number of categories, i.e. N = yN. Now both the overlaps between the output and the right category and with at least one of the other categories are finite. One could argue that category recognition is still possible because they become equal only for a cc, however, problems with numencal precision and the fact that now the hst of categones is 0(N) makes recognition impractical. Good category retrieval is possible thanks to a proper organization of the metastable states into classes. This can be checked by direct evaluation of the number N(m, m1, m2) of one spin flip stable states with overlap m with a pattern in category 1, and overlaps m1 and m2 with classes 1 and 2, respectively, The technique to count these states was first developed for the SK model [14] and it was ap-
model with couplings given by eq. (3.6) and a generalized version [22] where the coefficient of the second term is taken as a new~parameter. In this case the total number of metastable states can be expressed as
—~
f°J] N
N
=
0
dA, tr~6(h, AS.) —
i=i —
=
Jdm dm1 dm2N(m, m1, m2),
(3.8)
where the bar denotes the average over the branched stochastic process used to select the memorized patterns. The main result is that the distribution N(m, m1, m2) is dominated by a saddle point where m2 = 0 but m1 # 0, in agreement with the analysis of the dynamics [24]. 3.3. Limit cycles The simplest recipe to memorize a cycle of length L is [7] 1 ~
L
a±1 a
E~
=
~j,
(3.9)
a =1
with ~çL±1) = ~(1) However this does not work because it does not allow the pattern (~) to stabilize before the system goes to the next { ~a ±1), This problem can be solved by introducing two different processes, one acting in a short time scale which takes care of stabilizing the current pattern and a second one
N. Parga
/
Neural networks as models of associative memories
acting in a longer time scale which is responsible for the transition [27]. The corresponding synapses are, j(S)
— —
1 ~ t(a)t(a)
13 10
N a~i~’ ~ A
(L) =
N
V’
\
a1
(a±1) ~ ~
1ob~
13 ‘-
‘
‘
~‘
a—i
where S and L stand for the short and long time scales respectively. The corresponding internal fields are
83
have been observed. However, the authors of ref. [28] argue that the observed values can be taken as a diluted system and since neural networks are robust under dilution both the theoretical model and the real system exhibit similar behaviours. The reason why the theoretical model is able to describe the CPG of the Tritonia Diomedea probably is the simplicity of this system. There is no reason to expect that a more complicated CPG could be explained by this simple model. A detailed study of the Tritoma as well as other organisms would be necessary to elucidate this point.
N
h~(t)
=
h~(t)
=
~
(3.lla)
j(S)~(~)
3,4. Asymmetry ~ i=l
dt’w(t
J.(.L)[f
—
t’)~1(t’)].
-
(3.llb) The dynamics is still like in eq. (2.2) with h. h~+ h~.The function ~ is a monotonously decreasing one with characteristic time T. The relative strength of the two components is chosen such that the transition to the new pattern {~(a+i)) occurs after the system stayed at { Eta) for a time of O(T). An interesting application of this model was proposed by D.K. Kleinfeld and H. Sompolinsky [28] who used it to generate the rhythmic swimming movements of the Tritonia Diomedea. Biological studies [16] of this mollusk have determined the structure of the group of neurons which generate the neural activity necessary to produce the right movements. This set of neurons is the central pattern generator (CPG) of the Tritonia Diomedea. The firing patterns that are produced by the four premotor interneurons while swimming are known [16]. Using these in formulae (3.10) Kleinfeld and Sompolinsky could predict a synaptic matrix with short and long time scale components which are in good agreement with the experimental ones. Most of the fast synaptic components predicted by the theory are observed and have the right signs. For the slow components the theory predicts more synapses than those which =
The symmetry of the synaptic connections adopted by the Hopfield model is far from being realized in a biological system. In particular if neuron i sends its axon to the dendrites of neuron j, the inverse is not necessarily true; a case usually called asymmetric dilution. The number of metastable states in easily calculated using the techniques of ref. [14] for several types of asymmetry [25]. Surprisingly the number of non-retrieval states is not appreciably altered with respect to the symmetric model. This, however, does not say everything about the dynamics since the size and shape of the attraction basins is also important. It was found in ref. [29] that in spite of the large number of metastable states, the time necessary to reach them grows exponentially with the size of the system. The dynamic behaviour is chaotic. Apart from the description of models with asymmetric synaptic matrixes, a cognitive role of asymmetry was suggested by G. Parisi [30]. When the dynamics reaches a fixed point indicating that something has been recognized there is nothing in the Hopfield model which guarantees that it is one of the memorized patterns. Spin glass states are always present and they may confuse the system, since there is no way to distinguish them from the retrieval states, Then asymmetry helps: since those spurious states are replaced by chaotic trajectories the problem is automatically avoided.
84
N. Parga
/ Neural networks as models of associative memories
4. Conclusions
[4] D.O. Hebb, The Organization of Behaviour (Wiley, New York, 1949).
Some of the most recent contributions of physics to neural networks have been reviewed in this work. Starting from a fully connected model where a mean field analysis and the application of methods developed to solve disordered magnetic systems is straightforward, physicists could later introduce the effect of pattern correlations, non-linearities, dilution and asymmetry. At the same time different types of neural networks have been considered, usually in connection with their application to specific tasks [8]. These are multilayer models where neurons on a layer determine the activity of those on the next one through a non-linear (sigmoid) function of the activity field. These networks have been applied with some success to explain features of language acquisition [8], early vision [31], the contiguity problem [32], etc. Most of these works have been motivated by the ability of the multilayer non-linear networks to recognize the relevant features of a task after a set of examples have been presented to them during a learning session. Learning by examples allows the machine to find algorithms to solve the task. Until this moment, however, physics had little influence in the analysis of these networks. Even the questions. physics would try to answer about them should be different from those addressed up to now. Instead of bothering about how to construct a network useful for a specific task one should study properties which hold for most of them by considering ensembles of problems.
[5] F. Rosemblatt, Principles Books, New York, 1962). of Neurodynamics (Spartan [6] M. Minsky and S. Pappert, Perceptrons (MIT Press, Cambridge, MA, 1969). [7] J.J. Hopfield, Proc. Nat. Acad. Sci. USA 79 (1982) 2554. [8] D.E. Rumeihart and J.L. McCleiland, Parallel Distributed Processing, Vol. 1 and 2 (Bradford Books, Cambridge, ~ 1986). [9] A.N. Kolmogorov, Dokl. Akad. Nauk. SSSR 114 (1957) 953 (in Russian). AMS translating 2 (1957) 55. [10] D.J. Amit, H. Gutfreund and H. Sompolinsky, Phys. Rev. A32 (1985) 1007; Phys. Rev. Lett. 55 (1985) 1530; Ann. Phys. 173 (1987) 30. [11] D. Shemngton and S. Kirkpatrick, Phys. Rev. Lett. 35 (1975) 1792. [121G. Parisi, Phys. Rev. Lett. 43 (1979) 1754; J. Phys. A13 (1980) L-115, 1101, 1887. [13] G. Parisi, Phys. Rev. Lett. 50 (1983) 1946. [14] A.J. Bray and M.A. Moore, J. Phys. C13 (1980) L-469. [15] E. Gardner, J. Phys. A19 (1986) L-1047. [16] P.A. Getting and MS. Dekin, in: Model Neural Networks and Behavior, A.I. Selverston, ed. (Plenum Press, New York, 1985). [17] G. Parisi, J. Phys. A19 (1986) L-617. [18] J.P. Nadal, G. Toulouse, J.-P. Changeux and S. Dehaene, Europhys. Lett. 1 (1986) 535. [19] M. Mézard, J.P. Nadal and G. Toulouse, J. Physique 47 (1986) 1457. [201M. Mézard, G. Parisi, N. Sourlas, G. Toulouse and M. Virasoro, J. Physique 45 (1984) 843. [21] N. Parga and M.A. Virasoro, J. Physique 47 (1986) 1857. [22] M.V. Feigelman and L.B. loffe, mt. J. of Mod. Phys. Bi (1987) 51. [23] W. Kinzel, Z. Phys. B60 (1985) 205.
References
[28] print D. Kleinfeld and H. Sompolinsky, AlT Bell Labs. pre(1987), submitted to J. Neurosci. [29] A. Crisanti and H. Sompolinsky, Jerusalem preprint RI/87/89. [30] G. Parisi, J. Phys. A19 (1986) L675. [31] R. Linsker, Proc. Nat. Acad. Sci. USA 83 (1986) 7508, 8779, 8390. [32] 5. Solla, S. Ryckebush and D. Schwartz, to be published.
[1] D.J. Amit, in: Proc. Heidelberg Colloquium on Glassy Dynamics, J. van Hemmen and I. Morgenstern, eds. (Springer, Berlin, 1987). [2] M. Mézard, G. Parisi and M.A. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987). [3] W.S. McCulloch and W.A. Pitts, Bull. Math. Biophys. 5 (1943) 115.
[24] S. Bacci, J. Alfaro, C. Wiecko and N. Parga, J. Physique, to appear (April 1989). [25] A. Treves and D.J. Aniit, Jerusalem preprint RJ/88/119. [26] S. Bacci, G. Mato and N. Parga, Bariloche preprint. [27] D. Kleinfeld, Proc. Nat. Acad. Sci. USA 83 (1986) 9469; H. Sompolinsky and I. Kanter, Phys. Rev. Lett. 57 (1986) 2861.