Perceptron learning on Hopfield net

PERCEPTRON LEARNING ON HOPFIELD NET, ~ . National University of Singapore, Kent Ridge, Singapore 0511

Institute of Systems Science,

Hopfield introduced a powerful neural network which can be used as associative memory. The operations which are performed on the network can be summarized in the following two equations: y = Tx x = fo(Y) (1) where x is the current state of the neurons, T is the connection matrix, 0 is the threshold vector. In general, cq. (1) is iterated until a stable pattern is reached. It can be shown that Bidirectional Associative Memory (BAM) and multi-layer neural network can be expressed as special cases of Hopfield net. In a BAM, the vectors x and y arc partitioned into x A and xB; YA and YBrespectively, so the first part of eq.(1) becomes:

f*l

° .......

This can be generalized to multi-directional associative memory (MAM) where the diagonal sub-matrices denote self-associations and the off-diagonal sub-roan'ices deno~ e r o s s - ~ n s ~ ~es. A multi-layer neural net with hidden layers can also be expressed as a Hopfletd net, if the neural state vector x is arranged such that x = [ 0 1H I I ]t, where t denotes vector transpose, I denotes input neurons, 0 denotes output neurons and H denotes neurons in the hidden layers, then the matrix T can be ~ g e d as:

0

....i 0 . . . .

1

. . . . .

With the vector x so arranged, the upper triangular matrix is the feedforward path and the lower ~ matrix is the feedback path (which is zero). Note that for each iterationofeq, (I) ~ ~ u p m e level. A learning algorithm based on Perceptron is studied, in this case, ~ m ' e M ~ palm'ns {xt, x 2, .,. x M} and M target patterns {cI, c2.....CM} (Typically c i = x i for all i). The ~ l~g algm,ithm will iteratively find a better T and 0 such that the total error between all the input and target p a ~ s is minimized. From any initialcondition of T and 0, and for each pattern xi, the algorithm will firstcompute the error pattern e, where e = c i - f0 ( T x i )

(2)

then the new matrix T' and threshold 0' are updated as follow: T' = T + ot e xit

0' = 0 - [5 e

(3)

where cx and ~ arc constants. The process is iterateduntilthe totalerror is min~zed, In this study, ~ cases of ~ learning are considered, depending on whether the input or the target pattern sets are completely specified or partially specified. Case 1: Full input and target patterns This is the simplest case. After training, it will recall the correct target pattern when a full ~ ~ is presented. However, if a partial input pattern is presented, it may not be ~ibleto recall ~ era'feet ~ ~ e target p a l m . Case 2: Partial input patterns and full target patterns In this case, each input pattern vector is partitioned into categories, i.e. xi is partitioned into xia, XiB..... etc. Training is then done on the expanded set. Using this m e ~ the neural net can recall the correct and complete target pattern when (the same) ~ input ~ is ~ Case 3: Multi-layer neural network t h F o r the mulfi,lay~ ~ n n e t ~ the ~ t ~ ci consists o f o ~ the o ~ ~ . : A s a ~suk, • error p ~ is only ~ y naea. Mowev~, m e ~ ~ matrix Tin. When the ~ uiangular ~ (the ~ ~ ) il ~ ~ demur: (the feedforward path), the missing ~ t s in the ~ ~ ca.q be pattern • as x in e q . ( 1 ) . A ~ one iteration, the ~ s are :m compute the new mau-ix aria thresholds. It can be shown that the above a l ~ m is e q ~ to@teBackPropagation Training algorithm.

I98