- Email: [email protected]
Combination of the assembly neural network with a perceptron for recognition of handwritten digits arranged in numeral strings
Pattern Recognition 38 (2005) 315 – 322 www.elsevier.com/locate/patcog
Combination of the assembly neural network with a perceptron for recognition of handwritten digits arranged in numeral strings Alexander Goltsev∗ , Dmitri Rachkovskij Cybernetics Center of National Academy of Sciences of Ukraine, Pr. Glushkova 40, Kiev 03680, Ukraine
Abstract The purpose of the paper is to design and test neural network structures and mechanisms for making use of the information that is contained in the character strings for more correct recognition of the characters constituting these strings. Two neural networks are considered in the paper; both networks are combined into a joint recognition system. The first is the assembly neural network and the second is the neural network of a perceptron type. A computer simulation of the system is performed. The combined system solves the task of recognition of handwritten digits of the MNIST test set provided that the digits have been arranged in the numeral strings memorized in the system. During a recognition process of an input numeral string, the assembly neural network executes intermediate recognition of the digits basing on which a perceptron type network accomplishes the final choice among the limited combinations of strings memorized in the network. The experiments have demonstrated that the combined system is able to make use of the information that is contained in the strings for more correct recognition of digits of the MNIST test set. In particular, the experiments have shown that the combined system commits no errors in the recognition of MNIST test set on the condition that the digits of this set had been organized in the strings of more than 5 digits each. 䉷 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. Keywords: Neural network; A perceptron-type network; The assembly neural network; Learning; Connection; Recognition
1. Introduction The present paper falls into the category of artificial neural networks. Although the artificial neural networks are being intensively studied for many years, most of them may be classified as different variations of the perceptron (e.g., [1–6]). There exists a large amount of neurophysiological data that may be interpreted so that an early stage of visual information processing in the brain is performed by neural networks with a perceptron-type structure. Other neurophysiological data may be elucidated in favor of Hebb’s hypothesis about structural and functional
∗ Corresponding author. Fax: +380 44 266 15 70.
E-mail address: [email protected] (A. Goltsev).
organization of associative fields of the brain in the form of neural assemblies [7–9]. According to Hebb, integration of a group of neurons into a neural assembly is done with the aim of joint, simultaneous and mutually supporting activation of all neurons of the group as an informational and functional unit of the network. A recognition system is presented in the paper which is a combination of a simple neural network of a perceptron type and the assembly neural network described in Refs. [10–15]. The purpose of the paper is to design and test neural network structures and mechanisms with the aid of which it would be possible to make use of the information that is contained in the character strings for more correct recognition of the characters constituting these strings. This task originates from the problem of recognition of handwritten words. Actually, it is the problem of recognition of words
0031-3203/$30.00 䉷 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.patcog.2004.09.001
316
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
that is considered in a simplified form in the present work in order to evade a rather difficult problem of segmentation of handwritten words into separate letters. For this purpose, the character strings are generated by means of joining up separate handwritten digits of MNIST database [16]. A computer simulation of the system is performed. The system is tested on a task of recognition of handwritten digits of the MNIST test set on the condition that these digits have been previously arranged in the numeral strings and memorized in the system. During a recognition process of an input numeral string, the assembly neural network performs intermediate recognition of the digits basing on which a perceptron-type network accomplishes the final choice among the limited combinations of strings memorized in the network. The experiments have shown that the combined system commits no errors in the recognition of MNIST test set provided that the digits of this set had been organized in the strings of more than 5 digits each.
2. Assembly neural network The assembly neural network is named so because it is based on Hebb’s hypothesis about cell assemblies in the brain. The assembly neural network is intended to solve the task of classification of objects of a predefined number of classes. The classes of objects may be of different types: texture classes, character classes, etc. The learning process taking place in the assembly neural network is of a supervised learning type. Therefore, some numbers of training samples of each class must be given for the network’s learning. The task of the network is to classify test samples of the same classes. In many learning machines the features arise from the learning process itself and their formation are performed in parallel with the machine learning. Recognition devices of other types need to be provided with some pre-determined set of algorithms for the extraction of features from input images. The assembly neural network falls into the latter category, i.e. it is necessary for the network functioning that some features should be extracted from every input image according to a pre-determined set of algorithms. The feature set extracted from the input image may produce more or less adequate descriptions of the recognized objects. Detailed descriptions of the assembly neural network, which is a part of the combined recognition system, have already been presented in Refs. [11,12,14,15]. Thus, only a sketch of the assembly neural network is given in the paper as its details are not necessary for the understanding of the combined recognition system.
learning and recognition, is transformed into activation of a corresponding set of neurons in the network; this set of neurons is named as a pattern of initial neural activity in the further consideration. The assembly neural network has the following preorganization: it is artificially divided into several subnetworks. Every sub-network represents one recognized class. All sub-networks of the network consist of the same number of neurons; this number includes all neurons that are necessary to represent all features that could be extracted from input data during learning. Also, each sub-network includes all connections between all its neurons. Similar pre-organizations of neural networks are considered in Refs. [17–19]. 2.2. A process of primary learning of the assembly neural network A neural assembly is a definite group of neurons; all neurons of this group are bound into an assembly by means of mutual excitatory connections that connect all of them with one another. All neural assemblies are formed during the network’s primary learning. A procedure of binding of a group of neurons into the assembly is a learning step, which is carried out according to Hebb’s learning rule. Each learning step results in an increment of weights of all connections between those neurons that have been activated in the network in order to be bound into the assembly. The activated neurons are a neural encoding of the feature set extracted from a current training sample; these neurons are named above as a pattern of initial neural activity. In the model described in the paper, the network’s connections have analog weights. In a learning step, the weights of all connections between the activated neurons are increased by a small equal value. In the process of primary learning, all samples of the available training set are successively presented to the network to be memorized in it as neural assemblies. This process has the following peculiarity. Each sub-network is trained with the samples of the same class so that all neural assemblies representing training samples of the same class are formed in the same sub-network. On completion of the primary learning stage, connections in every sub-network will be of different weights because the formed neural assemblies have intersections between one another. In other words, the neural assemblies of each subnetwork fuse into some assemblage of neural assemblies in which every assembly, being a constituent of the bound connection structure, becomes a part of the integral description of the corresponding class. 2.3. Recognition algorithm of the assembly neural network
2.1. A general description of the assembly neural network Each neuron of the network is associated with a definite feature that may be extracted from input data. A set of features, which is extracted from an input sample as for
A recognition process begins with the same procedure of feature extraction from a test sample. In the recognition process, the pattern of initial neural activity is set in all sub-networks of the network in parallel to create equal
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
Fig. 1. Activities of 10 R-neurons in a recognition process taking place in the experiments on recognition of Arabic digits of MNIST database by the assembly neural network. Height of each vertical bar shows the activity of the corresponding R-neuron. This recognition procedure results in classification of the input sample by the assembly neural network as the digit ‘5’.
opportunities for the sub-networks to win a competition between them. The recognition algorithm is very simple. The weights of all connections between those neurons that constitute the pattern of initial neural activity are summarized separately in each sub-network. The maximum sum defines the sub-network winner and this “winner-takes-all” procedure determines the class of the test sample presented for recognition. A set of special R-neurons is used for the recognition process. Activity of each R-neuron represents the summarized weights of connections within the corresponding sub-network that have been computed according to the above description. Fig. 1 depicts an example of activities of 10 such R-neurons that correspond to 10 sub-networks of the assembly neural network in a recognition process taking place in the experiments described below. The height of each vertical bar in the figure shows the activity of the appropriate R-neuron. It is seen in the figure that the subnetwork number 5 becomes the winner in this recognition procedure. 2.4. Generalization of features in the assembly neural network A distinctive peculiarity of the considered assembly neural network is that the network is partitioned into several subnetworks each of which represents one recognized class. It is worth to emphasize that in the present work it is an innate artificial architecture of the network, which is not reached
317
due to any learning process. The evident disadvantage of this architecture is the necessity of proportionally increasing the number of neurons, when increasing the number of classes that the network has to recognize. However, owing to the partition, the network acquires three important properties. The first is the ability of the network to form complicated, non-linearly separable decision regions in the feature space. The second is the possibility to increase the network’s recognition capability by means of a procedure of differentiation, which is considered in the next subsection. And the third property is the network’s ability to generalize the description of each class separately within the corresponding sub-network. Let us characterize shortly the latter property of the network. During the process of primary learning, the neural assemblies formed in each sub-network have intersections between one another. Owing to these intersections, some bound neural structure is generated in every sub-network, which becomes a description of the corresponding class. The process of formation of such descriptions in all sub-networks is interpreted as the phenomenon of generalization of features. First of all, the network generalizes those features and their combinations that occur most frequently in the training sets of the corresponding classes. Experimental study of the generalization phenomenon and comparison between two assembly neural networks with binary connections, one of which is partitioned into subnetworks and the other not partitioned into them, are described in Refs. [11,14].
2.5. A differentiation procedure in the assembly neural network A process of secondary learning follows the procedure of primary learning. Another name for the secondary learning process is that of differentiation. The aim of this procedure is to increase the recognition capability of the network. The early version of the differentiation procedure is described in Refs. [10,13]. In the process of differentiation, the assembly neural network model repeatedly considers all available training samples in turn. During each consideration, the model recognizes the current training sample on the condition that the recognition process is counteracted with some artificial outer influence and then, in dependence on the recognition result, modifies or does not modify the weights of some connections of two relevant sub-networks. Thus, the differentiation procedure is performed by means of tuning the connection weights within the sub-networks. The procedure makes the connection structure of every sub-network more specific, irregular and containing many inhibitory connections. Due to the differentiation procedure the descriptions of classes contained in each sub-network becomes more adequate. Experimental study of the differentiation process, which has been used in the present work, is described in Ref. [12].
318
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
3. Recognition of digits by the assembly neural network The assembly neural network has been tested in experiments on the task of recognition of separate handwritten Arabic digits of MNIST database [16]. This database contains 60 000 samples of handwritten digits in the training set and 10 000 ones in the test set. Fig. 2 shows the samples 3001–3100 of the MNIST test set. In the experiments, each digit of MNIST database was processed with a feature extraction procedure before inputting to the assembly neural network. Description of the used feature extraction algorithms is presented in Ref. [12]. Therefore, only concise information about these algorithms is given in the paper which is as follows. As is mentioned above, the feature set extracted from a digit is represented in the network by means of activation of a certain pattern of initial neural activity in the sub-networks. In the experiments, every sub-network of the assembly neural network is arranged in a two-dimensional neural matrix divided into 32 neural columns of 32 neurons each. The features are produced by means of transformation of pixels of the original digit with a series of operations. The feature set consists of two parts. The first half of the feature set is generated from somewhat like a dilated image of the digit and the other half from the digit image itself. The first half of the set is represented in the upper 16 neurons of all neural columns of the neural matrix. The remaining lower 16 neurons of each neural column serve for representation of the second half of the feature set. Thirty-two radial scan lines of different orientations are used for extraction of the feature set. The same scan line serves for extraction of both halves of the feature set that are represented in the same neural column. The feature set takes into account only those pixels of the digit image that are crossed with the scan lines.
Fig. 2. Samples of separate handwritten digits with numbers 3001–3100 within the MNIST test set.
Fig. 3. A. An intermediate transformation, which the handwritten digit ‘5’ undergoes during the extraction of the first half of the feature set. B. Representation of the whole feature set extracted from the digit ‘5’ as a pattern of initial neural activity in a sub-network of the assembly neural network. The sub-network consists of 32 neural columns of 32 neurons each. The activated neurons are depicted with black color.
Fig. 3 illustrates this description by the example of the digit ‘5’. Fig. 3A shows an intermediate transformation, which the original digit undergoes during the extraction of the first half of the feature set. Fig. 3B demonstrates the neural encoding of the extracted feature set in the whole. The assembly neural network was divided into 10 subnetworks according to the number of classes of Arabic digits. In all stages of the experiments described below, the whole training and test sets of MNIST database were used. The experiments consisted in the following. First of all, the process of the network primary learning was carried out on all 60 000 training samples of MNIST database which resulted in formation of the bound neural structures of intersecting neural assemblies in all 10 sub-networks. Then, the differentiation process was performed which resulted in an increment of the recognition capability of the assembly neural network by 37% (see Ref. [12] for details). The minimum number of errors achieved in these experiments was 207 errors among 10 000 sample of the MNIST test set. There is a table of recognition percentages in Ref. [16]. This table contains the recognition percentages of different recognition devices on MNIST database. Now, the maximum recognition percentage recorded in the table is 99.33%. The accuracy rate of 97.93% of the assembly neural network is still low in comparison with the top rates of the table.
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
However, it is worth to note that the most of the top recognition percentages of the table have been achieved with the aid of multiple expansion of the MNIST training set by means of multiplex artificial distortions of the digits. The use of this method considerably increases the training set size and, therefore, the recognition percentage. The result of 97.93% of the assembly neural network has been achieved without any artificial expansion of the training set size, which should be taken into account for correct appraising of this result.
4. A task statement for the combined recognition system Many people have very illegible handwriting. Often, letters of their handwritten words have so distorted shapes that these letters being considered separately cannot be recognized correctly even by their authors. However, the same letters can be easily recognized in the context of the whole words (and sentences). Such a fact evidently testifies that the use of additional information about plausible letter compositions of the words considerably increases a probability of correct recognition of the distorted letters. A simple neural network of a perceptron type is presented in the next section. This neural network is attached to the assembly neural network so that both networks become a combined recognition system. The combined system is intended to solve the task of recognition of handwritten characters on condition that the recognized characters are arranged in strings of several characters each and that the symbolic compositions of all such strings are previously available for the system’s learning. In fact, this is a model for recognition of handwritten words. In the present work, the word recognition task is considered in this simplified form in order to evade the rather difficult problem of segmentation of handwritten words into separate letters. For this purpose, each string, which is input to the system, is produced by means of joining up several separate handwritten digits of the MNIST test set. The combined system uses the information that is contained in the strings of digits for more correct recognition of these digits. A wide use of MNIST database for comparison between different recognition techniques makes evident the advantages, which the combined system obtains from additional information encoded in the numeral strings. The combined system was tested in experiments with the numeral strings, which were generated by the simplest method. The strings are formed according to a sequential order of the digits in the MNIST test set as follows. The first H digits of the set are integrated into the first string, the second string is made up from the next H digits of the set (without overlap between the first one) and the subsequent strings are produced in the same way till the end of the test set. The numeral strings of different sizes were used in the experiments. Therefore, the total number of the strings generated by this procedure varied in different stages of the
319
experiments. However, the same number of 10 000 digits of the MNIST test set was used in each measurement of the system’s performance. This peculiarity of the experiments gives an opportunity to compare the recognition results achieved by the combined system with the results demonstrated by the devices listed in the table of Ref. [16].
5. A perceptron-type neural network A perceptron-type neural network serves for memorization and recognition of the numeral strings. In the learning stage, a perceptron-type neural network memorizes a symbolic order of the numeral strings. Let us name, thereafter, this network a “symbolic” neural network. One “symbolic” neural network is used to memorize only the strings of the same size. The “symbolic” neural network has the following structure. The network consists of two neural layers and connections that are directed from neurons of the first layer to neurons of the second layer. All connections of the network are binary; initially all of them have zero-valued weights. Neurons of the first layer are arranged in two-dimensional matrix. Let the network be intended to memorize the strings of H symbols each. Then, the neural matrix consists of H neural columns of M neurons each, where M is the number of the used symbolic classes (size of alphabet). For the considered case of the MNIST digits, M = 10 and, therefore, all neural columns of the first network layer contain 10 neurons each. Let us denote a total number of neurons in the first layer by N, N = MH. Let all neurons of every column be numerated separately within the column. Then, the neurons of different columns with the same intra-column’s numbers represent the same symbols (digits) of the alphabet. Let the columns be numerated as well. The columns’ numbers represent the order of digits in the string. Every string of H symbols is encoded in the neural matrix of the first layer by activation of a neural pattern, which consists of H neurons, so that each neural column contains one activated neuron. The neuron, which represents the first symbol of the string, is activated in the first neural column, the neuron, which represents the second symbol of the string, is activated in the second column and so on. Since every neural column represents all classes of digits, any string of H digits is unambiguously represented in the neural matrix by a unique pattern of activated neurons. Fig. 4 illustrates this description. It depicts the neural structure of the first layer of the “symbolic” neural network intended for memorization of the numeral strings of 6 digits. A procedure of memorization of a string is performed as follows. The memorized string is transformed into a certain pattern of a neural activity in the first layer. One neuron is allotted in the second layer to memorize this string. All connections that are directed from the activated neural pattern of the first layer to the allotted neuron of the second layer are set one-valued. Each neuron of the second layer
320
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
Fig. 4. The neural structure of the first layer of the “symbolic” neural network, which is intended for memorization and recognition of numeral strings of 6 digits.
represents one string. Thus, after memorization of J strings, J neurons arise in the second layer of the “symbolic” neural network and a multitude of one-valued connections is established between both network layers. Let us introduce a j two-dimensional binary matrix Wi for designation of these connections. In this matrix the subscript i indicates neurons of the first layer (i = 1, 2, 3, . . . , N) and the superscript j indicates neurons of the second layer (j = 1, 2, 3, . . . , J ). For this consideration, let the neurons of the first layer be numerated uninterruptedly all over the matrix. As it is mentioned above, the strings of different sizes were generated from digits of the MNIST test set to be used in the experiments described below. In the learning stage, all numerical strings of the same size were memorized in the appropriate “symbolic” neural network according to the above algorithm.
6. Functioning of the combined recognition system In a process of recognition, some sequence of H digit images of the MNIST test set is presented to the combined system. The combined system should recognize the digits that are drawn in all images of the sequence, provided that the corresponding numerical string has been previously memorized in the “symbolic” neural network among others. The combined system solves the task by means of recognition of the string, i.e. by means of choosing one neuron of the second layer of the “symbolic” neural network. The assembly neural network is the first constituent of the combined recognition system. The procedure of the network’s primary learning was previously performed on all
60 000 training samples of MNIST database which resulted in the formation of integral descriptions of the corresponding digits’ classes in all 10 sub-networks of the assembly neural network. Then, the recognition capability of the network was improved by means of implementation of the differentiation procedure. After that, the assembly neural network achieved 97.9% of correct recognition of the MNIST test set. The “symbolic” neural network is the second constituent of the combined recognition system. The procedure of the network’s learning was previously performed on all numeral strings of a certain size produced from 10 000 samples of the MNIST test set. This procedure resulted in memorization of all these numeral strings in the structure of the “symbolic” neural network. The combined system recognizes each sequence of H images as follows. Every image of the recognized sequence is processed according to the feature extraction algorithm. Then, the assembly neural network executes the process of recognition of the first input image of the sequence but does not accomplish the final operation of the recognition process, i.e. the network does not classify the digit drawn in the image. Instead of that, the activities of all 10 R-neurons of the assembly neural network are transmitted to the corresponding 10 neurons of the first column of the first layer of the “symbolic” neural network. These neurons retain their levels of activity during the whole process of recognition of the considered sequence. Then, the second image of the sequence is presented to the assembly neural network. Again, the assembly neural network performs almost all operations on recognition of this image that result in the appearance of some new pattern of activity of all R-neurons of the network. This new pattern of activity of R-neurons is transferred to the second neural column of the first layer of the “symbolic” neural network. After this successive analysis of all H images of the sequence by the assembly neural network is completed, every neuron of the first layer of the “symbolic” neural network has got some level of activity. These activity levels characterize probability of presence of the digits in the corresponding places of the numeral string. In the final stage of the string recognition process, the neural activity formed in the first layer of the “symbolic” neural network is distributed through the structure of onevalued connections directed to the neurons of the second network layer. This results in the formation of some pattern of neural activity in the second layer of the “symbolic” neural network. Let us represent the activity of all neurons of the second layer by analog vector Aj ; j = 1, 2, 3,…, J. Then, the activity of the j th neuron of the second layer may be expressed by the formula Aj =
N i=1
j
Ri W i ,
(1)
where vector R denotes activities of all N neurons of the first layer.
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
Fig. 5. Dependence of recognition percentage of the combined recognition system upon a number of digits contained in the strings. The string’s size is plotted on the X-axis. The percentages of recognition errors are plotted on the Y-axis.
The neuron with the maximum activity is found among the neurons of the second layer of the “symbolic” neural network. This neuron is a representative of a certain string. Evidently, this choice unambiguously determines the symbolic composition of the string presented for recognition. The strings of different sizes were used in the experiments. Fig. 5 shows in graphical form the percentages of errors committed by the combined recognition system during recognition of all images of the MNIST test set in dependence on the string’s size. As it follows from this figure, the error percentage decreases together with increment of the string’s size and reaches zero level when the size becomes equal to 6. It is evident that the longer strings are recognized with no errors as well. Fig. 5 demonstrates how much additional information can be elicited from organization of characters in the form of strings to improve the recognition of these characters. The more the string’s size, the more the amount of information that stores in the string. Also, the experiments have shown that the presented structure of the “symbolic” neural network can successfully solve the task of making use of the information that is contained in the character strings.
7. Discussion and conclusions Two neural networks combined into a joint recognition system are considered in the paper. The system solves the task of recognition of handwritten digits of the MNIST test
321
set that are arranged in the numeral strings. The first constituent of the combined system is the assembly neural network, which performs the task of preliminary recognition of the digits. The second constituent is the “symbolic” neural network of a perceptron type, whose task is to discover a symbolic composition of the numeral string containing the recognized digits. During recognition of every string, the assembly neural network provides the “symbolic” neural network with its intermediate recognition results for making the final procedure of classification. It is evinced in the paper that the combined system is able to make use of the information that is contained in the strings of separate characters for more correct recognition of these characters. The experiments have shown that the combined system commits no errors in recognition of all separate digits of the MNIST test set, provided that these digits are organized in the numeral strings of more than 5 symbols each. This result apparently demonstrates how much information is stored in organization of characters in the form of strings for the task of recognition of these characters. And this information is effectively elicited from the strings by means of a simple device, which is the “symbolic” neural network. It is evident from the above description that the assembly neural network can be substituted in the combined recognition system for any other recognition device. At the same time, the “symbolic” neural network is a necessary component of the system. A need in the combined recognition system arises in case if any recognition technique cannot solve the task of recognition of separate characters quite correctly. The most obvious example of such a task is recognition of handwritten words and sentences. As is mentioned above, the combined recognition system is a somewhat simplified model of the recognition of handwritten words. The term “simplified model” means that the system does not contain mechanisms for segmentation of handwritten words into separate letters. Nevertheless, the combined system could serve as a useful example and a prototype for designing neural network mechanisms for recognition of handwritten words in the future.
References [1] F. Rosenblatt, Principles of Neurodynamics. Perceptrons and Theory of Brain Mechanisms, Spartan Books, Washington DC, 1962. [2] G.D. Magoulas, M.N. Vrahatis, G.S. Androulakis, Effective backpropagation training with variable stepsize, Neural Networks 10 (1997) 69–82. [3] G.A. Carpenter, B.L. Milenova, B.W. Noeske, Distributed ARTMAP: a neural network for fast distributed supervised learning, Neural Networks 11 (1998) 793–813. [4] E. Kussul, T. Baidyk, L. Kasatkina, V. Lukovich, Rosenblatt perceptrons for handwritten digit recognition, Proceedings of IJCNN’01, Washington DC, USA, 2001, pp. 1516–1521.
322
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
[5] E. Kussul, T. Baidyk, Permutative coding technique for handwritten digit recognition system, Proceedings of IJCNN’03, Portland, OR, USA, 2003, pp. 2163–2168. [6] K. Fukushima, Neocognitron: a model for visual recognition, in: M.A. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, second ed., MIT Press, Cambridge, MA, London, England, 2003, pp. 715–719. [7] D.O. Hebb, The Organization of Behavior, Wiley, New York, 1949. [8] P.M. Milner, The cell assembly: mark 2, Psychol. Rev. 64 (1957) 242–252. [9] P.M. Milner, Physiological Psychology, Holt, Rinehart & Winston Inc., McGill University, 1970. [10] A. Goltsev, An assembly neural network for texture segmentation, Neural Networks 9 (4) (1996) 643–653. [11] A. Goltsev, A process of generalization in the assembly neural network, Proceedings of ICONIP’01, Shanghai, China, 2001, pp. 47–52. [12] A. Goltsev, Secondary learning in the assembly neural network, Neurocomputing (2004), accepted for publication.
[13] A. Goltsev, D.C. Wunsch, Inhibitory connections in the assembly neural network for texture segmentation, Neural Networks 11 (5) (1998) 951–962. [14] A. Goltsev, D.C. Wunsch, Generalization of features in the assembly neural networks, Int. J. Neural Systems (IJNS) 14 (1) (2004) 39–56. [15] A. Goltsev, D. Húsek, Some properties of the assembly neural networks, Neural Network World 12 (1) (2002) 15–32. [16] MNIST database http://yann.lecun.com/exdb/mnist/ [17] J.-D. Gascuel, B. Moobed, M. Weinfeld, An internal mechanism for detecting parasite attractors in a Hopfield network, Neural Comput. 6 (1994) 902–915. [18] H. Schwenk, M. Milgram, Structured diabolo-networks for hand-written character recognition, Proceedings of the International Conference on Artificial Neural Networks, vol. 2, Sorrento, Italy, 1994, pp. 985–988. [19] A.A. Ezhov, V.L. Vvedensky, Object generation with neural networks (when spurious memories are useful), Neural Networks 9 (9) (1996) 1491–1495.
About the Author—ALEXANDER GOLTSEV received the M.Sc. degree (with honors) in Physics and Electronics (Bionics) from the Dnepropetrovsk State University (Dnepropetrovsk, Ukraine) in 1971. He obtained the Ph.D. degree in Engineering from V.M. Glushkov Institute of Cybernetics of Ukrainian Academy of Sciences (Kiev, Ukraine) in 1976. From 1971 he has been working at the same Institute of Cybernetics (which is V.M. Glushkov Cybernetics Center now) holding the position of senior researcher at the Department of Neural Information Processing Systems of International Research and Training Centre of Information Technologies (which is a part V.M. Glushkov Cybernetics Center). He temporarily worked at Linz University (Linz, Austria, 1993–1994) and at German National Research Center for Information Technology (Sankt Augustin, Germany, 2001). His research interests are neural networks, image processing, computer vision, and robotics. The purposes of the current research work are to improve algorithms and structures of assembly neural networks and to construct a neural network of unsupervised learning type for texture segmentation. His whole list of publications consists of 64 items. About the Author—DMITRI A. RACHKOVSKIJ received the MS degree in radiophysics and electronics from Rostov State University, Rostov-on-Don, Russia, in 1983, and the Ph.D. degree from the V.M. Glushkov Cybernetics Center, Kiev, Ukraine, in 1990. From 1983 to 1987, he was an R&D engineer in computer-based systems at Rostov Special R&D Bureau. Since 1987, he worked under the supervision of Academician N.M. Amosov, who since the early 1960s has led one of the most interesting research programs in AI, and E.M. Kussul, who made a major contribution to that program by inventing the paradigm of Associative-Projective Neural Networks (APNNs). Currently, Dr. Rachkovskij is a senior researcher at the Cybernetics Center. His research is mainly connected with the schemes for distributed representations of various data types, their applications in the areas of classification, information retrieval, analogical reasoning, and comparison to stateof-the-art approaches. He (co)authored about 40 publications. E-mail: [email protected].
Combination of the assembly neural network with a perceptron for recognition of handwritten digits arranged in numeral strings Alexander Goltsev∗ , Dmitri Rachkovskij Cybernetics Center of National Academy of Sciences of Ukraine, Pr. Glushkova 40, Kiev 03680, Ukraine
Abstract The purpose of the paper is to design and test neural network structures and mechanisms for making use of the information that is contained in the character strings for more correct recognition of the characters constituting these strings. Two neural networks are considered in the paper; both networks are combined into a joint recognition system. The first is the assembly neural network and the second is the neural network of a perceptron type. A computer simulation of the system is performed. The combined system solves the task of recognition of handwritten digits of the MNIST test set provided that the digits have been arranged in the numeral strings memorized in the system. During a recognition process of an input numeral string, the assembly neural network executes intermediate recognition of the digits basing on which a perceptron type network accomplishes the final choice among the limited combinations of strings memorized in the network. The experiments have demonstrated that the combined system is able to make use of the information that is contained in the strings for more correct recognition of digits of the MNIST test set. In particular, the experiments have shown that the combined system commits no errors in the recognition of MNIST test set on the condition that the digits of this set had been organized in the strings of more than 5 digits each. 䉷 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. Keywords: Neural network; A perceptron-type network; The assembly neural network; Learning; Connection; Recognition
1. Introduction The present paper falls into the category of artificial neural networks. Although the artificial neural networks are being intensively studied for many years, most of them may be classified as different variations of the perceptron (e.g., [1–6]). There exists a large amount of neurophysiological data that may be interpreted so that an early stage of visual information processing in the brain is performed by neural networks with a perceptron-type structure. Other neurophysiological data may be elucidated in favor of Hebb’s hypothesis about structural and functional
∗ Corresponding author. Fax: +380 44 266 15 70.
E-mail address: [email protected] (A. Goltsev).
organization of associative fields of the brain in the form of neural assemblies [7–9]. According to Hebb, integration of a group of neurons into a neural assembly is done with the aim of joint, simultaneous and mutually supporting activation of all neurons of the group as an informational and functional unit of the network. A recognition system is presented in the paper which is a combination of a simple neural network of a perceptron type and the assembly neural network described in Refs. [10–15]. The purpose of the paper is to design and test neural network structures and mechanisms with the aid of which it would be possible to make use of the information that is contained in the character strings for more correct recognition of the characters constituting these strings. This task originates from the problem of recognition of handwritten words. Actually, it is the problem of recognition of words
0031-3203/$30.00 䉷 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.patcog.2004.09.001
316
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
that is considered in a simplified form in the present work in order to evade a rather difficult problem of segmentation of handwritten words into separate letters. For this purpose, the character strings are generated by means of joining up separate handwritten digits of MNIST database [16]. A computer simulation of the system is performed. The system is tested on a task of recognition of handwritten digits of the MNIST test set on the condition that these digits have been previously arranged in the numeral strings and memorized in the system. During a recognition process of an input numeral string, the assembly neural network performs intermediate recognition of the digits basing on which a perceptron-type network accomplishes the final choice among the limited combinations of strings memorized in the network. The experiments have shown that the combined system commits no errors in the recognition of MNIST test set provided that the digits of this set had been organized in the strings of more than 5 digits each.
2. Assembly neural network The assembly neural network is named so because it is based on Hebb’s hypothesis about cell assemblies in the brain. The assembly neural network is intended to solve the task of classification of objects of a predefined number of classes. The classes of objects may be of different types: texture classes, character classes, etc. The learning process taking place in the assembly neural network is of a supervised learning type. Therefore, some numbers of training samples of each class must be given for the network’s learning. The task of the network is to classify test samples of the same classes. In many learning machines the features arise from the learning process itself and their formation are performed in parallel with the machine learning. Recognition devices of other types need to be provided with some pre-determined set of algorithms for the extraction of features from input images. The assembly neural network falls into the latter category, i.e. it is necessary for the network functioning that some features should be extracted from every input image according to a pre-determined set of algorithms. The feature set extracted from the input image may produce more or less adequate descriptions of the recognized objects. Detailed descriptions of the assembly neural network, which is a part of the combined recognition system, have already been presented in Refs. [11,12,14,15]. Thus, only a sketch of the assembly neural network is given in the paper as its details are not necessary for the understanding of the combined recognition system.
learning and recognition, is transformed into activation of a corresponding set of neurons in the network; this set of neurons is named as a pattern of initial neural activity in the further consideration. The assembly neural network has the following preorganization: it is artificially divided into several subnetworks. Every sub-network represents one recognized class. All sub-networks of the network consist of the same number of neurons; this number includes all neurons that are necessary to represent all features that could be extracted from input data during learning. Also, each sub-network includes all connections between all its neurons. Similar pre-organizations of neural networks are considered in Refs. [17–19]. 2.2. A process of primary learning of the assembly neural network A neural assembly is a definite group of neurons; all neurons of this group are bound into an assembly by means of mutual excitatory connections that connect all of them with one another. All neural assemblies are formed during the network’s primary learning. A procedure of binding of a group of neurons into the assembly is a learning step, which is carried out according to Hebb’s learning rule. Each learning step results in an increment of weights of all connections between those neurons that have been activated in the network in order to be bound into the assembly. The activated neurons are a neural encoding of the feature set extracted from a current training sample; these neurons are named above as a pattern of initial neural activity. In the model described in the paper, the network’s connections have analog weights. In a learning step, the weights of all connections between the activated neurons are increased by a small equal value. In the process of primary learning, all samples of the available training set are successively presented to the network to be memorized in it as neural assemblies. This process has the following peculiarity. Each sub-network is trained with the samples of the same class so that all neural assemblies representing training samples of the same class are formed in the same sub-network. On completion of the primary learning stage, connections in every sub-network will be of different weights because the formed neural assemblies have intersections between one another. In other words, the neural assemblies of each subnetwork fuse into some assemblage of neural assemblies in which every assembly, being a constituent of the bound connection structure, becomes a part of the integral description of the corresponding class. 2.3. Recognition algorithm of the assembly neural network
2.1. A general description of the assembly neural network Each neuron of the network is associated with a definite feature that may be extracted from input data. A set of features, which is extracted from an input sample as for
A recognition process begins with the same procedure of feature extraction from a test sample. In the recognition process, the pattern of initial neural activity is set in all sub-networks of the network in parallel to create equal
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
Fig. 1. Activities of 10 R-neurons in a recognition process taking place in the experiments on recognition of Arabic digits of MNIST database by the assembly neural network. Height of each vertical bar shows the activity of the corresponding R-neuron. This recognition procedure results in classification of the input sample by the assembly neural network as the digit ‘5’.
opportunities for the sub-networks to win a competition between them. The recognition algorithm is very simple. The weights of all connections between those neurons that constitute the pattern of initial neural activity are summarized separately in each sub-network. The maximum sum defines the sub-network winner and this “winner-takes-all” procedure determines the class of the test sample presented for recognition. A set of special R-neurons is used for the recognition process. Activity of each R-neuron represents the summarized weights of connections within the corresponding sub-network that have been computed according to the above description. Fig. 1 depicts an example of activities of 10 such R-neurons that correspond to 10 sub-networks of the assembly neural network in a recognition process taking place in the experiments described below. The height of each vertical bar in the figure shows the activity of the appropriate R-neuron. It is seen in the figure that the subnetwork number 5 becomes the winner in this recognition procedure. 2.4. Generalization of features in the assembly neural network A distinctive peculiarity of the considered assembly neural network is that the network is partitioned into several subnetworks each of which represents one recognized class. It is worth to emphasize that in the present work it is an innate artificial architecture of the network, which is not reached
317
due to any learning process. The evident disadvantage of this architecture is the necessity of proportionally increasing the number of neurons, when increasing the number of classes that the network has to recognize. However, owing to the partition, the network acquires three important properties. The first is the ability of the network to form complicated, non-linearly separable decision regions in the feature space. The second is the possibility to increase the network’s recognition capability by means of a procedure of differentiation, which is considered in the next subsection. And the third property is the network’s ability to generalize the description of each class separately within the corresponding sub-network. Let us characterize shortly the latter property of the network. During the process of primary learning, the neural assemblies formed in each sub-network have intersections between one another. Owing to these intersections, some bound neural structure is generated in every sub-network, which becomes a description of the corresponding class. The process of formation of such descriptions in all sub-networks is interpreted as the phenomenon of generalization of features. First of all, the network generalizes those features and their combinations that occur most frequently in the training sets of the corresponding classes. Experimental study of the generalization phenomenon and comparison between two assembly neural networks with binary connections, one of which is partitioned into subnetworks and the other not partitioned into them, are described in Refs. [11,14].
2.5. A differentiation procedure in the assembly neural network A process of secondary learning follows the procedure of primary learning. Another name for the secondary learning process is that of differentiation. The aim of this procedure is to increase the recognition capability of the network. The early version of the differentiation procedure is described in Refs. [10,13]. In the process of differentiation, the assembly neural network model repeatedly considers all available training samples in turn. During each consideration, the model recognizes the current training sample on the condition that the recognition process is counteracted with some artificial outer influence and then, in dependence on the recognition result, modifies or does not modify the weights of some connections of two relevant sub-networks. Thus, the differentiation procedure is performed by means of tuning the connection weights within the sub-networks. The procedure makes the connection structure of every sub-network more specific, irregular and containing many inhibitory connections. Due to the differentiation procedure the descriptions of classes contained in each sub-network becomes more adequate. Experimental study of the differentiation process, which has been used in the present work, is described in Ref. [12].
318
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
3. Recognition of digits by the assembly neural network The assembly neural network has been tested in experiments on the task of recognition of separate handwritten Arabic digits of MNIST database [16]. This database contains 60 000 samples of handwritten digits in the training set and 10 000 ones in the test set. Fig. 2 shows the samples 3001–3100 of the MNIST test set. In the experiments, each digit of MNIST database was processed with a feature extraction procedure before inputting to the assembly neural network. Description of the used feature extraction algorithms is presented in Ref. [12]. Therefore, only concise information about these algorithms is given in the paper which is as follows. As is mentioned above, the feature set extracted from a digit is represented in the network by means of activation of a certain pattern of initial neural activity in the sub-networks. In the experiments, every sub-network of the assembly neural network is arranged in a two-dimensional neural matrix divided into 32 neural columns of 32 neurons each. The features are produced by means of transformation of pixels of the original digit with a series of operations. The feature set consists of two parts. The first half of the feature set is generated from somewhat like a dilated image of the digit and the other half from the digit image itself. The first half of the set is represented in the upper 16 neurons of all neural columns of the neural matrix. The remaining lower 16 neurons of each neural column serve for representation of the second half of the feature set. Thirty-two radial scan lines of different orientations are used for extraction of the feature set. The same scan line serves for extraction of both halves of the feature set that are represented in the same neural column. The feature set takes into account only those pixels of the digit image that are crossed with the scan lines.
Fig. 2. Samples of separate handwritten digits with numbers 3001–3100 within the MNIST test set.
Fig. 3. A. An intermediate transformation, which the handwritten digit ‘5’ undergoes during the extraction of the first half of the feature set. B. Representation of the whole feature set extracted from the digit ‘5’ as a pattern of initial neural activity in a sub-network of the assembly neural network. The sub-network consists of 32 neural columns of 32 neurons each. The activated neurons are depicted with black color.
Fig. 3 illustrates this description by the example of the digit ‘5’. Fig. 3A shows an intermediate transformation, which the original digit undergoes during the extraction of the first half of the feature set. Fig. 3B demonstrates the neural encoding of the extracted feature set in the whole. The assembly neural network was divided into 10 subnetworks according to the number of classes of Arabic digits. In all stages of the experiments described below, the whole training and test sets of MNIST database were used. The experiments consisted in the following. First of all, the process of the network primary learning was carried out on all 60 000 training samples of MNIST database which resulted in formation of the bound neural structures of intersecting neural assemblies in all 10 sub-networks. Then, the differentiation process was performed which resulted in an increment of the recognition capability of the assembly neural network by 37% (see Ref. [12] for details). The minimum number of errors achieved in these experiments was 207 errors among 10 000 sample of the MNIST test set. There is a table of recognition percentages in Ref. [16]. This table contains the recognition percentages of different recognition devices on MNIST database. Now, the maximum recognition percentage recorded in the table is 99.33%. The accuracy rate of 97.93% of the assembly neural network is still low in comparison with the top rates of the table.
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
However, it is worth to note that the most of the top recognition percentages of the table have been achieved with the aid of multiple expansion of the MNIST training set by means of multiplex artificial distortions of the digits. The use of this method considerably increases the training set size and, therefore, the recognition percentage. The result of 97.93% of the assembly neural network has been achieved without any artificial expansion of the training set size, which should be taken into account for correct appraising of this result.
4. A task statement for the combined recognition system Many people have very illegible handwriting. Often, letters of their handwritten words have so distorted shapes that these letters being considered separately cannot be recognized correctly even by their authors. However, the same letters can be easily recognized in the context of the whole words (and sentences). Such a fact evidently testifies that the use of additional information about plausible letter compositions of the words considerably increases a probability of correct recognition of the distorted letters. A simple neural network of a perceptron type is presented in the next section. This neural network is attached to the assembly neural network so that both networks become a combined recognition system. The combined system is intended to solve the task of recognition of handwritten characters on condition that the recognized characters are arranged in strings of several characters each and that the symbolic compositions of all such strings are previously available for the system’s learning. In fact, this is a model for recognition of handwritten words. In the present work, the word recognition task is considered in this simplified form in order to evade the rather difficult problem of segmentation of handwritten words into separate letters. For this purpose, each string, which is input to the system, is produced by means of joining up several separate handwritten digits of the MNIST test set. The combined system uses the information that is contained in the strings of digits for more correct recognition of these digits. A wide use of MNIST database for comparison between different recognition techniques makes evident the advantages, which the combined system obtains from additional information encoded in the numeral strings. The combined system was tested in experiments with the numeral strings, which were generated by the simplest method. The strings are formed according to a sequential order of the digits in the MNIST test set as follows. The first H digits of the set are integrated into the first string, the second string is made up from the next H digits of the set (without overlap between the first one) and the subsequent strings are produced in the same way till the end of the test set. The numeral strings of different sizes were used in the experiments. Therefore, the total number of the strings generated by this procedure varied in different stages of the
319
experiments. However, the same number of 10 000 digits of the MNIST test set was used in each measurement of the system’s performance. This peculiarity of the experiments gives an opportunity to compare the recognition results achieved by the combined system with the results demonstrated by the devices listed in the table of Ref. [16].
5. A perceptron-type neural network A perceptron-type neural network serves for memorization and recognition of the numeral strings. In the learning stage, a perceptron-type neural network memorizes a symbolic order of the numeral strings. Let us name, thereafter, this network a “symbolic” neural network. One “symbolic” neural network is used to memorize only the strings of the same size. The “symbolic” neural network has the following structure. The network consists of two neural layers and connections that are directed from neurons of the first layer to neurons of the second layer. All connections of the network are binary; initially all of them have zero-valued weights. Neurons of the first layer are arranged in two-dimensional matrix. Let the network be intended to memorize the strings of H symbols each. Then, the neural matrix consists of H neural columns of M neurons each, where M is the number of the used symbolic classes (size of alphabet). For the considered case of the MNIST digits, M = 10 and, therefore, all neural columns of the first network layer contain 10 neurons each. Let us denote a total number of neurons in the first layer by N, N = MH. Let all neurons of every column be numerated separately within the column. Then, the neurons of different columns with the same intra-column’s numbers represent the same symbols (digits) of the alphabet. Let the columns be numerated as well. The columns’ numbers represent the order of digits in the string. Every string of H symbols is encoded in the neural matrix of the first layer by activation of a neural pattern, which consists of H neurons, so that each neural column contains one activated neuron. The neuron, which represents the first symbol of the string, is activated in the first neural column, the neuron, which represents the second symbol of the string, is activated in the second column and so on. Since every neural column represents all classes of digits, any string of H digits is unambiguously represented in the neural matrix by a unique pattern of activated neurons. Fig. 4 illustrates this description. It depicts the neural structure of the first layer of the “symbolic” neural network intended for memorization of the numeral strings of 6 digits. A procedure of memorization of a string is performed as follows. The memorized string is transformed into a certain pattern of a neural activity in the first layer. One neuron is allotted in the second layer to memorize this string. All connections that are directed from the activated neural pattern of the first layer to the allotted neuron of the second layer are set one-valued. Each neuron of the second layer
320
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
Fig. 4. The neural structure of the first layer of the “symbolic” neural network, which is intended for memorization and recognition of numeral strings of 6 digits.
represents one string. Thus, after memorization of J strings, J neurons arise in the second layer of the “symbolic” neural network and a multitude of one-valued connections is established between both network layers. Let us introduce a j two-dimensional binary matrix Wi for designation of these connections. In this matrix the subscript i indicates neurons of the first layer (i = 1, 2, 3, . . . , N) and the superscript j indicates neurons of the second layer (j = 1, 2, 3, . . . , J ). For this consideration, let the neurons of the first layer be numerated uninterruptedly all over the matrix. As it is mentioned above, the strings of different sizes were generated from digits of the MNIST test set to be used in the experiments described below. In the learning stage, all numerical strings of the same size were memorized in the appropriate “symbolic” neural network according to the above algorithm.
6. Functioning of the combined recognition system In a process of recognition, some sequence of H digit images of the MNIST test set is presented to the combined system. The combined system should recognize the digits that are drawn in all images of the sequence, provided that the corresponding numerical string has been previously memorized in the “symbolic” neural network among others. The combined system solves the task by means of recognition of the string, i.e. by means of choosing one neuron of the second layer of the “symbolic” neural network. The assembly neural network is the first constituent of the combined recognition system. The procedure of the network’s primary learning was previously performed on all
60 000 training samples of MNIST database which resulted in the formation of integral descriptions of the corresponding digits’ classes in all 10 sub-networks of the assembly neural network. Then, the recognition capability of the network was improved by means of implementation of the differentiation procedure. After that, the assembly neural network achieved 97.9% of correct recognition of the MNIST test set. The “symbolic” neural network is the second constituent of the combined recognition system. The procedure of the network’s learning was previously performed on all numeral strings of a certain size produced from 10 000 samples of the MNIST test set. This procedure resulted in memorization of all these numeral strings in the structure of the “symbolic” neural network. The combined system recognizes each sequence of H images as follows. Every image of the recognized sequence is processed according to the feature extraction algorithm. Then, the assembly neural network executes the process of recognition of the first input image of the sequence but does not accomplish the final operation of the recognition process, i.e. the network does not classify the digit drawn in the image. Instead of that, the activities of all 10 R-neurons of the assembly neural network are transmitted to the corresponding 10 neurons of the first column of the first layer of the “symbolic” neural network. These neurons retain their levels of activity during the whole process of recognition of the considered sequence. Then, the second image of the sequence is presented to the assembly neural network. Again, the assembly neural network performs almost all operations on recognition of this image that result in the appearance of some new pattern of activity of all R-neurons of the network. This new pattern of activity of R-neurons is transferred to the second neural column of the first layer of the “symbolic” neural network. After this successive analysis of all H images of the sequence by the assembly neural network is completed, every neuron of the first layer of the “symbolic” neural network has got some level of activity. These activity levels characterize probability of presence of the digits in the corresponding places of the numeral string. In the final stage of the string recognition process, the neural activity formed in the first layer of the “symbolic” neural network is distributed through the structure of onevalued connections directed to the neurons of the second network layer. This results in the formation of some pattern of neural activity in the second layer of the “symbolic” neural network. Let us represent the activity of all neurons of the second layer by analog vector Aj ; j = 1, 2, 3,…, J. Then, the activity of the j th neuron of the second layer may be expressed by the formula Aj =
N i=1
j
Ri W i ,
(1)
where vector R denotes activities of all N neurons of the first layer.
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
Fig. 5. Dependence of recognition percentage of the combined recognition system upon a number of digits contained in the strings. The string’s size is plotted on the X-axis. The percentages of recognition errors are plotted on the Y-axis.
The neuron with the maximum activity is found among the neurons of the second layer of the “symbolic” neural network. This neuron is a representative of a certain string. Evidently, this choice unambiguously determines the symbolic composition of the string presented for recognition. The strings of different sizes were used in the experiments. Fig. 5 shows in graphical form the percentages of errors committed by the combined recognition system during recognition of all images of the MNIST test set in dependence on the string’s size. As it follows from this figure, the error percentage decreases together with increment of the string’s size and reaches zero level when the size becomes equal to 6. It is evident that the longer strings are recognized with no errors as well. Fig. 5 demonstrates how much additional information can be elicited from organization of characters in the form of strings to improve the recognition of these characters. The more the string’s size, the more the amount of information that stores in the string. Also, the experiments have shown that the presented structure of the “symbolic” neural network can successfully solve the task of making use of the information that is contained in the character strings.
7. Discussion and conclusions Two neural networks combined into a joint recognition system are considered in the paper. The system solves the task of recognition of handwritten digits of the MNIST test
321
set that are arranged in the numeral strings. The first constituent of the combined system is the assembly neural network, which performs the task of preliminary recognition of the digits. The second constituent is the “symbolic” neural network of a perceptron type, whose task is to discover a symbolic composition of the numeral string containing the recognized digits. During recognition of every string, the assembly neural network provides the “symbolic” neural network with its intermediate recognition results for making the final procedure of classification. It is evinced in the paper that the combined system is able to make use of the information that is contained in the strings of separate characters for more correct recognition of these characters. The experiments have shown that the combined system commits no errors in recognition of all separate digits of the MNIST test set, provided that these digits are organized in the numeral strings of more than 5 symbols each. This result apparently demonstrates how much information is stored in organization of characters in the form of strings for the task of recognition of these characters. And this information is effectively elicited from the strings by means of a simple device, which is the “symbolic” neural network. It is evident from the above description that the assembly neural network can be substituted in the combined recognition system for any other recognition device. At the same time, the “symbolic” neural network is a necessary component of the system. A need in the combined recognition system arises in case if any recognition technique cannot solve the task of recognition of separate characters quite correctly. The most obvious example of such a task is recognition of handwritten words and sentences. As is mentioned above, the combined recognition system is a somewhat simplified model of the recognition of handwritten words. The term “simplified model” means that the system does not contain mechanisms for segmentation of handwritten words into separate letters. Nevertheless, the combined system could serve as a useful example and a prototype for designing neural network mechanisms for recognition of handwritten words in the future.
References [1] F. Rosenblatt, Principles of Neurodynamics. Perceptrons and Theory of Brain Mechanisms, Spartan Books, Washington DC, 1962. [2] G.D. Magoulas, M.N. Vrahatis, G.S. Androulakis, Effective backpropagation training with variable stepsize, Neural Networks 10 (1997) 69–82. [3] G.A. Carpenter, B.L. Milenova, B.W. Noeske, Distributed ARTMAP: a neural network for fast distributed supervised learning, Neural Networks 11 (1998) 793–813. [4] E. Kussul, T. Baidyk, L. Kasatkina, V. Lukovich, Rosenblatt perceptrons for handwritten digit recognition, Proceedings of IJCNN’01, Washington DC, USA, 2001, pp. 1516–1521.
322
A. Goltsev, D. Rachkovskij / Pattern Recognition 38 (2005) 315 – 322
[5] E. Kussul, T. Baidyk, Permutative coding technique for handwritten digit recognition system, Proceedings of IJCNN’03, Portland, OR, USA, 2003, pp. 2163–2168. [6] K. Fukushima, Neocognitron: a model for visual recognition, in: M.A. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, second ed., MIT Press, Cambridge, MA, London, England, 2003, pp. 715–719. [7] D.O. Hebb, The Organization of Behavior, Wiley, New York, 1949. [8] P.M. Milner, The cell assembly: mark 2, Psychol. Rev. 64 (1957) 242–252. [9] P.M. Milner, Physiological Psychology, Holt, Rinehart & Winston Inc., McGill University, 1970. [10] A. Goltsev, An assembly neural network for texture segmentation, Neural Networks 9 (4) (1996) 643–653. [11] A. Goltsev, A process of generalization in the assembly neural network, Proceedings of ICONIP’01, Shanghai, China, 2001, pp. 47–52. [12] A. Goltsev, Secondary learning in the assembly neural network, Neurocomputing (2004), accepted for publication.
[13] A. Goltsev, D.C. Wunsch, Inhibitory connections in the assembly neural network for texture segmentation, Neural Networks 11 (5) (1998) 951–962. [14] A. Goltsev, D.C. Wunsch, Generalization of features in the assembly neural networks, Int. J. Neural Systems (IJNS) 14 (1) (2004) 39–56. [15] A. Goltsev, D. Húsek, Some properties of the assembly neural networks, Neural Network World 12 (1) (2002) 15–32. [16] MNIST database http://yann.lecun.com/exdb/mnist/ [17] J.-D. Gascuel, B. Moobed, M. Weinfeld, An internal mechanism for detecting parasite attractors in a Hopfield network, Neural Comput. 6 (1994) 902–915. [18] H. Schwenk, M. Milgram, Structured diabolo-networks for hand-written character recognition, Proceedings of the International Conference on Artificial Neural Networks, vol. 2, Sorrento, Italy, 1994, pp. 985–988. [19] A.A. Ezhov, V.L. Vvedensky, Object generation with neural networks (when spurious memories are useful), Neural Networks 9 (9) (1996) 1491–1495.
About the Author—ALEXANDER GOLTSEV received the M.Sc. degree (with honors) in Physics and Electronics (Bionics) from the Dnepropetrovsk State University (Dnepropetrovsk, Ukraine) in 1971. He obtained the Ph.D. degree in Engineering from V.M. Glushkov Institute of Cybernetics of Ukrainian Academy of Sciences (Kiev, Ukraine) in 1976. From 1971 he has been working at the same Institute of Cybernetics (which is V.M. Glushkov Cybernetics Center now) holding the position of senior researcher at the Department of Neural Information Processing Systems of International Research and Training Centre of Information Technologies (which is a part V.M. Glushkov Cybernetics Center). He temporarily worked at Linz University (Linz, Austria, 1993–1994) and at German National Research Center for Information Technology (Sankt Augustin, Germany, 2001). His research interests are neural networks, image processing, computer vision, and robotics. The purposes of the current research work are to improve algorithms and structures of assembly neural networks and to construct a neural network of unsupervised learning type for texture segmentation. His whole list of publications consists of 64 items. About the Author—DMITRI A. RACHKOVSKIJ received the MS degree in radiophysics and electronics from Rostov State University, Rostov-on-Don, Russia, in 1983, and the Ph.D. degree from the V.M. Glushkov Cybernetics Center, Kiev, Ukraine, in 1990. From 1983 to 1987, he was an R&D engineer in computer-based systems at Rostov Special R&D Bureau. Since 1987, he worked under the supervision of Academician N.M. Amosov, who since the early 1960s has led one of the most interesting research programs in AI, and E.M. Kussul, who made a major contribution to that program by inventing the paradigm of Associative-Projective Neural Networks (APNNs). Currently, Dr. Rachkovskij is a senior researcher at the Cybernetics Center. His research is mainly connected with the schemes for distributed representations of various data types, their applications in the areas of classification, information retrieval, analogical reasoning, and comparison to stateof-the-art approaches. He (co)authored about 40 publications. E-mail: [email protected].