- Email: [email protected]
A neural sorting network with O(1) time complexity
Information Elsevier
Processing
Letters
16 April
45 (1993) 309-313
1993
A neural sorting network with O( 1) time complexity Wen-Tsuen
Chen and Kuen-Rong Hsieh
Department of Computer Science, National Tsing Hua Uniuersity, Hsinchu 30043, Taiwan, ROC Communicated by K. Ikeda Received 17 April 1991 Revised 16 November 1992
Keywords:
Parallel
processing;
sorting;
neural
network
1. Introduction
Analog computation architectures such as artificial neural network models have received phenomenal attention lately. Massive parallel processing is natural in neural networks, which also meets the development trend of computer science. Because parallelism is a straight way to speed up computation. Thus, considering parallel algorithms on neural networks is quite reasonable. Neural network models have been shown to be able to find “good” solutions for some optimization problems in a short time [51. But, it cannot guarantee finding the real optimal solution except by techniques similar to simulated annealing. Under such circumstances, there is no systematic way to determine the annealing schedule and the time needed for convergence. Vergis et al. [ll] showed that analog computation can be simulated efficiently by a Turing machine in polynomial time. Unless P = NP, NP-complete problems cannot be solved by analog computation in polynomial time. But, from the view of time complexity, analog computation is better than digital computation in one perspective. The basic operation Correspondence to: Professor W.-T. Chen, Department of Computer Science, National Tsing Hua University, Hsinchu 30043, Taiwan, ROC. 0020-0190/93/f06.00
Q 1993 - Elsevier
Science
Publishers -
of neurons, namely the summation, has lower time complexity than that of digital computers. In digital computation, the time complexity to sum N numbers is O(log N) no matter how many processors are used. However, the time complexity is O(1) in analog computation. In neural network models, if any problem can be solved by only operations of summation of weighted sums, then it is highly promising to be solved by a neural network in constant time. As it seems unlikely that P = NP, it is unreliable to expect analog computation to solve NPcomplete problems in a complexity lower than exponential. But we are interested to see if some P problems can be solved by analog computation with lower time complexity. Since it can reduce summation from O(log N) to O(l), it is natural to first study the possibility of solving other O(log N) problems in O(1). Sorting is one of the most commonly studied P problems. Except by reconfigurable bus [2,121, its best parallel algorithms also have 00og N) complexity [l]. There have been many parallel sorting algorithms on different parallel architectures such as the shuffle-exchange [8], the binary cube [6], the mesh [lo], the cube-connected cycles [71, array processors [91, and processor array with reconfigurable bus [2,12]. In this paper, a multilayer feedforward neural network is proposed to solve the sorting problem. The network has O(N3> neurons and
B.V. - All rights reserved
309
Volume
45. Number
6
INFORMATION
PROCESSING
0) Fig. 1. The operation
of a neuron.
O(fV3) links. No matter what the input size is, the number of layers is fixed. Thus, the computation time complexity of the network is independent of input size, and so we have a sorting network of O(1).
2. Operation
LETTERS
16 April
1993
in parallel, the summation of weighted input also takes O(1) with O(N) links. (3) Application of the actication function. The application of the activation function in each neuron is usually implemented by an amplifier or diode. It takes only constant delay time to complete the function. According to the above description, if neurons are really implemented in analog mode, the operation in each neuron has O(1) complexity. In other words, no matter how many input links there are, it takes constant time to pass its output value to its output links. For a multilayer feedforward network, its computation time depends only on the number of layers in the network. Thus, if the number of layers is independent of input size, the time complexity of the network is O(1).
of a neuron 3. The sorting network
A neural network consists of neurons and links. Neurons can be considered as the processing elements (PEs) in the network and a link is the data path between two neurons. AI1 links are weighted, so that data is multiplied by the link weight when it passes through the link. The basic operations of neurons can be seen in Fig. 1. Each neuron sums all its inputs and passes the result through an activation function f. Typical activation functions are hard limiter, threshold logic, and sigmoid function. Here, we analyze the time complexity of the basic operation of a neuron. (1) Summation. From the view of practical implementation, the summation in each neuron can be considered as the summation of current from multiple links. Its time complexity can be reasonably considered to be O(1) with any number of links. We can imagine this as there being n small rivers pouring flows into a large river. No matter how many small rivers there are, the flow of the large river can reflect the sum of the small rivers in constant time. (2) Multiplication and summation. The data that passes through any link in a neural network is usually multiplied by the weight on that link. In practical implementation, each link is implemented by a variable resistor or amplifier to accomplish this. Since all multiplications are done 310
In this section, we present a neural sorting network which implements the rank finding strategy in 6-layer. Of course, the network sorts a set of numbers in analog representation. The block diagram of the sorting network is shown in Fig. 2. In Fig. 2, (Xi, X,, . . . , XJ is the input list. The sorted result is represented in the output list (S,, S,, . . . , S,_,) with S, being the
/ TjT KO
, K1
l
**
K
I
n-l
! i
\
x1
x”
x2
Fig. 2. The block diagram
of the neural
sorting
network.
Volume 45, Number 6
INFORMATION
16 April 1993
PROCESSING LETTERS
smallest value. The sorting strategy used here consists of the following two steps which are implemented by rank layer and selection layer respectively. Step 1: Find the rank of each input data. The rank Ri of Xi is defined to be the position of Xi in the sorted list minus one. For example, if (9, 6, 2, 4) is the input list, then the sorted list is (2, 4, 6, 91, and R, = 3, R, = 2, R, = 2, R,= 1. After the rank of each input data is determined, we know its position in the sorted list. Step 2: Pass the input data to a proper output position according to its rank. This step works as a multiplexer or a router, which selects some data to pass through some path. That is, let Si = Xi for all i and certain j.
Rj
xj Fig. 3. The connections inside module K, of the rank layer.
The rank layer. The rank layer is composed
of n modules K,, K,, . . . , K,. Module Ki is to compute the rank Ri of input Xi. From the definition of rank, Ri can be computed by the following equation:
Ri =
2
sgn(Xi -X,),
j= l,j+l
sgn(x) =
1 0
if x20, otherwise.
According to the above equation, there is a simple 2-layer network to compute Ri. The detailed connection of module Ki in rank layer is shown in Fig. 3. The activation functions of neurons in each layer are also depicted in the figure. The selection
constitute the sorted list. There will be only one nonzero value among the n links connected to Si. The test module q in Lj tests if Ri = j. If yes, it outputs 0, else output 1. The arrangement of the weights is to ensure that when some X,, is not the correct value for Sj, it will be inhibited. The detailed connections of test module I;. in Lj are shown in Fig. 5. In the test module 7, it actually tests if Ri is between j - 1 and j + 1. When input data are ready, the sorting network can report the sorted list at the output nodes S,, S,, . . . , S, _ i after a short delay time. S
layer. After the R,‘s are computed,
we know the position of each input data in the sorted list. The selection layer is constructed to select which Xi should be passed through which output node so as to match their correct position. The layer is composed of module La, L,,. . ., L,_ 1. Each module Li selects some Xi such that Rj = i to pass through it and inhibits all other input values. This function can be implemented by the network in Fig. 4. In Fig. 4, the output of the test module 5 determines the connection weight between Xi and Si. If 7;. outputs 0 (1) then the connection weight is 0 (1). The output nodes S,, S,, . . . , S,_,
A
Fig. 4. The connections
inside module Li of the selection layer. 311
Volume 4jq Number 6
INFORMATION Ti
I
Rj Fig. 5. The connections inside test module ?;.
The total delay time is the delay time of a single neuron multiplied by the number of layers in the network. When the input size gets larger and larger, the network extends horizontally, and the delay time of the network is not changed. Thus, the network is a sorting network with O(1) time complexity. The nodes and links of each module K and L are both O(N’), thus the nodes and links of the sorting network are both O(N’).
4. Conclusion Our sorting strategy is based on rank sorting which is similar to that of Wang et al. 1121. The existing constant time sorting algorithms are based on an assumption, viz., broadcasting on a reconfigurable bus takes constant time. Such an assumption is not very realistic when the number of processing elements is large. Our basic assumption is that the summation of current is constant time which is considered to be more realistic. Analog computation, of course, has to sacrifice precision in data representation. The practical use of the scheme depends on analog VLSI technology. It demands more precise implementation of amplifiers and more connections in a chip. We cannot use this sorting network in practice until these requirements are fulfilled. 312
PROCESSING LEl-I’ERS
16 April 1993
In computation theory, problems in P are divided into NC problems and log-space complete problems. Whether both classes of problems are equivalent is not proved. NC problems have efficient parallel algorithms in O(log N), while logspace complete problems are unlikely to find O(logkN) parallel algorithms. It has been shown that if any log-space complete problems can be solved in polylog time complexity (i.e. O(logkN)), then P = NC. Sorting problems are in the NC class; meanwhile the time complexity of some problems are bounded by that of sorting. That is, some problems can be solved in the same time complexity with sorting such as convex hull finding. We conjecture that many problems with O(log N) as lower bound can find O(1) algorithms in neural network models. For instance, under this computation model, the inner product of two vectors has O(1) complexity, thus the matrix multiplication problem can also be solved in O(1) with O(N’) neurons and O(N2> links. According to Kolmogorov’s theorem 131, given any continuous function 4 &1:[0, l]n--+R.M,
4(x)
=y,
there exists a three-layer neural network to compute 4, with n neurons in the input layer, 2n + 1 neurons in the hidden layer, and m neurons in the output layer. If we can find such a 3-layer network, the computation complexity of any problem is bounded by the complexity of finding such a mapping. The sorting network proposed in this paper is not a direct demonstration of Kolmogorov’s theorem. Rather, it is to show that some NC problems can actually be solved in constant time under the analog computation model. From the Kolmogorov’s theorem and our result, it is possible to find more problems in P which have more efficient computation complexity under neural network analog computation model.
References [l] M. Aigner, Parallel complexity of sorting problems, /. Algorithms
3 (1982) 79-89.
Volume
45. Number
6
INFORrMATION
PROCESSING
[2] Y. Ben-.&her. D. Peleg. R. Ramaswami and A. Schuster, The power of reconfiguration, /. Parulfel Distributed Compuf. 13 (1991) 139-153. [3] R. Hecht-Nielsen, Kolmogorov’s mapping neural network e.xistence theorem, in: Proc. Internat. ConJ on Neural Neverworks, Vol. 3 (1987) 11-14. 141 G.E. Hinton and T.J. Sejnowski, Optimal perceptual inference, in: Proc. IEEE Computer Vtsion and Pattern Recognition Conf., Washington, DC (1983) 448-453. [5] J.J. Hopfield and D.W. Tank, Neural computation of decisions in optimization problems, Biol. Cybernet. 52 (198.5) 141-132. [6] MC. Pease, The indirect binary n-cube microprocessor array, IEEE Trans. Comprct. 26 (1977) 458-473. [7] E.P. Preparata and J. Vuilemin, The cube-connected
[S] [9] [lo] [ll]
[12]
LETTERS
16 April 1993
cycles: a versatile network for parallel computation, Comm. ACM 2-l (1981) 300-309. H.S. Stone, Parallel processing with the perfect shuffle, IEEE Trans. Comput. 20 (1971) 153-161. H.S. Stone, Sorting on STAR, /EEE Trans. Software Engrg. 4 (1978) 138-146. C.D. Thompson and H.T. Kung. Sorting on a mesh-connected computer, Comm. ACM 20 (1977) 263-271. A. Vergis. K. Steiglitz and B. Dickinson. The complexity of analog computation, Muth. Compxt. Simulation 28 (1986) 91-113. B.F. Wang, G.H. Chen and F.C. Lin. Constant time sorting on a processor array with a reconfigure bus system, Inform. Process. Lett. 34 (1990) 187-192.
Processing
Letters
16 April
45 (1993) 309-313
1993
A neural sorting network with O( 1) time complexity Wen-Tsuen
Chen and Kuen-Rong Hsieh
Department of Computer Science, National Tsing Hua Uniuersity, Hsinchu 30043, Taiwan, ROC Communicated by K. Ikeda Received 17 April 1991 Revised 16 November 1992
Keywords:
Parallel
processing;
sorting;
neural
network
1. Introduction
Analog computation architectures such as artificial neural network models have received phenomenal attention lately. Massive parallel processing is natural in neural networks, which also meets the development trend of computer science. Because parallelism is a straight way to speed up computation. Thus, considering parallel algorithms on neural networks is quite reasonable. Neural network models have been shown to be able to find “good” solutions for some optimization problems in a short time [51. But, it cannot guarantee finding the real optimal solution except by techniques similar to simulated annealing. Under such circumstances, there is no systematic way to determine the annealing schedule and the time needed for convergence. Vergis et al. [ll] showed that analog computation can be simulated efficiently by a Turing machine in polynomial time. Unless P = NP, NP-complete problems cannot be solved by analog computation in polynomial time. But, from the view of time complexity, analog computation is better than digital computation in one perspective. The basic operation Correspondence to: Professor W.-T. Chen, Department of Computer Science, National Tsing Hua University, Hsinchu 30043, Taiwan, ROC. 0020-0190/93/f06.00
Q 1993 - Elsevier
Science
Publishers -
of neurons, namely the summation, has lower time complexity than that of digital computers. In digital computation, the time complexity to sum N numbers is O(log N) no matter how many processors are used. However, the time complexity is O(1) in analog computation. In neural network models, if any problem can be solved by only operations of summation of weighted sums, then it is highly promising to be solved by a neural network in constant time. As it seems unlikely that P = NP, it is unreliable to expect analog computation to solve NPcomplete problems in a complexity lower than exponential. But we are interested to see if some P problems can be solved by analog computation with lower time complexity. Since it can reduce summation from O(log N) to O(l), it is natural to first study the possibility of solving other O(log N) problems in O(1). Sorting is one of the most commonly studied P problems. Except by reconfigurable bus [2,121, its best parallel algorithms also have 00og N) complexity [l]. There have been many parallel sorting algorithms on different parallel architectures such as the shuffle-exchange [8], the binary cube [6], the mesh [lo], the cube-connected cycles [71, array processors [91, and processor array with reconfigurable bus [2,12]. In this paper, a multilayer feedforward neural network is proposed to solve the sorting problem. The network has O(N3> neurons and
B.V. - All rights reserved
309
Volume
45. Number
6
INFORMATION
PROCESSING
0) Fig. 1. The operation
of a neuron.
O(fV3) links. No matter what the input size is, the number of layers is fixed. Thus, the computation time complexity of the network is independent of input size, and so we have a sorting network of O(1).
2. Operation
LETTERS
16 April
1993
in parallel, the summation of weighted input also takes O(1) with O(N) links. (3) Application of the actication function. The application of the activation function in each neuron is usually implemented by an amplifier or diode. It takes only constant delay time to complete the function. According to the above description, if neurons are really implemented in analog mode, the operation in each neuron has O(1) complexity. In other words, no matter how many input links there are, it takes constant time to pass its output value to its output links. For a multilayer feedforward network, its computation time depends only on the number of layers in the network. Thus, if the number of layers is independent of input size, the time complexity of the network is O(1).
of a neuron 3. The sorting network
A neural network consists of neurons and links. Neurons can be considered as the processing elements (PEs) in the network and a link is the data path between two neurons. AI1 links are weighted, so that data is multiplied by the link weight when it passes through the link. The basic operations of neurons can be seen in Fig. 1. Each neuron sums all its inputs and passes the result through an activation function f. Typical activation functions are hard limiter, threshold logic, and sigmoid function. Here, we analyze the time complexity of the basic operation of a neuron. (1) Summation. From the view of practical implementation, the summation in each neuron can be considered as the summation of current from multiple links. Its time complexity can be reasonably considered to be O(1) with any number of links. We can imagine this as there being n small rivers pouring flows into a large river. No matter how many small rivers there are, the flow of the large river can reflect the sum of the small rivers in constant time. (2) Multiplication and summation. The data that passes through any link in a neural network is usually multiplied by the weight on that link. In practical implementation, each link is implemented by a variable resistor or amplifier to accomplish this. Since all multiplications are done 310
In this section, we present a neural sorting network which implements the rank finding strategy in 6-layer. Of course, the network sorts a set of numbers in analog representation. The block diagram of the sorting network is shown in Fig. 2. In Fig. 2, (Xi, X,, . . . , XJ is the input list. The sorted result is represented in the output list (S,, S,, . . . , S,_,) with S, being the
/ TjT KO
, K1
l
**
K
I
n-l
! i
\
x1
x”
x2
Fig. 2. The block diagram
of the neural
sorting
network.
Volume 45, Number 6
INFORMATION
16 April 1993
PROCESSING LETTERS
smallest value. The sorting strategy used here consists of the following two steps which are implemented by rank layer and selection layer respectively. Step 1: Find the rank of each input data. The rank Ri of Xi is defined to be the position of Xi in the sorted list minus one. For example, if (9, 6, 2, 4) is the input list, then the sorted list is (2, 4, 6, 91, and R, = 3, R, = 2, R, = 2, R,= 1. After the rank of each input data is determined, we know its position in the sorted list. Step 2: Pass the input data to a proper output position according to its rank. This step works as a multiplexer or a router, which selects some data to pass through some path. That is, let Si = Xi for all i and certain j.
Rj
xj Fig. 3. The connections inside module K, of the rank layer.
The rank layer. The rank layer is composed
of n modules K,, K,, . . . , K,. Module Ki is to compute the rank Ri of input Xi. From the definition of rank, Ri can be computed by the following equation:
Ri =
2
sgn(Xi -X,),
j= l,j+l
sgn(x) =
1 0
if x20, otherwise.
According to the above equation, there is a simple 2-layer network to compute Ri. The detailed connection of module Ki in rank layer is shown in Fig. 3. The activation functions of neurons in each layer are also depicted in the figure. The selection
constitute the sorted list. There will be only one nonzero value among the n links connected to Si. The test module q in Lj tests if Ri = j. If yes, it outputs 0, else output 1. The arrangement of the weights is to ensure that when some X,, is not the correct value for Sj, it will be inhibited. The detailed connections of test module I;. in Lj are shown in Fig. 5. In the test module 7, it actually tests if Ri is between j - 1 and j + 1. When input data are ready, the sorting network can report the sorted list at the output nodes S,, S,, . . . , S, _ i after a short delay time. S
layer. After the R,‘s are computed,
we know the position of each input data in the sorted list. The selection layer is constructed to select which Xi should be passed through which output node so as to match their correct position. The layer is composed of module La, L,,. . ., L,_ 1. Each module Li selects some Xi such that Rj = i to pass through it and inhibits all other input values. This function can be implemented by the network in Fig. 4. In Fig. 4, the output of the test module 5 determines the connection weight between Xi and Si. If 7;. outputs 0 (1) then the connection weight is 0 (1). The output nodes S,, S,, . . . , S,_,
A
Fig. 4. The connections
inside module Li of the selection layer. 311
Volume 4jq Number 6
INFORMATION Ti
I
Rj Fig. 5. The connections inside test module ?;.
The total delay time is the delay time of a single neuron multiplied by the number of layers in the network. When the input size gets larger and larger, the network extends horizontally, and the delay time of the network is not changed. Thus, the network is a sorting network with O(1) time complexity. The nodes and links of each module K and L are both O(N’), thus the nodes and links of the sorting network are both O(N’).
4. Conclusion Our sorting strategy is based on rank sorting which is similar to that of Wang et al. 1121. The existing constant time sorting algorithms are based on an assumption, viz., broadcasting on a reconfigurable bus takes constant time. Such an assumption is not very realistic when the number of processing elements is large. Our basic assumption is that the summation of current is constant time which is considered to be more realistic. Analog computation, of course, has to sacrifice precision in data representation. The practical use of the scheme depends on analog VLSI technology. It demands more precise implementation of amplifiers and more connections in a chip. We cannot use this sorting network in practice until these requirements are fulfilled. 312
PROCESSING LEl-I’ERS
16 April 1993
In computation theory, problems in P are divided into NC problems and log-space complete problems. Whether both classes of problems are equivalent is not proved. NC problems have efficient parallel algorithms in O(log N), while logspace complete problems are unlikely to find O(logkN) parallel algorithms. It has been shown that if any log-space complete problems can be solved in polylog time complexity (i.e. O(logkN)), then P = NC. Sorting problems are in the NC class; meanwhile the time complexity of some problems are bounded by that of sorting. That is, some problems can be solved in the same time complexity with sorting such as convex hull finding. We conjecture that many problems with O(log N) as lower bound can find O(1) algorithms in neural network models. For instance, under this computation model, the inner product of two vectors has O(1) complexity, thus the matrix multiplication problem can also be solved in O(1) with O(N’) neurons and O(N2> links. According to Kolmogorov’s theorem 131, given any continuous function 4 &1:[0, l]n--+R.M,
4(x)
=y,
there exists a three-layer neural network to compute 4, with n neurons in the input layer, 2n + 1 neurons in the hidden layer, and m neurons in the output layer. If we can find such a 3-layer network, the computation complexity of any problem is bounded by the complexity of finding such a mapping. The sorting network proposed in this paper is not a direct demonstration of Kolmogorov’s theorem. Rather, it is to show that some NC problems can actually be solved in constant time under the analog computation model. From the Kolmogorov’s theorem and our result, it is possible to find more problems in P which have more efficient computation complexity under neural network analog computation model.
References [l] M. Aigner, Parallel complexity of sorting problems, /. Algorithms
3 (1982) 79-89.
Volume
45. Number
6
INFORrMATION
PROCESSING
[2] Y. Ben-.&her. D. Peleg. R. Ramaswami and A. Schuster, The power of reconfiguration, /. Parulfel Distributed Compuf. 13 (1991) 139-153. [3] R. Hecht-Nielsen, Kolmogorov’s mapping neural network e.xistence theorem, in: Proc. Internat. ConJ on Neural Neverworks, Vol. 3 (1987) 11-14. 141 G.E. Hinton and T.J. Sejnowski, Optimal perceptual inference, in: Proc. IEEE Computer Vtsion and Pattern Recognition Conf., Washington, DC (1983) 448-453. [5] J.J. Hopfield and D.W. Tank, Neural computation of decisions in optimization problems, Biol. Cybernet. 52 (198.5) 141-132. [6] MC. Pease, The indirect binary n-cube microprocessor array, IEEE Trans. Comprct. 26 (1977) 458-473. [7] E.P. Preparata and J. Vuilemin, The cube-connected
[S] [9] [lo] [ll]
[12]
LETTERS
16 April 1993
cycles: a versatile network for parallel computation, Comm. ACM 2-l (1981) 300-309. H.S. Stone, Parallel processing with the perfect shuffle, IEEE Trans. Comput. 20 (1971) 153-161. H.S. Stone, Sorting on STAR, /EEE Trans. Software Engrg. 4 (1978) 138-146. C.D. Thompson and H.T. Kung. Sorting on a mesh-connected computer, Comm. ACM 20 (1977) 263-271. A. Vergis. K. Steiglitz and B. Dickinson. The complexity of analog computation, Muth. Compxt. Simulation 28 (1986) 91-113. B.F. Wang, G.H. Chen and F.C. Lin. Constant time sorting on a processor array with a reconfigure bus system, Inform. Process. Lett. 34 (1990) 187-192.