A discrete-time dynamic K-winners-take-all neural circuit

ARTICLE IN PRESS Neurocomputing 72 (2009) 3191–3202

Contents lists available at ScienceDirect

Neurocomputing journal homepage: www.elsevier.com/locate/neucom

A discrete-time dynamic K-winners-take-all neural circuit Pavlo V. Tymoshchuk Institute of Computer Science & Information Technology, Lviv Polytechnic National University, 12, S. Bandera Str., Lviv 79013, Ukraine

a r t i c l e in fo

abstract

Article history: Received 27 July 2007 Received in revised form 1 March 2009 Accepted 1 March 2009 Communicated by T. Heskes Available online 2 April 2009

A discrete-time mathematical model of K-winners-take-all (KWTA) neural circuit that can quickly identify the K-winning from N neurons, where 1pKoN, whose input signals are larger than that of remaining NK neurons, is given and analyzed. A functional block scheme of the circuit is presented. For N competitors, such circuit is composed of N feedforward and one feedback hard-limiting neurons that are used to determine the dynamic shift of input signals. The circuit has low computational and hardware implementation complexity, high speed of signal processing, can process signals of any finite range, possesses signal order preserving property and does not require resetting and corresponding supervisory circuit that additionally increases a speed of signal processing. & 2009 Elsevier B.V. All rights reserved.

Keywords: Discrete-time mathematical model K-winners-take-all neural circuit Functional block scheme Hard-limiting neuron Dynamic shift

1. Introduction The K-winners-take all (KWTA) neural networks are known to perform selection of K from N elements with larger activation functions than in the remaining N  K elements. When K is equal to unity, the KWTA network is the winner-take all (WTA) one, that can distinguish the maximum activation neuron [8,10,15]. Selection of K largest elements from a data set of N real numbers is a key task in decision making, pattern recognition, associative memories and competitive learning networks [16,20]. Tasks of such type are naturally met in classification problems and applied for the neural network classifier development, for problem solving of pattern recognition and pattern classification [7]. The KWTA networks are applied in telecommunications, particularly to control data packet switches [1]. The KWTA operation has important applications in machine learning, such as k nearest neighbours classification, k-means clustering, etc. [9]. Different kinds of neural networks have been proposed to solve the K-winners-take-all problem. Specifically, a KWTA mechanism that uses the continuous-time Hopfield neural network for reliable K-winners convergence is suggested in [10]. It has been verified that the network converges to the stable equilibria. The network which uses the Hopfield model (the Grossberg additive model) is characterized by mutually inhibitory interconnections for which a sigmoid gain of activation function must be large enough. Equilibrium states of the network are asymptotically stable. In the steady state, the network demonstrates K

E-mail address: [email protected] 0925-2312/$ - see front matter & 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.neucom.2009.03.001

components with values a40 and NK components with bo0. A special class of mutually inhibitory networks for designing a KWTA network by using interactive activations is described in [17]. When parameters are chosen properly, the interactive activations of K-winners-take-all network has been proven to be a duality of the network [10]. Most of the existing KWTA networks perform the so-called mutual inhibition for winning competition and require transient dynamics or settling period with a precise reset of initial states for repetitive use. Associated energy functions of such networks have many local minima and they do not have a unique global minimum. Therefore, the network outputs can converge to different steady states that prevents its application for real time signal processing [2,5,11,13,17,19]. A dynamical system with global convergence to a unique stable equilibrium states is proposed in [12]. The analog circuit realization is of the neural network type, with N cells represented by amplifiers, global feedback, and at most 2N interconnections, where N is the number of inputs. An application (called ‘‘the Kselector’’) which signals the ranks of the K largest elements of input list and, in parallel the rank of the (K+1)th element, is designed and numerically tested. For a given density of the input elements, one obtains feasible separation intervals of output signals. The circuit requires high-gain amplifiers, an appropriate control source, suitable scaling of the input data, and output signal resetting. Analog neural networks capable of solving the KWTA problem are formulated in [6,9]. The KWTA operation is firstly converted to an equivalent constrained quadratic optimization problem and then the simplified and improved dual neural networks are tailored to KWTA networks which are designed and simulated in

ARTICLE IN PRESS 3192

P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

[6,9] correspondingly. In contrast to the most recurrent neural networks, which are full connectivity the networks have good scalability performance. The networks are globally Lyapunov stable and globally convergent to the KWTA operation. Numerical results demonstrate the performance of the neural networks. In particular, the convergence rate of the network from [9] is independent of the number of inputs, i.e. the problem scale; it is inversely proportional to the network resolution and equal to the network transient. If the network resolution rises, the speed of signal processing falls. Moreover to increase a resolution ability of the network a value of gain of analog amplifiers should be increased. This value is limited by problems of electronic circuit implementation of high gain. Therefore, a network resolution should be chosen as a compromise between a precision of signal processing from one side and processing speed, and hardware restrictions of the network from another one. The simulation results for the network from [6] show that the convergence time of the network decreases when the problem size increases on average. The networks are successfully tested for dynamical timevarying inputs when a speed of the network transient is much larger than a speed of input signal change. Discrete-time KWTA neural networks comparatively to continuous-time analogs are more reliable and demonstrate a more high precision of signal processing [3]. A simple and fast discretetime KWTA neural network which does not utilize the concept of mutual inhibition for a large number of competitors has been proposed in [18]. The network with one-layer structure determines the dynamic thresholding of the desired number of winners to achieve the winning process. The network has limited range of signal processing, requires a resetting of the initial dynamic threshold to the center of the change range of inputs and needs additional supervisory circuit. In this paper, discrete-time mathematical model and corresponding functional block scheme of KWTA neural circuit built in one-layer competitive architecture performing the dynamic shifting of input signals to obtain K winners are proposed. The circuit which is composed of N feedforward neurons and one feedback hard-limiting neuron used to determine the desired shift of inputs can be implemented in a digital hardware by adders, digital integrator, switches and external sources of voltage or current, which are appropriate for real time signal processing using VLSI technologies [3]. Proposed circuit has low computational and hardware implementation complexity, high speed of signal processing, is capable for signal processing of any finite range, and possesses signal order preserving property. The circuit does not require resetting and corresponding supervisory circuit that simplifies the hardware and increases a speed of signal processing. In this paper, the term ‘‘dynamic’’ refers to the neural network which is modeled by difference equation that determines the evolution of a state variable or simply the state. The evolution occurs in the discrete-time state space, and a function that describes this evolution in the state-space is referred to as a (state) trajectory or solution. Discrete-time dynamic neural network has feedback connection that is implemented on the basis of time delay. A special case of dynamic neural network is static neural network. The term ‘‘static neural network’’ defines the neural network functioning in static mode. Static neural network is modeled by algebraic equation; therefore, a trajectory of its solution does not have any transient. There is no any feedback connection in static neural network.

2. The problem statement Let us be given N real numbers from a1 to aN , N41, i.e. a1 ; a2 ; . . . ; aN as instant values of unknown input signals and we

wish to choose the K largest of them, which are referred to as the winners, where 1pKoN is a nonnegative integer. Assume that the given numbers are distributed in a known range a 2 ðAmin ; Amax Þ. Let us take them firstly to be non-equal (or distinct) and arranged in the descending order of magnitude to satisfy inequalities: a1 4a2 4    4aN ,

(1)

where indexes 1; 2; . . . ; N can differ in general from the original numbers of inputs meaning that the components of vector a ¼ ½a1 ; . . . ; aN  are ordered. Let it be necessary to design a discretetime neural circuit that processes an input vector a to obtain, after finite number of iterations, such outputs b ¼ ½b1 ; . . . ; bN  of the circuit that satisfies inequalities: bi 40; i 2 1; 2; . . . ; K;

bj o0; j 2 K þ 1; K þ 2; . . . ; N.

(2)

Inequalities (2) express the KWTA property, i.e. that the outputs from b1 to bK ‘‘win’’ the competition, and the fact that they are the only positive components of b shows that the inputs from a1 to aK are the K largest components of vector a. In other words, it is necessary to design a simple neural classifier that selects the K largest inputs from a vector a. An iterative neural circuit of such a classifier is presented in the next sections and it is referred to as KWTA circuit.

3. A discrete-time KWTA neural circuit 3.1. Mathematical model of the circuit Let us preprocess given vector a of inputs having subtracted from all its components a value Amin to obtain additional signals: c1 4c2 4    4cN ,

(3)

where cn ¼ an  Amin , n ¼ 1; 2; . . . ; N. It is easy to see that signals (3) are located in the range ð0; AÞ, where A ¼ Amax  Amin 40, i.e. c 2 ð0; AÞ, where c ¼ ½c1 ; c2 ; . . . ; cN . Since inputs (1) are distinct and distributed in known range, signals (3) are also distinct and limited in range ð0; AÞ. Thus, there exists for any 1pKoN such value of x 2 <, that the following inequalities are satisfied: ci 4x; i 2 1; 2; . . . ; K;

cj ox; j 2 K þ 1; K þ 2; . . . ; N.

(4)

Subtracting x from inequalities (4) gives ci  x40; i 2 1; 2; . . . ; K;

cj  xo0; j 2 K þ 1; K þ 2; . . . ; N.

(5)

As it can be seen from inequalities (5) signals given by cn  x, where n ¼ 1; 2; . . . ; N, possess KWTA property. Therefore, such signals can be used as outputs of designing KWTA neural circuit, i.e. equalities bi ¼ ci  x; i 2 1; 2; . . . ; K;

bj ¼ cj  x; j 2 K þ 1; K þ 2; . . . ; N (6)

can be written. In order to design desirable circuit it is necessary to develop a procedure of finding a value of scalar dynamic shift x of inputs which satisfies inequality (4). Let us use for this purpose the requirement that such shift, in the steady state, should be located in the range ð0; AÞ. Design discrete-time trajectory xðkÞ, where k ¼ 1; 2; . . . ; m is a number of iteration to reaching a steady state, which can pass all range ð0; AÞ. Let such a trajectory be a solution of the corresponding difference equation xðkþ1Þ ¼ jðxðkÞ Þ with an initial condition xð1Þ , where jðxðkÞ Þ is a certain function to be determined. Suppose that in a certain moment of discrete-time t ðmÞ , a variable xðkÞ accepts a steady state value xðkÞ ¼ xðmÞ that satisfies inequality (4). In order to stop a computational process in a moment tðmÞ , determine a condition that controls quantities of winners and losers in each discrete-time point during a

ARTICLE IN PRESS P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

computational process. Let us use for this purpose the condition given by RðxðkÞ Þ ¼ 2K  N 

N X

ðkÞ

sgnðbn Þ,

(7)

n¼1 ðkÞ

where RðxðkÞ Þ is a kth discrete value of the residual, bn ¼ cn  xðkÞ is a value of nth output of designed circuit on a kth iteration: 8 ðkÞ 1 if bn 40; > > < ðkÞ ðkÞ if bn ¼ 0; sgnðbn Þ ¼ 0 (8) > > : 1 if bðkÞ o0 n

PN ðkÞ is a signum (hard limiting) function, n¼1 sgnðbn Þ is the difference between actual quantities of winners and losers. A signum function carries out the comparison between a kth ðkÞ ðkÞ discrete value of nth output bn and zero. If bn 40, then the nth ðkÞ ðkÞ signum function gives output sgnðbn Þ ¼ 1 if bn ¼ 0, then the ðkÞ output of nth signum function is given by sgnðbn Þ ¼ 0, otherwise ðkÞ sgnðbn Þ ¼ 1. Consider a form of function RðxðkÞ Þ, for instance, for input vector a ¼ ð0:5; 0:3; 0:7; 0:9; 0:2; 0:8; 0:4; 0:1; 0:6Þ, i.e. for N ¼ 9 inputs and for K ¼ 1; 4; 5; 8 winners, where k ¼ 1; 2; . . . ; 500 which is shown in Fig. 1. As one can see, RðxðkÞ Þ is a scalar integer-valued stepping function. This function accepts zero values when shift x is located in range 0:8oxo0:9 for K ¼ 1, in range 0:5oxo0:6 for K ¼ 4, in range 0:4oxo0:5 for K ¼ 5 and in range 0:1oxo0:2 for K ¼ 8 which correspond to the desired quantities of winners in accordance with given inputs and Eq. (7). An equality RðxðkÞ Þ ¼ 0 is satisfied each time when inequalities cKþ1 oxðkÞ ocK hold for each K ¼ 1; 2; . . . ; N  1. Let us update a dynamic shift xðkÞ by the following recursive algorithm: xðkþ1Þ ¼ xðkÞ  ADxðkÞ ,

(9)

where DxðkÞ ¼ sgnðRðxðkÞ ÞÞak , a is an iteratively updated parameter that, as it is shown below, guarantees a convergence of the algorithm to a correct solution; 0pxð1Þ pA is an initial condition; m is a number of iterations until reaching a convergence of search process to a steady state. Let us describe the mathematical model of the circuit by difference equation (9) and equalities (6). According to (9), if a value of R(x) is less than zero, then the dynamic shift xðkÞ must be increased. In contrary, if a value of R(x) becomes greater than zero, xðkÞ should be decreased. In the steady state, when R(x) accurately equals to zero, xðkÞ must not be

R

0 K=1

-10 -20 0

0.1

0.2

0.3

0.4

0.5 x

0.6

0.7

0.8

0.9

1

0.9

1

0.9

1

0.9

1

10 R

0

K=4

-10 0

0.1

0.2

0.3

0.4

0.5 x

0.6

0.7

0.8

R

10 K=5

0 -10 0

0.1

0.2

0.3

0.4

0.5 x

0.6

0.7

0.8

R

20 10 K=8

0 0

0.1

0.2

0.3

0.4

0.5 x

0.6

0.7

0.8

Fig. 1. Graph of function RðxðkÞ Þ for N ¼ 9 inputs and K ¼ 1; 4; 5; 8 winners.

3193

changed further. If the K winners are desired, then such the model should progressively provide a proper shift xðkÞ , that gradually approaches and finally falls into the range between the Kth and the (K+1)st maximum values of signals cn, i.e. cKþ1 oxðkÞ ocK . Once the shift xðkÞ is in the range between cK and cKþ1 , then the outputs ðkÞ bn exactly provide the KWTA results. The most important issue developed in this paper is how to design and analyze a circuit, that can direct xðkÞ to fall into the range ðcKþ1 ; cK Þ. Existence and uniqueness of the steady state solutions. Let us analyze an existence and uniqueness of steady state solutions of difference equation (9). For this purpose formulate the following theorem. Theorem 1. The steady state KWTA solutions of difference equation (9) exist for any parameter 0oao1 and for each initial condition xð1Þ 2 ½0; A. Proof. The steady state solution xðmÞ of Eq. (9) can be given by xðmÞ ¼ xð1Þ  A

m 1 X

sgnðRðxðkÞ ÞÞak .

(10)

k¼1

The first term of right hand side of (10) is located in the range 0pxð1Þ pA. Consider the second term of this equation and the numerical row which presents a nonfinite geometric progression with a first term Aa0 and with a denominator a. If aa1, then a sum of first m þ 1 terms of such row be given by smþ1 ¼

A  Aamþ1 . 1a

(11)

If jajo1 and m ! 1, then amþ1 ! 0 yielding   A  Aamþ1 A Aamþ1 A lim smþ1 ¼ lim ¼ ¼ lim  . m!1 m!1 m!1 1  a 1a 1a 1a (12) ðkÞ

Since a variable sgnðRðx ÞÞ of the second term of Eq. (10) can be 1, 0, 1; therefore, the following inequalities hold: 

m X A A oA sgnðRðxðkÞ ÞÞak o 1a 1a k¼1

(13)

if 0oao1. Summing inequalities (13) with inequalities 0pxð1Þ pA results in 

A A oxðmÞ oA þ . 1a 1a

(14)

ðmÞ

Thus, a solution x of Eq. (9) can possess any finite value in range ðA=ð1  aÞ; A þ A=ð1  aÞÞ if 0oao1. Since inputs are distinct and located in the range [0,A], there exists such value of xðmÞ 2 ½0; A which satisfies inequalities (5). However range [0,A] is a sub region of range ðA=ð1  aÞ; A þ A=ð1  aÞÞ if 0oao1. Therefore, a solution xðmÞ can take on any finite value in range [0,A], including a value satisfying inequalities (4) and (5). This assures an existence of steady state KWTA solutions of difference equation (9) for each 0oao1 and for any initial condition xð1Þ 2 ½0; A. & Corollary. It follows from Theorem 1 and equalities (6), that the KWTA outputs bn ¼ cn  xðkÞ , n ¼ 1; 2; . . . ; N, given by Eq. (9) and equalities (6), exist. Note that a solution of Eq. (9) xðmÞ can take on any finite value in the range [0,A] that satisfies equality RðxÞ ¼ 0 This solution is not unique, since it can take on any finite value in range ðcKþ1 ; cK Þ. Therefore, values of outputs bk ¼ ck  x ðk ¼ 1; 2; . . . ; N  1Þ are not unique either. However, according to inequalities (5), the KWTA property of the model described by Eq. (9) and equalities (6) is determined by signs of outputs bn, n ¼ 1; 2; . . . ; N but not by their values. These signs are unchanged for any xðkÞ 2 ðcKþ1 ; cK Þ. Therefore, the model described by Eq. (9) and equalities (6)

ARTICLE IN PRESS 3194

P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

possesses KWTA property (2) for each 0oao1 and for any xð1Þ 2 ½0; A. Consider the steady state solution xðmÞ of Eq. (9) and output signals bn ¼ cn  x, n ¼ 1; 2; . . . ; N, described by equalities (6). If the components of input vector a are distinct, then the steady state solution xðmÞ of (9) and output signals determined by (6) possess the following two properties that makes designing corresponding KWTA neural circuit possible: (1) The steady state solution xðmÞ of Eq. (9) is such that inequalities (4) are satisfied. (2) The K largest components of input vector a correspond to the K positive components of output vector b and the N  K rest components of input vector a correspond to the N  K negative components of output vector b.

The right hand side of Eq. (9) is constant for each O. A steady state solution of (9) is a scalar xðmÞ 2 < such that xðmÞ 2 O. Let us consider a module of shift change jDxðkÞ j ¼ jxðkþ1Þ  xðkÞ j. Let us formulate the following lemma.

is monotonic non-increasing along each non-constant discretetime trajectory xðkÞ, k ¼ 1; 2; . . . ; m. It is bounded below, convex with zero minimum. Thus, function (16) converges to a limit WðxðkÞ Þ ¼ 0 and its change DWðxðkÞ Þ converges to zero. The change DWðxðkÞ Þ is strictly less than zero except the steady state where it vanishes. This fact assures that the minimum of function (16) is stable. A state variable xðkÞ is changed in a given limited range [0, A] until reaching a steady state value xðkÞ ¼ xðmÞ . Stability follows from the positive definiteness of WðxðkÞ Þ that is below bounded and from the fact that DWðxðkÞ Þo0, 8DxðkÞ a0. Consider a static part of Eq. (9), when the condition DxðkÞ ¼ 0 is met describing a static mode with stationary state satisfying Eq. (7). After reaching zero value by DxðkÞ, the trajectory of Eq. (9) passes in the static mode. A variable xðkÞ accepts finite value xðkÞ ¼ xðmÞ and is not changed further keeping an obtained value thereafter. Thus, for monotonically decreasing bounded below function (16), i.e. for DWðxÞp0 with DW ¼ 0 for Dx ¼ 0, when the solution of Eq. (9) is changed, it follows as a sufficient condition 0oao1, under which the solutions xðkÞ are Lyapunov stable and always convergent to a stable state that corresponds to a minimum of the function WðxðkÞ Þ. In other words, the model (9) is stable according to Lyapunov stability theory. &

Lemma 1. A function jDxðkÞ j, where k ¼ 1; 2; . . . ; m is monotonically decreasing discrete-time function for each 0oao1.

In order to analyze a convergence of difference equation (9) solutions let us formulate the following lemma.

3.2. Dynamics of the model states Let us define a certain set O as

O :¼ fxðkÞ : RðxðkÞ Þ ¼ 0g.

(15)

Proof. Using difference equation (9) and the condition 0oao1, a change jDxðkþ1Þ j  jDxðkÞ j can be expressed as jDxðkþ1Þ j  jDxðkÞ j ¼ jakþ1 sgnðRðxðkþ1Þ ÞÞj  jak sgnðRðxðkÞ ÞÞj ¼ ak ðajsgnðRðxðkþ1Þ ÞÞj  jsgnðRðxðkÞ ÞÞjÞ. According to (9) a variable sgnðRðxðkÞ ÞÞ can accept for each k ¼ 1; 2; . . . ; m one of the following three possible values: 1, 0 or 1. If sgnðRðxðkÞ ÞÞ ¼ 1, then a variable sgnðRðxðkþ1Þ ÞÞ can obtain value 1, 0 or 1. A change in jDxðkÞ j is jDxðkþ1Þ j  jDxðkÞ jp0 in this case, since ajsgnðRðxðkþ1Þ ÞÞjpjsgnðRðxðkÞ ÞÞjÞ for each 0oao1. In the case, when sgnðRðxðkÞ ÞÞ ¼ 0, a value of sgnðRðxðkþ1Þ ÞÞ can be only equal to zero. Therefore, in this case, (9) is reduced to xðkþ1Þ ¼ xðkÞ , since as it follows from (7) Rðxðkþ1Þ Þ ¼ RðxðkÞ Þ ¼ 0 for each kXm. Thus, in the case when sgnðRðxðkÞ ÞÞ ¼ 0 a change in jDxðkÞ j is equal to zero. However, it means that a change in jDxðkÞ j, where k ¼ 1; 2; . . . ; m is monotonically decreasing discrete-time function for each 0oao1. & Let us prove a stability of model (9) in the sense of Lyapunov. The following lemma can be stated. Lemma 2. The solutions of difference equation (9) under the condition 0oao1 are stable in the sense of Lyapunov and converge to finite values. Proof. Accomplish the stability analysis of dynamic part of difference equation (9) using Lyapunov direct method [14]. For this part of (9) the Lyapunov function can be given by ðkÞ

ðkÞ

Wðx Þ ¼ jDx j.

(16) ðkÞ

Since function (16) is changed from Wðx Þ to Wðx change in function (16) can be expressed as

DWðxðkÞ Þ ¼ Wðxðkþ1Þ Þ  WðxðkÞ Þ ¼ jDxðkþ1Þ j  jDxðkÞ j.

ðkþ1Þ

Þ, a resulting (17)

According to Lemma 1, a variable jDxðkÞ j, k ¼ 1; 2; . . . ; m is monotonically decreasing function of discrete-time for each 0oao1. Therefore, inequality jDxðkþ1Þ jpjDxðkÞ j is satisfied for each 0oao1. In other words, the change jDxðkþ1Þ j  jDxðkÞ j is negative semi-definite for each 0oao1. Therefore, function (16)

Lemma 3. If the condition 0:5oao1 is satisfied and inputs (3) are distinct, then the trajectories of states of difference equation (9) converge to the solution satisfying inequality (4) in finite number of iterations and remain there thereafter. Proof. In order to reach a steady state KWTA solution of Eq. (9) for any initial condition xð1Þ 2 ½0; A, a shift xðkÞ should be able to pass all the range [0,A]. Taking into account equality (10) this demand can be expressed as xðmÞ  xð1Þ ¼ A

m1 X

sgnðRðxðkÞ ÞÞak 4A.

(18)

k¼1

One can write using (11) that A

m 1 X

sgnðRðxðkÞ ÞÞak ¼ A

k¼1

m 1 X k¼1

ak ¼

A  Aam  A. 1a

Therefore, the following inequality can be written: A  Aam  A4A. 1a

(19)

Multiplying the inequality (19) by 1  a and taking the logarithm with base a for 0oao1 results in the bounding below for a required number of iterations: m4loga ð2a  1Þ.

(20)

The right hand side of (20) is finite if 2a  140; therefore, the inferior limit for a parameter a is a40:5. In order K winners to be found a variable xðkÞ should fall into the range ½cKþ1 ; cK . For this purpose a value of Aak should become smaller than this range. After the first iteration, the dynamic shift xðkÞ performs either addition or subtraction by Aak. Therefore, the shift accepts values xð1Þ ¼ Aa, xð2Þ ¼ Aað1  aÞ or xð2Þ ¼ Aað1 þ aÞ. After m1 iterations, the dynamic shift becomes P ðkÞ k ðkÞ xðmÞ ¼ xð1Þ  A m1 k¼1 sgnðRðx ÞÞa , where sgnðRðx ÞÞ can be +1 or 1 if the KWTA process has not converged. Thus, the searched P ðkÞ k k shifts include A m1 k¼1 sgnðRðx ÞÞa . When a value of Aa becomes

ARTICLE IN PRESS P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

3195

smaller than the range ½cKþ1 ; cK , then K winners can be found if a variable xðkÞ falls in this range. Consider difference equation (9), change (17) of (16) and the set O, defined by (15). For the solutions of (9), let us analyze firstly the case, when the trajectory has not reached some O. The aim is to show that there exists a scalar 40 such that DWðxðkÞ Þp   and that O is an invariant set. A change in DWðxðkÞ Þ ¼ jDxðkþ1Þ j  jDxðkÞ j ¼ Aakþ1 jsgnðRðxðkþ1Þ ÞÞj  Aak jsgnðRðxðkÞ Þj ¼ Aak ða  1ÞoAam ða  1Þ ¼ o0 for any RðxðkÞ Þa0 and for each 0oao1, since m is a finite number. Hence, WðxðkÞ Þ converges to zero after finite number of iterations m. Therefore, we conclude that the trajectory of solutions of difference equation (9) converges to the set O after finite quantity of iterations m. It remains to show that this trajectory stays in O, i.e. that O is an invariant set. If xðkÞ 2 O, then WðxÞ ¼ 0 and DWðxÞ ¼ 0. If for some tðkÞ a variable xðkÞ lives D, then DWðYÞo0 and WðxðkÞ Þ40, that is a contradiction, since WðxðkÞ Þ is non-increasing along the discrete-time trajectory of Eq. (9). Thus, the trajectory of solutions of (9) reaches O and remains in this set. &

2K  2N þ 2o0 and so on, only if equality Rðxðqþ1Þ Þ ¼ 0 is not satisfied, i.e. if KaN  1. Otherwise, if K ¼ N  1 and therefore Rðxðqþ1Þ Þ ¼ 0, then Eq. (9) is transformed to xðkþ1Þ ¼ xðkÞ and the shift xðkÞ is not changing thereafter accepting constant value xðkÞ ¼ xðmÞ . As a result of such dynamics the function RðxðkÞ Þ gradually accepts not positive values 2K  2N; 2K  2N þ 1; 2K  2N þ 2; . . . ; 0. Therefore, the function jRðxðkÞ Þj ¼ RðxðkÞ Þ, k ¼ 1; 2; . . . ; m in this case is monotonically decreasing to zero for each 0oao1. Thus, the energy function jRðxðkÞ Þj, k ¼ 1; 2; . . . ; m is monotonically decreasing to zero discrete-time function for each 0pxð1Þ pA, and for any 0oao1. &

Consider a dynamics of the model described by difference equation (9) and equalities (6). Let us formulate the following lemma.

The right hand side of (21) is finite for each cK  cKþ1 a0, i.e. for each distinct inputs. The algorithm described by difference equation (9) and equality (6) has the logarithmic convergence speed, as it follows from (21). Note, that if it is necessary to identify the K winners in the case when the kth largest and (k+1)th largest signals are hard to discriminate, i.e. if the range ½cKþ1 ; cK  is very small, then a number of iterations m needed for a variable xðkÞ to fall into this range will be finite, but it can be very large. However, the property of monotony and decreasing to zero of discrete-time energy function jRðxðkÞ Þj commonly with the logarithmic convergence speed of the algorithm described by difference equation (9) and equality (6), yield in finding K1 winners after much less than m quantity of iterations if the gap ½cK ; cK1  is not very narrow. Using (21), it can be stated, that to reach a condition RðxðkÞ Þ ¼ 0 in not more than m iterations for any given inputs an ; n ¼ 1; 2; . . . ; N, a value of decaying coefficient rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi c  cKþ1 (22) a4 m K A

Lemma 4. The energy function jRðxðkÞ Þj, where k ¼ 1; 2; . . . ; m is monotonically decreasing to zero discrete-time function for each 0oao1 and for any 0pxð1Þ pA. Proof. Let the initial point be xð1Þ ¼ A. Then the outputs accept ð1Þ values bn ¼ cn  xð1Þ o0 for each n ¼ 1; 2; . . . ; N and the function P ð1Þ ðkÞ Rðx Þ gets value 2K  N  N n¼1 sgnðbn Þ ¼ 2K40. Therefore, the ðkþ1Þ ðkÞ k ¼ x  Aa holds in this point and the difference equation x shift xðkÞ will be exponentially decreased. At the point xðlÞ ¼ c1 , where c1 is a maximal input in accordance with (3), a value of ðlÞ ðlÞ output b1 ¼ c1  xðlÞ ¼ 0 and values of outputs bj o0, j ¼ 2; 3; . . . ; N according to (5) and (6). Therefore, the function RðxðkÞ Þ P ðlÞ obtains value 2K  N  N n¼1 sgnðbn Þ ¼ 2K  140 and the shift xðkÞ will continue exponentially decreasing according to equation xðkþ1Þ ¼ xðkÞ  Aak . At the interval c1 4xðlþ1Þ 4c2 , where c2 is a ðlþ1Þ ¼ c1  xðlþ1Þ 40 and the second maximal input, the output b1 ðlþ1Þ ðlþ1Þ ¼ cj  x o0, j ¼ 2; 3; . . . ; N. In this interval, rest of outputs bj P ðlþ1Þ Þ ¼ 2K  the function RðxðkÞ Þ gets value 2K  N  N k¼1 sgnðbn 240 and so on, if only equality Rðxðlþ1Þ Þ ¼ 0 is not satisfied, i.e. if Ka1. Otherwise, if K ¼ 1 and therefore Rðxðlþ1Þ Þ ¼ 0, then Eq. (9) accepts a form xðkþ1Þ ¼ xðkÞ and the shift xðkÞ is not changed thereafter keeping constant value xðkþ1Þ ¼ xðmÞ . As a result of such dynamics, the function RðxðkÞ Þ sequentially obtains nonnegative values 2K; 2K  1; 2K  2; . . . ; 0. This means that the function jRðxðkÞ Þj ¼ RðxðkÞ Þ, k ¼ 1; 2; . . . ; m in this case is monotonically decreasing to zero for each 0oao1. If the initial point is xð1Þ ¼ 0, then the outputs obtain values ð1Þ bn ¼ cn  xð1Þ 40, n ¼ 1; 2; . . . ; N. The function RðxðkÞ Þ gets value P ð1Þ 2K  N  N n¼1 sgnðbn Þ ¼ 2K  2No0. Therefore, the difference equation (9) accepts in this point takes the form xðkþ1Þ ¼ xðkÞ þ Aak and the shift xðkÞ will be exponentially increasing. At the point xðqÞ ¼ cN , where cN is a minimal input according to (3), an output ðqÞ ðqÞ ðqÞ bN is bN ¼ cN  xðqÞ ¼ 0 and outputs bj , j ¼ 1; 2; . . . ; N  1 are ðqÞ bj 40 according to (5) and (6). Therefore, the function RðxðkÞ Þ runs P ðqÞ up value 2K  N  N n¼1 sgnðbn Þ ¼ 2K  2N þ 1o0 and the shift xðkÞ continues to increase exponentially according to equation xðkþ1Þ ¼ xðkÞ þ Aak . At the range cN oxðqþ1Þ ocN1 , where cN1 is a ðqþ1Þ ¼ cN  xðqþ1Þ o0 and the second minimal input, the output bN ðqþ1Þ ðqþ1Þ ¼ cj  x 40, j ¼ 1; 2; . . . ; N  1. In this range rest outputs bj P ðqþ1Þ Þ¼ the function RðxðkÞ Þ runs up value RðxÞ ¼ 2K  N  N k¼1 sgnðbn

To guarantee that the model described by difference equation (9) and equalities (6) find the K winners, the inequality Aam ocK  cKþ1 should be satisfied. By taking the logarithm of last inequality to the base a for 0:5oao1, one can obtain the following inferior limit for a number of iterations for the trajectories of Eq. (9) solutions to achieving the convergence: m4loga

cK  cKþ1 . A

(21)

should be set. A repetition period of inputs an ; n ¼ 1; 2; . . . ; N which can be processed correctly by the model described by difference equation (9) and equalities (6) should satisfy inequality T4loga

cK  cKþ! t, A

(23)

where t ¼ t ðkþ1Þ  t ðkÞ is a sampling period. Assume that all inputs are randomly distributed in [0,A]. As it was shown above, if condition 0:5oao1 is satisfied and inputs are distinct, then it can be assured that the exponential search can reach any value from [0,A] for each xð1Þ 2 ½0; A in finite number of iterations. From order statistics [4,18], the expected value of cKþ1 from N uniformly distributed inputs denoted by mKþ1;N for 0pcKþ1 pA can be determined as

mKþ1;N ¼

ðN  KÞA . Nþ1

(24)

By the same reasoning, the expected Kth maximum input is mK;N ¼ ðN  K þ 1ÞA=N þ 1. The difference in the expected Kth maximum input and the expected (K+1)st maximum input is mK;N  mKþ1;N ¼ A=N þ 1. If a module of DxðkÞ becomes smaller than the expected difference such that the search points can fall into the range ðmKþ1;N ; mK;N Þ, the convergence is assured in average sense. Replacing in (21) cK  cKþ1 with A=N þ 1, the expected

ARTICLE IN PRESS 3196

P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

number of iterations to achieve a convergence for the uniformly distributed inputs can be determined as m4loga

1 . Nþ1

(25)

For the uniformly distributed inputs, the expected number of iterations to achieve the convergence is reduced with reducing a problem scale N, since   d 1 1 40 loga ¼ dN Nþ1 ðN þ 1Þ ln a for each 0:5oao1, N40. On the other hand, for the uniformly distributed inputs, the expected quantity of iterations required for the convergence is decreased with decreasing a value of parameter a, since   d 1 aðlnðN þ 1ÞÞ3 loga 40. ¼ da Nþ1 ðln aÞ2 Eventually, for the uniformly distributed inputs, the expected number of iterations for the convergence is not dependent on the number of winners, since d=dKðloga 1=N þ 1Þ ¼ 0. In the case when the inputs are randomly selected from the other distribution, the number of iterations required for the model convergence dependent on the expected resolution of ðmKþ1;N ; mK;N Þ can be given by m4loga

mK;N  mKþ1;N A

,

(26)

where mKþ1;N and mK;N are the statistical averages. Note, that algorithm (9) can be considered as the exponential search for determining the dynamic shift xðkÞ . This algorithm can be simplified by replacing in (9) ak on a, where a parameter a should satisfy inequality aor=A to ensure falling a shift x into the range ðxðkÞ ; xðkþ1Þ Þ. The algorithm obtained in such a way will be a particular case of (9). The procedure of determination of a shift xðkÞ in this case is a linear search process resulting in a slow convergence speed. It can be stated on the basis of the above derivation that there exists such t ðmÞ 40 that inequalities ðkÞ

ðkÞ

ðkÞ

ðkÞ

b1 ðt Þ4    4bK ðt Þ404bKþ1 ðt Þ4    4bN ðt Þ

(27)

are satisfied for each t ðkÞ 4tðmÞ , where bn ðt ðkÞ Þ ¼ cn  xðt ðkÞ Þ, n ¼ 1; 2; . . . ; N, since a number of winners K ¼ 1; 2; . . . ; N  1 and an input vector a is ordered. This means that a variable xðkÞ of Eq. (9) starts from an initial value xð1Þ , and it finishes in the state xðmÞ that corresponds to the components of output vector b split into positive and negative plane in accordance with (2). At any time after t ðmÞ, the outputs, given by Eq. (9) and equalities (6), possess the KWTA property. Note, that for any k40 inequalities (27) describe signal order preserving property, i.e. for any given inputs an ; n ¼ 1; 2; . . . ; N, the ordering of outputs is the same as the ordering of inputs. Such ðkÞ property is evident, since the condition bn ¼ cn  xðkÞ ¼ an  ðkÞ Amin  x , n ¼ 1; 2; . . . ; N is satisfied for each t ðkÞ X0. In other words, outputs are equal to inputs minus the dynamic shift which is the same for all inputs; therefore, it does not change the ordering of signals. This leads to satisfying inequalities (27), since inequalities (1) hold. Thus, the ordering of the model outputs and the ordering of corresponding inputs are the same for each xðkÞ 2 ½0; A. The results derived in this section, based on Lemmas 1–3, ensure that difference equation (9) and equalities (6) describe the model of discrete-time K-winners-take-all circuit. This fact can be summarized as the following theorem. Theorem 2. Let a decaying coefficient a of difference equation (9) satisfies bound 0:5oao1. Then, this equation and equalities (6)

describe the model of discrete-time KWTA neural circuit for a given input vector a 2 <, with distinct components, and a positive integer 1pKoN. Perturbations of nonlinearities. For difference equation (9), it is natural to consider small changes of its right hand side, specifically for the solution of this equation if perturbations in ðkÞ the signum nonlinearities sgnðRðxðkÞ ÞÞ and sgnðbn Þ occur. Such perturbations can lead to deviation of the nonlinearities from its nominal values and it is necessary to provide conditions under which algorithm (9) will be robust to such perturbations. Consider the case when these nonlinearities are not ideal, but vary discretely from 1 to 1 over a small interval. Determine maximal ðkÞ allowable errors of functions sgnðRðxðkÞ ÞÞ and sgnðbn Þ under which these functions still operate properly. Let a perturbed function sgnðRðxðkÞ ÞÞ be given by 8 if RðxðkÞ Þ40; > <1 þ  if RðxðkÞ Þ ¼ 0; (28) psnðRðxðkÞ ÞÞ ¼ 0 > : 1 þ  if RðxðkÞ Þo0: As one can see from the stability results derived above a value of e can be limited in range jjo1 to guarantee a robustness of nonlinearity sgnðRðxðkÞ ÞÞ. ðkÞ Let us describe a perturbed function sgnðbn Þ as 8 ðkÞ 1þs if bn 40; > > < ðkÞ ðkÞ if bn ¼ 0; psnðbn Þ ¼ 0 (29) > > : 1 þ s if bðkÞ o0: n

ðkÞ

In order to round values of function sgnðbn Þ correctly, a value of s must be constrained in range jsjo0:5=ðN  1Þ. Note, that the results derived above are valid for sets of timeconstant inputs (1), if its repetition period is limited by inequality (26). However, the model described by difference equation (9) and equalities (6) can be generalized for applying in the case of timevarying input signals an ðt ðkÞ Þ ¼ aðkÞ n , n ¼ 1; 2; . . . ; N. In this case condition ðkþ1Þ  aðkÞ  xðkÞ j jaðkþ1Þ n n j5jx

(30)

should be satisfied for each k ¼ 1; 2; . . . ; m and a shift change DxðkÞ of Eq. (9) must be updated by equality DxðkÞ ¼ sgnðRðxðkÞ ÞÞaðkÞ, ( aðkÞ að1Þ if RðxðkÞ Þa0; ðkþ1Þ a ¼ að1Þ if RðxðkÞ Þ ¼ 0: m1  aðkÞ  am should be This means that inequality jaðkþ1Þ n n j5a satisfied for each 0oao1, k ¼ 1; 2; . . . ; m, n ¼ 1; 2; . . . ; N. The energy function jRðxðkÞ Þj, where k ¼ 1; 2; . . . ; m, will be monotonically decreasing to zero discrete-time function for each 0oao1 and 0pxð1Þ pA. In other words, in this case output signals of the model should be changed much more faster than input signals during transients. Then, the model described by difference equation (9) and equalities (6) will be functioning correctly as the model of KWTA neural circuit in the case of time-varying signals. Computer simulations of performance of the model described by difference equation (9) and equalities (6) in the case of time-varying input signals are given in Section 5. Note, that the problem of application of neural circuit described by difference equation (9) and equalities (6) for time-varying signal processing is the subject of our further investigations.

3.3. Digital implementation of the circuit The functional block-scheme of discrete-time dynamical KWTA (DTDKWTA) neural circuit described by Eq. (9) and equalities (6) is presented in Fig. 2, where xðkÞ ¼ xððk  1ÞtÞ is an input signal

ARTICLE IN PRESS P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

dynamic shift at the discrete-time moment t ðkÞ ¼ ðk  1Þt. The scheme contains blocks of discrete-time summing S, multiplication  , integration z1 , signum function sgn (hardlimiter), external sources of constant signals A; 2K; N; xð1Þ and external source of controlled signal ak . Thus, the DTDKWTA circuit can be implemented in a modern hardware using such traditional digital components as adders, hard-limiting quantizers (switches), digital integrator, and external sources of voltage or current. Note, that output signals of N blocks sgn could be also used as outputs b of the circuit. However, in this case the only number K of winners from N inputs would be determined. No information concerning ordering of inputs which can be used further, for instance, for solving problems of sorting, classification, clustering, etc. would be available. As one can see, from a hardware implementation complexity point of view, the proposed circuit for processing input signals located in range a 2 ð0; 1Þ should contain one multiplier, N þ 2 adders, N+1 hard-limiting quantizers, one digital integrator, three external sources of constant signals and one external source of variable signal. Note that in particular case of algorithm (9) when ak in DxðkÞ is replaced with a, then an external source of variable signal is substituted by a source of constant signal that additionally simplifies the circuit. For comparison, one of the most simple and fast DTKWTA network [18] for processing such inputs requires two multipliers, N þ 4 summers, N þ 2 hardlimiting quantizers, one digital integrator, five external sources of constant signals and one external source of variable signal. One of the most simple continuous-time analog from [6] needs N amplifiers, N+1 adders, one integrator, N hard-limiting quantizers, and two external sources of constant signals. Thus, proposed circuit has slightly less hardware implementation complexity than other analogs. From a computational complexity point of view the DTDKWTA circuit in the case of input signals, located in range a 2 ð0; 1Þ requires a consecutive performing two multiplications, N þ 4 additions/subtractions, two logic hard-limiting operations and one integrating operation on each iteration. If in algorithm (9) ak is replaced with a, then the DTDKWTA circuit requires only one multiplication instead of two. Competitive DTKWTA network requires in this case, a consecutive fulfilling of two multiplica-

(k) (k) b 1 b2

a1 a2

+ _

(k) bN N

sgn

Σ _

_

+ _

Σ

sgn

_

_

_

aN

+

sgn

Σ _ _

amin

αk

x(1) x(k) z

−1

x(k+1)

+

Σ

tions, 3N þ 2 additions/subtractions, two signum function operations and one integrating operation for each iteration. Therefore, the computational time of the DTDKWTA circuit is less than that of the DTKWTA, CHKWTA [10] and IAKWTA [17] networks. A resolution ability of the circuit is infinite theoretically, i.e. if inputs are distinct, then the circuit can always identify them according to KWTA property (2). Practical resolution ability can be limited by precision of the circuit hardware implementation. Since proposed circuit can process correctly any distinct inputs, the accuracy performance of the circuit is the same as that of other analogs. A functioning of the circuit is independent on initial condition xð1Þ which can accept arbitrary value in range [0,A]. Therefore, the circuit does not require periodical resetting xð1Þ for repetitive signal processing, additional supervisory circuit for resetting, and spend additional processing time on this operation [9,11,12,18]. This allows to simplify the hardware and increase a speed of signal processing that is important for real time operation. An important advantage of proposed KWTA neural circuit is that in contrast to other analogs [6,9,18] it possesses an order preserving property of input signals.

4. Equality of inputs of the circuit Consider the case when two or more inputs of the circuit are equal one to other. If equal inputs are located among K winners or among N  K losers, then outputs of the circuit described by Eq. (9) and equalities (6) satisfy KWTA property (2) in the steady states. However, if the circuit should distinguish equal inputs and split them into positive and negative planes when these inputs are located among winners and their quantity is greater than K or these inputs are placed among losers and its number is greater than N  K, then various spurious trajectories of the circuit outputs which do not have KWTA property will be obtained. Specifically, dynamic shift xðkÞ of the circuit inputs will demonstrate decaying oscillations around the point ae  Amin of equal inputs. Therefore, outputs bn of the circuit will have decaying oscillations around the points an  Amin  x. In other words, outputs corresponding to equal inputs will have decaying oscillations around zero values and the rest outputs will have decaying oscillations in positive or negative plane. Thus, it can be concluded that if the K largest inputs of vector a contain 1oppK equal inputs or if the N  K smallest inputs of vector a include K þ 1oqpN equal inputs, then it may be guaranteed that the circuit described by Eq. (9) and equalities (6) will always identify them in accordance with KWTA property (2).

R(x(k))

Σ

2K +

3197

sgn

+ A Fig. 2. Functional block diagram of iterative KWTA neural circuit described by difference equation (9) and equality (6).

5. Simulation results In order to illustrate the theoretical results, presented in this paper, consider some examples with corresponding computer simulations which demonstrate signal processing by proposed discrete-time model of KWTA neural circuit. The first simulation will be for the case of unequal inputs. Example 1. Let it be necessary to identify the four largest inputs, that is K ¼ 4, from vector a ¼ ½1:1; 1:7; 0:3; 1:8; 0:2, i.e. N ¼ 5 having been used the model described by difference equation (9) and equalities (6). Let us take for this model Amin ¼ 2, A ¼ 4, an initial condition xð1Þ ¼ A and a decaying coefficient a ¼ 0:6 chosen from the bound of Lemma 3. Let us determine the discrete-time ðkÞ trajectories of the shift xðkÞ and the outputs bi ; i ¼ 1; 2; 3; 4; 5 according to difference equation (9) and equalities (6). Such

ARTICLE IN PRESS P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

4

10

3

8 Trajectories x and b1 to b5

Trajectories x and b1 to b5

3198

2 1 0 −1 x b1 b2 b3 b4 x b5

−2 −3 −4 1

2

3

4 k

5

6

2 0 −2 −4 −6

−10 1

7

6

0.5

5

0

4 x(k+1)−x(k)

−0.5 −1

−2

0

−2.5 2

2.5

3

3.5

5

6 k

7

8

9

10

11

2 1

1.5

4

3

−1.5

1

3

Fig. 5. The discrete-time trajectories of the shift xðkÞ and the outputs bi ; i ¼ 1; 2; 3; 4; 5 of the model described by difference equation (9) and equalities (6) in the case of equal inputs—Example 2.

1

0.5

2

ðkÞ

ðkÞ

x(k+1)−x(k)

4

−8

Fig. 3. The discrete-time trajectories of the shift xðkÞ and the outputs bi ; i ¼ 1; 2; 3; 4; 5 presenting KWTA property of the model described by difference equation (9) and equalities (6)—Example 1.

0

x b1 b2 b3 b4 b5

6

4

x(k)

−1 0

1

2

3

4

5 x

6

7

8

9

10

(k)

Fig. 4. The phase curve of the discrete-time trajectory of the shift xðkÞ ; k ¼ 1; 2; . . . ; m of Example 1.

Fig. 6. The phase curve of the shift xðkÞ ; k ¼ 1; 2; . . . ; 1 of Example 2.

trajectories in normalized units are shown in Fig. 3. As one can see, in the steady state, the components b1 40; b2 40; b3 40, b5 40 correspond to the four largest components of vector a-the winners and the remaining one, b4 o0 corresponds to the loser according to KWTA property (2). A convergence of the search process to a steady state is achieved in m ¼ 5 iterations in accordance with valuing (23). The phase curve of the discrete-time trajectory of the shift xðkÞ ; k ¼ 1; 2; . . . ; m is presented in Fig. 4. As it can be seen, the phase curve of a xðkÞ is the finite twisting spiral of piecewise-linear form that assures a stable dynamics of the shift xðkÞ . The shift change DxðkÞ ¼ xðkþ!Þ  xðkÞ demonstrates jumps in the beginning and in the end of each horizontal sector. Now we will demonstrate functioning of the proposed model of discrete-time KWTA neural circuit in the case of equal inputs.

xð1Þ ¼ 0. The discrete-time trajectories of the shift xðkÞ and the ðkÞ outputs bi ; i ¼ 1; 2; 3; 4; 5 are shown in Fig. 5, where xðkÞ accomplishes infinite decaying oscillations around the point ðkÞ x ¼ 9. The outputs bi ; i ¼ 1; 2; 3; 4; 5 fulfill infinite decaying oscillations around the point bi ¼ ai  Amin  x, i.e. the components b1 ; b3 ; b4 are imposed one on another. The phase curve of the shift xðkÞ ; k ¼ 1; 2; . . . ; 1 is depicted in Fig. 6. This phase curve is the infinite twisting spiral of piecewiselinear form that guarantees a stable dynamics of the shift xðkÞ . A change xðkþ!Þ  xðkÞ of xðkÞ ; k ¼ 1; 2; . . . ; 1 has jumps in the beginning and in the end of each horizontal sector. Let us compare the KWTA performance of well-known KWTA networks, specifically MAXNET, IAKWTA, CHKWTA, DTKWTA networks [8,18] and DTDKWTA neural circuit.

Example 2. Let us try to extract two largest inputs, that is K ¼ 2 from vector a ¼ ½4; 1; 4; 4; 5 by proposed model of KWTA neural circuit for parameters a ¼ 0:5, Amin ¼ 5, A ¼ 10 and

Example 3. Let the inputs be uniformly distributed in [0,1] and randomly generated by Monte Carlo experiments. Let the parameters a ¼ 0:5; C ¼ 50 be used for the CHKWTA network. In the IAKWTA network, let us choose X min ¼ 0:428571; X max ¼

ARTICLE IN PRESS P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

3.5

3.5

3

3 2.5 IAKWTA CHKWTA DTKWTA DTDKWTA

2

log10m

2.5 log10m

3199

MAXNET DTDKWTA

2

1.5

1.5

1

1 0.5

0.5 10

20

30

40

50

60

70

80

90

200 400 600

100

N

80 α=0.55 α=0.65 α=0.75 α=0.85 α=0.95

70 60 50 m

þ1; g ¼ 0:02; and take Ei ¼ 0. Set for the DTKWTA network parameters is m ¼ 0:5, M k ð0Þ ¼ 0:5, and d ¼ 0:5. Let us choose for the DTDKWTA circuit parameters a ¼ 0:5, x0 ¼ 0:5 and A ¼ 1. The finite-difference equations instead of differential equations to realize the CHKWTA network iteratively are used for correct comparison. Sampling period of difference equation (9) is Dt ¼ 1. Let us simulate 1000 independent runs. The average numbers of iterations required for the convergence of three WTA by DTDKWTA circuit and other KWTA networks for various quantities of inputs are shown in Fig. 7. The completion of convergence for the IAKWTA network is defined as identifying K neurons with maximum activation values which are greater than the threshold X min þ X max =2. Similarly, for the CHKWTA network, the threshold is chosen as X min þ X max =2. The DTKWTA network and the DTDKWTA circuit attain K neurons with greater inputs and (NK) neurons with smaller inputs for the completion of convergence. From Fig. 7, the CHKWTA and IAKWTA networks are converged nearly proportional to the number of inputs. The DTKWTA network and the DTDKWTA circuit achieve the logarithmic convergence in accordance with the theoretical predictions. For each network, the average number of iterations required for the convergence rises if a quantity of inputs is increased. According to simulation results the DTDKWTA circuit has averaged number of iterations required for the completion of convergence much less than that of the CHKWTA and IAKWTA networks and near to that of the DTKWTA network. Fig. 8 shows the average number of iterations required for the convergence of well-known WTA network MAXNET and DTDKWTA circuit for K ¼ 1. As one can see the average number of iterations of DTDKWTA circuit is much less than that of MAXNET especially for large N. For the case of uniformly distributed inputs if N ¼ 100, Fig. 9 shows the average quantities of iterations required for the convergence of DTDKWTA circuit for A ¼ 10 and for different values of parameter a if various numbers of winners are identified. As one can see from Fig. 9, the average number of iterations, required for the convergence falls, if a value of a decreases in accordance with theoretical predictions. The convergence speed of DTDKWTA circuit is minimal and

Fig. 8. Simulation results of Example 3, showing averaged numbers of iterations required for convergence of MAXNET neural network and DTDKWTA neural circuit for K ¼ 1.

40 30 20 10 0 10

20

30

40

50 K

60

70

80

90

Fig. 9. Simulation results of Example 3, presenting averaged numbers of iterations required for convergence of DTDKWTA neural circuit for N ¼ 100 and for various values of K and a.

60

α=0.55 α=0.65 α=0.75 α=0.85 α=0.95

50 40 m

Fig. 7. Simulation results of Example 3, presenting averaged numbers of iterations required for convergence of IAKWTA, CHKWTA, DTKWTA neural networks and DTDKWTA neural circuit for K ¼ 3.

800 1000 1200 1400 1600 1800 2000 N

30 20 10 0 10

20

30

40

50

60

70

80

90

100

N Fig. 10. Simulations of Example 3, showing averaged numbers of iterations needed for convergence of DTDKWTA neural circuit for K ¼ 5 and for different values of N and a.

ARTICLE IN PRESS 3200

P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

independent on the number of desired winners if a value of a is close to 0.5. Fig. 10 demonstrates average numbers of iterations required for the convergence of DTDKWTA circuit for various quantities of uniformly distributed inputs and for different values of parameter a if the number of desired winners is K ¼ 5 and A ¼ 10. As it can be observed, the average quantity of iterations falls if the number of inputs decreases. The average number of iterations is reduced if a value of parameter a is decreased according to theoretical predictions.

10 9 8

m

7 6 5 TNI UDRI NDRI

4 3 50

100

150

200

250 N

300

350

400

450

500

b3

b2

b1

x

a

Fig. 11. Theoretical numbers of iterations (TNI) and averaged simulated numbers of iterations required for convergence of DTDKWTA neural circuit with various numbers of uniformly distributed random inputs (UDRI) and with normally distributed random inputs (NDRI)—Example 3.

With a number of desired winners K ¼ 5, Fig. 11 shows that for random inputs, chosen from a uniform distribution on the interval [0,1], the simulation results are close to the theoretical convergence performance of DTDKWTA circuit given in (23). If the inputs are random and chosen from a normal distribution with mean 0.5, variance 0.5 and standard deviation 0.5, then the averaged numbers of iterations required for the convergence are less than that obtained by the theoretical predictions. The convergence speed decreases slowly with raising the quantity of inputs. Example 4. Consider a set of three discrete-time sinusoidal input signals with following instantaneous values: aðkÞ i ¼ 1:2 sin½2pð0:001k þ 2ði  1ÞÞ (i ¼ 1; 2; 3, i.e. N ¼ 3). Such input signals are time-varying and the corresponding KWTA problem is also time-varying. Fig. 12 illustrates the dynamics of these input signals, shift xðkÞ and corresponding output signals of model described by difference equation (9) and equalities (6) with K ¼ 2, Amin ¼ 1:2, A ¼ 2.4, að1Þ ¼ 0:7 and xð1Þ ¼ 0. According to Fig. 12, a shift xðkÞ has two updating (or searching) processes with duration which is less than a period of oscillations of input signals. Constraint (30) is satisfied for each k ¼ kj ; kj þ 1; . . . ; kj þ mj , where j ¼ 1,2 is a number of updating process. Two largest input signals are identified in each discrete-time point kakj ; kj þ 1; . . . ; kj þ mj . The dynamics of given three input signals, shift xðkÞ and corresponding output signals of the model described by difference equation (9) and equalities (6), where DxðkÞ ¼ sgnðRðxðkÞ ÞÞa, a ¼ 0:1 are presented in Fig. 13. As one can see, a trajectory of shift xðkÞ has an updating part in the neighborhood of discrete-time point k ¼ 9000 and three parts with a phenomenon of chattering with duration which is less than a period of oscillations of input signals. The chattering of shift xðkÞ exists in discrete-time points of trajectory xðkÞ where condition jDxðkÞ j ¼ jxðkþ1Þ  xðkÞ joaK  aKþ1

a1 a2 a3

1 0 −1 100

200

300

400

500 k

600

700

800

900

1000

100

200

300

400

500 k

600

700

800

900

1000

100

200

300

400

500 k

600

700

800

900

1000

100

200

300

400

500 k

600

700

800

900

1000

100

200

300

400

500 k

600

700

800

900

1000

1 0.5 0

2 1 0

2 1 0

2 1 0

Fig. 12. Plots of inputs and corresponding outputs of KWTA neural circuit of Example 4 with DxðkÞ ¼ sgnðRðxðkÞ ÞÞaðkÞ , where a ¼ 0:7.

ARTICLE IN PRESS 3201

a1 a2 a

1 0 −1

3

1000

2000

3000

4000

5000 k

6000

7000

8000

9000

10000

1000

2000

3000

4000

5000 k

6000

7000

8000

9000

10000

1000

2000

3000

4000

5000 k

6000

7000

8000

9000

10000

1000

2000

3000

4000

5000 k

6000

7000

8000

9000

10000

1000

2000

3000

4000

5000 k

6000

7000

8000

9000

10000

0.8 0.6 0.4

b3

b2

b1

x

a

P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

2 1 0

2 1 0

2 1 0

Fig. 13. Simulation results of KWTA neural circuit of Example 4 with DxðkÞ ¼ sgnðRðxðkÞ ÞÞa, where a ¼ 0:1.

is violated in each discrete-time point k ¼ kl ; kl þ 1; . . . ; kl þ ml , where l ¼ 1,2,3,4 is a number of updating or chattering processes. Two largest input signals are determined in each discrete-time point kakl ; kl þ 1; . . . ; kl þ ml . Thus, the simulation results show that proposed model of KWTA neural circuit can effectively determine largest signals not only from time-constant but also from time-varying signals in real time. In other words, the simulation results demonstrate good matching with the theoretical predictions. 6. Concluding remarks In this paper, new discrete-time K-winners-take-all neural circuit designed on the basis of input signal dynamic shifting is proposed. The circuit is stable and convergent to KWTA operation in finite number of iterations. The results of comparison testify that the circuit achieves the speed of signal processing not lower than other analogs. The hardware implementation complexity of the circuit is slightly less than that of other KWTA networks. The circuit can process any finite value distinct signals and possesses signal order preserving property. Periodical resetting and corresponding supervisory circuit for repetitive signal processing are not necessary. Proposed circuit can be better to use if it is necessary to have simple KWTA circuit with a high resolution ability, high speed of signal processing of wide range, independency on initial conditions and signal ordering preserving property. Thus, the proposed circuit is suitable for various applications.

Acknowledgments The author would like to thank each of the reviewers, editors, Prof. E. Kaszkurewicz, Prof. A. Bhaya and L.V. Ferreira

for their helpful comments and suggestions for improving the paper.

References [1] L.N. Bihn, H.C. Chong, A neural-network contention controller for packet switching networks, IEEE Transactions on Neural Networks 6 (1995) 1402–1410. [2] B.D. Calvert, C.A. Marinov, Another K-winners-take-all analog neural network, IEEE Transactions on Neural Networks 4 (2000) 829–838. [3] A. Cichocki, R. Unbehauen, Neural Networks for Optimization and Signal Processing, Wiley, New York, 1993. [4] H.A. David, Order Statistics, second ed., Wiley, New York, 1980. [5] J.J. Hopfield, Neurons with graded response have collective computational properties like those of two-state neurons, in: Proceedings of the National Academy of Sciences, vol. 81, USA, 1984, pp. 3088–3092. [6] X. Hu, J. Wang, An improved dual neural network for solving a class of quadratic programming problems and its k-winners-take-all application, IEEE Transactions on Neural Networks 19 (2008) 2022–2031. [7] T.M. Kwon, M. Zervakis, A parallel sorting network without comparators: a neural-network approach, in: Proceedings of International Joint Conference on Neural Networks, vol. 1, 1992, pp. 701–706. [8] R.P. Lippmann, B. Gold, M.L. Malpass, A comparison of Hamming and Hopfield neural nets for pattern classification, MIT Lincoln Laboratory Technical Report TR-769, 1987, pp. 1–37. [9] S. Liu, J. Wang, A simplified dual neural network for quadratic programming with its KWTA application, IEEE Transactions on Neural Networks 17 (2006) 1500–1510. [10] E. Majani, R. Erlanson, Y. Abu-Mostafa, On the K-winners-take-all network, in: D.S. Touretzky (Ed.), Advances in Neural Information Processing Sysems, vol. 1, Kaufmann, San Mateo, 1989, pp. 634–642. [11] C.A. Marinov, B.D. Calvert, Performance analysis for a K-winners-take-all analog neural network: basic theory, IEEE Transactions on Neural Networks 14 (2003) 766–780. [12] C.A. Marinov, J.J. Hopfield, Stable computational dynamics for a class of circuits with 0(N) interconnections capable of KWTA and rank extractions, IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 52 (2005) 949–959. [13] R. Perfetti, On the robust design of k-winners-take-all networks, IEEE Transactions on Circuits and Systems II: Analog and Digit. Sign. Process CAS-42 (1995) 55–58.

ARTICLE IN PRESS 3202

P.V. Tymoshchuk / Neurocomputing 72 (2009) 3191–3202

[14] S.K. Persidskii, . Problem of absolute stability, Automation and Remote Control 12 (1969) 1889–1895. [15] P. Tymoshchuk, E. Kaszkurewicz, A winner-take-all circuit using neural networks as building blocks, Neurocomputing 64 (2005) 375–396. [16] K. Urahama, T. Nagao, K-winner-take-all circuit with 0(n) complexity, IEEE Transactions on Neural Networks 6 (1995) 776–778. [17] W.J. Wolfe, D. Mathis, C. Anderson, J. Rothman, M. Gotler, G. Bragy, R. Walker, G. Duane, G. Alaghband, K-winner networks, IEEE Transactions on Neural Networks 2 (1991) 310–315. [18] J.F. Yang, C.M. Chen, A dynamic K-winners-take-all neural network, IEEE Transactions on Systems, Man and Cybernetics 27 (1997) 523–526. [19] J. C. Yen, S. Chang, A new first-k-winners neural network, in: Proceedings of the ISANN, 1997, D-01–D-06. [20] J.C. Yen, J.I. Guo, H.-C. Chen, A new k-winners-take all neural network and its array architecture, IEEE Transactions on Neural Networks 9 (1998) 901–912.

Pavlo V. Tymoshchuk received his M.Sc. and Ph.D. degrees in Electrical Engineering from Lviv National Polytechnic University, respectively, in 1982 and 1992 and his D.Sc. degree in Radio Engineering & TV Systems from Odesa National Academy of Communication, Ukraine, in 2005. From 2001 to 2002 he was a Visiting Researcher at Federal University of Rio de Janeiro, Brazil. Presently he is a Professor at the Institute of Computer Science and Information Technology of Lviv Polytechnic National University. His current research interests are related to designing continuous-time and discrete-time neural networks and its applications. He is a co-author of the book Theory Bases of Designing Neural Networks: Manual (in Ukrainian) published in 2007 by Lviv Polytechnic National University Press.