Survey of random neural network applications

European Journal of Operational Research 126 (2000) 319±330

www.elsevier.com/locate/dsw

Survey of random neural network applications Hakan Bakõrcõo glu *, Tasßkõn Kocßak Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708-0291, USA

Abstract This paper consists of a survey of various engineering applications based on the random neural network (RNN) model [Neural Computation 1(4) (1989) 502±511; 2(2) (1990) 239±247; Comptes-Rendus Acad. Sci. t. Serie II 310(2) (1990) 177±180; Applied Math. Modelling 15 (1991) 534±541; Neural Computation 5(1) (1993) 154±164], and also a summary of the recent image processing techniques such as still image compression, image enlargement, and image fusion. The advantage of the RNN model is that it is closer to biophysical reality and mathematically more tractable than standard neural methods, especially when used as a recurrent structure. Ó 2000 Published by Elsevier Science B.V. All rights reserved. Keywords: Random neural network model; Still image compression; Image enlargement and enhancement; Sensor image fusion; Automatic target recognition

1. Introduction Neural networks and related techniques are powerful tools that prove ecient in real-world engineering applications such as associative memory, computer communications [6,7], combinatorial optimization problems, system identi®cation and control, function approximation, pattern recognition, signal processing [8], where problems are awkwardly de®ned or dicult to formulate. Neural networks possess properties which give them

* Corresponding author. Tel.: +1-919-5442; fax: +1-919-6605293. E-mail addresses: [email protected] (H. BakõrcõogÆlu), [email protected] (T. Kocßak).

advantages over other methods. Because of their parallel architecture, they can overcome most of the limiting computational diculties. Since they are trained on example data, they are more adaptable to changes in the input data by allowing the training to continue during the processing of new information. The high degree of connectivity allows neural networks to self-organize, which is important when the structure of the data is not known beforehand. And ®nally, due to the analogy between neural networks and neurobiological systems, current biological networks could be used as models for designing arti®cial neural networks. This paper will touch upon previous applications of the random neural network (RNN) model which is described in detail in [1±5], together with a brief description of the recent image processing

0377-2217/00/$ - see front matter Ó 2000 Published by Elsevier Science B.V. All rights reserved. PII: S 0 3 7 7 - 2 2 1 7 ( 9 9 ) 0 0 4 8 1 - 6

320

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

methods [9,30], also based on the same model. Hence, it would be helpful to explain the dynamics of the model ®rst. 1.1. The random neural network model In the random neural network model by Gelenbe [1,2,4], signals in the form of impulses which have unit amplitude travel among the neurons. Positive signals represent excitation, whereas negative signals represent inhibition to the receiving neuron. Thus, an excitatory impulse is interpreted as a ``+1'' signal, while an inhibitory impulse is interpreted as a ``ÿ1'' signal. Each neuron i has a state ki …t†, which is its potential at time t, represented by a non-negative integer. When the potential of neuron i is positive, it is referred to as being `excited', and it can transmit impulses (®re). The impulses will be sent out at a Poisson rate ri , with independent, identical exponentially distributed inter-impulse intervals. The impulses transmitted will arrive at neuron j as excitation signals with probability pij‡ , and as inhibitory signals with probability pijÿ . A neuron's transmitted impulse may also leavePthe network n with probability di , therefore, di ‡ jˆ1 ‰pij‡ ‡ pijÿ Š ˆ 1. To make these probabilities easier to work ‡ ÿ wÿ with, let w‡ ij ˆ ri pij , and P ij ˆ ri pij ; then ®ring n ‡ ÿ rate of neuron i, ri , is jˆ1 ‰wij ‡ wij Š. The w matrices can be viewed as being analogous to the synaptic weights in connectionist models, though they speci®cally represent rates of excitatory and inhibitory impulse emission. Since the w matrices are formed through a product of rates and probabilities, they are guaranteed to be nonnegative. Exogenous excitatory and inhibitory signals, meaning those arriving to the neuron from a source outside of the network, also arrive to neuron i at rates Ki and ki , respectively. These are analogous to the input received by the input neurons in a connectionist model, again however, they represent rates. Fig. 1 shows the representation of a neuron in the RNN using the model parameters that have been de®ned above. In this ®gure, only the transitions to and from a single neuron i are considered

Fig. 1. Representation of a neuron in the RNN.

in a recurrent fashion. All the other neurons can be interpreted as the replicates of neuron i. At this point, it is necessary to consider the dynamics of the RNN model by analyzing the possible state transitions. Within a time interval of Dt, several transitions can occur which change a neuron's state ki …t†: · The potential ki …t† of a neuron will decrease by one whenever it ®res, regardless of the type of the signal emitted (excitation or inhibition). Also, when an exogenous inhibitory signal arrives from outside the network to neuron i, its potential drops to ki …t† ÿ 1 at time t ‡ Dt. Moreover, neuron i might receive an inhibitory impulse from another neuron j, whose e€ect will again be to decrement the value of ki at time t by one. · Arrival of an exogenous excitatory signal from outside, or an excitatory impulse from another neuron within the network will result in incrementing the neuron potential by one, yielding ki …t† ‡ 1. · Needless to say, the value of ith neuron's state remains unchanged when none of the events described above occur.

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

In the case when self-inhibition is allowed, the value of the neuron's state can drop by two units in a single time step, however this case will not be considered in the following expressions. Also in this model, self-excitation is not of interest because in its presence, the potential of the neuron may increase without bound which would lead to instability. There are also some boundary conditions which prevent some of the transitions from occurring. First of all, a neuron can ®re only when it has a positive potential as explained above. Second, when the neuron has a potential of zero, the arrival of new inhibitory signals does not decrease its value further. All of these constraints will be uni®ed in a single expression when the state transitions are expressed in mathematical form. Let k…t† ˆ k1 …t†; . . . ; kn …t† be the vector of signal potentials at time t, and k ˆ k1 ; . . . ; kn be a particular value of the vector, and lets de®ne the probability p…k; t† ˆ Pr‰k…t† ˆ kŠ. The behavior of the probability distribution of the network state can be derived through the following equations. Since k…t† : t P 0 is a continuous time Markov chain, it satis®es an in®nite system of Chapman± Kolmogorov equations. " X p…ki‡ ; t†r…i†d…i†Dt p…k; t ‡ Dt† ˆ i

‡

p…ki‡ ; t†k…i†Dt

 …1 ÿ r…i†Dt† 1‰ki …t† > 0Š Xn p…kij‡ÿ ; t†r…i†p‡ …i; j†Dt 1‰kj …t† ‡ j

> 0Š ‡ p…kij‡‡ ; t†r…i†pÿ …i; j†Dt

‡ o…Dt†; where 1‰xŠ ˆ



1 0

if x is true; otherwise:

ˆ

i

p…ki‡ †r…i†d…i† ‡ p…kiÿ †K…i†1‰ki > 0Š

‡ p…ki‡ †k…i† ‡ ‡ ‡

X fp…kij‡ÿ †r…i†p‡ …i; j†1‰kj > 0Š j

p…kij‡‡ †r…i†pÿ …i; j† p…ki‡ †r…i†pÿ …i; j†1‰kj

 ˆ 0Šg :

The stationary probability distribution associated with the model is the value which will be taken to be the output of the network, and is given by qi ˆ lim Pr‰ki …t† > 0Š; t!1

i ˆ 1; . . . ; n;

ˆ 0Š

# o

…1†

which reduces to the form qi ˆ k‡ …i†=‰r…i† ‡ kÿ …i†Š;

…2†

where the k‡ …i†; kÿ …i† for i ˆ 1; . . . ; n satisfy the system of non-linear simultaneous equations X qj w‡ k‡ …i† ˆ ji ‡ K…i†; X j

 …1 ÿ K…i†Dt†…1 ÿ k…i†Dt†

‡

i

X

kÿ …i† ˆ

‡ p…ki ; t†

p…ki‡ ; t†r…i†pÿ …i; j†Dt 1‰kj …t†

For steady state analysis, let p…k† denote the stationary probability distribution which is equal to limt!1 Pr‰k…t† ˆ kŠ if it exists. Thus, in steady state, stationary probability distribution, p…k†, must satisfy the global balance equations: X p…k† ‰K…i† ‡ ‰k…i† ‡ r…i†Š1‰ki > 0ŠŠ

j

‡ p…kiÿ ; t†K…i†Dt 1‰ki …t† > 0Š

321

qj wÿ ji ‡ k…i†:

…3†

To put Eq. (2) into words, the steady state probability that the neuron i is excited is simply equal to the ratio of the sum of all the rates of arriving excitatory signals to the sum of the rates of arriving inhibitory signals together with the ®ring rate of that particular neuron. 1.2. Random neural network learning algorithm The algorithm chooses the set of network parameters W in order to learn the training set of K input±output pairs (i, Y) where the set of successive inputs is denoted i ˆ fi1 ; . . . ; iK g, and ik ˆ …Kk ; kk † are pairs of excitation and inhibition

322

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

signal ¯ow rates entering each neuron from outside of the network: Kk ˆ ‰Kk …1†; . . . ; Kk …n†Š;

kk ˆ ‰kk …1†; . . . ; kk …n†Š:

The successive desired outputs are the vectors Y ˆ fy1 ; . . . ; yK g, where each vector yk ˆ …y1k ; . . . ; ynk †, whose elements yik 2 ‰0; 1Š correspond to the desired output values for each neuron. The network adjusts its parameters to produce the set of desired output vectors in a manner that minimizes a cost function Ek : Ek ˆ

n 1X ai …qi ÿ yik †2 ; 2 iˆ1

ai P 0:

In this network, all neurons are generalized to be output neurons, therefore, if it is desired that a neuron j is to be removed from the network output, and therefore the error function, it suces to set aj ˆ 0. Recall that the steady state output rate of all neurons in the network is given by Eqs. (2) and (3). Both of the n by n weight matrices W‡ k ˆ ‡ ÿ ˆ fw …i; j†g must be adjusted fwk …i; j†g and Wÿ k k after each input is presented, by computing for ÿ each input ik ˆ …Kk ; kk †, a new value W‡ k and Wk of the weight matrices. Since the weight matrices represent a rate times a probability, only solutions for which all values in the matrices are positive are valid. Let w…u; v† denote any weight term, which would be either w…u; v†  wÿ …u; v†, or w…u; v†  w‡ …u; v†. The weights will be updated using gradient descent method: wnew …u; v† ˆ wold …u; v† ÿ goE=ow…u; v†: The partial derivative of the cost function can be computed and substituted to obtain the update di€erence equation wk …u; v† ˆ wkÿ1 …u; v† n X ÿ g ai …qik ÿ yik †‰oqi =ow…u; v†Šk ;

…4†

iˆ1

where g > 0 is the learning parameter which is constant over each iteration of training, and 1. qik is calculated using the input ik and w…u; v† ˆ wkÿ1 …u; v†, in Eqs. (2) and (3).

2. ‰oqi =ow…u; v†Šk is evaluated at the values qi ˆ qik , ÿ ÿ w‡ …u; v† ˆ w‡ kÿ1 …u; v† and w …u; v† ˆ wkÿ1 …u; v†. To compute ‰oqi =ow…u; v†Šk the following equation is derived from expressions (2) and (3): oqi =ow…u;v† X oqj =ow…u;v†‰w‡ …j;i†ÿwÿ …j;i†qi Š=…r…i†‡kÿ …i†† ˆ j

ÿ1‰uˆiŠqi =…r…i†‡kÿ …i†† ‡1‰w…u;v†w‡ …u;i†Šqu =…r…i†‡kÿ …i†† ÿ1‰w…u;v†wÿ …u;i†Šqu qi =…r…i†‡kÿ …i††: Let q ˆ …q1 ; . . . ; qn †; and de®ne the n  n matrix  W ˆ ‰w‡ …i; j† ÿ wÿ …i; j†qj Š=…r…j† ‡ kÿ …j†† ; i; j ˆ 1; . . . ; n: The vector equations can now be written as oq=ow‡ …u; v† ˆ oq=ow‡ …u; v†W ‡ c‡ …u; v†qu ; oq=owÿ …u; v† ˆ oq=owÿ …u; v†W ‡ cÿ …u; v†qu ; where the elements of the n-vectors ‡ c‡ …u; v† ˆ ‰c‡ 1 …u; v†; . . . ; cn …u; v†Š

and ÿ cÿ …u; v† ˆ ‰cÿ 1 …u; v†; . . . ; cn …u; v†Š

are 8 ÿ1=…r…i† ‡ kÿ …i†† > > > > > > < if u ˆ i; v 6ˆ i; ‡ ci …u; v† ˆ ‡1=…r…i† ‡ kÿ …i†† > > > if u 6ˆ i; v ˆ i; > > > : 0 for all other values of …u; v†; 8 ÿ…1 ‡ qi †=…r…i† ‡ kÿ …i†† > > > > > if u ˆ i; v ˆ i; > > > > ÿ > > < ÿ1=…r…i† ‡ k …i†† if u ˆ i; v 6ˆ i; cÿ i …u; v† ˆ > > > ÿqi =…r…i† ‡ kÿ …i†† > > > > > > if u 6ˆ i; v ˆ i; > > : 0 for all other values of …u; v†:

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

2. Survey of previous RNN applications

Notice that oq=ow‡ …u; v† ˆ c‡ …u; v†qu ‰I ÿ WŠÿ1 ; oq=owÿ …u; v† ˆ cÿ …u; v†qu ‰I ÿ WŠÿ1 ;

323

…5†

where I denotes the n by n identity matrix. Hence the main computational e€ort in this algorithm is ÿ1 to obtain ‰I ÿ WŠ . This is of time complexity 3 2 O…n †, or O…mn † if an m-step relaxation method is used. From the above, the complete learning algorithm for the network can be given. First initialize ÿ the matrices W‡ 0 and W0 in some appropriate manner. This initiation can be made at random if no better method can be determined. Choose a value of g, and then for each successive value of k, starting with k ˆ 1 proceed as follows: 1. Set the input values to ik ˆ …Kk ; kk †. 2. Solve the system of non-linear equations given in (2) and (3) with these values, perhaps by using an iterative method such as Gauss± Seidel. 3. Solve the system of linear equations (5) with the results of (2). 4. Using Eq. (4) and the results of (2) and (3), upÿ date the matrices W‡ k and Wk . Since the ``best'' matrices (in terms of gradient descent of the quadratic cost function) which satisfy the nonnegativity constraint are sought, in any step k of the algorithm, if the iteration yields a negative value of a term, there are two alternatives: (a) Set the term to zero, and stop the iteration for this term in this term in this step k, in the next step, k ‡ 1, iterate on this term with the same rule starting from its current zero value. (b) Go back to the previous value of the term and iterate with a smaller value of g. This general scheme can be specialized to feedforward networks by noting that the matrix ‰I ÿ WŠ will be triangular, yielding a computational complexity of O…n2 †, rather than O…n3 †, for each gradient iteration. Furthermore, in a feedforward network, the equations given in (2) and (3) are simpli®ed in that qi is only dependent upon qj for j < i. This reduces the computational e€ort required to solve (2) and (3).

The RNN model has been proven to be successful in a variety of applications when used either in a feed-forward or a fully recurrent architecture. In most problems, RNN yields strong generalization capabilities, even when the training data set is relatively small compared to the actual testing data. The model also achieves fast learning due to its computational simplicity for weight updating process. In the following, we will brie¯y describe the past work related to RNN. Associative memory: In [10,11], the network's ability of acting as autoassociative memory is examined and a technique for reconstructing distorted patterns is developed, which is based on properties of the network. The performance of the resulting approach has been investigated through experiments, which yielded promising results. Also, in [12], the author shows how distributed associative memory can be used to compute membership functions for decision-making under uncertainty. Optimization: The traveling salesman problem (TSP) is commonly considered as a benchmark case for heuristic methods among hard combinatorial optimization problems. It is shown in [13] that the dynamical RNN yields solutions to the TSP which are close to the optimal in a majority of instances tested. Yet another application is the vertex covering problem, which is designated as NP-complete. In [14,15], authors compare the performances of the RNN, the conventional Greedy Algorithm, the Hop®eld network, and simulated annealing, when applied to the same problem. Results reveal that the RNN heuristic is superior to the others in terms of overall optimization. Texture generation: Generation of arti®cial textures is a useful function in image synthesis systems. Authors in [16,17] describe the use of the RNN model to generate various textures having di€erent characteristics. Numerical iterations of the ®eld equations of the model, starting with a randomly generated gray-level image, are shown to produce textures having di€erent desirable features such as granularity, inclination, and randomness. Their experimental evaluation shows

324

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

that the RNN provides good results, at a computational cost less than that of other approaches such as Markov random ®elds (MRF). Magnetic resonance imaging (MRI): Brain MR images contain massive information requiring lengthy and complex interpretation (as in the identi®cation of signi®cant portions of the image), quantitative evaluation (as in the determination of the size of certain signi®cant regions), and sophisticated interpretation (as in determining any image portions which indicate signs of lesions or of disease). In [18], RNNs are used to extract precise morphometric information from MRI scans of the human brain. A method for classi®cation of gray matter from MR images is proposed, and the classi®cation performance is shown to be very similar to those that are known to be obtained by a human expert carrying out manual volumetric analysis of brain MR images. Function approximation: In [19], approximation of arbitrary continuous functions on ‰0; 1ŠS using the ``Gelenbe'' random neural network (GNN) is studied. It is shown that the clamped GNN and the bipolar GNN have the universal approximation property, using a constructive method which exhibits networks constructed from a polynomial approximation of the function to be approximated. There are no restrictions on the structure of the networks except for limiting them to being feedforward. In [20], the design of GNN approximators with a bounded number of layers is discussed. It is shown that the feedforward CGNN and BGNN with s hidden layers (total of s ‡ 2 layers) can uniformly approximate continuous functions of s variables. Mine detection: In [21], authors introduce an RNN approach to mine detection which provides a robust non-parametric method, based on training the network using data from a previously calibrated portion of the mine®eld, or from a similar mine®eld. This approach is shown to be very effective for detecting mines and rejecting false alarms. Experimental evidence indicates that the neural network trained for mine detection and false alarm rejection on a small calibration site, can be e€ective on geographical locations which are distinct and far removed from the locations where training of the network takes place. It is also

shown that the RNN trained for a speci®c EMI instrument can be e€ective when it produces decisions based on data from a di€erent EMI sensor instrument. Automatic target recognition (ATR): ATR and object recognition is important not only in military applications but also in a variety of other contexts such as medical imaging [18,22]. A neural approach based on the RNN model is proposed in [23], to detect shaped targets with the help of multiple neural networks whose outputs are combined for making decisions. Radial features are proposed and used in order to train a set of learning RNNs in [24]. These RNNs are then used to detect targets within clutter with high accuracy, and to classify the targets or man-made objects from natural clutter. 3. Summary of recent image processing techniques Applications involving images require a huge number of operations. For image processing applications, this suggests that the most ecient computational model would be a parallel, highly interconnected neural network. In the following sections, we will summarize our recent research in image processing (i.e., on still image compression, image enlargement, and sensor image fusion), where the RNN is the common tool that is deployed. We will illustrate our results and evaluate the performances of the proposed methods. 3.1. Still image compression One of the common neural approaches in image compression is to train a network to encode and decode the input data [25,26], so that the resulting di€erence between input and output images is minimized. The network consists of an input layer and an output layer of equal sizes, with an intermediate layer of smaller size in between. The ratio of the size of the input layer to the size of the intermediate layer is the compression ratio. The network is usually trained on one or more images so that it develops an internal representation cor-

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

325

Fig. 2. (a) Compression of an arbitrarily large image using a neural encoder/decoder. (b) A neural network compression/decompression pair.

responding not to the image itself, but rather to the relevant features of a class of images. In our approach [27,28], both the input, intermediate and output image is subdivided into equal-sized blocks of 8  8 or 4  4, and compression is carried out block-by-block (see Fig. 2). We use a feed-forward RNN with one intermediate layer as shown in Fig. 2 The weights between the input layer and the intermediate layer correspond to the compression process, while the weights from the intermediate to the output layer correspond to decompression process. The results in Table 1 were obtained by compressing and decompressing with a neural network trained on the ``Lena'' image. See Fig. 3 for an example. We test the robustness of the still image compression network with respect to weight changes. A certain percentage of the trained weights were selected at random, and the values were changed within some speci®ed tolerance ranges. The results obtained by training the neural network on the ``Lena'' image, and testing the performance on the ``Peppers'' image with 8:1 compression ratio are Table 1 Image quality for varying levels of compression on the trained and untrained images, using the RNN method Compression ratio 4:1 8:1 16:1 32:1

Lena

Peppers

Girl

33.606 31.002 29.301 27.266

32.162 29.231 28.236 25.995

33.240 29.685 28.309 26.141

Fig. 3. Test results for 16:1 compression (0.5 bits/pixel) with the RNN.

shown in Fig. 4. As expected, the altered weights yielded smaller PSNR. Still the compressor/decompresser pair was observed to perform satisfactorily both technically and visually. The performance of the RNN compression/decompression algorithm was compared with other state-of-the-art techniques such as JPEG and Wavelet. Fig. 4 shows a plot PSNR against compression level on the ``Lena'' image. Both JPEG and the Wavelet approach yield a PSNR of 10 dB higher on the average than the RNN approach. On the other hand, several other important points that need to be kept in mind when comparing these three techniques: · The coding/decoding time required for the neural network is much smaller compared to the other methods. Once the o€-line training is complete, the compressing and decompressing process is signi®cantly faster with the RNN.

326

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

Fig. 4. (i) Performance for various tolerances. (ii) PSNR plots for comparison.

· The parallel processing capability of neural networks makes them superior to the other approaches in terms of hardware implementation. · Finally, the three techniques need to be compared in terms of global performance ± including run-times ± in the context of video compression. Video compression using in parts the novel neural approach presented here is described in [27,29]. Very high video compression ratios are achieved with a combination of neural networks with temporal subsampling. 3.2. Enhanced image enlargement Let …Rx ; Ry † be the enlargement factor desired between the given ``small'' input image I and the output image O. Thus in the process of enlargement of I, one pixel in the original image will be mapped into a rectangle of Rx  Ry pixels in the x and y coordinates. The original image is assumed to be of size Ix  Iy while the enhanced and enlarged image O is of size Rx Ix  Ry Iy . Often we will deal with an enlargement which is homogenous in both coordinates with R ˆ Rx ˆ Ry . In that case, we will say that the Enlargement ratio is R2 . For a ``small'' image Ii , let Sint;i be the zero order interpolated enlarged image. Then Tdiff;i is the di€erence image such that for each pixel posi-

tion …u; v† in the images, between the zero-order interpolated enlarged image and the hypothetical enhanced image Oi : Oi …u; v† ˆ Sint;i …u; v† ‡ Tdiff;i …u; v†:

…6†

The procedure we propose for training the three layer feed-forward network is as shown in Fig. 5. RNN is trained using the learning algorithm in [5] to produce at its output the di€erence image Tdiff;i which minimizes the square of the sum of pixel errors: Eˆ

1X 2 fSi …u; v† ÿ ‰Tdiff;i …u; v† ‡ Sint;i …u; v†Šg 2 u;v

…7†

when the network is presented with the training image Ii . Fig. 6 presents the generalization capability of the algorithm by displaying the training image, zero-order interpolated version and the enhanced enlarged image for `peppers'. The network was trained using ``Lena'', however the comparisons were carried out both for ``Lena'' and for the ``Peppers'' image, they are summarized in Table 2. 3.3. Sensor image fusion Sensor fusion can be ``model based'', i.e., using an a priori known representation of the object(s)

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

327

Fig. 5. Block diagram of the procedure.

Fig. 6. Results of RNN enlargement on a test image (peppers) for enlargement ratio R2 ˆ 4, training image (left), zero-order PSNR ˆ 29:10 (middle), enhanced enlarged PSNR ˆ 30.87 (right).

Table 2 Comparison of zero-order interpolation and RNN in terms of PSNR (dB) for R2 ˆ 4 Method

Lena

Peppers

Zero-order RNN

29.68 32.49

29.10 30.87

which are being viewed via di€erent sensors, or it can be ``learning based'', i.e., based on a training set of multi-sensor observations and of the resulting ``true'' output. The block diagram for the learning based sensor fusion is given in Fig. 7. Training of the network is carried out as follows. First, a set of K examples of N  N ``true ''

Fig. 7. The block diagram for the learning based sensor fusion.

and di€erent images are created. Let us denote the set of ``real'' or ``true'' data fRk …i; j†g; i; j ˆ 1; . . . ; N ; k ˆ 1; . . . ; K, where …i; j† is used for the

328

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

Fig. 8. Network architecture.

pixel plane position. For each Rk the outputs of all M sensors are obtained. We will denote the mth sensor outputs which correspond to any given Rk by rk;m ˆ frk;m …i; j†g; i; j ˆ 1; . . . ; N . Clearly m ˆ 1; . . . ; M. We will denote by qk the multisensory input set rk;m ; m ˆ 1; . . . ; M. The three layer feed-forward network NN shown in Fig. 8 is now trained as follows. The output of the network has one output neuron for each position …i; j† in the pixel plane. Let fi;j …qk † denote the value of the network's output neuron …i; j† when the network is presented with the multisensory data set qk . The weights are adjusted so as to minimize the cost function Ek ˆ

N X N X



iˆ1 jˆ1

2 Rk …i; j† ÿ fi;j …qk †

…8†

for each successive k ˆ 1; . . . ; K. As shown in Table 3, the PSNR values for the fused images are higher than those for the sensor images for 8 out of 10 images used in our experiment. Hence, we achieved this aim by using our novel approach. Furthermore, we thresholded the fused image with gray-value 128 such that if the grayvalue of a pixel is greater than 128 then its gray-

value is set to 255 otherwise its gray-value is set to 0. Hence 8-bit gray-valued image is turned into 2-color image as real image. The real, fused, sensor and thresholded images for the image #1 are shown in Fig. 9. 4. Conclusions We have surveyed the wide range of engineering applications of the RNN model [1±5], and dem-

Table 3 PSNR (dB) values of sensor outputs and fused images for 10 di€erent images Image #

Sensor 1

Sensor 2

Fused

1 2 3 4 5 6 7 8 9 10

8.06 6.30 5.76 6.92 6.30 6.92 7.27 5.76 7.27 6.92

12.08 14.02 13.34 12.91 10.96 12.09 16.13 14.06 12.55 12.15

15.78 16.12 15.87 12.96 13.86 16.40 13.68 13.36 14.05 14.67

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330

329

Fig. 9. Real, fused, sensor output 1, sensor output 2 for image 1, thresholded.

onstrated some of the recent image processing techniques based on the model. For still image compression, it is shown that the proposed network model is successful in compressing and decompressing images that it did not see before. The obtained visual and PSNR quality is acceptable. The robustness analysis reveals the applicability of the RNN compressor/decompressor for hardware implementation. For enlarging images, the RNN method is experimentally shown to perform quantitatively better than the zero-order interpolation technique. As future work, there is need to compare our approach with higher-order interpolation methods. We believe that the neural network method can be used to obtain improvements, via the computation of a non-parametric di€erence image with respect to higher-order interpolated images. In the fusion method, RNN is used to fuse data such that the output of the networks is as close as possible to the real data, once it is presented with the outputs of the sensors. Experimental results reveal that the proposed method is a promising approach to the sensor image fusion problem.

Acknowledgements This research was supported by the Oce of Naval Research under Grant No. N00014-97-10112, the Army Research Oce under Grant No. DAAH04-96-1-0388 and the IBM Corporation with an equipment grant and a research grant to Duke University.

References [1] E. Gelenbe, Random neural networks with negative and positive signals and product form solution, Neural Computation 1 (4) (1989) 502±511. [2] E. Gelenbe, Stability of the random neural network model, Neural Computation 2 (2) (1990) 239±247. [3] E. Gelenbe, Reseaux neuronaux aleatoires stables, Comptes-Rendus Acad. Sci. t. Serie II 310 (2) (1990) 177±180. [4] E. Gelenbe, A. Stafylopatis, Global behaviour of homogeneous random neural systems, Applied Mathematical Modelling 15 (1991) 534±541. [5] E. Gelenbe, Learning in the recurrent random neural network, Neural Computation 5 (1) (1993) 154±164. [6] E. Gelenbe, Neural Nets and Teletrac: Adapting the Interface, Keynote Address, Fifth International Conference on Telecommunication Systems, Nashville, March 1997. [7] E. Gelenbe, User oriented video: Quality, trac and computational trade-o€s, Invited plenary paper, in: Proceedings of the ICCC'97 (International Council for Computer Communication), Cannes, November 1997. [8] E. Gelenbe, K. Harmanci, J. Krolik, Learning neural networks for detection and classi®cation of synchronous recurrent transient signals, Signal Processing 64 (3) (1998). [9] H. Bakõrcõo glu, E. Gelenbe, T. Kocßak, Image processing with the random neural network model, Invited paper, Proceedings of the 13th International Conference on Digital Signal Processing (DSP97), Santorini, Greece, July 1997.  [10] J. Bourrely, E. Gelenbe, Memoires associatives: Evaluation et architectures, Comptes-Rendus Acad. Sci. Paris t. Serie II 309 (2) (1989) 523±526. [11] E. Gelenbe, A. Stafylopatis, A. Likas, An extended random network model with associative memory capabilities, in: Proceedings of the International Conference on Arti®cial Neural Networks (ICANN'91), June 1991, pp. 307±312. [12] E. Gelenbe, Distributed associative memory and the computation of membership functions, Information Sciences 57±58 (1991) 171±180. [13] E. Gelenbe, V. Koubi, F. Pekergin, Dynamical random neural network approach to the traveling salesman prob-

330

[14]

[15]

[16]

[17]

[18]

[19] [20] [21]

H. Bakõrcõo glu, T. Kocßak / European Journal of Operational Research 126 (2000) 319±330 lem ELEKTRIK, Turkish Journal of Electrical Engineering and Computer Sciences 2 (1994) 1±10. E. Gelenbe, F. Batty, Minimum cost graph covering with the random neural network, in: O. Balci, R. Sharda, S. Zenios (Eds.), Computer Science and Operations Research, Pergamon Press, New York, 1992, pp. 139±147. A. Ghanwani, A qualitative comparison of neural network models applied to the vertex covering problem ELEKTRIK, Turkish Journal of Electrical Engineering and Computer Sciences 2 (1994) 11±19. V. Atalay, E. Gelenbe, N. Yalabõk, The random neural network model for texture generation, International Journal of Pattern Recognition and Arti®cial Intelligence 6 (1) (1992) 131±141. V. Atalay, E. Gelenbe, Parallel algorithm for colour texture generation using the random neural network model, International Journal of Pattern Recognition and Arti®cial Intelligence 6 (2) (1992) 437±446. E. Gelenbe, Y. Feng, K. Krishnan, Neural network methods for volumetric magnetic resonance imaging of the human brain, Proceedings of the IEEE 84 (1996) 1488± 1496. E. Gelenbe, Z.H. Mao, Y.D. Li, Function approximation with the random neural network, IEEE Transactions on Neural Networks, to be submitted. E. Gelenbe, Z.H. Mao, Y.D. Li, Function approximation by random neural networks with a bounded number of layers. E. Gelenbe, T. Kocßak, Mine detection using local area information and the random neural network, IEEE Transactions on PAMI, to be submitted.

[22] E. Gelenbe, G. Shih, L.P. Clarke, R.P. Velthuizen, Random neural networks for brain tumor quantitation from MR imaging, submitted for publication. [23] H. Bakõrcõo glu, E. Gelenbe, Random neural network recognition of shaped objects in strong clutter, in: Proceedings of the International Conference on Arti®cial Neural Networks, ICANN '97, Lausanne, Switzerland, 1997, pp. 961±966. [24] H. Bakõrcõo glu, E. Gelenbe, Feature-based RNN target recognition, to be published in Aerosense '98. [25] N. Sonehara, M. Kawato, S. Miyake, K. Nakane, Image data compression using a neural network model, in: Proceedings of the International Joint Conference on Neural Networks, IEEE Press, New York, 1989, pp. 35±41. [26] G. Cottrell, P. Munro, D. Zipser, Image compression by backpropagation: an example of extensional programming, in: N. Sharkey (Ed.), Models of cognition: a review of cognition science, NJ, Norwood, 1989. [27] C. Cramer, E. Gelenbe, H. Bakõrcõo glu, Low bit rate video compression with neural networks and temporal subsampling, in: Proceedings of the IEEE (Special Issue) on Neural Networks, accepted for publication . [28] C. Cramer, E. Gelenbe, P. Gelenbe Image and video compression, IEEE Potentials, February/March 1998. [29] E. Gelenbe, C. Cramer, M. Sungur, P. Gelenbe, Trac and video quality in adaptive neural compression, Multimedia Systems 4 (1996) 357±369. [30] H. Bakõrcõo glu, E. Gelenbe, T. Kocßak, Image processing with the random neural network model ELEKTRIK, Turkish Journal of Electrical Engineering and Computer Sciences 5 (1) (1997) 65±77.