A Photosynthetic Learning Algorithm for the Training of Neural Network

Copynght © IFAC Artificial Intelligence In Agriculture, Makuhan, Chiba, Japan, 1998

A PHOTOSYNTHETIC LEARNING ALGORITHM FOR THE TRAINING OF NEURAL NETWORK Haruhiko Murase* and Akira Wadano**

*Lab. of Phytotronics and SensibilitY Engineering Dept. of Regional Environmental Science ** Lab. of Biosignal Network Dept. of Applied Biochemistry Osaka Prefecture University 1-1. Gakuen. Sakai 599-8531 JAPAN

Abstract:The process of the reaction of carbon molecules in the dark reaction of photosynthesis was chosen as a BOA (Biosystem-Oerived Algorithm). In this paper, the BOA was referred to as the 'Photosynthetic Learning Algorithm (PLA),. The PLA utilizes the rules governing the conversion of carbon molecules from one substance to another in the Benson-Calvin cycle and Photorespiration reactions . In this paper, the principles of the PLA are presented in detail. Neural network training was selected as a typical optimization or parameter estimation problem to demonstrate the performance of the PLA. The neural net work model for the logical functions was successfully trained CopyrIght © 1998 IFAC Keywords :photosynthetic algorithms, biosystem originated algorithm, dark reaction, Benson-Calvin cycle, photorespiration, local minimum trap, optimization methods.

I. INTRODUCTION

fitting quantitative models eventually require some treatment or processing by adaptive search procedures or optimization techniques. There are many search techniques such as exhaustive techniques (random walk), calculus-based techniques (gradient methods), partial knowledge techniques (hilI climbing), knowledge-based techniques (production rule systems, heuristic methods), stochastic techniques (simulated annealing) and biosystem originated algorithms (genetic algorithm, immune system algorithm). In realistic systems, the interactions between the parameters are not generally amenable to analytical treatment, and the researcher must resort to appropriate search techniques. Recently, genetic algorithms (GA) and immune system algorithms are receiving attention, due to their ability to locate very good approximate solutions in extremely large search spaces with a reasonable amount of computational effort. It is interesting to note that if one carefully tries to look at plant systems or phytosystems, many different biosystem(BOA) can be found. In thee derived algorithms

A number of problems found in the agricultural engineering discipline involve various types of optimization of operating systems such as drainage and irrigation systems, crop scheduling, handling and blending of materials. Such operating systems typically depend upon decision parameters that can be chosen by the system designer or operator. An inappropriate choice of deciSIOn parameters causes the performance of such a system to be seriously flawed , as measured by some relevant objective or fitness function . Another problem often encountered in the discipline of agricultural engineering lies in the field of testing and fitting quantitative models . Engineering or scientIfic research in any problem area classically consists of the iterative process of building explanatory or descriptive models, collectmg data, testIng the models and, when discrepancies are found, modifying the models and then repeating the process until the problem is solved. TIle problems that deal with optimizing operating systems and

103

phytosystem, photosynthesis is one of the most important biochemical pathways. The most interesting reactions of photosynthesis consist of the set known as the 'dark reactions'. The dark reactions comprise a biochemical process consisting of a combination of the Benson-Calvin cycle and photorespiration. The product of the dark reactions is carbohydrate. The biochemical balance between the Benson-Calvin cycle and photorespiration can be viewed as a natural implementation of an optimization procedure that maximizes the efficiency of sugar production under the continuous variation of energy input from the sun. It is sometimes maintamed that a plant is not optimized by nature to functIOn as an energy conversion device, because of its very low energy conversion efficiency of about three percent. ComparIsons are made to man-made devices such as photovoltaic cells and photoelectrochemical cells, which transform the sun's energy into an electrical current, or chemical fuels with an effiCiency as high as 25 percent. This IS not a fair comparIson, however, because plants are under heavy functional constraints to maintain the diverse set of biological activities nCl:cssary for their survival and for the preservation of their species. A fair comparison would require that only those biodlemical pathways in the plant directly related to energy conversIOn be considered when l:a\culating energy conversion efficiency. In the present study the process of the reactIOn of carbon molecules in the dark reaction of photosynthesis IS chosen as a BOA. The BOA is referred to as the 'Photosynthetic Learning Algorithm (PLA)'. In the next section, the prinCiples of the PLA are presented. Subsequently, the application of the PLA to a typical learning problem is discussed.

2. PRINCIPLE OF PLA Figure I shows a diagram of the Benson-Calvin cycle (Bowyer and Leegood ,1997). In thiS diagram, each line represents the conversion of one molecule of each metaholitc . The cycle can he divided mto three phases. The first phase of the Benson-Calvin Cycle is a carhoxylation, l:atalyzed by Ribulose Bisphosphate Carboxylase (Rubisl:O). The second phase is the reductive phase in which glycerate-3-P is reduced to trIose-P by the actIOns of glycerate-3-P kinase and NADP-dependent gIYl:eraldehyde-P dehydrogenase . The third phase involves the regeneration of the acceptor, ribulose-I ,5-P of the sugar phosphate shuffle , in which five C 3 units are converted into

of RuBP leads to the production of one molecule of glycerate-3-P (3PG) and one of glycolate-2-P. Figure 2 shows the part of the photorespiratory pathway that is dependent upon Rubisco. ~ ribulose·S.P

-XIuIoS.

J

/ /

GAP

S.

3 ATP

~3AOP rjbulOS.~ 3 CO,

P

sedoheptulose·7·P

M'~

" glycarate-3.P

PI

sedoheptulose-l ,7 -P 2

\\\\\C

~

DHAP

,

erythrose·4·P xylulose·S·P

>-<

GAP

6ATP

-6ADP

glycerale -l ,3·P 2

truetose·6·P

t.,.. p ,

6 NADPH

H,01 tructose -, .s-P2

/'---. DHAP (product or leedback)

Fig. I Benson-Calvin Cycle. The PLA utilizes the rules governing the conversion of carbon molecules from one substance to another in the Benson-Calvin cycle and Photorespiration reactions . Figure 3 illustrates the variation of recombmation of carbon molecules appearing in the PLA. The first phase of the Benson-Calvin cycle and the reaction taking place m the chloroplast subcellular compartment for photorespirauon were utilized for the algorithm . The product of photosynthesIs, OHAP as shown in Fig. I, serves to provide the knowledge strings of the algorithm. . Optimization is attained when the quality of the products no longer improves. The quality of a product is evaluatcd based on the fitness value. The fitness value can be obtained by calculating the difference between the output value of the system using parameters currently given by the PLA and the training output data. Figure 4 shows a flow diagram indicating the calculation process of the PLA. Onc of the unique features of the algorithm IS that the stimulation function is provided. The stimulation occurs due to randomly changing light intensity which alters the degree of influence on renewing the elements of RuBP by photorespiration . The frequency of the stimulation cycle by photorespiration can be calculated by the CO 2 fixation rate given by Eq.1. C =V max / ( I + AIL ) where: C: CO 2 fixation rate

three C 5 units.

Vmax:Maxmum CO 2 fixation rate A:Affimty of CO 2

Rublsco IS a bifunctlOnal enzyme which catalyzes both the carboxylation and the oxygenatIOn of RuBP. Oxygenation

LLight intensity.

104

(I)

Alternatively, the actual li ght intensity varyi ng with through an on- line measuring system might be used. stimu lation is effective in reducing the occurrence of minimum traps in the search process . The

CHLOROPLAST

r;;3 :;~TP::: ~ \ RuBP

r

r~

concentration in the leaf vanes depending on the CO 2 fixation rate . The ratio of 02 concentration and CO 2

PEROXISOME

-p

time The local CO 2

L--_----l

concentration is evaluated to determine the ratio of the calculation frequency of the Benson-Calvin cycle to that of the photoresplration cycle.

glycolate-2-P

O2

,

atmosphere

.-------,

Fig. 2 Photorespiratlon.

Light (Stimulation)

I I I I I I DRuBPTI

I I I I I I I 11

rsedoheptulose-7 -P I I

I 11

[I

1 1 I 1 I I

'-I I

n

3C0 2

I I

I I I I GAP

I

xylulose-5-p

_'111

rT I

+ t,1/{,1

• •

I I

I I I I I I

I

I

lerythrose-4-P ...

[ I I I

I~~~h~~~discard

I

.. "

'I "l xl

DHAP

U ""I fructose1,6-P

IT I

I

I I I I I I I I

I I I I

CO 2

Fig. 4 The photosynthetIc learning algorithm .

3. NEURAL NETWORK TRAINING BY PLA

,.~

.

I

I I

glycerate

: I

Neural network (NN) traming IS an example of a typical optimIzation or parameter estimation problem. The back propagation algorithm IS often used for neural network training . Genetic Algorithms may also be used for the training , even though thIS is rather a tough task . In this section, the performance of the PLA in neural network training is discussed . The logical functIOns can be mode led by a simple neural network, as shown in Fig. 5. The PLA is expected to search optllnum values of six synapse weights (W I - W 6 ) of the NN to model the logical

glycerate

I I I IT] + CDIIII glycolate .91YCOlate

(8-C cycle) I

,-- --------

functions. Parameters used for this test were as follows: the affinity of CO was set to 10000, the maximum light II1tenslty varied 2 from 10000 to 50000 lux (increase in light mtenslty Implies an Increase In the chance to activate photorespiratlOn) , the maximum CO 2 fixat ion speed was

Fig. :I Recombmation or carhon molecules in the B-C cycle (ahove) and photoresplratlon (below).

The parameters II1 vo lved In Eq. I are all measurable, but these values can be assIgned wl\hln a realistic range and need not be empIrical. Whcn executmg the PLA, the light intensity (L) should be gencrated randomly by computer.

30, and the maXln1Um number of cycles for the BensonCalvln and for photoresplration were 30 and 45, respectively, per search iteratIOn.

105

Read in Initial Conditions: Parameters for CC2 fixation (Affinity, Max. fixation rate, etc.) Parameters for NN calculations No of la ers, No of units, etc.

Read in NN Training Data (AND, OR , XOR, etc.)

Fig. 5

Random Generation of Light Intensity Level

Logical function neural network model.

Calculation of CO 2 Concentraion

Figure 6 illustrates the neural network training procedure using the PLA . As shown in Fig. 6, neural network evaluation appears in the process of the fitness check of the knowledge strings (DHAP) indicated at the end of the flow diagram . Each of six estimated synapse weights is coded in a 16 bit DHAP molecule . After converting them into decimal numbers, neural network outputs are calculated using estimated sy napse weights and the training input data. The training output data are compared with the calculated output data to obtain the fitness of the estimates. When a se t of estimates with better fitness than previous data are obtained, the best data set is stored in the DHAP reservoir for the next comparison. After the completion of one cycle of the search process with a predetermined frequency for the Benson-Calvin and Photorespiratory cycle, the process is then directed to the random generation of light intensity level for the next iteration with a renewed photosynthetic frequency condition.

Determination of 0 2 Concentration ratio

END)

Complete Benson-Calvin Cycle (Renewed Synapse Weight)

Input

AND

OR

XOR

y 0

Output Z

I I

0

0 0 0

I

I

0 0

0

0

I

I I

0

I I I

0 0

0 I

0 I

I I

0 I

0

0 0

I

I

Photo respiration Cycle (Renewed Synapse Weight)

Evaluation of DHAP (Calculation of NN Output using new DHAP) (Comparison of Trainin Data and Estimates)

TABLE I Inputs and outputs of logical functions

X

CO 2

Determination of 8-C / Photoresp. frequency ratio

Logical AND function, logical OR function and logical XOR function were tested to evaluate the PLA convergence properties. The input-output relatIOnships of these logical functions are summarized in Table I . For comparison the genetic algorithm (Murase, et aI., 1994) was also applied to the test of NN trainlOg of logical functions.

FunctIOns

/

Fig. 6 Flow diagram of NN training by PLA.

4. RESULTS AND DISCUSSION

The training of the neural network model for logical AND and OR function s using the PLA was very satisfactory. Figures 7 and 8 show the convergence properties of the learning. After 20 iterations for AND training, the PLA converged at an absolute error level of about 6.0 x 10-5 .

I

106

domain speCific knowledge, or measures of goal distance. Moreover, the PLA has general characteristics that enable it to be independent of the techniques of model-building using parameters, requiring only a measure of performance, referred to as the degree of fitness, for any given parameter combination.

However, the GA converged at an absolute error level of about 1.6 x 10- 1. GA training was terminated at 500 Iterations. No improvement was observed within the 500 iterations. As shown in Fig. 8, a sharp dec1ine in the error level at the beginning of PLA training for the OR function was observed. After 6 iterations for OR training, the PLA converged at an absolute error level of about 1.2 x 10- 11 .

o

On the other hand, the GA converged at an absolute error level of about 3.8 x 10-4 . GA training for the OR function

Ll:i

Cl)

.0

«

~ -10 0>

o

....I

PLA

I

o

1

2

4

3

5

6

Fig. 8 Convergence properties of the PLA and the GA for OR.

-e

o

..... .....

::l

-3

w


I

-0.5

S

(5 C/)

.0

PLA

~ ~

0)

""

I I

'-

GA

\.

" ..........

lP~

-1.0

.3 20

:J

Iteration mumber

.$

-5

-

,I

-15

w -2

....I

.-

I

GA

Se-4 0> o

\

::J

'-

g .J:l «

-5

oC/)

Or-~-----r--~~--~-

e

... Gl.

\ "'

e

was also terminated at 500 iterations. No improvement was observed within the 500 iterations. The convergence properties of the training for logical XOR are shown in Fig. 9 . Higher error levels at convergence were obtained for the XOR function than for the other logic functions. This result is due to the fact that finding a separation plane for the exclusive OR in the state space is very difficult task. It was not possible to reduce the error by an appreciable amount unless photorespiration was applied during the PLA process. The influence of a light intensity level of more than 50000 lux did not have any significant influence on the convergence characteristics.

-.:::- -1

r\

-.:::-

I

30

I

Iteration mumber

-1.5

Fig . 7 Convergence properties of the PLA and the GA for AND.

1

100 10 Iteration mumber

1000

Fig. 9 Convergence properties of the PLA and the GA for XOR.

The results show that the PLA is capable of traInIng a neural network. The PLA can locate very good approximate solutions in extremely large search spaces with a reasonable amount of computational effort. The PLA searches from a number of search points (RuBP), which make it an ideal candidate for parallel processing implementation, and is far more efficient than exhaustive techniques. Since it uses the recombmation of existing high quality knowledge strings (DHAP) and a method of measuring current performance, the PLA does not require complex surface descriptions,

5. CONCLUSION The effectiveness of the PLA in neuron traInIng was demonstrated in this study. The Photosynthetic algorithm is a newly developed algorithm that can be used for solving problems in search, optimization and machine learning. 1be importance of this study IS that the Photosynthetic

107

algorithm is denved from a blOsystem . In our biosystem there are many different natural phenomena, many of which are very peculiar and impressive. The genetic algorithms, the Immune system algorithms and the photosynthetic algorithms are all biosystem-derived algorithms. There are surely many other algorithms represented in biosystems that might prove useful in engineering applications. Seeking useful engineering principles exemplified in hiosystems is likely to be a fruitful path to advancements in bioengineering.

6. REFERENCES Bowyer, J.B. and R. C. Leegood (1997). Photosynthesis , In Plant Biochemistry (P .M. Dey and J. B. Harborne , eds), pp.49-110. Academic Press, San Diego. Farmer, J. D .. N. H. Packard and A. S. Perelson (1986). The immune system, adaptation , and machine learning. Physic(1 D22, 187-204. Goldberg, D . E . (1989). Genetic Algorithms in Search, Optimi-:.ation and Machine Learning. Addison Wesley, Reading, Massachusetts, USA. Murase , H., S. Koyama and S. Koyama (1994). Ka/man Nellro Computing by Personal Computers. Morikllashuppan, Tokyo, Japan.

108