- Email: [email protected]
Chaos in a three-neuron hysteresis Hopfield-type neural network☆
9 July 2001
Physics Letters A 285 (2001) 368–372 www.elsevier.com/locate/pla
Chaos in a three-neuron hysteresis Hopfield-type neural network ✩ Chunguang Li ∗ , Juebang Yu, Xiaofeng Liao Department of Opto-Electronics Technology, University of Electronics Science and Technology of China, Chengdu, 610054, Sichuan, PR China Received 24 January 2001; received in revised form 29 May 2001; accepted 4 June 2001 Communicated by A.R. Bishop
Abstract A three-neuron hysteresis Hopfield-type neural network was constructed and its dynamics studied. Chaotic behavior and strange attractors were observed. Lyapunov exponents and power spectra are presented for these attractors 2001 Published by Elsevier Science B.V. Keywords: Chaos; Hysteresis; Neural network; Lyapunov exponent; Power spectrum
1. Introduction In last decade, much attention has been drawn on the research of chaotic artificial neural networks (CANN). A number of reports in this field revealed that complicate dynamics responsible for pattern recognition (such as pattern perception under attention mechanism in retina-brain system [1]), memory searching and learning [2–13] in biological neural systems can be simulated by CANN. In this research field, most works are focused on delayed-type neural networks [14,15] and hysteretic-type neural networks [2–7]. It is well known that hysteresis is defined as a lagging effect due to a change of force acting on a body. ✩
This work is supported by the Ministry of Information Industry of China under the Major Science and Technology Development Project, Grant No. 98048, and National Communication Key Labs, under the Project Grants No. 1DZ0207 and 1DZ0208. * Corresponding author. E-mail address: [email protected] (C. Li).
Hysteresis manifests itself in the structures of many cooperative dynamical systems, and is observed in animal’s (such as frogs and crayfish) behavior. Many engineering systems also display hysteresis. In [2–7], the authors studied some kinds of binary hysteresis neural networks, they observed and analyzed the bifurcation phenomenon and chaotic behavior of these neural networks. A simple hysteresis neural network was theoretically and experimentally studied in [5], where the network contains two neurons with binary hysteresis. In this Letter, we will report chaotic behavior and an interesting phenomenon called “order– chaos switching” caused by varying a network parameter in an HHNN (hysteresis Hopfield neural network [8]) containing three neurons with smooth hysteresis as shown in Fig. 1. The HHNN we studied can also exhibit various kinds of attractors such as stable equilibrium, limit cycle, torus and chaos. To the best of our knowledge, this is the first observation of chaotic behavior in a smooth hysteresis neural network as well as the report on such switching phenomenon in an HHNN.
0375-9601/01/$ – see front matter 2001 Published by Elsevier Science B.V. PII: S 0 3 7 5 - 9 6 0 1 ( 0 1 ) 0 0 3 8 1 - 4
C. Li et al. / Physics Letters A 285 (2001) 368–372
369
Fig. 1. (a) Hysteresis neuron, (b) hysteresis activation function.
2. Network (HHNN) model The HHNN [8] consists of a system of N neurons, each characterized by a neuron potential xi , such that Ci
xi (t) λi (x(t dxi (t) ˙ − δt)) =− + dt Ri Ri +
N
wij yj + Ii ,
i = 1, 2, . . . , N,
neuron gain is described as [8] y x(t)x(t ˙ − δ) = ϕ x(t) − λ x(t ˙ − δ) , where ϕ(p) = tanh(γp), and ˙ − δt) 0, γα , x(t γ x(t ˙ − δt) = γβ , x(t ˙ − δt) < 0, −α, x(t ˙ − δt) 0, λ x(t ˙ − δt) = β, x(t ˙ − δt) < 0,
(3)
(4) (5)
where β > −α, and (γα , γβ ) > 0, and
j =i=1
(1) where Ci is the input capacitance of neuron i, wij is the coupling constants (connection weights) between neurons i and j , and Ri is the membrane resistance. yi is a shorthand notation for y(xi |x˙i (t − δt)), given later. Such systems have been studied in the case of symmetric couplings [8]. However, our focus has been on small numbers of neurons connected in an asymmetric way with differing connection strengths and with self-connection weights. In the present study, we examine the dynamics of a simple, autonomous HHNN consisting of three neurons. In this case, (1) reduces to wij yj + wii yi , i = 1, 2, 3. x˙i (t) = −xi (t) + j =i=1
(2) As mentioned before, the hysteresis neurons of the network shown in Fig. 1(a) have the same activation function given in Fig. 1(b) and corresponding hysteresis
x(t) − x(t − δt) dx(t − δt) = lim . δt →0 dt δt Note that in special case when α = β, and γα = γβ , the activation function becomes the conventional sigmoid function. Note also the neuron output not only depends on its input, x(t), but also on the latter derivative information, x(t ˙ − δt). Therefore the HHNN exhibits memory behavior. The main purpose of our studies is to investigate how the dynamics such as bifurcations and chaos depend on the weights wij in the coupling matrix W = {wij }. The results presented in this Letter are for parameters choices as shown below with w33 taken as the bifurcation parameter: 1.95 −1.5 −0.3 W = 1.08 (6) 2 0.4 . 1.1 −0.5 w33 x(t ˙ − δt) =
It should be noted that several other coupling matrices have been examined and shown to exhibit similar
370
C. Li et al. / Physics Letters A 285 (2001) 368–372
dynamics. Due to limitation of space, such results will not be presented here.
3. Numerical studies and simulation results To solve Eqs. (1)–(6), a fourth-order Runge–Kutta method used with time step h = 0.01 is implemented. Some random values of xi (0) are chosen as the initial values. A time series of N = 100000 are analyzed. The power spectra are calculated using the general FFT. In order to calculate the largest Lyapunov exponent, we applied the method introduced by Wolf et. al [16]. Lyapunov exponents can be defined as the long-term evolution of the axes of an infinitesimal sphere of states. To estimate the largest Lyapunov exponent, we in effect monitor the long-term evolution of a single pair of nearby orbits. And the largest Lyapunov exponent is defined by λ=
L (tk ) 1 , M log2 tM − t0 L(tk−1 )
We start from the case when w33 = 1.6, under this condition, the neural network produces chaotic output and the dimensionless largest Lyapunov exponent λ = 0.0398. Fig. 2 shows the attractor in the threedimensional phase space. And the power spectra of the first neuron output are presented in Fig. 3. The spectrum is truncated at v = 1 Hz. The attractor and power spectra of the first neuron output when w33 = 2.0 and w33 = 2.2 are shown in Figs. 4, 5 and Figs. 6, 7, respectively. We can see clearly that Fig. 2 and Fig. 4 show two different type of strange attractors. The curve given in Fig. 8 shows our results for the largest Lyapunov exponent as a function of the parameter w33 . It is seems that λ is an irregular function of the parameter w33 . By varying w33 , it is possible to control the largest Lyapunov exponent of the HHNN outputs. Thereby we can also produce “chaos–order”, “order–chaos” and “chaos–chaos” transition with dif-
(7)
k=1
where M is the total number of replacement steps, L(tk−1 ) is the distance between the two initial points in the time point tk−1 , and after a time step ∆ = tk − tk−1 , the initial length will have evolved to length L (tk ). For a detail, please refer to [14]. The Lyapunov exponent λ is calculated from a time series consisting of N = 100000 points. The calculation carried out show that an asymmetric HHNN demonstrates chaotic outputs. We have examined a range of parameters and noted that the appearance of at least two types of strange attractors.
Fig. 2. Three-dimensional phase space sections of the dynamics of the three-neuron HHNN when w33 = 1.6.
Fig. 3. Power spectrum of the time series of x1 when w33 = 1.6.
Fig. 4. Three-dimensional phase space sections of the dynamics of the three-neuron HHNN when w33 = 2.0.
C. Li et al. / Physics Letters A 285 (2001) 368–372
Fig. 5. Power spectrum of the time series of x1 when w33 = 2.0.
371
Fig. 8. The largest Lyapunov exponent with w33 ranging from 1.4 to 2.3.
4. Concluding remarks From the numerical simulation results given in Section 3 we can conclude the following:
Fig. 6. Three-dimensional phase space sections of the dynamics of the three-neuron HHNN when w33 = 2.2.
(i) HHNN can be served as one of the rich resources for generating sophisticate nonlinear dynamics beyond DHNN, such as those reported in [14]. (ii) HHNN employing smoothly hysteretic neurons is also able to exhibit various attractors: stable equilibrium, limit cycle, torus and chaos like those generated by the HHNN with binary hysteresis as reported in [5]. (iii) By merely changing one bifurcation parameter w33 between 1.4 and 2.5 one can observe interesting “order–chaos” mutually switching phenomena which would be of interest in the future research work. Future works regarding the HHNN model described by (2) with both theoretical and practical significance are suggested as below:
Fig. 7. Power spectrum of the time series of x1 when w33 = 2.2.
ferent quantitative characteristics in the HHNN model. Thus we can control the chaos by adjust parameter w33 .
(1) The aforementioned switching phenomena observed from numerical simulation results (Fig. 8) might be attributed to the bifurcation effect, thus the qualitative analysis of the bifurcation occurred in this three-neuron HHNN model is worthy being done in the future. (2) In practice, an inductor with magnetic core shows a smoothly hysteretic characteristics provided the core is made of “soft” magnetic materials, hence it is feasible to implement the neuron model depicted by (3)–(5) thereby to construct prototype
372
C. Li et al. / Physics Letters A 285 (2001) 368–372
circuitry of the HHNN model (2) and investigate further its engineering applications. (3) Since by simply controlling w33 one can obtain different chaotic attractors, it is therefore worthy studying applications of this HHNN model such as associate memory, shift-keying modulation in chaotic communication systems.
References [1] M. Livingstone, D. Hubel, Science 240 (1988) 740. [2] Y. Takefuji, K.C. Lee, Biol. Cybern. 64 (1991) 353. [3] T. Saito, M. Oikawa, IEEE Trans. Neural Networks 4 (1993) 43. [4] K. Jin’no, T. Saito, Electron. Comm. Jpn. 78 (1995) 76. [5] K. Jin’no, IEICE Trans. Fund. 79 (1996) 402.
[6] K. Jin’no, T. Saito, IEICE Trans. Fund. 79 (1996) 873. [7] K. Jin’no, T. Nakamura, T. Saito, IEEE Trans. CAS-I 46 (1999) 851. [8] S. Bharitkar, J.M. Mendel, IEEE Trans. NN 11 (2000) 879. [9] P.K. Das II, W.C. Schieve, Z.J. Zeng, Phys. Lett. A 161 (1991) 60. [10] K. Aihara, T. Takabe, M. Toyoda, Phys. Lett. A 144 (1990) 333. [11] K.E. Kurten, J.W. Clark, Phys. Lett. A 114 (1986) 413. [12] V.E. Bondarenko, Phys. Lett. A 236 (1997) 513. [13] T.B. Kepler, S. Datt, R.B. Meyer, H.L. Abbott, Physica D 46 (1990) 449. [14] C.M. Marcus, F.R. Waugh, R.M. Westervelt, Physica D 51 (1991) 234. [15] X. Liao, Z. Wu, J. Yu, IEEE Trans. SMC A 29 (1998) 692. [16] A. Wolf, J.B. Swift, H.L. Swinney, J.A. Vastano, Physica D 16 (1985) 285.
Physics Letters A 285 (2001) 368–372 www.elsevier.com/locate/pla
Chaos in a three-neuron hysteresis Hopfield-type neural network ✩ Chunguang Li ∗ , Juebang Yu, Xiaofeng Liao Department of Opto-Electronics Technology, University of Electronics Science and Technology of China, Chengdu, 610054, Sichuan, PR China Received 24 January 2001; received in revised form 29 May 2001; accepted 4 June 2001 Communicated by A.R. Bishop
Abstract A three-neuron hysteresis Hopfield-type neural network was constructed and its dynamics studied. Chaotic behavior and strange attractors were observed. Lyapunov exponents and power spectra are presented for these attractors 2001 Published by Elsevier Science B.V. Keywords: Chaos; Hysteresis; Neural network; Lyapunov exponent; Power spectrum
1. Introduction In last decade, much attention has been drawn on the research of chaotic artificial neural networks (CANN). A number of reports in this field revealed that complicate dynamics responsible for pattern recognition (such as pattern perception under attention mechanism in retina-brain system [1]), memory searching and learning [2–13] in biological neural systems can be simulated by CANN. In this research field, most works are focused on delayed-type neural networks [14,15] and hysteretic-type neural networks [2–7]. It is well known that hysteresis is defined as a lagging effect due to a change of force acting on a body. ✩
This work is supported by the Ministry of Information Industry of China under the Major Science and Technology Development Project, Grant No. 98048, and National Communication Key Labs, under the Project Grants No. 1DZ0207 and 1DZ0208. * Corresponding author. E-mail address: [email protected] (C. Li).
Hysteresis manifests itself in the structures of many cooperative dynamical systems, and is observed in animal’s (such as frogs and crayfish) behavior. Many engineering systems also display hysteresis. In [2–7], the authors studied some kinds of binary hysteresis neural networks, they observed and analyzed the bifurcation phenomenon and chaotic behavior of these neural networks. A simple hysteresis neural network was theoretically and experimentally studied in [5], where the network contains two neurons with binary hysteresis. In this Letter, we will report chaotic behavior and an interesting phenomenon called “order– chaos switching” caused by varying a network parameter in an HHNN (hysteresis Hopfield neural network [8]) containing three neurons with smooth hysteresis as shown in Fig. 1. The HHNN we studied can also exhibit various kinds of attractors such as stable equilibrium, limit cycle, torus and chaos. To the best of our knowledge, this is the first observation of chaotic behavior in a smooth hysteresis neural network as well as the report on such switching phenomenon in an HHNN.
0375-9601/01/$ – see front matter 2001 Published by Elsevier Science B.V. PII: S 0 3 7 5 - 9 6 0 1 ( 0 1 ) 0 0 3 8 1 - 4
C. Li et al. / Physics Letters A 285 (2001) 368–372
369
Fig. 1. (a) Hysteresis neuron, (b) hysteresis activation function.
2. Network (HHNN) model The HHNN [8] consists of a system of N neurons, each characterized by a neuron potential xi , such that Ci
xi (t) λi (x(t dxi (t) ˙ − δt)) =− + dt Ri Ri +
N
wij yj + Ii ,
i = 1, 2, . . . , N,
neuron gain is described as [8] y x(t)x(t ˙ − δ) = ϕ x(t) − λ x(t ˙ − δ) , where ϕ(p) = tanh(γp), and ˙ − δt) 0, γα , x(t γ x(t ˙ − δt) = γβ , x(t ˙ − δt) < 0, −α, x(t ˙ − δt) 0, λ x(t ˙ − δt) = β, x(t ˙ − δt) < 0,
(3)
(4) (5)
where β > −α, and (γα , γβ ) > 0, and
j =i=1
(1) where Ci is the input capacitance of neuron i, wij is the coupling constants (connection weights) between neurons i and j , and Ri is the membrane resistance. yi is a shorthand notation for y(xi |x˙i (t − δt)), given later. Such systems have been studied in the case of symmetric couplings [8]. However, our focus has been on small numbers of neurons connected in an asymmetric way with differing connection strengths and with self-connection weights. In the present study, we examine the dynamics of a simple, autonomous HHNN consisting of three neurons. In this case, (1) reduces to wij yj + wii yi , i = 1, 2, 3. x˙i (t) = −xi (t) + j =i=1
(2) As mentioned before, the hysteresis neurons of the network shown in Fig. 1(a) have the same activation function given in Fig. 1(b) and corresponding hysteresis
x(t) − x(t − δt) dx(t − δt) = lim . δt →0 dt δt Note that in special case when α = β, and γα = γβ , the activation function becomes the conventional sigmoid function. Note also the neuron output not only depends on its input, x(t), but also on the latter derivative information, x(t ˙ − δt). Therefore the HHNN exhibits memory behavior. The main purpose of our studies is to investigate how the dynamics such as bifurcations and chaos depend on the weights wij in the coupling matrix W = {wij }. The results presented in this Letter are for parameters choices as shown below with w33 taken as the bifurcation parameter: 1.95 −1.5 −0.3 W = 1.08 (6) 2 0.4 . 1.1 −0.5 w33 x(t ˙ − δt) =
It should be noted that several other coupling matrices have been examined and shown to exhibit similar
370
C. Li et al. / Physics Letters A 285 (2001) 368–372
dynamics. Due to limitation of space, such results will not be presented here.
3. Numerical studies and simulation results To solve Eqs. (1)–(6), a fourth-order Runge–Kutta method used with time step h = 0.01 is implemented. Some random values of xi (0) are chosen as the initial values. A time series of N = 100000 are analyzed. The power spectra are calculated using the general FFT. In order to calculate the largest Lyapunov exponent, we applied the method introduced by Wolf et. al [16]. Lyapunov exponents can be defined as the long-term evolution of the axes of an infinitesimal sphere of states. To estimate the largest Lyapunov exponent, we in effect monitor the long-term evolution of a single pair of nearby orbits. And the largest Lyapunov exponent is defined by λ=
L (tk ) 1 , M log2 tM − t0 L(tk−1 )
We start from the case when w33 = 1.6, under this condition, the neural network produces chaotic output and the dimensionless largest Lyapunov exponent λ = 0.0398. Fig. 2 shows the attractor in the threedimensional phase space. And the power spectra of the first neuron output are presented in Fig. 3. The spectrum is truncated at v = 1 Hz. The attractor and power spectra of the first neuron output when w33 = 2.0 and w33 = 2.2 are shown in Figs. 4, 5 and Figs. 6, 7, respectively. We can see clearly that Fig. 2 and Fig. 4 show two different type of strange attractors. The curve given in Fig. 8 shows our results for the largest Lyapunov exponent as a function of the parameter w33 . It is seems that λ is an irregular function of the parameter w33 . By varying w33 , it is possible to control the largest Lyapunov exponent of the HHNN outputs. Thereby we can also produce “chaos–order”, “order–chaos” and “chaos–chaos” transition with dif-
(7)
k=1
where M is the total number of replacement steps, L(tk−1 ) is the distance between the two initial points in the time point tk−1 , and after a time step ∆ = tk − tk−1 , the initial length will have evolved to length L (tk ). For a detail, please refer to [14]. The Lyapunov exponent λ is calculated from a time series consisting of N = 100000 points. The calculation carried out show that an asymmetric HHNN demonstrates chaotic outputs. We have examined a range of parameters and noted that the appearance of at least two types of strange attractors.
Fig. 2. Three-dimensional phase space sections of the dynamics of the three-neuron HHNN when w33 = 1.6.
Fig. 3. Power spectrum of the time series of x1 when w33 = 1.6.
Fig. 4. Three-dimensional phase space sections of the dynamics of the three-neuron HHNN when w33 = 2.0.
C. Li et al. / Physics Letters A 285 (2001) 368–372
Fig. 5. Power spectrum of the time series of x1 when w33 = 2.0.
371
Fig. 8. The largest Lyapunov exponent with w33 ranging from 1.4 to 2.3.
4. Concluding remarks From the numerical simulation results given in Section 3 we can conclude the following:
Fig. 6. Three-dimensional phase space sections of the dynamics of the three-neuron HHNN when w33 = 2.2.
(i) HHNN can be served as one of the rich resources for generating sophisticate nonlinear dynamics beyond DHNN, such as those reported in [14]. (ii) HHNN employing smoothly hysteretic neurons is also able to exhibit various attractors: stable equilibrium, limit cycle, torus and chaos like those generated by the HHNN with binary hysteresis as reported in [5]. (iii) By merely changing one bifurcation parameter w33 between 1.4 and 2.5 one can observe interesting “order–chaos” mutually switching phenomena which would be of interest in the future research work. Future works regarding the HHNN model described by (2) with both theoretical and practical significance are suggested as below:
Fig. 7. Power spectrum of the time series of x1 when w33 = 2.2.
ferent quantitative characteristics in the HHNN model. Thus we can control the chaos by adjust parameter w33 .
(1) The aforementioned switching phenomena observed from numerical simulation results (Fig. 8) might be attributed to the bifurcation effect, thus the qualitative analysis of the bifurcation occurred in this three-neuron HHNN model is worthy being done in the future. (2) In practice, an inductor with magnetic core shows a smoothly hysteretic characteristics provided the core is made of “soft” magnetic materials, hence it is feasible to implement the neuron model depicted by (3)–(5) thereby to construct prototype
372
C. Li et al. / Physics Letters A 285 (2001) 368–372
circuitry of the HHNN model (2) and investigate further its engineering applications. (3) Since by simply controlling w33 one can obtain different chaotic attractors, it is therefore worthy studying applications of this HHNN model such as associate memory, shift-keying modulation in chaotic communication systems.
References [1] M. Livingstone, D. Hubel, Science 240 (1988) 740. [2] Y. Takefuji, K.C. Lee, Biol. Cybern. 64 (1991) 353. [3] T. Saito, M. Oikawa, IEEE Trans. Neural Networks 4 (1993) 43. [4] K. Jin’no, T. Saito, Electron. Comm. Jpn. 78 (1995) 76. [5] K. Jin’no, IEICE Trans. Fund. 79 (1996) 402.
[6] K. Jin’no, T. Saito, IEICE Trans. Fund. 79 (1996) 873. [7] K. Jin’no, T. Nakamura, T. Saito, IEEE Trans. CAS-I 46 (1999) 851. [8] S. Bharitkar, J.M. Mendel, IEEE Trans. NN 11 (2000) 879. [9] P.K. Das II, W.C. Schieve, Z.J. Zeng, Phys. Lett. A 161 (1991) 60. [10] K. Aihara, T. Takabe, M. Toyoda, Phys. Lett. A 144 (1990) 333. [11] K.E. Kurten, J.W. Clark, Phys. Lett. A 114 (1986) 413. [12] V.E. Bondarenko, Phys. Lett. A 236 (1997) 513. [13] T.B. Kepler, S. Datt, R.B. Meyer, H.L. Abbott, Physica D 46 (1990) 449. [14] C.M. Marcus, F.R. Waugh, R.M. Westervelt, Physica D 51 (1991) 234. [15] X. Liao, Z. Wu, J. Yu, IEEE Trans. SMC A 29 (1998) 692. [16] A. Wolf, J.B. Swift, H.L. Swinney, J.A. Vastano, Physica D 16 (1985) 285.