- Email: [email protected]
Multistability of neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays
Accepted Manuscript Multistability of neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays Xiaobing Nie, Wei Xing Zheng PII: DOI: Reference:
S0893-6080(15)00031-3 http://dx.doi.org/10.1016/j.neunet.2015.01.007 NN 3440
To appear in:
Neural Networks
Received date: 23 September 2014 Revised date: 22 December 2014 Accepted date: 25 January 2015 Please cite this article as: Nie, X., & Zheng, W. X. Multistability of neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays. Neural Networks (2015), http://dx.doi.org/10.1016/j.neunet.2015.01.007 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Multistability of Neural Networks with Discontinuous Non-monotonic Piecewise Linear Activation Functions and Time-Varying Delays Xiaobing Niea,b , Wei Xing Zhengb,∗ Department of Mathematics, Southeast University, Nanjing 210096, China School of Computing, Engineering and Mathematics, University of Western Sydney, Sydney NSW 2751, Australia a
b
Abstract This paper is concerned with the problem of coexistence and dynamical behaviors of multiple equilibrium points for neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays. The fixed point theorem and other analytical tools are used to develop certain sufficient conditions that ensure that the n-dimensional discontinuous neural networks with time-varying delays can have at least 5n equilibrium points, 3n of which are locally stable and the others are unstable. The importance of the derived results is that it reveals that the discontinuous neural networks can have greater storage capacity than the continuous ones. Moreover, different from the existing results on multistability of neural networks with discontinuous activation functions, the 3n locally stable equilibrium points obtained in this paper are located in not only saturated regions, but also unsaturated regions, due to the non-monotonic structure of discontinuous activation functions. A numerical simulation study is conducted to illustrate and support the derived theoretical results. Keywords: Neural networks, Multistability, Instability, Discontinuous non-monotonic piecewise linear activation functions, Time-varying delays.
Corresponding author. Tel.: +61 2 47360608; fax: +61 2 47360374. Email addresses: [email protected] (Xiaobing Nie), [email protected] (Wei Xing Zheng) ∗
Preprint submitted to Neural Networks
January 30, 2015
1. Introduction The coexistence of multiple equilibrium points and their local stability is referred to as the “multistability” of neural networks. Multistability is necessary whenever neural networks are used for implementing an associative memory or for solving in real time other tasks in the field of combinatorial optimization, pattern recognition and image processing (Cohen & Grossberg , 1983; Thiran, Setti & Hasler , 1998). For example, in the application of associative memory, the addressable memories are usually stored as stable equilibrium points and the process of memory attainment is that the network converges to a certain stable equilibrium point. The number of locally stable equilibrium points corresponds to the storage capacity of neural networks. Thus, it is more desirable that neural networks can have multiple locally stable equilibrium points. The past few years have witnessed a growing research interest in multistability of neural networks and some results on this topic have been reported. In Cheng, Lin and Shih (2007), the existence of 2n stable equilibrium points for general n-dimensional delayed neural networks with two classes of activation functions was presented through formulating parameter conditions motivated by a geometrical observation. Based on decomposition of the state space, Cao, Feng and Wang (2008) investigated the multistability of Cohen-Grossberg neural networks with nondecreasing saturated activation functions with two corner points. In Lin and Shih (2007); Wang, Lu and Chen (2009), the multistability of almost-periodic solution in delayed neural networks was studied. In Huang and Cao (2010), a delay-dependent multistability criterion on recurrent neural networks was derived by using the matrix inequality technique and constructing Lyapunov functional. Kaslik & Sivasundaram (2011) were the first who revealed the effect of impulse on the multistability of neural networks. In Huang, Song and Feng (2010); Nie and Cao (2011), high-order synaptic connectivity was introduced into neural networks and the multistability was considered for high-order neural networks based on decomposition of state space, Cauchy convergence principle and inequality technique. Using the theory of monotone semiflows, the multistability of cooperative neural networks with piecewise linear activation functions was investigated in Di Marco, Forti, Grazzini and Pancioni (2012). In order to increase storage capacity, the Hopfield neural networks and competitive neural networks with nondecreasing piecewise linear activation functions with 2 r corner points were considered in Wang, Lu and 2
Chen (2010); Nie, Cao and Fei (2013), respectively. It was proved that under some conditions, the n-neuron neural networks have exactly (2 r + 1)n equilibrium points, (r + 1)n of which are locally exponentially stable and the others are unstable. In Zeng and Zheng (2012), the multistability of neural networks with k + m step stair activation functions was discussed based on an appropriate partition of the n-dimensional state space and characteristics of nonlinear activation functions. It was shown therein that the n-neuron neural networks can have (2k + 2m − 1)n equilibrium points, (k + m)n of which are locally exponentially stable. In particular, the case of k = m was previously discussed in Zeng, Huang and Zheng (2010). In Zhang, Zhang and Li (2008), some multistability properties for a class of delayed recurrent neural networks with unsaturating piecewise linear transfer functions were studied based on local inhibition. For more related work, see Pan, Wang, Gao, Li and Du (2010); Huang, Wang and Feng (2010); Huang, Feng and Mohamad (2012); Zhou and Song (2013); Zeng and Zheng (2013); Wang and Chen (2014); Huang, Raffoul and Cheng (2014) and the references therein. One underlying assumption adopted for all the aforementioned multistability results is that the activation functions are continuous. However, as pointed out in Forti and Nistri (2003), neural networks with discontinuous activation functions are important and frequently do arise in practice. When dealing with dynamical systems possessing high-slope nonlinear elements, it is often advantageous to model them with a system of differential equations with discontinuous right-hand side. As a result, the mono-stability of neural networks with discontinuous activation functions has been studied intensively over the last decade (Forti, Nistri & Papini , 2005; Lu & Chen , 2005, 2008; Wang, Huang & Guo , 2009; Liu & Cao , 2009; Nie & Cao , 2012; Huang, Cai & Zhang , 2013; Wang & Huang , 2014). However, to the best of our knowledge, there have been only a few papers investigating the multistability of neural networks with discontinuous activation functions (see Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b)). Furthermore, only piecewise constants activation functions were used for the discontinuous activation functions utilized in these works, which are simple and monotonically increasing. It has been well recognized that the multistability analysis of neural networks critically depends upon the type of activation functions used. In the above-mentioned works as well as most existing works, the activation functions employed in multistability analysis, including continuous activation 3
functions and discontinuous activation functions, were mainly focused on sigmoidal activation functions, nondecreasing saturated activation functions and piecewise constants activation functions, which are all monotonically increasing. Recent works by Wang and Chen (2012), and Nie, Cao and Fei (2014) have started a new research line concerning the study of multistability for neural networks with continuous non-monotonic piecewise linear activation functions, which are defined by ui , −∞ < x < pi , li,1 x + ci,1 , pi ≤ x ≤ ri , fi (x) = (1) li,2 x + ci,2 , ri < x ≤ qi , ui , qi < x < +∞,
where pi , ri , qi , ui , li,1 , li,2 , ci,1 , ci,2 are constants with −∞ < pi < ri < qi < +∞, li,1 > 0, li,2 < 0, and ui = fi (pi ) = fi (qi ). Under the assumption that the index set {1, 2, · · · , n} can be decomposed into four subsets with respect to different external input ranges, it was shown in Wang and Chen (2012) that the n-neuron neural networks without delays have 3]N2 equilibrium points in all, and 2]N2 of them are locally stable, where ]N2 denotes the number of elements in the second index subset N2 . Motivated by the above continuous activation functions, in this paper, we introduce a general class of discontinuous activation functions, which are defined as follows (see Fig. 1): ui , −∞ < x < pi , li,1 x + ci,1 , pi ≤ x ≤ ri , (2) fi (x) = li,2 x + ci,2 , ri < x ≤ qi , vi , qi < x < +∞,
where pi , ri , qi , ui , vi , li,1 , li,2 , ci,1 , ci,2 are constants with −∞ < pi < ri < qi < +∞, li,1 > 0, li,2 < 0, ui = fi (pi ) = fi (qi ), fi (ri ) = li,2 ri + ci,2 and vi > fi (ri ) i = 1, 2, · · · , n. Obviously, fi (x) is continuous in < except at the point of discontinuity x = qi , where there exist finite right and left limits fi (qi+ ) and fi (qi− ), respectively, with fi (qi+ ) = vi > ui = f (qi− ). Note that activation functions (2) are discontinuous and non-monotonically increasing, which are totally different from those employed in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where the activation functions are piecewise constants and monotonically increasing. Hence, the existing methods cannot be applied directly to activation 4
8
6
f(x)
4
2
0
−2
−4 −5
0
5
10
15
x
Figure 1: Discontinuous non-monotonic piecewise linear activation functions (2).
functions (2). Besides, it is easy to see that activation functions (1) and (2) are almost the same except for the value in interval (qi , +∞). Thus, some nontrivial questions arise naturally: which type of activation functions will lead to both more total equilibrium points and more locally stable equilibrium points? how does the discontinuity of activation function affect the multistability of neural networks? In this paper, we will give a definitive answer to these questions. Inspired by these concerns, in this paper, we will investigate the dynamical behaviors of multiple equilibrium points for neural networks with discontinuous activation functions (2) and time-varying delays. Note that time delays are inevitable in many real-world neural networks due to, for example, signal transmission, and time delays are often the source of oscillation and instability (Marcus & Westervelt , 1989). More precisely, the main contributions of this paper are as follows: • By means of the known fixed point theorem, we develop sufficient conditions under which the n-neuron neural networks with discontinuous activation functions (2) and time-varying delays have at least 5n equilibrium points in
• We make rigorous mathematical analysis of the dynamical behaviors of each equilibrium point for the corresponding discontinuous neural networks with time-varying delays, and prove that the addressed neural networks can have 5n equilibrium points, 3n of them are locally stable and the others are unstable. • Compared with the result in Wang and Chen (2012), the neural networks with discontinuous activation functions (2) have both more total equilibrium points and more locally stable equilibrium points than the same ones with continuous activation functions (1). Furthermore, different from results in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where locally stable equilibrium points are only located in saturated regions, the 3n locally stable equilibrium points given in this paper are located in not only saturated regions, but also unsaturated regions, due to the non-monotonic structure of discontinuous activation functions (2). • We reveal the effect of discontinuity of activation functions on the coexistence and local stability of multiple equilibrium points for the discontinuous neural networks and arrive at an important insight: discontinuous neural networks may have greater storage capacity than the continuous ones. • We substantiate the theoretical findings by an illustrative example with comprehensive computer simulations. 2. Model formulation and preliminaries 2.1. Notation Let x(t) ˙ denote the derivative of x(t). Given a set Ω ⊂
and Ωn =
n Q
i=1
(i)
(i)
(i)
(−∞, pi )δ1 × [pi , ri ]δ2 × (ri , qi )δ3 × (i)
(i)
(i)
(i)
(i)
(qi , +∞)δ4 , (δ1 , δ2 , δ3 , δ4 ) = (1, 0, 0, 0)
(3)
or (0, 1, 0, 0) or (0, 0, 1, 0) or (0, 0, 0, 1) .
It is easy to see that Ωn is composed of 4n regions. 2.2. Model Consider a general class of recurrent neural networks with time-varying delays described by the following equations n
n
X X dxi (t) = − di xi (t) + aij fj (xj (t)) + bij fj (xj (t − τij (t))) + Ii , dt j=1 j=1
(4)
where i = 1, 2, · · · , n, x(t) = (x1 (t), · · · , xn (t))T ∈ 0, A = (aij ) and B = (bij ) are connection weight matrices that are not assumed to be symmetric, I = (I1 , I2 , · · · , In )T is an input vector, fj (·) is the activation function defined in (2), τij (t) is the time-varying delay that satisfies 0 ≤ τij (t) ≤ τ = max1≤i,j≤n {sup{τij (t), t ≥ 0}}, and τ is a constant number. 2.3. Definitions Since neural networks (4) with activation functions (2) are systems of differential equations with discontinuous right-hand side, we need to specify what is meant by a solution of (4). Let K[f (x)] = (K[f1 (x1 )], K[f2 (x2 )] · · · , + K[fn (xn )])T and K[fi (xi )] = [fi (x− i ), fi (xi )]. Definition 1. A function x(t) = (x1 (t), · · · , xn (t))T : [−τ, T ) →
Definition 2. An equilibrium point of (4) is a vector x∗ ∈
n X
j6=i,j=1
− di ri + (aii + bii ) fi (ri ) +
− di qi + (aii + bii ) vi +
max (aij + bij ) uj , (aij + bij ) vj + Ii < 0,
(5)
n X
j6=i,j=1
n X
j6=i,j=1
min (aij + bij ) uj , (aij + bij ) vj + Ii > 0,
(6)
min (aij + bij ) uj , (aij + bij ) vj + Ii > 0,
(7)
then neural networks (4) with activation functions (2) have 4n equilibrium points located in Ωn . Proof. Arbitrarily pick a region from the set Ωn as Y Y Y Y en = (qi , +∞) ⊂ Ωn , (ri , qi ) × [pi , ri ] × (−∞, pi ) × Ω i∈N1
i∈N3
i∈N2
i∈N4
where Ni (i = 1, 2, 3, 4) are subsets of {1, 2, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {1, 2, · · · , n}, Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 2, 3, 4). We will show that neural 8
networks (4) with activation functions (2) have at least an equilibrium point en. located in Ω Note that, the equilibrium points of neural networks (4) are the same as the equilibrium points of the following neural networks without delay n
X dxi (t) = −di xi (t) + (aij + bij )fj (xj (t)) + Ii , i = 1, 2, · · · , n. dt j=1
(8)
e n and index i = 1, 2, · · · , n, fix x1 , · · · , xi−1 , For any point (x1 , x2 , · · · , xn )T ∈ Ω xi+1 , · · · , xn except for xi , and define Fi (x) = −di x + (aii + bii )fi (x) +
n X
(aij + bij )fj (xj ) + Ii .
(9)
j6=i,j=1
Then there are four possible cases for us to discuss. Case 1. i ∈ N1 . Noting that ui = fi (pi ) and uj ≤ fj ≤ vj , from (5) and (9), we have lim Fi (x) = +∞, Fi (pi ) < 0. x→−∞
Then due to the continuity of Fi (x) (x ∈ (−∞, pi ]), we can find an x¯i ∈ (−∞, pi ) such that Fi (¯ xi ) = 0. Case 2. i ∈ N2 . From (5), (6) and (9), we get Fi (pi ) < 0,
Fi (ri ) > 0.
Then we can find an x¯i ∈ (pi , ri ) such that Fi (¯ xi ) = 0. Case 3. i ∈ N3 . Note that fi (qi ) = ui and pi < qi . It follows from (5) that n P (aij + bij )fj (xj ) + Ii Fi (qi ) = −di qi + (aii + bii )fi (qi ) + j6=i,j=1
< −di pi + (aii + bii )ui + < 0.
n P
j6=i,j=1
max (aij + bij ) uj , (aij + bij ) vj + Ii
Then we can find an x¯i ∈ (ri , qi ) such that Fi (¯ xi ) = 0, owing to that Fi (ri ) > 0, Fi (qi ) < 0. Case 4. i ∈ N4 . By virtue of (7) and (9), we obtain lim Fi (x) > 0,
x→qi+
lim Fi (x) = −∞.
x→+∞
9
Then we can also find an x¯i ∈ (qi , +∞) such that Fi (¯ xi ) = 0, because of the continuity of Fi (x) (x ∈ (qi , +∞)). en → Ω e n by H(x1 , x2 , · · · , xn ) = (¯ Define a map H : Ω x1 , x¯2 , · · · , x¯n ). It is clear that the map H is continuous. By Brouwer’s fixed point theorem, e n , which is also the there exists one fixed point x∗ = (x∗1 , · · · , x∗n ) of H in Ω e n . As Ωn is divided into equilibrium point of neural networks (8) and (4) in Ω e n , neural networks (4) discontinuous activation 4n parts, by arbitrariness of Ω n functions (2) have 4 equilibrium points located in Ωn . Remark 1. Inequalities (5)–(7) imply that aii + bii > 0, vi > ui , i = 1, 2, · · · , n. In fact, from (5) and (6), we obtain −di pi + (aii + bii ) ui < −di ri + (aii + bii ) fi (ri ), i.e., 0 < di (ri − pi ) < (aii + bii )(fi (ri ) − ui ), which means that aii + bii > 0, due to fi (ri ) − ui > 0, i = 1, 2, · · · , n. Similarly, it follows from (5) and (7) that −di pi + (aii + bii ) ui < −di qi + (aii + bii ) vi , i.e., 0 < di (qi − pi ) < (aii + bii )(vi − ui ). Thus, we get that vi > ui , due to aii + bii > 0, i = 1, 2, · · · , n. Lemma 1 given above only considers the case where all components of equilibrium points are the points of continuity of activation functions. It is important to know if there exist some equilibrium points whose components are located at the points of discontinuity of activation functions. This issue is addressed in the following lemma. Lemma 2. Under the conditions of Lemma 1, neural networks (4) with discontinuous activation functions (2) and time-varying delays have at least Pn j n−j equilibrium points which are located at the points of discontij=1 Cn 4 nuity of function f (x) = (f1 (x1 ), · · · , fn (xn ))T , where Cnj = n(n−1)···(n−j+1) . j(j−1)···2·1
10
Proof. For any point (x1 , x2 , · · · , xn )T ∈
n X
j=2 n X
(a1j + b1j )fj (xj ) + I1 ,
(10)
(aij + bij )fj (xj ) + Ii ,
j=2
i = 2, 3, · · · , n.
(11)
We first prove that (10) holds for any (x2 , x3 , · · · , xn )T ∈ 0 (Remark 1), we have (a11 +b11 )[u1 , v1 ] = [(a11 +b11 ) u1, (a11 + b11 ) v1 ]. It follows from uj ≤ fj ≤ vj , (5) and (7) that − d1 q1 + (a11 + b11 ) u1 + < − d1 p1 + (a11 + b11 ) u1 + < 0, − d1 q1 + (a11 + b11 ) v1 + ≥ − d1 q1 + (a11 + b11 ) v1 + > 0,
n X
(a1j + b1j )fj (xj ) + I1
j=2 n X j=2
n X
j=2 n X j=2
max{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1 (12)
(a1j + b1j )fj (xj ) + I1 min{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1 (13)
which imply that (10) holds.
11
Arbitrarily pick a region from the set Ωn−1 as Y Y Y Y e n−1 = (qi , +∞), (ri , qi ) × [pi , ri ] × (−∞, pi ) × Ω i∈N1
i∈N4
i∈N3
i∈N2
where Ni (i = 1, 2, 3, 4) are subsets of {2, 3, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {2, 3, · · · , n}, Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 2, 3, 4). We will show e n−1 such that (11) holds. For any point that there exists (x∗2 , · · · , x∗n )T ∈ Ω T e n−1 and index i = 2, · · · , n, fix x2 , · · · , xi−1 , xi+1 , · · · , xn (x2 , · · · , xn ) ∈ Ω except xi , and define Fi (x) = − di x + (aii + bii ) fi (x) + a ˜i1 + ˜bi1 n X + (aij + bij )fj (xj ) + Ii ,
(14)
j6=i,j=2
where i = 2, · · · , n, a ˜i1 + ˜bi1 ∈ (ai1 + bi1 )[u1 , v1 ]. It is obvious that min{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 } ≤ a˜i1 + ˜bi1 ≤ max{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 }. Then there are four possible cases for us to discuss. (a) i ∈ N1 . Note that fi (pi ) = ui , uj ≤ fj ≤ vj and a ˜i1 + ˜bi1 ≤ max{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 }. It follows from (5) that Fi (pi ) = − di pi + (aii + bii ) ui + a˜i1 + ˜bi1 + ≤ − di pi + (aii + bii ) ui + < 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj ) + Ii
j6=i,j=2
max{(aij + bij ) uj , (aij + bij ) vj } + Ii
Then due to limx→−∞ Fi (x) = +∞ and the continuity of Fi (x) (x 6= qi ), we can find an x¯i ∈ (−∞, pi ) such that Fi (¯ xi ) = 0.
12
(b) i ∈ N2 . From inequality a ˜i1 + ˜bi1 ≥ min{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 } and (6), we get Fi (ri ) = − di ri + (aii + bii ) fi (ri ) + a ˜i1 + ˜bi1 + + Ii ≥ − di ri + (aii + bii ) fi (ri ) + + Ii > 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj )
j6=i,j=2
min{(aij + bij ) uj , (aij + bij ) vj }
Then we can find an x¯i ∈ (pi , ri ) such that Fi (¯ xi ) = 0, because of Fi (pi ) < 0. (c) i ∈ N3 . It follows from (5) that lim− Fi (x) = − di qi + (aii + bii ) ui + a ˜i1 + ˜bi1 +
x→qi
+ Ii < − di pi + (aii + bii ) ui + + Ii < 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj )
j6=i,j=2
max{(aij + bij ) uj , (aij + bij ) vj }
Then we can find an x¯i ∈ (ri , qi ) such that Fi (¯ xi ) = 0, owing to Fi (ri ) > 0. (d) i ∈ N4 . By virtue of (7), we obtain that lim+ Fi (x) = − di qi + (aii + bii ) vi + a˜i1 + ˜bi1 +
x→qi
≥ − di qi + (aii + bii ) vi + + Ii > 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj ) + Ii
j6=i,j=2
min{(aij + bij ) uj , (aij + bij ) vj }
Then we can also find an x¯i ∈ (qi , +∞) such that Fi (¯ xi ) = 0, in view of the continuity of Fi (x) (x 6= qi ) and limx→+∞ Fi (x) = −∞. 13
e n−1 → Ω e n−1 by H(x2 , · · · , xn ) = (¯ Define a map H : Ω x2 , · · · , x¯n ). It is clear that the map H is continuous. By Brouwer’s fixed point theorem, e n−1 , which also satisfies there exists one fixed point (x∗2 , · · · , x∗n )T of H in Ω ∗ ∗ T (11). Thus, we obtain that (q1 , x2 , · · · , xn ) is the equilibrium point of neural e n−1 , networks (4). As
(15)
j=3
0 ∈ − d2 q2 + (a21 + b21 )[u1 , v1 ] + (a22 + b22 )[u2 , v2 ] n X (a2j + b2j )fj (xj ) + I2 , +
(16)
0 ∈ − di xi + (ai1 + bi1 )[u1 , v1 ] + (ai2 + bi2 )[u2 , v2 ] n X + (aij + bij )fj (xj ) + Ii , i = 3, 4, · · · , n.
(17)
j=3
j=3
We first prove that (15) holds for any (x3 , · · · , xn )T ∈ 0 (Remark 1), we get that (a11 + b11 )[u1 , v1 ] = [(a11 + b11 ) u1 , (a11 + b11 ) v1 ], (a12 + b12 )[u2 , v2 ] = [a12 + b12 , a ¯12 + ¯b12 ], where a12 + b12 = min{(a12 + b12 ) u2, (a12 + b12 ) v2 }, a ¯12 + ¯b12 = max{(a12 + b12 ) u2 , (a12 + b12 ) v2 }. It follows
14
from (5) and (7) that − d1 q1 + (a11 + b11 ) u1 + min{(a12 + b12 ) u2 , (a12 + b12 ) v2 } n X (a1j + b1j )fj (xj ) + I1 + j=3
< − d1 p1 + (a11 + b11 ) u1 + < 0,
n X j=2
max{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1
− d1 q1 + (a11 + b11 ) v1 + max{(a12 + b12 ) u2 , (a12 + b12 ) v2 } n X + (a1j + b1j )fj (xj ) + I1 j=3
≥ − d1 q1 + (a11 + b11 ) v1 + > 0.
n X j=2
min{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1
The above two inequalities imply that (15) holds for any (x3 , · · · , xn )T ∈
i∈N3
i∈N2
i∈N4
where Ni (i = 1, 2, 3, 4) are subsets of {3, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {3, · · · , n}, Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 2, 3, 4). Similar to Case 1, we can e n−2 such that (17) holds. By arbiprove that there exists (x∗3 , · · · , x∗n )T ∈ Ω n−2 e n−2 , we can obtain 4 trariness of Ω equilibrium points of neural networks (4), and the first two components of these equilibrium points have the values xi = qi , i = 1, 2.. For an n−dimensional vector (x1 , · · · , xn )T , it is easy to see that the number of ways choosing xi = qi and xj = qj (i 6= j) is Cn2 . Other Cn2 − 1 subcases can be proved similarly. Therefore, for Case 2, there are at least Cn2 · 4n−2 equilibrium points of neural networks (4). Case 3. For other cases of δ( = )גj, j = 3, · · · , n, similar arguments can be employed to prove that the number of equilibrium points is at least Cnj · 4n−j , j = 3, · · · , n. 15
To sum up, neural (4) with discontinuous activation functions Pn networks j n−j equilibrium points which are the points of (2) have at least j=1 Cn 4 discontinuity of function f (x) = (f1 (x1 ), · · · , fn (xn ))T . P From the fact that 4n + nj=1 Cnj 4n−j = (4 + 1)n = 5n , and combining Lemma 1 and Lemma 2, we immediately obtain the following theorem on the coexistence of multiple equilibrium points of neural networks (4). Theorem 1. Suppose that the conditions (5), (6) and (7) hold. Then neural networks (4) with discontinuous activation functions (2) and time-varying delays can have at least 5n equilibrium points in
n Q
(i)
(i)
(i)
(−∞, pi )δ1 × (ri , qi )δ2 × (qi , +∞)δ3 ,
i=1
(i) (i) (i) (δ1 , δ2 , δ3 )
Φ2 = Ωn − Φ1 .
= (1, 0, 0) or (0, 1, 0) or (0, 0, 1) ,
It is easy to see that Φ1 is composed of 3n regions, Φ2 is composed of 4n − 3n regions. Next, we will precisely figure out all equilibrium points of neural networks (4) and analyze the dynamical behaviors of each equilibrium point. The following theorem shows that under some conditions, neural networks (4) with discontinuous activation functions (2) and time-varying delays can have at least 5n equilibrium points, and 3n equilibrium points located in Φ1 are locally stable.
16
Theorem 2. Suppose that the following conditions − di pi + aii ui +
n X
j6=i,j=1
− di ri + aii fi (ri ) +
− di qi + aii vi +
max{aij uj , aij vj } +
n X
j6=i,j=1
n X
j6=i,j=1
n X j=1
min{aij uj , aij vj } +
min{aij uj , aij vj } +
n X j=1
max{bij uj , bij vj } + Ii < 0, (18)
n X j=1
min{bij uj , bij vj } + Ii > 0, (19)
min{bij uj , bij vj } + Ii > 0 (20)
hold for all i = 1, 2, · · · , n. If there are positive constants ξ1 , ξ2 , · · · , ξn such that n n X X ξj |bij ||lj,2| < 0 (21) ξj |aij ||lj,2| + (−di + aii li,2 ) ξi + j=1
j6=i,j=1
hold for all i = 1, 2, · · · , n, then neural networks (4) with discontinuous activation functions (2) and time-varying delays have at least 5n equilibrium points, 3n of which located in Φ1 are locally stable. Proof. First of all, it is easy to see that conditions (18)–(20) imply the conditions (5)–(7) in Theorem 1 hold. Thus, according to Theorem 1, the coexistence of 5n equilibrium points for neural networks (4) can be guaranteed under the conditions of Theorem 2. In the following, we will prove the local stability of 3n equilibrium points in two steps. Step I. Arbitrarily pick a region from the set Φ1 as Y Y Y e1 = (qi , +∞) ⊂ Φ1 , (ri , qi ) × (−∞, pi ) × Φ i∈N1
i∈N4
i∈N3
where N1 , N3 , N4 are subsets of {1, 2, · · · , n}, and N1 ∪N3 ∪N4 = {1, 2, · · · , n}, e 1 is an invariant set of Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 3, 4). We will show that Φ e 1 ), neural networks (4). That is, for any initial condition φ(θ) ∈ C([−τ, 0], Φ e 1 for all t ≥ 0. If this is not true, then we claim that the solution x(t; φ) ∈ Φ there are three possible cases to discuss. 17
Case 1. There exists a component xi (t) of x(t; φ) which is the first (or Q one of the firsts) escaping from i∈N1 (−∞, pi ). Restated, there exist some i ∈ N1 and t∗ > 0 such that xi (t∗ ) = pi − ( is a small enough positive number), x˙ i (t∗ ) > 0, xi (t) ≤ pi − for −τ ≤ t ≤ t∗ . It follows from (4), (18) and the definition of fi that x˙ i (t∗ ) = −di xi (t∗ ) + aii fi (xi (t∗ )) + n P
+
j=1
n P
aij f (xj (t∗ ))
j6=i,j=1
bij fj (xj (t − τij (t∗ ))) + Ii + di
n P
max bij uj , bij vj + Ii + di
∗
≤ −di pi + aii ui + j=1
≤ 0,
j6=i,j=1
n P
j=1
+
aij fj (xj (t∗ ))
bij fj (xj (t∗ − τij (t∗ ))) + Ii
= −di pi + aii ui + +
n P
n P
j6=i,j=1
max aij uj , aij vj
which is a contradiction. Case 2. There exists a component xi (t) of x(t; φ) which is the first (or one Q of the firsts) escaping from i∈N3 (ri , qi ). Restated, there exist some i ∈ N3 and t∗ > 0 such that either xi (t∗ ) = ri + ( is a small enough positive number), x˙ i (t∗ ) < 0, xi (t) ∈ [ri +, qi ] for −τ ≤ t ≤ t∗ or xi (t∗ ) = qi − ( is a small enough positive number), x˙ i (t∗ ) > 0, xi (t) ∈ (ri , qi −] for −τ ≤ t ≤ t∗ . For the first case, from (4), (19) and the continuity of fi (x) (x ∈ [ri , ri + ]) we derive that x˙ i (t∗ ) ≥ − di ri + aii fi (ri + ) + +
n X j=1
≥ 0,
n X
j6=i,j=1
min aij uj , aij vj
min bij uj , bij vj + Ii − di
18
which is a contradiction. Similarly, for the second case, by virtue of ui = fi (qi ), pi < qi , (4), (18) and the continuity of fi (x) (x ∈ [qi − , qi ]), we get ∗
x˙ i (t ) ≤ − di qi + aii fi (qi − ) + +
n X j=1
≤ 0,
n X
j6=i,j=1
max aij uj , aij vj
max bij uj , bij vj + Ii + di
which is also a contradiction. Case 3. There exists a component xi (t) of x(t; φ) which is the first (or Q one of the firsts) escaping from i∈N4 (qi , +∞). Restated, there exist some i ∈ N4 and t∗ > 0 such that xi (t∗ ) = qi + ( is a small enough positive number), x˙ i (t∗ ) < 0, xi (t) ≥ qi + for −τ ≤ t ≤ t∗ . Then we get from (4) and (20) that ∗
x˙ i (t ) ≥ − di qi + aii vi + +
n X j=1
≥ 0,
n X
j6=i,j=1
min aij uj , aij vj
min bij uj , bij vj + Ii − di
which contradicts x˙ i (t∗ ) < 0. From the above three cases, we know that the solution x(t; φ) will never e 1 for all t ≥ 0. That is, Φ e 1 is an invariant set of model (4). escape from Φ e 1 is Step II. We will prove the equilibrium point x∗ of system (4) in Φ locally stable. Let x(t) = x(t; φ) be a solution of system (4) with initial e 1 ). By the positive invariance of Φ e , we know condition φ(θ) ∈ C([−τ, 0], Φ P 1 e e that P x(t) ∈ Φ1 for all t ≥ 0. Note that P when x(t) ∈ Φ1 , P j∈N1 fj (xj (t)) ∗ and i.e., j∈N4 fj (xj (t)) j∈N1 fj (xj (t)) = j∈N1 fj (xj ), P Pare constants, ∗ ∗ j∈N4 fj (xj (t)) = j∈N4 fj (xj ). Let yi (t) = xi (t) − xi . Then it follows
19
from model (4) that y˙ i (t) = − di yi (t) + +
n X j=1
bij
n X
aij
j=1
fj (x∗j
= − di yi (t) +
X
fj (x∗j
+ yj (t)) −
fj (x∗j )
+ yj (t − τij (t))) − aij lj,2 yj (t) +
X
bij lj,2 yj (t − τij (t)).
j∈N3
j∈N3
fj (x∗j )
(22)
From (21), there exists a positive constant ε small enough such that (−di + aii li,2 + ε) ξi +
n X
j6=i,j=1
ετ
ξj |aij | |lj,2| + e
n X j=1
ξj |bij | |lj,2| < 0.
(23)
Define zi (t) = eεt yi (t) and −1 M(t) = sup max ξi |zi (s)| , t ≥ 0. s≤t
i∈N3
(24)
In the following, we claim that M(t) is bounded. More precisely, for all t ≥ 0, we have M(t) = M(0). In fact, for any t0 ≥ 0, there are two possible cases: Case 1. maxi∈N3 ξi−1 |zi (t0 )| < M(t0 ). In this case, there exists a δ > 0 such that maxi∈N3 ξi−1 |zi (t)| < M(t0 ) for t ∈ (t0 , t0 + δ). Case 2. maxi∈N3 ξi−1 |zi (t0 )| = M(t0 ). In this case, let it0 = it0 (t0 ) ∈ N3 be such an index that −1 (t )| = max |z ξi−1 ξ |z (t )| . 0 i i 0 t i t0 0 i∈N3
20
By using (22) and (23), we derive that d|zit0 (t)| dt t=t0 " εt0 − dit0 yit0 (t0 ) + ait0 it0 lit0 ,2 yit0 (t0 ) = ε|zit0 (t0 )| + sign zit0 (t0 ) e X
+
ait0 j lj,2 yj (t0 ) +
X
j∈N3
j6=it0 ,j∈N3
bit0 j lj,2 yj (t0 − τit0 j (t0 ))
≤(−dit0 + ait0 it0 lit0 ,2 + ε)|zit0 (t0 )| + +
X
j∈N3
X
j∈N3
+e
X
j∈N3
≤ 0.
|ait0 j | |lj,2| |zj (t0 )|
X
j6=it0 ,j∈N3
ξj |ait0 j | |lj,2| · max ξi−1 |zi (t0 )|
ξj |bit0 j | |lj,2| · max ξi−1 |zi (t0 − τit0 j (t0 ))| i∈N3
≤ (−dit0 + ait0 it0 lit0 ,2 + ε)ξit0 + ετ
j6=it0 ,j∈N3
eετit0 j (t0 ) |bit0 j | |lj,2| |zj (t0 − τit0 j (t0 ))|
≤ (−dit0 + ait0 it0 lit0 ,2 + ε)ξit0 + + eετ
X
#
#
X
j6=it0 ,j∈N3
i∈N3
ξj |ait0 j | |lj,2|
ξj |bit0 j | |lj,2| M(t0 )
Then there exists a δ1 > 0 such that M(t) = M(t0 ) for t ∈ (t0 , t0 + δ1 ). Therefore, we can conclude that M(t) = M(0) for all t ≥ 0, which implies that max ξi−1 |zi (t)| ≤ M(0). i∈N3
Then we can arrive at
|yi (t)| ≤ M(0) ξi e−εt ≤ Me−εt , i ∈ N3 ,
21
where M = M(0) maxi∈N3 {ξi }. Therefore, we obtain that limt→+∞ |yi (t)| = 0 (i ∈ N3 ). Thus, for any ε1 > 0, there exists a constant T > 0 such that X X |bij ||lj,2||yj (t − τij (t))| < ε1 , t ≥ T. |aij ||lj,2||yj (t)| + j∈N3
j∈N3
Then it follows from (22) that d|yi (t)| ≤ −di |yi (t)| + ε1 , t ≥ T, i ∈ N1 ∪ N4 , dt which implies that |yi (t)| ≤ e−di (t−T ) |yi (T )| +
ε1 1 − e−di (t−T ) , i ∈ N1 ∪ N4 . di
That is, limt→+∞ |yi (t)| = 0 (i ∈ N1 ∪ N4 ). To sum up, for all i = 1, 2, · · · , n, we have limt→+∞ |yi (t)| = 0. That is, ∗ e 1. x is locally stable in Φ e 1 ⊂ Φ1 is chosen arbitrarily, we conclude that in each subset Because Φ of Φ1 , there is a locally stable equilibrium point. Therefore, neural networks (4) have 3n locally stable equilibrium points, and the proof is complete. Remark 2. If neural networks (4) are without time delays (i.e., bij = 0, i, j = 1, 2, · · · , n), then it is easy to see that conditions (5)–(7) in Theorem 1 are the same as conditions (18)–(20) in Theorem 2. However, if neural networks (4) are with time delays, then conditions (18)–(20) in Theorem 2 are stronger than conditions (5)–(7) in Theorem 1. This implies that more restrictive conditions are necessary to stabilize neural networks (4) with time delays. Remark 3. Inequalities (18) and (19) imply that −di + aii li,1 > 0, −di + aii li,2 < 0, aii > 0, i = 1, 2, · · · , n. In fact, from (18) and (19), we get −di pi + aii ui < −di ri + aii fi (ri ).
(25)
Noting that pi < ri and substituting ui = li,1 pi + ci,1 and fi (ri ) = li,1 ri + ci,1 into (25), we derive that −di + aii li,1 > 0, which implies that aii > 0, due to li,1 > 0. Thus, we get that −di + aii li,2 < 0, due to li,2 < 0. 22
Remark 4. As revealed in Wang and Chen (2012), under some conditions, neural networks (4) without delay (i.e., bij = 0) and with continuous activation functions (1) have at most 3n equilibrium points in all and at most 2n locally stable equilibrium points. Compared with the result reported in Wang and Chen (2012), it can be seen that neural networks (4) with discontinuous activation functions (2) now have both more total equilibrium points and more locally stable equilibrium points. This supports a very important viewpoint that discontinuous neural networks can have greater storage capacity. Remark 5. To the best of our knowledge, multistability of neural networks with discontinuous activation functions was investigated only in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b). It is worth mentioning that activation functions (2) employed in this paper are not monotonically increasing, which are totally different from those employed in the above-mentioned literature, where the activation functions are piecewise constants and monotonically increasing. Hence, the existing methods cannot be applied directly to activation functions (2). In addition, different from the results in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where locally stable equilibrium points are only located in saturated regions, the 3n locally stable equilibrium points obtained in this paper are located in not only saturated regions, but also unsaturated regions, owing to the non-monotonic structure of discontinuous activation functions (2). The following theorem shows that under some conditions, the other 5n −3n equilibrium points of neural networks (4) are unstable. Theorem 3. Suppose that the conditions in Theorem 2 hold. Furthermore, if there are also positive constants ζ1 , · · · , ζn such that (−di + aii li,1 ) ζi −
n X
j6=i,j=1
ζj | aij | γj −
where γj = max{|lj,1|, |lj,2|} and ( λ , max
1≤i≤n
(−di + aii li,2 ) ζi +
n X
j6=i,j=1
23
n X j=1
ζj | bij | γj > max{λ, 0},
(26)
)
(27)
ζj |aij | γj +
n X j=1
ζj |bij | γj ,
then neural networks (4) with discontinuous activation functions (2) and time-varying delays can have at least 5n equilibrium points, 3n of which are locally stable and the others are unstable. Proof. (i) Firstly, we will prove that the 4n − 3n equilibrium points located in Φ2 are unstable. Arbitrarily pick a region from the set Φ2 as Y Y Y Y e2 = (qi , +∞) ⊂ Φ2 , (ri , qi ) × [pi , ri ] × (−∞, pi ) × Φ i∈N2
i∈N1
i∈N4
i∈N3
where Ni (i = 1, 2, 3, 4) are subsets of {1, 2, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {1, 2, · · · , n}, Ni ∩ Nj = ∅ (i 6= j) and N2 6= ∅. Let x(t) be a solution of neural networks (4) with initial condition φ(θ) ∈ e 2 ) and x∗ be the equilibrium point of neural networks (4) located C([−τ, 0], Φ e 2 . We will prove that the equilibrium point x∗ is unstable. Without loss in Φ e 2 for all t ≥ 0. Let y(t) = x(t) − x∗ . of generality, we assume that x(t) ∈ Φ Then we get from model (4) that X X aij lj,2 yj (t) aij lj,1 yj (t) + y˙ i (t) = − di yi (t) + +
X
j∈N2
j∈N3
j∈N2
bij lj,1yj (t − τij (t)) +
X
j∈N3
bij lj,2 yj (t − τij (t)).
(28)
Define −1 G1 (t) = sup max ζi |yi (s)| , t ≥ 0, t−τ ≤s≤t i∈N3 −1 G2 (t) = sup max ζi |yi (s)| , t ≥ 0, t−τ ≤s≤t i∈N2 −1 G(t) = sup max ζi |yi (s)| , t ≥ 0. t−τ ≤s≤t
i∈N2 ∪N3
Let G1 (0) = maxi∈N3 ζi−1|yi (0)| , G2 (0) = maxi∈N2 ζi−1|yi (0)| , and G2 (0) > G1 (0) > 0. If max ζi−1 |yi (t)| = G2 (t) = G(t) i∈N2
24
|yit (t)| = max ζi−1 |yi (t)| . holds, then denote the index it ∈ N2 such that ζi−1 t i∈N2
Differentiating |yit (t)| at time t, then it follows from (26) and (28) that d|yit (t)| dt
"
= sign (yit (t)) − dit yit (t) + ait it lit ,1 yit (t) + +
X
ait j lj,2 yj (t) +
X
j∈N2
j∈N3
X
j∈N2
X
j6=it ,j∈N2
|bit j ||lj,1||yj (t − τit j (t))| −
≥ (−dit + ait it lit ,1 ) ζit G2 (t) − − "
"
X
j∈N3
ζj |ait j ||lj,2| +
X
j∈N3
≥ (−dit + ait it lit ,1 ) ζit − −
X
j∈N2
ζj |bit j ||lj,1| −
> max{λ, 0}G(t),
"
j∈N3
X
bit j lj,2yj (t − τit j (t))
j∈N3
|ait j ||lj,1||yj (t)| −
X
j∈N3
X
j6=it ,j∈N2
X
j∈N3
#
|ait j ||lj,2||yj (t)|
|bit j ||lj,2||yj (t − τit j (t))| ζj |ait j ||lj,1| + #
X
j∈N2
#
ζj |bit j ||lj,1| G2 (t)
ζj |bit j ||lj,2| G1 (t)
X
j6=it ,j∈N2
X
ait j lj,1yj (t)
j6=it ,j∈N2
bit j lj,1yj (t − τit j (t)) +
≥ (−dit + ait it lit ,1 ) |yit (t)| − −
X
ζj |ait j ||lj,1| − #
X
j∈N3
ζj |ait j ||lj,2|
ζj |bit j ||lj,2| G(t) (29)
which implies that there exists a δ > 0 such that |yit (s)| > |yit (t)| and G2 (s) ≥ G2 (t) for all s ∈ (t, t + δ). On the other hand, if there exists t1 such that G1 (t1 ) = G2 (t1 ) = G(t1 ) (without loss of generality, assume that t1 is the first time point satisfying this equation), then it is easy to see that G1 (t1 ) = max ζi−1|yi (t1 )| . i∈N3
Denote the index i1 ∈ N3 such that
ζi−1 |yi1 (t1 )| = max ζi−1 |yi (t1 )|. 1 i∈N3
25
Differentiating |yi1 (t)| at time point t1 , then we get from (27) and (28) that d|yi1 (t)| dt t=t1 X ai1 j lj,1yj (t1 ) = sign (yi1 (t1 )) (−di1 + ai1 i1 li1 ,2 )yi1 (t1 ) + X
+
ai1 j lj,2 yj (t1 ) +
+
j∈N3
bi1 j lj,2yj (t1 − τi1 j (t1 ))
≤(−di1 + ai1 i1 li1 ,2 )|yi1 (t1 )| + X
+
j6=i1 ,j∈N3
+
X
j∈N3
X
j∈N2
|ai1 j ||lj,2||yj (t1 )| +
+ "
X
j6=i1 ,j∈N3
ζj |ai1 j ||lj,2| +
≤ (−di1 + ai1 i1 li1 ,2 ) ζi1 + +
X
j∈N2
|ai1 j ||lj,1||yj (t1 )|
X
j∈N2
|bi1 j ||lj,1||yj (t1 − τi1 j (t1 ))|
|bi1 j ||lj,2||yj (t1 − τi1 j (t1 ))|
≤ (−di1 + ai1 i1 li1 ,2 ) ζi1 G1 (t1 ) + "
bi1 j lj,1yj (t1 − τi1 j (t1 ))
j∈N2
j6=i1 ,j∈N3
X
j∈N2
X
ζj |bi1 j ||lj,1| +
≤ λ G(t1 ).
X
j∈N3
X
j∈N2
X
j∈N3
"
X
j∈N2
ζj |ai1 j ||lj,1| + #
X
j∈N2
#
ζj |bi1 j ||lj,1| G2 (t1 )
ζj |bi1 j ||lj,2| G1 (t1 )
ζj |ai1 j ||lj,1| + #
X
j6=i1 ,j∈N3
ζj |ai1 j ||lj,2|
ζj |bi1 j ||lj,2| G(t1 ) (30)
Meanwhile, similar to the derivation of (29), we can obtain that d|yit1 (t)| > max{λ, 0}G(t1), dt t=t1
|yit1 (t1 )| = max ζi−1 |yi (t1 )|. in which it1 ∈ N2 is the index such that ζi−1 t1 i∈N2
26
(31)
Therefore, we can conclude that G(t) = G2 (t) ≥ G2 (0) holds for all t ≥ 0, and there exists an increasing time sequence {tk }∞ k=1 with limk→∞ tk = +∞ such that G2 (tk ) = max ζi−1 |yi (tk )|. i∈N2
Thus, we can find an index i0 ∈ N2 and an increasing time subsequence −1 {tkl }∞ l=1 with liml→∞ tkl = +∞ such that ζi0 |yi0 (tkl )| = G2 (tkl ) ≥ G2 (0) > 0, l = 1, 2, · · · , which implies that yi0 (t) would not converge to 0 when t tends to +∞. In other words, the equilibrium point x∗ is unstable. (ii) Secondly, we will show that the other 5n −4n equilibrium points are unstable, which are the points of discontinuity of function f (x) = (f1 (x1 ), · · · , fn (xn ))T . From (18) and (20), there exists a positive number ε0 such that − di qi + aii ui + (di − aii li,2 ) ε0 + +
n X j=1
+
j=1
j6=i,j=1
max{aij uj , aij vj }
max{bij uj , bij vj } + Ii ≤ 0,
− di qi + aii vi − di ε0 + n X
n X
n X
j6=i,j=1
(32)
min{aij uj , aij vj }
min{bij uj , bij vj } + Ii ≥ 0,
(33)
hold for all i = 1, 2, · · · , n. Let x∗ = (x∗1 , · · · , x∗n )T be any one of the 5n − 4n equilibrium points, and x(t) = (x1 (t), · · · , xn (t))T be a solution of neural networks (4) with initial condition φ(θ) ∈ C([−τ, 0],
Then there are two cases for us to discuss.
27
1≤i≤n
(a) qi0 − ε0 ≤ xi0 (0) < qi0 . When qi0 − ε0 ≤ xi0 (t) < qi0 , we get from Definition 1, ai0 i0 > 0 (Remark 3), equality li0 ,2 qi0 + ci0 ,2 = ui0 and (32) that x˙ i0 (t) = − di0 xi0 (t) + ai0 i0 fi0 (xi0 (t)) + +
n X j=1
n X
ai0 j γj (t)
j6=i0 ,j=1
bi0 j γj (t − τi0 j (t)) + Ii0
≤ − di0 (qi0 − ε0 ) + ai0 i0 [li0 ,2 (qi0 − ε0 ) + ci0 ,2 ] n n X X + max ai0 j uj , ai0 j vj + max bi0 j uj , bi0 j vj + Ii0 j=1
j6=i0 ,j=1
≤ 0.
Combining with Case 2 in Step I of the proof of Theorem 2, we can conclude that xi0 (t) ≤ xi0 (0) for all t ≥ 0. (b) qi0 < xi0 (0) ≤ qi0 + ε0 . When qi0 < xi0 (t) ≤ qi0 + ε0 , it follows from Definition 1 and (33) that x˙ i0 (t) ≥ − di0 (qi0 + ε0 ) + ai0 i0 vi0 + +
n X j=1
≥ 0.
n X
j6=i0 ,j=1
min bi0 j uj , bi0 j vj + Ii0
min ai0 j uj , ai0 j vj
Combining with Case 3 in Step I of the proof of Theorem 2, we obtain that xi0 (t) ≥ xi0 (0) for all t ≥ 0. From cases (a) and (b), we get that for all t ≥ 0, |xi0 (t) − x∗i0 | = |xi0 (t) − qi0 | ≥ |xi0 (0) − qi0 | > 0, which implies that xi0 (t) would not converge to x∗i0 when t tends to +∞. In other words, the equilibrium point x∗ is unstable. Summarizing cases (i) and (ii), neural networks (4) have (4n − 3n ) + (5n − 4n ) = 5n − 3n equilibrium points which are unstable, thereby completing the proof.
28
S1
2
S4
S7
S2
3
S5
S8
y 10
y 1
6
4
2
7
y! 2
S3 x! 2
S9
S6
5
x
2
x 10
Figure 2: <2 is composed of 16 regions.
15
5
1
x (t)
10
0
−5
−10 0
2
4
6
8
10
t
Figure 3: Transient behavior of x1 in Example 1.
29
4. An Illustrative Example Example 1. For illustration convenience, we consider the following twodimensional neural networks with time-varying delays: 1 1 −t x ˙ f 1 (t) = − x1 (t) + 2 f1 (x1 (t)) − 2 (x2 (t)) + f1 (x1 (t − e )) 6 6 1 − f2 (x2 (t − e−t )) + 1, 6 (34) x˙ 2 (t) = − x2 (t) + 1 f1 (x1 (t)) + 2 f2 (x2 (t)) − 1 f1 (x1 (t − e−t )) 6 6 1 − f2 (x2 (t − e−t )) + 1, 6 where fi (x) is defined as follows: −3, 2 x + 1, fi (x) = −x + 7, 6,
−∞ < x < −2, −2 ≤ x ≤ 2, 2 < x ≤ 10, 10 < x < +∞.
(i = 1, 2).
It is easy to verify that the conditions of Theorem 3 are satisfied with ξ1 = ξ2 , ζ1 = ζ2 . Thus, according to Theorem 3, neural networks (34) can have 52 = 25 equilibrium points. Among them, 32 = 9 equilibrium points xSi (i = 1, 2, · · · , 9) are locally stable, and the other 52 − 32 = 16 equilibrium points xΞj (j = 1, 2, · · · , 7) and xΛk (k = 1, 2, · · · , 9) are unstable. In fact, by direct computations, we can obtain all the sixteen equilibrium points as follows xS1 xS3 xS5 xS7 xS9 xΞ2 xΞ4 xΞ6
= (−15/2, 12)T , = (−9/2, −9/2)T , = (1577/323, 83/17)T , = (12, 12)T , = (15, −9/2)T , = (−1/20, 12)T , = (−17/16, −17/16)T , = (389/76, −17/16)T ,
xS2 xS4 xS6 xS8 xΞ1 xΞ3 xΞ5 xΞ7
= (−211/34, 83/17)T , = (85/19, 12)T , = (103/19, −9/2)T , = (226/17, 83/17)T , = (−41/8, −17/16)T , = (−251/340, 83/17)T , = (−5/4, −9/2)T , = (115/8, −17/16)T ,
which are located in regions Si , (i = 1, 2, · · · , 9) and Ξj (j = 1, 2, · · · , 7) (see Fig. 2) and are the points of continuity of activation functions. Moreover, the
30
other nine equilibrium points which are located at the points of discontinuity of activation functions are found to be xΛ1 = (10, −9/2)T , xΛ2 = (10, −17/16)T ,
xΛ3 = (10, 83/17)T , xΛ4 = (10, 12)T ,
xΛ5 = (xΛ1 5 , 10)T , xΛ1 5 ∈ [−15/2, −9/2],
xΛ6 = (xΛ1 6 , 10)T , xΛ1 6 ∈ [−5/4, −7/20],
xΛ7 = (xΛ1 7 , 10)T , xΛ1 7 ∈ [85/19, 103/19], xΛ8 = (xΛ1 8 , 10)T , xΛ1 8 ∈ [12, 15], xΛ9 = (10, 10)T .
From Figs. 3–5, it can be seen that the nine equilibrium points xSi (i = 1, 2, · · · , 9) are locally stable. Furthermore, Figs. 6–21 confirm that the other sixteen equilibrium points xΞi (i = 1, 2, · · · , 7) and xΛk (k = 1, 2, · · · , 9) are unstable. Remark 6. Discontinuous neural networks (34) have nine locally stable equilibrium points. Among them, equilibrium points xSi (i = 1, 3, 7, 9) are located in saturated regions, whereas equilibrium points xSi (i = 2, 4, 5, 6, 8) are located in unsaturated regions, which is significantly different from Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where all locally stable equilibrium points are located only in saturated regions. 5. Conclusion In this paper, we have investigated the dynamical behaviors of multiple equilibrium points for neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays. Specifically, we have shown that this type of n-neuron neural networks can have at least 5n equilibrium points, 3n of them are locally stable and the others are unstable. The key to achieve these considerably improved results over the existing multistability results reported in the literature has been the use of discontinuous non-monotonic activation functions. Because increasing storage capacity is a fundamental problem in associative memories, the work of this paper has provided an efficient way for increasing the storage capacity of the neural networks by virtue of discontinuous non-monotonic activation functions. 31
15
5
2
x (t)
10
0
−5
0
2
4
6
8
10
t
Figure 4: Transient behavior of x2 in Example 1.
14 12 10 8
2
x (t)
6 4 2 0 −2 −4 −6 −10
−5
0
5 x1(t)
10
15
20
Figure 5: Phase plot of state variable (x1 , x2 )T in Example 1.
32
−4
−4.5
x1(t)
−5
−5.5
−6
−6.5
−7
0
2
4
6
8
10
t
Figure 6: Transient behavior of x1 near the equilibrium point (−41/8, −17/16)T in Example 1.
6 4
2
1
x (t)
0
−2
−4
−6 −8 0
2
4
6
8
10
t
Figure 7: Transient behavior of x1 near the equilibrium point (−1/20, 12)T in Example 1.
33
6
4
2
1
x (t)
0
−2
−4
−6
−8
0
2
4
6
8
10
t
Figure 8: Transient behavior of x1 near the equilibrium point (−251/340, 83/17)T in Example 1.
6
4
0
1
x (t)
2
−2
−4
−6 0
2
4
6
8
10
t
Figure 9: Transient behavior of x1 near the equilibrium point (−17/16, −17/16)T in Example 1.
34
6
4
1
x (t)
2
0
−2
−4
−6
0
2
4
6
8
10
t
Figure 10: Transient behavior of x1 near the equilibrium point (−5/4, −9/2)T in Example 1.
5.5 5.4 5.3
1
x (t)
5.2 5.1 5 4.9 4.8 4.7
0
2
4
6
8
10
t
Figure 11: Transient behavior of x1 near the equilibrium point (389/76, −17/16)T in Example 1.
35
15
x1(t)
14.5
14
13.5
13
0
2
4
6
8
10
t
Figure 12: Transient behavior of x1 near the equilibrium point (115/8, −17/16)T in Example 1.
16
14
x1(t)
12
10
8
6
4
0
2
4
6
8
10
t
Figure 13: Transient behavior of x1 near the equilibrium point (10, −9/2)T in Example 1.
36
16
14
x1(t)
12
10
8
6
4 0
2
4
6
8
10
t
Figure 14: Transient behavior of x1 near the equilibrium point (10, −17/16)T in Example 1.
14 13 12 11
x1(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
Figure 15: Transient behavior of x1 near the equilibrium point (10, 83/17)T in Example 1.
37
13 12 11 10
x1(t)
9 8 7 6 5 4 3
0
2
4
6
8
10
t
Figure 16: Transient behavior of x1 near the equilibrium point (10, 12)T in Example 1.
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 5 Figure 17: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
38
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 6 Figure 18: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 7 Figure 19: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
39
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 8 Figure 20: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
14
12
x1(t)
10
8
6
4 0
2
4
6
8
10
t
Figure 21: Transient behavior of x1 near the equilibrium point (10, 10)T in Example 1.
40
Acknowledgments This work was supported in part by a research grant from the Australian Research Council, the National Natural Science Foundation of China under Grant 61203300, the Specialized Research Fund for the Doctoral Program of Higher Education under Grant 20120092120029, the Natural Science Foundation of Jiangsu Province of China under Grant BK2012319, the China Postdoctoral Science Foundation funded project under Grant 2012M511177, and State Scholarship Fund. References Bao, G., & Zeng, Z. (2012). Analysis and design of associative memories based on recurrent neural network with discontinuous activation functions. Neurocomputing, 77, 101–107. Cao, J., Feng, G., & Wang, Y. (2008). Multistability and multiperiodicity of delayed Cohen-Grossberg neural networks with a general class of activation functions. Physica D, 237(13), 1734–1749. Cheng, C., Lin, K., & Shih, C. (2007). Multistability and convergence in delayed neural networks. Physica D, 225(1), 61–74. Cohen, M., & Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man, and Cybernetics, 28(4), 815–825. Di Marco, M., Forti, M., Grazzini, M. & Pancioni, L. (2012). Limit set dichotomy and multistability for a class of cooperative neural networks with delays. IEEE Transactions on Neural Networks and Learning Systems, 23(9), 1473–1485. Forti, M., & Nistri, P. (2003). Global convergence of neural networks with discontinuous neuron activations. IEEE Transactions on Circuits and Systems I: Fundamenral Theory and Applications, 50(11), 1421–1435. Forti, M., Nistri, P., & Papini, D. (2005). Global exponential stability and global convergence in ginite time of delayed neural networks with infinite gain. IEEE Transactions on Neural Networks, 16(6), 1449–1463.
41
Huang, G., & Cao, J. (2008). Multistability of neural networks with discontinuous activation function. Communications in Nonlinear Science and Numerical Simulation, 13(10), 2279–2289. Huang, G., & Cao, J. (2010). Delay-dependent multistability in recurrent neural networks. Neural Networks, 23(2), 201–209. Huang, L., Cai, Z., & Zhang, L. (2013). Dynamical behaviors for discontinuous and delayed neural networks in the framework of Filippov differential inclusions. Neural Networks, 48, 180–194. Huang, Y., Zhang, H., & Wang, Z. (2012a). Dynamical stability analysis of multiple equilibrium points in time-varying delayed recurrent neural networks with discontinuous activation functions. Neurocomputing, 91, 21– 28. Huang, Y., Zhang, H., & Wang, Z. (2012b). Multistability and multiperiodicity of delayed bidirectional associative memory neural networks with discontinuous activation functions. Applied Mathematics and Computation, 219(3), 899–910. Huang, Z., Song, Q., & Feng, C. (2010). Multistability in networks with self-excitation and high-order synaptic connectivity. IEEE Transactions on Circuits and Systems I, 57(8), 2144–2155. Huang, Z., Wang, X., & Feng, C. (2010). Multiperiodicity of periodically oscillated discrete-time neural networks with transient excitatory selfconnections and sigmoidal nonlinearities. IEEE Transactions on Neural Networks, 21(10), 1643–1655. Huang, Z., Feng, C., & Mohamad, S. (2012). Multistability analysis for a general class of delayed Cohen-Grossberg neural networks. Information Sciences, 187, 233–244. Huang, Z., Raffoul, Y., & Cheng, C. (2014). Scale-limited activating sets and multiperiodicity for threshold networks on time scales. IEEE Transactions on Cybernetics, 44(4), 488–499. Kaslik, E., & Sivasundaram, S. (2011). Impulsive hybrid discrete-time Hopfield neural networks with delays and multistability analysis. Neural Networks, 24(4), 370–377. 42
Lin, K., & Shih, C. (2007). Multiple almost periodic solutions in nonautonomous delayed neural networks. Neural Computation, 19(12), 3392– 3420. Liu, X., & Cao, J. (2009). On periodic solutions of neural networks via differential inclusions. Neural Networks, 22(4), 329–334. Lu, W., & Chen, T. (2005). Dynamical behaviors of Cohen-Grossberg neural networks with discontinuous activation functions. Neural Networks, 18(3), 231–242. Lu, W., & Chen, T. (2008). Almost periodic dynamics of a class of delayed neural networks with discontinuous activations. Neural Computation, 20(4), 1065–1090. Marcus, C. M., & Westervelt, R. M. (1989). Stability of analog neural networks with delay. Physical Review A, 39(1), 347–359. Nie, X., & Cao, J. (2011). Multistability of second-order competitive neural networks with nondecreasing saturated activation functions. IEEE Transactions on Neural Networks, 22(11), 1694–1708. Nie, X., & Cao, J. (2012). Existence and global stability of equilibrium point for delayed competitive neural networks with discontinuous activation functions. International Journal of Systems Science, 43(3), 459–474. Nie, X., Cao, J., & Fei, S. (2013). Multistability and instability of delayed competitive neural networks with nondecreasing piecewise linear activation functions. Neurocomputing, 119, 281–291. Nie, X., Cao, J., & Fei, S. (2014). Multistability and instability of competitive neural networks with Mexican-hat-type activation functions. Abstract and Applied Analysis, 2014, Article ID 901519, 20 pages. Pan, W., Wang, Z., Gao, H., Li, Y., & Du, M. (2010). On multistability of delayed genetic regulatory networks with multivariable regulation functions. Mathematical Biosciences, 228, 100–109. Thiran, P., Setti, G., & Hasler, M. (1998). An approach to information propagation in 1-D cellular neural networks-part I: local diffusion. IEEE Transactions on Circuits and Systems I: Fundamenral Theory and Applications, 45(8), 777–789. 43
Wang, D., & Huang, L. (2014). Periodicity and global exponential stability of generalized Cohen-Grossberg neural networks with discontinuous activations and mixed delays. Neural Networks, 51, 80–95. Wang, J., Huang, L., & Guo, Z. (2009). Global asymptotic stability of neural networks with discontinuous activations. Neural Networks, 22(7), 931–937. Wang, L., Lu, W., & Chen, T. (2009). Multistability and new attraction basins of almost-periodic solutions of delayed neural networks. IEEE Transactions on Neural Networks, 20(10), 1581–1593. Wang, L., Lu, W., & Chen, T. (2010). Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions. Neural Networks, 23(2), 189–200. Wang, L., & Chen, T. (2012). Multistability of neural networks with Mexicanhat-type activation functions. IEEE Transactions on Neural Networks and Learning Systems, 23(11), 1816–1826. Wang, L., & Chen, T. (2014). Multiple µ-stability of neural networks with unbounded time-varying delays. Neural Networks, 53, 109–118. Zeng, Z., Huang, T., & Zheng, W. X. (2010). Multistability of recurrent neural networks with time-varying delays and the piecewise linear activation function. IEEE Transactions on Neural Networks, 21(8), 1371–1377. Zeng, Z., & Zheng, W. X. (2012). Multistability of neural networks with timevarying delays and concave-convex characteristics. IEEE Transactions on Neural Networks and Learning Systems, 23(2), 293–305. Zeng, Z., & Zheng, W. X. (2013). Multistability of two kinds of recurrent neural networks with activation functions symmetrical about the origin on the phase plane. IEEE Transactions on Neural Networks and Learning Systems, 24(11), 1749–1762. Zhang, L., Zhang, Y., & Li, J. (2008). Multiperiodicity and attractivity of delayed recurrent neural networks with unsaturating piecewise linear transfer functions. IEEE Transactions on Neural Networks, 19(1), 158–167. Zhou, B., & Song, Q. (2013). Boundedness and complete stability of complexvalued neural networks with time delay. IEEE Transactions on Neural Networks and Learning Systems, 24(8), 1227–1238. 44
Highlights • Discontinuous non-monotonic piecewise linear activation functions are introduced. • Multiple equilibriums of neural networks with time-varying delays are investigated. • n-dimensional discontinuous neural networks with delays can have 5^n equilibriums. • 3^n equilibriums are locally stable and the others 5^n - 3^n are unstable. • Discontinuous NNs can have greater storage capacity than continuous NNs.
S0893-6080(15)00031-3 http://dx.doi.org/10.1016/j.neunet.2015.01.007 NN 3440
To appear in:
Neural Networks
Received date: 23 September 2014 Revised date: 22 December 2014 Accepted date: 25 January 2015 Please cite this article as: Nie, X., & Zheng, W. X. Multistability of neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays. Neural Networks (2015), http://dx.doi.org/10.1016/j.neunet.2015.01.007 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Multistability of Neural Networks with Discontinuous Non-monotonic Piecewise Linear Activation Functions and Time-Varying Delays Xiaobing Niea,b , Wei Xing Zhengb,∗ Department of Mathematics, Southeast University, Nanjing 210096, China School of Computing, Engineering and Mathematics, University of Western Sydney, Sydney NSW 2751, Australia a
b
Abstract This paper is concerned with the problem of coexistence and dynamical behaviors of multiple equilibrium points for neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays. The fixed point theorem and other analytical tools are used to develop certain sufficient conditions that ensure that the n-dimensional discontinuous neural networks with time-varying delays can have at least 5n equilibrium points, 3n of which are locally stable and the others are unstable. The importance of the derived results is that it reveals that the discontinuous neural networks can have greater storage capacity than the continuous ones. Moreover, different from the existing results on multistability of neural networks with discontinuous activation functions, the 3n locally stable equilibrium points obtained in this paper are located in not only saturated regions, but also unsaturated regions, due to the non-monotonic structure of discontinuous activation functions. A numerical simulation study is conducted to illustrate and support the derived theoretical results. Keywords: Neural networks, Multistability, Instability, Discontinuous non-monotonic piecewise linear activation functions, Time-varying delays.
Corresponding author. Tel.: +61 2 47360608; fax: +61 2 47360374. Email addresses: [email protected] (Xiaobing Nie), [email protected] (Wei Xing Zheng) ∗
Preprint submitted to Neural Networks
January 30, 2015
1. Introduction The coexistence of multiple equilibrium points and their local stability is referred to as the “multistability” of neural networks. Multistability is necessary whenever neural networks are used for implementing an associative memory or for solving in real time other tasks in the field of combinatorial optimization, pattern recognition and image processing (Cohen & Grossberg , 1983; Thiran, Setti & Hasler , 1998). For example, in the application of associative memory, the addressable memories are usually stored as stable equilibrium points and the process of memory attainment is that the network converges to a certain stable equilibrium point. The number of locally stable equilibrium points corresponds to the storage capacity of neural networks. Thus, it is more desirable that neural networks can have multiple locally stable equilibrium points. The past few years have witnessed a growing research interest in multistability of neural networks and some results on this topic have been reported. In Cheng, Lin and Shih (2007), the existence of 2n stable equilibrium points for general n-dimensional delayed neural networks with two classes of activation functions was presented through formulating parameter conditions motivated by a geometrical observation. Based on decomposition of the state space, Cao, Feng and Wang (2008) investigated the multistability of Cohen-Grossberg neural networks with nondecreasing saturated activation functions with two corner points. In Lin and Shih (2007); Wang, Lu and Chen (2009), the multistability of almost-periodic solution in delayed neural networks was studied. In Huang and Cao (2010), a delay-dependent multistability criterion on recurrent neural networks was derived by using the matrix inequality technique and constructing Lyapunov functional. Kaslik & Sivasundaram (2011) were the first who revealed the effect of impulse on the multistability of neural networks. In Huang, Song and Feng (2010); Nie and Cao (2011), high-order synaptic connectivity was introduced into neural networks and the multistability was considered for high-order neural networks based on decomposition of state space, Cauchy convergence principle and inequality technique. Using the theory of monotone semiflows, the multistability of cooperative neural networks with piecewise linear activation functions was investigated in Di Marco, Forti, Grazzini and Pancioni (2012). In order to increase storage capacity, the Hopfield neural networks and competitive neural networks with nondecreasing piecewise linear activation functions with 2 r corner points were considered in Wang, Lu and 2
Chen (2010); Nie, Cao and Fei (2013), respectively. It was proved that under some conditions, the n-neuron neural networks have exactly (2 r + 1)n equilibrium points, (r + 1)n of which are locally exponentially stable and the others are unstable. In Zeng and Zheng (2012), the multistability of neural networks with k + m step stair activation functions was discussed based on an appropriate partition of the n-dimensional state space and characteristics of nonlinear activation functions. It was shown therein that the n-neuron neural networks can have (2k + 2m − 1)n equilibrium points, (k + m)n of which are locally exponentially stable. In particular, the case of k = m was previously discussed in Zeng, Huang and Zheng (2010). In Zhang, Zhang and Li (2008), some multistability properties for a class of delayed recurrent neural networks with unsaturating piecewise linear transfer functions were studied based on local inhibition. For more related work, see Pan, Wang, Gao, Li and Du (2010); Huang, Wang and Feng (2010); Huang, Feng and Mohamad (2012); Zhou and Song (2013); Zeng and Zheng (2013); Wang and Chen (2014); Huang, Raffoul and Cheng (2014) and the references therein. One underlying assumption adopted for all the aforementioned multistability results is that the activation functions are continuous. However, as pointed out in Forti and Nistri (2003), neural networks with discontinuous activation functions are important and frequently do arise in practice. When dealing with dynamical systems possessing high-slope nonlinear elements, it is often advantageous to model them with a system of differential equations with discontinuous right-hand side. As a result, the mono-stability of neural networks with discontinuous activation functions has been studied intensively over the last decade (Forti, Nistri & Papini , 2005; Lu & Chen , 2005, 2008; Wang, Huang & Guo , 2009; Liu & Cao , 2009; Nie & Cao , 2012; Huang, Cai & Zhang , 2013; Wang & Huang , 2014). However, to the best of our knowledge, there have been only a few papers investigating the multistability of neural networks with discontinuous activation functions (see Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b)). Furthermore, only piecewise constants activation functions were used for the discontinuous activation functions utilized in these works, which are simple and monotonically increasing. It has been well recognized that the multistability analysis of neural networks critically depends upon the type of activation functions used. In the above-mentioned works as well as most existing works, the activation functions employed in multistability analysis, including continuous activation 3
functions and discontinuous activation functions, were mainly focused on sigmoidal activation functions, nondecreasing saturated activation functions and piecewise constants activation functions, which are all monotonically increasing. Recent works by Wang and Chen (2012), and Nie, Cao and Fei (2014) have started a new research line concerning the study of multistability for neural networks with continuous non-monotonic piecewise linear activation functions, which are defined by ui , −∞ < x < pi , li,1 x + ci,1 , pi ≤ x ≤ ri , fi (x) = (1) li,2 x + ci,2 , ri < x ≤ qi , ui , qi < x < +∞,
where pi , ri , qi , ui , li,1 , li,2 , ci,1 , ci,2 are constants with −∞ < pi < ri < qi < +∞, li,1 > 0, li,2 < 0, and ui = fi (pi ) = fi (qi ). Under the assumption that the index set {1, 2, · · · , n} can be decomposed into four subsets with respect to different external input ranges, it was shown in Wang and Chen (2012) that the n-neuron neural networks without delays have 3]N2 equilibrium points in all, and 2]N2 of them are locally stable, where ]N2 denotes the number of elements in the second index subset N2 . Motivated by the above continuous activation functions, in this paper, we introduce a general class of discontinuous activation functions, which are defined as follows (see Fig. 1): ui , −∞ < x < pi , li,1 x + ci,1 , pi ≤ x ≤ ri , (2) fi (x) = li,2 x + ci,2 , ri < x ≤ qi , vi , qi < x < +∞,
where pi , ri , qi , ui , vi , li,1 , li,2 , ci,1 , ci,2 are constants with −∞ < pi < ri < qi < +∞, li,1 > 0, li,2 < 0, ui = fi (pi ) = fi (qi ), fi (ri ) = li,2 ri + ci,2 and vi > fi (ri ) i = 1, 2, · · · , n. Obviously, fi (x) is continuous in < except at the point of discontinuity x = qi , where there exist finite right and left limits fi (qi+ ) and fi (qi− ), respectively, with fi (qi+ ) = vi > ui = f (qi− ). Note that activation functions (2) are discontinuous and non-monotonically increasing, which are totally different from those employed in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where the activation functions are piecewise constants and monotonically increasing. Hence, the existing methods cannot be applied directly to activation 4
8
6
f(x)
4
2
0
−2
−4 −5
0
5
10
15
x
Figure 1: Discontinuous non-monotonic piecewise linear activation functions (2).
functions (2). Besides, it is easy to see that activation functions (1) and (2) are almost the same except for the value in interval (qi , +∞). Thus, some nontrivial questions arise naturally: which type of activation functions will lead to both more total equilibrium points and more locally stable equilibrium points? how does the discontinuity of activation function affect the multistability of neural networks? In this paper, we will give a definitive answer to these questions. Inspired by these concerns, in this paper, we will investigate the dynamical behaviors of multiple equilibrium points for neural networks with discontinuous activation functions (2) and time-varying delays. Note that time delays are inevitable in many real-world neural networks due to, for example, signal transmission, and time delays are often the source of oscillation and instability (Marcus & Westervelt , 1989). More precisely, the main contributions of this paper are as follows: • By means of the known fixed point theorem, we develop sufficient conditions under which the n-neuron neural networks with discontinuous activation functions (2) and time-varying delays have at least 5n equilibrium points in
• We make rigorous mathematical analysis of the dynamical behaviors of each equilibrium point for the corresponding discontinuous neural networks with time-varying delays, and prove that the addressed neural networks can have 5n equilibrium points, 3n of them are locally stable and the others are unstable. • Compared with the result in Wang and Chen (2012), the neural networks with discontinuous activation functions (2) have both more total equilibrium points and more locally stable equilibrium points than the same ones with continuous activation functions (1). Furthermore, different from results in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where locally stable equilibrium points are only located in saturated regions, the 3n locally stable equilibrium points given in this paper are located in not only saturated regions, but also unsaturated regions, due to the non-monotonic structure of discontinuous activation functions (2). • We reveal the effect of discontinuity of activation functions on the coexistence and local stability of multiple equilibrium points for the discontinuous neural networks and arrive at an important insight: discontinuous neural networks may have greater storage capacity than the continuous ones. • We substantiate the theoretical findings by an illustrative example with comprehensive computer simulations. 2. Model formulation and preliminaries 2.1. Notation Let x(t) ˙ denote the derivative of x(t). Given a set Ω ⊂
and Ωn =
n Q
i=1
(i)
(i)
(i)
(−∞, pi )δ1 × [pi , ri ]δ2 × (ri , qi )δ3 × (i)
(i)
(i)
(i)
(i)
(qi , +∞)δ4 , (δ1 , δ2 , δ3 , δ4 ) = (1, 0, 0, 0)
(3)
or (0, 1, 0, 0) or (0, 0, 1, 0) or (0, 0, 0, 1) .
It is easy to see that Ωn is composed of 4n regions. 2.2. Model Consider a general class of recurrent neural networks with time-varying delays described by the following equations n
n
X X dxi (t) = − di xi (t) + aij fj (xj (t)) + bij fj (xj (t − τij (t))) + Ii , dt j=1 j=1
(4)
where i = 1, 2, · · · , n, x(t) = (x1 (t), · · · , xn (t))T ∈
Definition 2. An equilibrium point of (4) is a vector x∗ ∈
n X
j6=i,j=1
− di ri + (aii + bii ) fi (ri ) +
− di qi + (aii + bii ) vi +
max (aij + bij ) uj , (aij + bij ) vj + Ii < 0,
(5)
n X
j6=i,j=1
n X
j6=i,j=1
min (aij + bij ) uj , (aij + bij ) vj + Ii > 0,
(6)
min (aij + bij ) uj , (aij + bij ) vj + Ii > 0,
(7)
then neural networks (4) with activation functions (2) have 4n equilibrium points located in Ωn . Proof. Arbitrarily pick a region from the set Ωn as Y Y Y Y en = (qi , +∞) ⊂ Ωn , (ri , qi ) × [pi , ri ] × (−∞, pi ) × Ω i∈N1
i∈N3
i∈N2
i∈N4
where Ni (i = 1, 2, 3, 4) are subsets of {1, 2, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {1, 2, · · · , n}, Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 2, 3, 4). We will show that neural 8
networks (4) with activation functions (2) have at least an equilibrium point en. located in Ω Note that, the equilibrium points of neural networks (4) are the same as the equilibrium points of the following neural networks without delay n
X dxi (t) = −di xi (t) + (aij + bij )fj (xj (t)) + Ii , i = 1, 2, · · · , n. dt j=1
(8)
e n and index i = 1, 2, · · · , n, fix x1 , · · · , xi−1 , For any point (x1 , x2 , · · · , xn )T ∈ Ω xi+1 , · · · , xn except for xi , and define Fi (x) = −di x + (aii + bii )fi (x) +
n X
(aij + bij )fj (xj ) + Ii .
(9)
j6=i,j=1
Then there are four possible cases for us to discuss. Case 1. i ∈ N1 . Noting that ui = fi (pi ) and uj ≤ fj ≤ vj , from (5) and (9), we have lim Fi (x) = +∞, Fi (pi ) < 0. x→−∞
Then due to the continuity of Fi (x) (x ∈ (−∞, pi ]), we can find an x¯i ∈ (−∞, pi ) such that Fi (¯ xi ) = 0. Case 2. i ∈ N2 . From (5), (6) and (9), we get Fi (pi ) < 0,
Fi (ri ) > 0.
Then we can find an x¯i ∈ (pi , ri ) such that Fi (¯ xi ) = 0. Case 3. i ∈ N3 . Note that fi (qi ) = ui and pi < qi . It follows from (5) that n P (aij + bij )fj (xj ) + Ii Fi (qi ) = −di qi + (aii + bii )fi (qi ) + j6=i,j=1
< −di pi + (aii + bii )ui + < 0.
n P
j6=i,j=1
max (aij + bij ) uj , (aij + bij ) vj + Ii
Then we can find an x¯i ∈ (ri , qi ) such that Fi (¯ xi ) = 0, owing to that Fi (ri ) > 0, Fi (qi ) < 0. Case 4. i ∈ N4 . By virtue of (7) and (9), we obtain lim Fi (x) > 0,
x→qi+
lim Fi (x) = −∞.
x→+∞
9
Then we can also find an x¯i ∈ (qi , +∞) such that Fi (¯ xi ) = 0, because of the continuity of Fi (x) (x ∈ (qi , +∞)). en → Ω e n by H(x1 , x2 , · · · , xn ) = (¯ Define a map H : Ω x1 , x¯2 , · · · , x¯n ). It is clear that the map H is continuous. By Brouwer’s fixed point theorem, e n , which is also the there exists one fixed point x∗ = (x∗1 , · · · , x∗n ) of H in Ω e n . As Ωn is divided into equilibrium point of neural networks (8) and (4) in Ω e n , neural networks (4) discontinuous activation 4n parts, by arbitrariness of Ω n functions (2) have 4 equilibrium points located in Ωn . Remark 1. Inequalities (5)–(7) imply that aii + bii > 0, vi > ui , i = 1, 2, · · · , n. In fact, from (5) and (6), we obtain −di pi + (aii + bii ) ui < −di ri + (aii + bii ) fi (ri ), i.e., 0 < di (ri − pi ) < (aii + bii )(fi (ri ) − ui ), which means that aii + bii > 0, due to fi (ri ) − ui > 0, i = 1, 2, · · · , n. Similarly, it follows from (5) and (7) that −di pi + (aii + bii ) ui < −di qi + (aii + bii ) vi , i.e., 0 < di (qi − pi ) < (aii + bii )(vi − ui ). Thus, we get that vi > ui , due to aii + bii > 0, i = 1, 2, · · · , n. Lemma 1 given above only considers the case where all components of equilibrium points are the points of continuity of activation functions. It is important to know if there exist some equilibrium points whose components are located at the points of discontinuity of activation functions. This issue is addressed in the following lemma. Lemma 2. Under the conditions of Lemma 1, neural networks (4) with discontinuous activation functions (2) and time-varying delays have at least Pn j n−j equilibrium points which are located at the points of discontij=1 Cn 4 nuity of function f (x) = (f1 (x1 ), · · · , fn (xn ))T , where Cnj = n(n−1)···(n−j+1) . j(j−1)···2·1
10
Proof. For any point (x1 , x2 , · · · , xn )T ∈
n X
j=2 n X
(a1j + b1j )fj (xj ) + I1 ,
(10)
(aij + bij )fj (xj ) + Ii ,
j=2
i = 2, 3, · · · , n.
(11)
We first prove that (10) holds for any (x2 , x3 , · · · , xn )T ∈
n X
(a1j + b1j )fj (xj ) + I1
j=2 n X j=2
n X
j=2 n X j=2
max{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1 (12)
(a1j + b1j )fj (xj ) + I1 min{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1 (13)
which imply that (10) holds.
11
Arbitrarily pick a region from the set Ωn−1 as Y Y Y Y e n−1 = (qi , +∞), (ri , qi ) × [pi , ri ] × (−∞, pi ) × Ω i∈N1
i∈N4
i∈N3
i∈N2
where Ni (i = 1, 2, 3, 4) are subsets of {2, 3, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {2, 3, · · · , n}, Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 2, 3, 4). We will show e n−1 such that (11) holds. For any point that there exists (x∗2 , · · · , x∗n )T ∈ Ω T e n−1 and index i = 2, · · · , n, fix x2 , · · · , xi−1 , xi+1 , · · · , xn (x2 , · · · , xn ) ∈ Ω except xi , and define Fi (x) = − di x + (aii + bii ) fi (x) + a ˜i1 + ˜bi1 n X + (aij + bij )fj (xj ) + Ii ,
(14)
j6=i,j=2
where i = 2, · · · , n, a ˜i1 + ˜bi1 ∈ (ai1 + bi1 )[u1 , v1 ]. It is obvious that min{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 } ≤ a˜i1 + ˜bi1 ≤ max{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 }. Then there are four possible cases for us to discuss. (a) i ∈ N1 . Note that fi (pi ) = ui , uj ≤ fj ≤ vj and a ˜i1 + ˜bi1 ≤ max{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 }. It follows from (5) that Fi (pi ) = − di pi + (aii + bii ) ui + a˜i1 + ˜bi1 + ≤ − di pi + (aii + bii ) ui + < 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj ) + Ii
j6=i,j=2
max{(aij + bij ) uj , (aij + bij ) vj } + Ii
Then due to limx→−∞ Fi (x) = +∞ and the continuity of Fi (x) (x 6= qi ), we can find an x¯i ∈ (−∞, pi ) such that Fi (¯ xi ) = 0.
12
(b) i ∈ N2 . From inequality a ˜i1 + ˜bi1 ≥ min{(ai1 + bi1 ) u1, (ai1 + bi1 ) v1 } and (6), we get Fi (ri ) = − di ri + (aii + bii ) fi (ri ) + a ˜i1 + ˜bi1 + + Ii ≥ − di ri + (aii + bii ) fi (ri ) + + Ii > 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj )
j6=i,j=2
min{(aij + bij ) uj , (aij + bij ) vj }
Then we can find an x¯i ∈ (pi , ri ) such that Fi (¯ xi ) = 0, because of Fi (pi ) < 0. (c) i ∈ N3 . It follows from (5) that lim− Fi (x) = − di qi + (aii + bii ) ui + a ˜i1 + ˜bi1 +
x→qi
+ Ii < − di pi + (aii + bii ) ui + + Ii < 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj )
j6=i,j=2
max{(aij + bij ) uj , (aij + bij ) vj }
Then we can find an x¯i ∈ (ri , qi ) such that Fi (¯ xi ) = 0, owing to Fi (ri ) > 0. (d) i ∈ N4 . By virtue of (7), we obtain that lim+ Fi (x) = − di qi + (aii + bii ) vi + a˜i1 + ˜bi1 +
x→qi
≥ − di qi + (aii + bii ) vi + + Ii > 0.
n X
j6=i,j=1
n X
(aij + bij )fj (xj ) + Ii
j6=i,j=2
min{(aij + bij ) uj , (aij + bij ) vj }
Then we can also find an x¯i ∈ (qi , +∞) such that Fi (¯ xi ) = 0, in view of the continuity of Fi (x) (x 6= qi ) and limx→+∞ Fi (x) = −∞. 13
e n−1 → Ω e n−1 by H(x2 , · · · , xn ) = (¯ Define a map H : Ω x2 , · · · , x¯n ). It is clear that the map H is continuous. By Brouwer’s fixed point theorem, e n−1 , which also satisfies there exists one fixed point (x∗2 , · · · , x∗n )T of H in Ω ∗ ∗ T (11). Thus, we obtain that (q1 , x2 , · · · , xn ) is the equilibrium point of neural e n−1 , networks (4). As
(15)
j=3
0 ∈ − d2 q2 + (a21 + b21 )[u1 , v1 ] + (a22 + b22 )[u2 , v2 ] n X (a2j + b2j )fj (xj ) + I2 , +
(16)
0 ∈ − di xi + (ai1 + bi1 )[u1 , v1 ] + (ai2 + bi2 )[u2 , v2 ] n X + (aij + bij )fj (xj ) + Ii , i = 3, 4, · · · , n.
(17)
j=3
j=3
We first prove that (15) holds for any (x3 , · · · , xn )T ∈
14
from (5) and (7) that − d1 q1 + (a11 + b11 ) u1 + min{(a12 + b12 ) u2 , (a12 + b12 ) v2 } n X (a1j + b1j )fj (xj ) + I1 + j=3
< − d1 p1 + (a11 + b11 ) u1 + < 0,
n X j=2
max{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1
− d1 q1 + (a11 + b11 ) v1 + max{(a12 + b12 ) u2 , (a12 + b12 ) v2 } n X + (a1j + b1j )fj (xj ) + I1 j=3
≥ − d1 q1 + (a11 + b11 ) v1 + > 0.
n X j=2
min{(a1j + b1j ) uj , (a1j + b1j ) vj } + I1
The above two inequalities imply that (15) holds for any (x3 , · · · , xn )T ∈
i∈N3
i∈N2
i∈N4
where Ni (i = 1, 2, 3, 4) are subsets of {3, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {3, · · · , n}, Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 2, 3, 4). Similar to Case 1, we can e n−2 such that (17) holds. By arbiprove that there exists (x∗3 , · · · , x∗n )T ∈ Ω n−2 e n−2 , we can obtain 4 trariness of Ω equilibrium points of neural networks (4), and the first two components of these equilibrium points have the values xi = qi , i = 1, 2.. For an n−dimensional vector (x1 , · · · , xn )T , it is easy to see that the number of ways choosing xi = qi and xj = qj (i 6= j) is Cn2 . Other Cn2 − 1 subcases can be proved similarly. Therefore, for Case 2, there are at least Cn2 · 4n−2 equilibrium points of neural networks (4). Case 3. For other cases of δ( = )גj, j = 3, · · · , n, similar arguments can be employed to prove that the number of equilibrium points is at least Cnj · 4n−j , j = 3, · · · , n. 15
To sum up, neural (4) with discontinuous activation functions Pn networks j n−j equilibrium points which are the points of (2) have at least j=1 Cn 4 discontinuity of function f (x) = (f1 (x1 ), · · · , fn (xn ))T . P From the fact that 4n + nj=1 Cnj 4n−j = (4 + 1)n = 5n , and combining Lemma 1 and Lemma 2, we immediately obtain the following theorem on the coexistence of multiple equilibrium points of neural networks (4). Theorem 1. Suppose that the conditions (5), (6) and (7) hold. Then neural networks (4) with discontinuous activation functions (2) and time-varying delays can have at least 5n equilibrium points in
n Q
(i)
(i)
(i)
(−∞, pi )δ1 × (ri , qi )δ2 × (qi , +∞)δ3 ,
i=1
(i) (i) (i) (δ1 , δ2 , δ3 )
Φ2 = Ωn − Φ1 .
= (1, 0, 0) or (0, 1, 0) or (0, 0, 1) ,
It is easy to see that Φ1 is composed of 3n regions, Φ2 is composed of 4n − 3n regions. Next, we will precisely figure out all equilibrium points of neural networks (4) and analyze the dynamical behaviors of each equilibrium point. The following theorem shows that under some conditions, neural networks (4) with discontinuous activation functions (2) and time-varying delays can have at least 5n equilibrium points, and 3n equilibrium points located in Φ1 are locally stable.
16
Theorem 2. Suppose that the following conditions − di pi + aii ui +
n X
j6=i,j=1
− di ri + aii fi (ri ) +
− di qi + aii vi +
max{aij uj , aij vj } +
n X
j6=i,j=1
n X
j6=i,j=1
n X j=1
min{aij uj , aij vj } +
min{aij uj , aij vj } +
n X j=1
max{bij uj , bij vj } + Ii < 0, (18)
n X j=1
min{bij uj , bij vj } + Ii > 0, (19)
min{bij uj , bij vj } + Ii > 0 (20)
hold for all i = 1, 2, · · · , n. If there are positive constants ξ1 , ξ2 , · · · , ξn such that n n X X ξj |bij ||lj,2| < 0 (21) ξj |aij ||lj,2| + (−di + aii li,2 ) ξi + j=1
j6=i,j=1
hold for all i = 1, 2, · · · , n, then neural networks (4) with discontinuous activation functions (2) and time-varying delays have at least 5n equilibrium points, 3n of which located in Φ1 are locally stable. Proof. First of all, it is easy to see that conditions (18)–(20) imply the conditions (5)–(7) in Theorem 1 hold. Thus, according to Theorem 1, the coexistence of 5n equilibrium points for neural networks (4) can be guaranteed under the conditions of Theorem 2. In the following, we will prove the local stability of 3n equilibrium points in two steps. Step I. Arbitrarily pick a region from the set Φ1 as Y Y Y e1 = (qi , +∞) ⊂ Φ1 , (ri , qi ) × (−∞, pi ) × Φ i∈N1
i∈N4
i∈N3
where N1 , N3 , N4 are subsets of {1, 2, · · · , n}, and N1 ∪N3 ∪N4 = {1, 2, · · · , n}, e 1 is an invariant set of Ni ∩ Nj = ∅ (i 6= j, i, j = 1, 3, 4). We will show that Φ e 1 ), neural networks (4). That is, for any initial condition φ(θ) ∈ C([−τ, 0], Φ e 1 for all t ≥ 0. If this is not true, then we claim that the solution x(t; φ) ∈ Φ there are three possible cases to discuss. 17
Case 1. There exists a component xi (t) of x(t; φ) which is the first (or Q one of the firsts) escaping from i∈N1 (−∞, pi ). Restated, there exist some i ∈ N1 and t∗ > 0 such that xi (t∗ ) = pi − ( is a small enough positive number), x˙ i (t∗ ) > 0, xi (t) ≤ pi − for −τ ≤ t ≤ t∗ . It follows from (4), (18) and the definition of fi that x˙ i (t∗ ) = −di xi (t∗ ) + aii fi (xi (t∗ )) + n P
+
j=1
n P
aij f (xj (t∗ ))
j6=i,j=1
bij fj (xj (t − τij (t∗ ))) + Ii + di
n P
max bij uj , bij vj + Ii + di
∗
≤ −di pi + aii ui + j=1
≤ 0,
j6=i,j=1
n P
j=1
+
aij fj (xj (t∗ ))
bij fj (xj (t∗ − τij (t∗ ))) + Ii
= −di pi + aii ui + +
n P
n P
j6=i,j=1
max aij uj , aij vj
which is a contradiction. Case 2. There exists a component xi (t) of x(t; φ) which is the first (or one Q of the firsts) escaping from i∈N3 (ri , qi ). Restated, there exist some i ∈ N3 and t∗ > 0 such that either xi (t∗ ) = ri + ( is a small enough positive number), x˙ i (t∗ ) < 0, xi (t) ∈ [ri +, qi ] for −τ ≤ t ≤ t∗ or xi (t∗ ) = qi − ( is a small enough positive number), x˙ i (t∗ ) > 0, xi (t) ∈ (ri , qi −] for −τ ≤ t ≤ t∗ . For the first case, from (4), (19) and the continuity of fi (x) (x ∈ [ri , ri + ]) we derive that x˙ i (t∗ ) ≥ − di ri + aii fi (ri + ) + +
n X j=1
≥ 0,
n X
j6=i,j=1
min aij uj , aij vj
min bij uj , bij vj + Ii − di
18
which is a contradiction. Similarly, for the second case, by virtue of ui = fi (qi ), pi < qi , (4), (18) and the continuity of fi (x) (x ∈ [qi − , qi ]), we get ∗
x˙ i (t ) ≤ − di qi + aii fi (qi − ) + +
n X j=1
≤ 0,
n X
j6=i,j=1
max aij uj , aij vj
max bij uj , bij vj + Ii + di
which is also a contradiction. Case 3. There exists a component xi (t) of x(t; φ) which is the first (or Q one of the firsts) escaping from i∈N4 (qi , +∞). Restated, there exist some i ∈ N4 and t∗ > 0 such that xi (t∗ ) = qi + ( is a small enough positive number), x˙ i (t∗ ) < 0, xi (t) ≥ qi + for −τ ≤ t ≤ t∗ . Then we get from (4) and (20) that ∗
x˙ i (t ) ≥ − di qi + aii vi + +
n X j=1
≥ 0,
n X
j6=i,j=1
min aij uj , aij vj
min bij uj , bij vj + Ii − di
which contradicts x˙ i (t∗ ) < 0. From the above three cases, we know that the solution x(t; φ) will never e 1 for all t ≥ 0. That is, Φ e 1 is an invariant set of model (4). escape from Φ e 1 is Step II. We will prove the equilibrium point x∗ of system (4) in Φ locally stable. Let x(t) = x(t; φ) be a solution of system (4) with initial e 1 ). By the positive invariance of Φ e , we know condition φ(θ) ∈ C([−τ, 0], Φ P 1 e e that P x(t) ∈ Φ1 for all t ≥ 0. Note that P when x(t) ∈ Φ1 , P j∈N1 fj (xj (t)) ∗ and i.e., j∈N4 fj (xj (t)) j∈N1 fj (xj (t)) = j∈N1 fj (xj ), P Pare constants, ∗ ∗ j∈N4 fj (xj (t)) = j∈N4 fj (xj ). Let yi (t) = xi (t) − xi . Then it follows
19
from model (4) that y˙ i (t) = − di yi (t) + +
n X j=1
bij
n X
aij
j=1
fj (x∗j
= − di yi (t) +
X
fj (x∗j
+ yj (t)) −
fj (x∗j )
+ yj (t − τij (t))) − aij lj,2 yj (t) +
X
bij lj,2 yj (t − τij (t)).
j∈N3
j∈N3
fj (x∗j )
(22)
From (21), there exists a positive constant ε small enough such that (−di + aii li,2 + ε) ξi +
n X
j6=i,j=1
ετ
ξj |aij | |lj,2| + e
n X j=1
ξj |bij | |lj,2| < 0.
(23)
Define zi (t) = eεt yi (t) and −1 M(t) = sup max ξi |zi (s)| , t ≥ 0. s≤t
i∈N3
(24)
In the following, we claim that M(t) is bounded. More precisely, for all t ≥ 0, we have M(t) = M(0). In fact, for any t0 ≥ 0, there are two possible cases: Case 1. maxi∈N3 ξi−1 |zi (t0 )| < M(t0 ). In this case, there exists a δ > 0 such that maxi∈N3 ξi−1 |zi (t)| < M(t0 ) for t ∈ (t0 , t0 + δ). Case 2. maxi∈N3 ξi−1 |zi (t0 )| = M(t0 ). In this case, let it0 = it0 (t0 ) ∈ N3 be such an index that −1 (t )| = max |z ξi−1 ξ |z (t )| . 0 i i 0 t i t0 0 i∈N3
20
By using (22) and (23), we derive that d|zit0 (t)| dt t=t0 " εt0 − dit0 yit0 (t0 ) + ait0 it0 lit0 ,2 yit0 (t0 ) = ε|zit0 (t0 )| + sign zit0 (t0 ) e X
+
ait0 j lj,2 yj (t0 ) +
X
j∈N3
j6=it0 ,j∈N3
bit0 j lj,2 yj (t0 − τit0 j (t0 ))
≤(−dit0 + ait0 it0 lit0 ,2 + ε)|zit0 (t0 )| + +
X
j∈N3
X
j∈N3
+e
X
j∈N3
≤ 0.
|ait0 j | |lj,2| |zj (t0 )|
X
j6=it0 ,j∈N3
ξj |ait0 j | |lj,2| · max ξi−1 |zi (t0 )|
ξj |bit0 j | |lj,2| · max ξi−1 |zi (t0 − τit0 j (t0 ))| i∈N3
≤ (−dit0 + ait0 it0 lit0 ,2 + ε)ξit0 + ετ
j6=it0 ,j∈N3
eετit0 j (t0 ) |bit0 j | |lj,2| |zj (t0 − τit0 j (t0 ))|
≤ (−dit0 + ait0 it0 lit0 ,2 + ε)ξit0 + + eετ
X
#
#
X
j6=it0 ,j∈N3
i∈N3
ξj |ait0 j | |lj,2|
ξj |bit0 j | |lj,2| M(t0 )
Then there exists a δ1 > 0 such that M(t) = M(t0 ) for t ∈ (t0 , t0 + δ1 ). Therefore, we can conclude that M(t) = M(0) for all t ≥ 0, which implies that max ξi−1 |zi (t)| ≤ M(0). i∈N3
Then we can arrive at
|yi (t)| ≤ M(0) ξi e−εt ≤ Me−εt , i ∈ N3 ,
21
where M = M(0) maxi∈N3 {ξi }. Therefore, we obtain that limt→+∞ |yi (t)| = 0 (i ∈ N3 ). Thus, for any ε1 > 0, there exists a constant T > 0 such that X X |bij ||lj,2||yj (t − τij (t))| < ε1 , t ≥ T. |aij ||lj,2||yj (t)| + j∈N3
j∈N3
Then it follows from (22) that d|yi (t)| ≤ −di |yi (t)| + ε1 , t ≥ T, i ∈ N1 ∪ N4 , dt which implies that |yi (t)| ≤ e−di (t−T ) |yi (T )| +
ε1 1 − e−di (t−T ) , i ∈ N1 ∪ N4 . di
That is, limt→+∞ |yi (t)| = 0 (i ∈ N1 ∪ N4 ). To sum up, for all i = 1, 2, · · · , n, we have limt→+∞ |yi (t)| = 0. That is, ∗ e 1. x is locally stable in Φ e 1 ⊂ Φ1 is chosen arbitrarily, we conclude that in each subset Because Φ of Φ1 , there is a locally stable equilibrium point. Therefore, neural networks (4) have 3n locally stable equilibrium points, and the proof is complete. Remark 2. If neural networks (4) are without time delays (i.e., bij = 0, i, j = 1, 2, · · · , n), then it is easy to see that conditions (5)–(7) in Theorem 1 are the same as conditions (18)–(20) in Theorem 2. However, if neural networks (4) are with time delays, then conditions (18)–(20) in Theorem 2 are stronger than conditions (5)–(7) in Theorem 1. This implies that more restrictive conditions are necessary to stabilize neural networks (4) with time delays. Remark 3. Inequalities (18) and (19) imply that −di + aii li,1 > 0, −di + aii li,2 < 0, aii > 0, i = 1, 2, · · · , n. In fact, from (18) and (19), we get −di pi + aii ui < −di ri + aii fi (ri ).
(25)
Noting that pi < ri and substituting ui = li,1 pi + ci,1 and fi (ri ) = li,1 ri + ci,1 into (25), we derive that −di + aii li,1 > 0, which implies that aii > 0, due to li,1 > 0. Thus, we get that −di + aii li,2 < 0, due to li,2 < 0. 22
Remark 4. As revealed in Wang and Chen (2012), under some conditions, neural networks (4) without delay (i.e., bij = 0) and with continuous activation functions (1) have at most 3n equilibrium points in all and at most 2n locally stable equilibrium points. Compared with the result reported in Wang and Chen (2012), it can be seen that neural networks (4) with discontinuous activation functions (2) now have both more total equilibrium points and more locally stable equilibrium points. This supports a very important viewpoint that discontinuous neural networks can have greater storage capacity. Remark 5. To the best of our knowledge, multistability of neural networks with discontinuous activation functions was investigated only in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b). It is worth mentioning that activation functions (2) employed in this paper are not monotonically increasing, which are totally different from those employed in the above-mentioned literature, where the activation functions are piecewise constants and monotonically increasing. Hence, the existing methods cannot be applied directly to activation functions (2). In addition, different from the results in Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where locally stable equilibrium points are only located in saturated regions, the 3n locally stable equilibrium points obtained in this paper are located in not only saturated regions, but also unsaturated regions, owing to the non-monotonic structure of discontinuous activation functions (2). The following theorem shows that under some conditions, the other 5n −3n equilibrium points of neural networks (4) are unstable. Theorem 3. Suppose that the conditions in Theorem 2 hold. Furthermore, if there are also positive constants ζ1 , · · · , ζn such that (−di + aii li,1 ) ζi −
n X
j6=i,j=1
ζj | aij | γj −
where γj = max{|lj,1|, |lj,2|} and ( λ , max
1≤i≤n
(−di + aii li,2 ) ζi +
n X
j6=i,j=1
23
n X j=1
ζj | bij | γj > max{λ, 0},
(26)
)
(27)
ζj |aij | γj +
n X j=1
ζj |bij | γj ,
then neural networks (4) with discontinuous activation functions (2) and time-varying delays can have at least 5n equilibrium points, 3n of which are locally stable and the others are unstable. Proof. (i) Firstly, we will prove that the 4n − 3n equilibrium points located in Φ2 are unstable. Arbitrarily pick a region from the set Φ2 as Y Y Y Y e2 = (qi , +∞) ⊂ Φ2 , (ri , qi ) × [pi , ri ] × (−∞, pi ) × Φ i∈N2
i∈N1
i∈N4
i∈N3
where Ni (i = 1, 2, 3, 4) are subsets of {1, 2, · · · , n}, and N1 ∪ N2 ∪ N3 ∪ N4 = {1, 2, · · · , n}, Ni ∩ Nj = ∅ (i 6= j) and N2 6= ∅. Let x(t) be a solution of neural networks (4) with initial condition φ(θ) ∈ e 2 ) and x∗ be the equilibrium point of neural networks (4) located C([−τ, 0], Φ e 2 . We will prove that the equilibrium point x∗ is unstable. Without loss in Φ e 2 for all t ≥ 0. Let y(t) = x(t) − x∗ . of generality, we assume that x(t) ∈ Φ Then we get from model (4) that X X aij lj,2 yj (t) aij lj,1 yj (t) + y˙ i (t) = − di yi (t) + +
X
j∈N2
j∈N3
j∈N2
bij lj,1yj (t − τij (t)) +
X
j∈N3
bij lj,2 yj (t − τij (t)).
(28)
Define −1 G1 (t) = sup max ζi |yi (s)| , t ≥ 0, t−τ ≤s≤t i∈N3 −1 G2 (t) = sup max ζi |yi (s)| , t ≥ 0, t−τ ≤s≤t i∈N2 −1 G(t) = sup max ζi |yi (s)| , t ≥ 0. t−τ ≤s≤t
i∈N2 ∪N3
Let G1 (0) = maxi∈N3 ζi−1|yi (0)| , G2 (0) = maxi∈N2 ζi−1|yi (0)| , and G2 (0) > G1 (0) > 0. If max ζi−1 |yi (t)| = G2 (t) = G(t) i∈N2
24
|yit (t)| = max ζi−1 |yi (t)| . holds, then denote the index it ∈ N2 such that ζi−1 t i∈N2
Differentiating |yit (t)| at time t, then it follows from (26) and (28) that d|yit (t)| dt
"
= sign (yit (t)) − dit yit (t) + ait it lit ,1 yit (t) + +
X
ait j lj,2 yj (t) +
X
j∈N2
j∈N3
X
j∈N2
X
j6=it ,j∈N2
|bit j ||lj,1||yj (t − τit j (t))| −
≥ (−dit + ait it lit ,1 ) ζit G2 (t) − − "
"
X
j∈N3
ζj |ait j ||lj,2| +
X
j∈N3
≥ (−dit + ait it lit ,1 ) ζit − −
X
j∈N2
ζj |bit j ||lj,1| −
> max{λ, 0}G(t),
"
j∈N3
X
bit j lj,2yj (t − τit j (t))
j∈N3
|ait j ||lj,1||yj (t)| −
X
j∈N3
X
j6=it ,j∈N2
X
j∈N3
#
|ait j ||lj,2||yj (t)|
|bit j ||lj,2||yj (t − τit j (t))| ζj |ait j ||lj,1| + #
X
j∈N2
#
ζj |bit j ||lj,1| G2 (t)
ζj |bit j ||lj,2| G1 (t)
X
j6=it ,j∈N2
X
ait j lj,1yj (t)
j6=it ,j∈N2
bit j lj,1yj (t − τit j (t)) +
≥ (−dit + ait it lit ,1 ) |yit (t)| − −
X
ζj |ait j ||lj,1| − #
X
j∈N3
ζj |ait j ||lj,2|
ζj |bit j ||lj,2| G(t) (29)
which implies that there exists a δ > 0 such that |yit (s)| > |yit (t)| and G2 (s) ≥ G2 (t) for all s ∈ (t, t + δ). On the other hand, if there exists t1 such that G1 (t1 ) = G2 (t1 ) = G(t1 ) (without loss of generality, assume that t1 is the first time point satisfying this equation), then it is easy to see that G1 (t1 ) = max ζi−1|yi (t1 )| . i∈N3
Denote the index i1 ∈ N3 such that
ζi−1 |yi1 (t1 )| = max ζi−1 |yi (t1 )|. 1 i∈N3
25
Differentiating |yi1 (t)| at time point t1 , then we get from (27) and (28) that d|yi1 (t)| dt t=t1 X ai1 j lj,1yj (t1 ) = sign (yi1 (t1 )) (−di1 + ai1 i1 li1 ,2 )yi1 (t1 ) + X
+
ai1 j lj,2 yj (t1 ) +
+
j∈N3
bi1 j lj,2yj (t1 − τi1 j (t1 ))
≤(−di1 + ai1 i1 li1 ,2 )|yi1 (t1 )| + X
+
j6=i1 ,j∈N3
+
X
j∈N3
X
j∈N2
|ai1 j ||lj,2||yj (t1 )| +
+ "
X
j6=i1 ,j∈N3
ζj |ai1 j ||lj,2| +
≤ (−di1 + ai1 i1 li1 ,2 ) ζi1 + +
X
j∈N2
|ai1 j ||lj,1||yj (t1 )|
X
j∈N2
|bi1 j ||lj,1||yj (t1 − τi1 j (t1 ))|
|bi1 j ||lj,2||yj (t1 − τi1 j (t1 ))|
≤ (−di1 + ai1 i1 li1 ,2 ) ζi1 G1 (t1 ) + "
bi1 j lj,1yj (t1 − τi1 j (t1 ))
j∈N2
j6=i1 ,j∈N3
X
j∈N2
X
ζj |bi1 j ||lj,1| +
≤ λ G(t1 ).
X
j∈N3
X
j∈N2
X
j∈N3
"
X
j∈N2
ζj |ai1 j ||lj,1| + #
X
j∈N2
#
ζj |bi1 j ||lj,1| G2 (t1 )
ζj |bi1 j ||lj,2| G1 (t1 )
ζj |ai1 j ||lj,1| + #
X
j6=i1 ,j∈N3
ζj |ai1 j ||lj,2|
ζj |bi1 j ||lj,2| G(t1 ) (30)
Meanwhile, similar to the derivation of (29), we can obtain that d|yit1 (t)| > max{λ, 0}G(t1), dt t=t1
|yit1 (t1 )| = max ζi−1 |yi (t1 )|. in which it1 ∈ N2 is the index such that ζi−1 t1 i∈N2
26
(31)
Therefore, we can conclude that G(t) = G2 (t) ≥ G2 (0) holds for all t ≥ 0, and there exists an increasing time sequence {tk }∞ k=1 with limk→∞ tk = +∞ such that G2 (tk ) = max ζi−1 |yi (tk )|. i∈N2
Thus, we can find an index i0 ∈ N2 and an increasing time subsequence −1 {tkl }∞ l=1 with liml→∞ tkl = +∞ such that ζi0 |yi0 (tkl )| = G2 (tkl ) ≥ G2 (0) > 0, l = 1, 2, · · · , which implies that yi0 (t) would not converge to 0 when t tends to +∞. In other words, the equilibrium point x∗ is unstable. (ii) Secondly, we will show that the other 5n −4n equilibrium points are unstable, which are the points of discontinuity of function f (x) = (f1 (x1 ), · · · , fn (xn ))T . From (18) and (20), there exists a positive number ε0 such that − di qi + aii ui + (di − aii li,2 ) ε0 + +
n X j=1
+
j=1
j6=i,j=1
max{aij uj , aij vj }
max{bij uj , bij vj } + Ii ≤ 0,
− di qi + aii vi − di ε0 + n X
n X
n X
j6=i,j=1
(32)
min{aij uj , aij vj }
min{bij uj , bij vj } + Ii ≥ 0,
(33)
hold for all i = 1, 2, · · · , n. Let x∗ = (x∗1 , · · · , x∗n )T be any one of the 5n − 4n equilibrium points, and x(t) = (x1 (t), · · · , xn (t))T be a solution of neural networks (4) with initial condition φ(θ) ∈ C([−τ, 0],
Then there are two cases for us to discuss.
27
1≤i≤n
(a) qi0 − ε0 ≤ xi0 (0) < qi0 . When qi0 − ε0 ≤ xi0 (t) < qi0 , we get from Definition 1, ai0 i0 > 0 (Remark 3), equality li0 ,2 qi0 + ci0 ,2 = ui0 and (32) that x˙ i0 (t) = − di0 xi0 (t) + ai0 i0 fi0 (xi0 (t)) + +
n X j=1
n X
ai0 j γj (t)
j6=i0 ,j=1
bi0 j γj (t − τi0 j (t)) + Ii0
≤ − di0 (qi0 − ε0 ) + ai0 i0 [li0 ,2 (qi0 − ε0 ) + ci0 ,2 ] n n X X + max ai0 j uj , ai0 j vj + max bi0 j uj , bi0 j vj + Ii0 j=1
j6=i0 ,j=1
≤ 0.
Combining with Case 2 in Step I of the proof of Theorem 2, we can conclude that xi0 (t) ≤ xi0 (0) for all t ≥ 0. (b) qi0 < xi0 (0) ≤ qi0 + ε0 . When qi0 < xi0 (t) ≤ qi0 + ε0 , it follows from Definition 1 and (33) that x˙ i0 (t) ≥ − di0 (qi0 + ε0 ) + ai0 i0 vi0 + +
n X j=1
≥ 0.
n X
j6=i0 ,j=1
min bi0 j uj , bi0 j vj + Ii0
min ai0 j uj , ai0 j vj
Combining with Case 3 in Step I of the proof of Theorem 2, we obtain that xi0 (t) ≥ xi0 (0) for all t ≥ 0. From cases (a) and (b), we get that for all t ≥ 0, |xi0 (t) − x∗i0 | = |xi0 (t) − qi0 | ≥ |xi0 (0) − qi0 | > 0, which implies that xi0 (t) would not converge to x∗i0 when t tends to +∞. In other words, the equilibrium point x∗ is unstable. Summarizing cases (i) and (ii), neural networks (4) have (4n − 3n ) + (5n − 4n ) = 5n − 3n equilibrium points which are unstable, thereby completing the proof.
28
S1
2
S4
S7
S2
3
S5
S8
y 10
y 1
6
4
2
7
y! 2
S3 x! 2
S9
S6
5
x
2
x 10
Figure 2: <2 is composed of 16 regions.
15
5
1
x (t)
10
0
−5
−10 0
2
4
6
8
10
t
Figure 3: Transient behavior of x1 in Example 1.
29
4. An Illustrative Example Example 1. For illustration convenience, we consider the following twodimensional neural networks with time-varying delays: 1 1 −t x ˙ f 1 (t) = − x1 (t) + 2 f1 (x1 (t)) − 2 (x2 (t)) + f1 (x1 (t − e )) 6 6 1 − f2 (x2 (t − e−t )) + 1, 6 (34) x˙ 2 (t) = − x2 (t) + 1 f1 (x1 (t)) + 2 f2 (x2 (t)) − 1 f1 (x1 (t − e−t )) 6 6 1 − f2 (x2 (t − e−t )) + 1, 6 where fi (x) is defined as follows: −3, 2 x + 1, fi (x) = −x + 7, 6,
−∞ < x < −2, −2 ≤ x ≤ 2, 2 < x ≤ 10, 10 < x < +∞.
(i = 1, 2).
It is easy to verify that the conditions of Theorem 3 are satisfied with ξ1 = ξ2 , ζ1 = ζ2 . Thus, according to Theorem 3, neural networks (34) can have 52 = 25 equilibrium points. Among them, 32 = 9 equilibrium points xSi (i = 1, 2, · · · , 9) are locally stable, and the other 52 − 32 = 16 equilibrium points xΞj (j = 1, 2, · · · , 7) and xΛk (k = 1, 2, · · · , 9) are unstable. In fact, by direct computations, we can obtain all the sixteen equilibrium points as follows xS1 xS3 xS5 xS7 xS9 xΞ2 xΞ4 xΞ6
= (−15/2, 12)T , = (−9/2, −9/2)T , = (1577/323, 83/17)T , = (12, 12)T , = (15, −9/2)T , = (−1/20, 12)T , = (−17/16, −17/16)T , = (389/76, −17/16)T ,
xS2 xS4 xS6 xS8 xΞ1 xΞ3 xΞ5 xΞ7
= (−211/34, 83/17)T , = (85/19, 12)T , = (103/19, −9/2)T , = (226/17, 83/17)T , = (−41/8, −17/16)T , = (−251/340, 83/17)T , = (−5/4, −9/2)T , = (115/8, −17/16)T ,
which are located in regions Si , (i = 1, 2, · · · , 9) and Ξj (j = 1, 2, · · · , 7) (see Fig. 2) and are the points of continuity of activation functions. Moreover, the
30
other nine equilibrium points which are located at the points of discontinuity of activation functions are found to be xΛ1 = (10, −9/2)T , xΛ2 = (10, −17/16)T ,
xΛ3 = (10, 83/17)T , xΛ4 = (10, 12)T ,
xΛ5 = (xΛ1 5 , 10)T , xΛ1 5 ∈ [−15/2, −9/2],
xΛ6 = (xΛ1 6 , 10)T , xΛ1 6 ∈ [−5/4, −7/20],
xΛ7 = (xΛ1 7 , 10)T , xΛ1 7 ∈ [85/19, 103/19], xΛ8 = (xΛ1 8 , 10)T , xΛ1 8 ∈ [12, 15], xΛ9 = (10, 10)T .
From Figs. 3–5, it can be seen that the nine equilibrium points xSi (i = 1, 2, · · · , 9) are locally stable. Furthermore, Figs. 6–21 confirm that the other sixteen equilibrium points xΞi (i = 1, 2, · · · , 7) and xΛk (k = 1, 2, · · · , 9) are unstable. Remark 6. Discontinuous neural networks (34) have nine locally stable equilibrium points. Among them, equilibrium points xSi (i = 1, 3, 7, 9) are located in saturated regions, whereas equilibrium points xSi (i = 2, 4, 5, 6, 8) are located in unsaturated regions, which is significantly different from Huang and Cao (2008); Bao and Zeng (2012); Huang, Zhang and Wang (2012a,b), where all locally stable equilibrium points are located only in saturated regions. 5. Conclusion In this paper, we have investigated the dynamical behaviors of multiple equilibrium points for neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays. Specifically, we have shown that this type of n-neuron neural networks can have at least 5n equilibrium points, 3n of them are locally stable and the others are unstable. The key to achieve these considerably improved results over the existing multistability results reported in the literature has been the use of discontinuous non-monotonic activation functions. Because increasing storage capacity is a fundamental problem in associative memories, the work of this paper has provided an efficient way for increasing the storage capacity of the neural networks by virtue of discontinuous non-monotonic activation functions. 31
15
5
2
x (t)
10
0
−5
0
2
4
6
8
10
t
Figure 4: Transient behavior of x2 in Example 1.
14 12 10 8
2
x (t)
6 4 2 0 −2 −4 −6 −10
−5
0
5 x1(t)
10
15
20
Figure 5: Phase plot of state variable (x1 , x2 )T in Example 1.
32
−4
−4.5
x1(t)
−5
−5.5
−6
−6.5
−7
0
2
4
6
8
10
t
Figure 6: Transient behavior of x1 near the equilibrium point (−41/8, −17/16)T in Example 1.
6 4
2
1
x (t)
0
−2
−4
−6 −8 0
2
4
6
8
10
t
Figure 7: Transient behavior of x1 near the equilibrium point (−1/20, 12)T in Example 1.
33
6
4
2
1
x (t)
0
−2
−4
−6
−8
0
2
4
6
8
10
t
Figure 8: Transient behavior of x1 near the equilibrium point (−251/340, 83/17)T in Example 1.
6
4
0
1
x (t)
2
−2
−4
−6 0
2
4
6
8
10
t
Figure 9: Transient behavior of x1 near the equilibrium point (−17/16, −17/16)T in Example 1.
34
6
4
1
x (t)
2
0
−2
−4
−6
0
2
4
6
8
10
t
Figure 10: Transient behavior of x1 near the equilibrium point (−5/4, −9/2)T in Example 1.
5.5 5.4 5.3
1
x (t)
5.2 5.1 5 4.9 4.8 4.7
0
2
4
6
8
10
t
Figure 11: Transient behavior of x1 near the equilibrium point (389/76, −17/16)T in Example 1.
35
15
x1(t)
14.5
14
13.5
13
0
2
4
6
8
10
t
Figure 12: Transient behavior of x1 near the equilibrium point (115/8, −17/16)T in Example 1.
16
14
x1(t)
12
10
8
6
4
0
2
4
6
8
10
t
Figure 13: Transient behavior of x1 near the equilibrium point (10, −9/2)T in Example 1.
36
16
14
x1(t)
12
10
8
6
4 0
2
4
6
8
10
t
Figure 14: Transient behavior of x1 near the equilibrium point (10, −17/16)T in Example 1.
14 13 12 11
x1(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
Figure 15: Transient behavior of x1 near the equilibrium point (10, 83/17)T in Example 1.
37
13 12 11 10
x1(t)
9 8 7 6 5 4 3
0
2
4
6
8
10
t
Figure 16: Transient behavior of x1 near the equilibrium point (10, 12)T in Example 1.
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 5 Figure 17: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
38
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 6 Figure 18: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 7 Figure 19: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
39
13 12 11
x2(t)
10 9 8 7 6 5 4
0
2
4
6
8
10
t
T 8 Figure 20: Transient behavior of x2 near the equilibrium point (xΛ 1 , 10) in Example 1.
14
12
x1(t)
10
8
6
4 0
2
4
6
8
10
t
Figure 21: Transient behavior of x1 near the equilibrium point (10, 10)T in Example 1.
40
Acknowledgments This work was supported in part by a research grant from the Australian Research Council, the National Natural Science Foundation of China under Grant 61203300, the Specialized Research Fund for the Doctoral Program of Higher Education under Grant 20120092120029, the Natural Science Foundation of Jiangsu Province of China under Grant BK2012319, the China Postdoctoral Science Foundation funded project under Grant 2012M511177, and State Scholarship Fund. References Bao, G., & Zeng, Z. (2012). Analysis and design of associative memories based on recurrent neural network with discontinuous activation functions. Neurocomputing, 77, 101–107. Cao, J., Feng, G., & Wang, Y. (2008). Multistability and multiperiodicity of delayed Cohen-Grossberg neural networks with a general class of activation functions. Physica D, 237(13), 1734–1749. Cheng, C., Lin, K., & Shih, C. (2007). Multistability and convergence in delayed neural networks. Physica D, 225(1), 61–74. Cohen, M., & Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man, and Cybernetics, 28(4), 815–825. Di Marco, M., Forti, M., Grazzini, M. & Pancioni, L. (2012). Limit set dichotomy and multistability for a class of cooperative neural networks with delays. IEEE Transactions on Neural Networks and Learning Systems, 23(9), 1473–1485. Forti, M., & Nistri, P. (2003). Global convergence of neural networks with discontinuous neuron activations. IEEE Transactions on Circuits and Systems I: Fundamenral Theory and Applications, 50(11), 1421–1435. Forti, M., Nistri, P., & Papini, D. (2005). Global exponential stability and global convergence in ginite time of delayed neural networks with infinite gain. IEEE Transactions on Neural Networks, 16(6), 1449–1463.
41
Huang, G., & Cao, J. (2008). Multistability of neural networks with discontinuous activation function. Communications in Nonlinear Science and Numerical Simulation, 13(10), 2279–2289. Huang, G., & Cao, J. (2010). Delay-dependent multistability in recurrent neural networks. Neural Networks, 23(2), 201–209. Huang, L., Cai, Z., & Zhang, L. (2013). Dynamical behaviors for discontinuous and delayed neural networks in the framework of Filippov differential inclusions. Neural Networks, 48, 180–194. Huang, Y., Zhang, H., & Wang, Z. (2012a). Dynamical stability analysis of multiple equilibrium points in time-varying delayed recurrent neural networks with discontinuous activation functions. Neurocomputing, 91, 21– 28. Huang, Y., Zhang, H., & Wang, Z. (2012b). Multistability and multiperiodicity of delayed bidirectional associative memory neural networks with discontinuous activation functions. Applied Mathematics and Computation, 219(3), 899–910. Huang, Z., Song, Q., & Feng, C. (2010). Multistability in networks with self-excitation and high-order synaptic connectivity. IEEE Transactions on Circuits and Systems I, 57(8), 2144–2155. Huang, Z., Wang, X., & Feng, C. (2010). Multiperiodicity of periodically oscillated discrete-time neural networks with transient excitatory selfconnections and sigmoidal nonlinearities. IEEE Transactions on Neural Networks, 21(10), 1643–1655. Huang, Z., Feng, C., & Mohamad, S. (2012). Multistability analysis for a general class of delayed Cohen-Grossberg neural networks. Information Sciences, 187, 233–244. Huang, Z., Raffoul, Y., & Cheng, C. (2014). Scale-limited activating sets and multiperiodicity for threshold networks on time scales. IEEE Transactions on Cybernetics, 44(4), 488–499. Kaslik, E., & Sivasundaram, S. (2011). Impulsive hybrid discrete-time Hopfield neural networks with delays and multistability analysis. Neural Networks, 24(4), 370–377. 42
Lin, K., & Shih, C. (2007). Multiple almost periodic solutions in nonautonomous delayed neural networks. Neural Computation, 19(12), 3392– 3420. Liu, X., & Cao, J. (2009). On periodic solutions of neural networks via differential inclusions. Neural Networks, 22(4), 329–334. Lu, W., & Chen, T. (2005). Dynamical behaviors of Cohen-Grossberg neural networks with discontinuous activation functions. Neural Networks, 18(3), 231–242. Lu, W., & Chen, T. (2008). Almost periodic dynamics of a class of delayed neural networks with discontinuous activations. Neural Computation, 20(4), 1065–1090. Marcus, C. M., & Westervelt, R. M. (1989). Stability of analog neural networks with delay. Physical Review A, 39(1), 347–359. Nie, X., & Cao, J. (2011). Multistability of second-order competitive neural networks with nondecreasing saturated activation functions. IEEE Transactions on Neural Networks, 22(11), 1694–1708. Nie, X., & Cao, J. (2012). Existence and global stability of equilibrium point for delayed competitive neural networks with discontinuous activation functions. International Journal of Systems Science, 43(3), 459–474. Nie, X., Cao, J., & Fei, S. (2013). Multistability and instability of delayed competitive neural networks with nondecreasing piecewise linear activation functions. Neurocomputing, 119, 281–291. Nie, X., Cao, J., & Fei, S. (2014). Multistability and instability of competitive neural networks with Mexican-hat-type activation functions. Abstract and Applied Analysis, 2014, Article ID 901519, 20 pages. Pan, W., Wang, Z., Gao, H., Li, Y., & Du, M. (2010). On multistability of delayed genetic regulatory networks with multivariable regulation functions. Mathematical Biosciences, 228, 100–109. Thiran, P., Setti, G., & Hasler, M. (1998). An approach to information propagation in 1-D cellular neural networks-part I: local diffusion. IEEE Transactions on Circuits and Systems I: Fundamenral Theory and Applications, 45(8), 777–789. 43
Wang, D., & Huang, L. (2014). Periodicity and global exponential stability of generalized Cohen-Grossberg neural networks with discontinuous activations and mixed delays. Neural Networks, 51, 80–95. Wang, J., Huang, L., & Guo, Z. (2009). Global asymptotic stability of neural networks with discontinuous activations. Neural Networks, 22(7), 931–937. Wang, L., Lu, W., & Chen, T. (2009). Multistability and new attraction basins of almost-periodic solutions of delayed neural networks. IEEE Transactions on Neural Networks, 20(10), 1581–1593. Wang, L., Lu, W., & Chen, T. (2010). Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions. Neural Networks, 23(2), 189–200. Wang, L., & Chen, T. (2012). Multistability of neural networks with Mexicanhat-type activation functions. IEEE Transactions on Neural Networks and Learning Systems, 23(11), 1816–1826. Wang, L., & Chen, T. (2014). Multiple µ-stability of neural networks with unbounded time-varying delays. Neural Networks, 53, 109–118. Zeng, Z., Huang, T., & Zheng, W. X. (2010). Multistability of recurrent neural networks with time-varying delays and the piecewise linear activation function. IEEE Transactions on Neural Networks, 21(8), 1371–1377. Zeng, Z., & Zheng, W. X. (2012). Multistability of neural networks with timevarying delays and concave-convex characteristics. IEEE Transactions on Neural Networks and Learning Systems, 23(2), 293–305. Zeng, Z., & Zheng, W. X. (2013). Multistability of two kinds of recurrent neural networks with activation functions symmetrical about the origin on the phase plane. IEEE Transactions on Neural Networks and Learning Systems, 24(11), 1749–1762. Zhang, L., Zhang, Y., & Li, J. (2008). Multiperiodicity and attractivity of delayed recurrent neural networks with unsaturating piecewise linear transfer functions. IEEE Transactions on Neural Networks, 19(1), 158–167. Zhou, B., & Song, Q. (2013). Boundedness and complete stability of complexvalued neural networks with time delay. IEEE Transactions on Neural Networks and Learning Systems, 24(8), 1227–1238. 44
Highlights • Discontinuous non-monotonic piecewise linear activation functions are introduced. • Multiple equilibriums of neural networks with time-varying delays are investigated. • n-dimensional discontinuous neural networks with delays can have 5^n equilibriums. • 3^n equilibriums are locally stable and the others 5^n - 3^n are unstable. • Discontinuous NNs can have greater storage capacity than continuous NNs.