Neural network for nonsmooth pseudoconvex optimization with general constraints

Neurocomputing 131 (2014) 336–347

Contents lists available at ScienceDirect

Neurocomputing journal homepage: www.elsevier.com/locate/neucom

Neural network for nonsmooth pseudoconvex optimization with general constraints$ Qingfa Li, Yaqiu Liu n, Liangkuan Zhu School of Information and Computer Engineering, Northeast Forestry University, Harbin 150040, PR China

art ic l e i nf o

a b s t r a c t

Article history: Received 19 April 2013 Received in revised form 6 August 2013 Accepted 3 October 2013 Communicated by Y. Liu Available online 23 October 2013

In this paper, a one-layer recurrent projection neural network is proposed for solving pseudoconvex optimization problems with general convex constraints. The proposed network in this paper deals with the constraints into two parts, which brings the network simpler structure and better properties. By the Tikhonov-like regularization method, the proposed network need not estimate the exact penalty parameter in advance. Moreover, comparing with some existing neural networks, the proposed network can solve more general constrained pseudoconvex optimization problems. When the solution of the proposed network is bounded, it converges to the optimal solution set of considered optimization problem, which may be nonsmooth and nonconvex. Meantime, some sufficient conditions are presented to guarantee the boundedness of the solution of the proposed network. Numerical examples with simulation results are given to illustrate the effectiveness and good characteristics of the proposed network for solving constrained pseudoconvex optimization. & 2013 Published by Elsevier B.V.

Keywords: Neural network Nonsmooth optimization Pseudoconvex function Differential inclusion Regularization method

1. Introduction In this paper, we consider the following constrained nonsmooth optimization problem: min f ðxÞ s:t: x A Ω; g i ðxÞ r 0;

i ¼ 1; …; m;

ð1Þ

where x A R , Ω is a nonempty, closed and convex subset of Rn , g i ði ¼ 1; 2; …; mÞ : Rn -R is convex, not necessarily smooth, f : Rn -R is regular, not necessarily smooth or convex. The optimization problem (1) considered in this paper is a general model in optimization with convex constraints. Constrained optimization problems arise in a variety of scientific and engineering contexts, including filter design, signal processing, system identification, robot control, game theory and so on. Some real-world applications are often modeled by nonsmooth or even nonconvex optimization problems, where real-time online solutions of optimization problems are desired in many engineering and scientific applications. One possible and very promising approach to solve the real-time optimization problems is to apply artificial neural networks [1–5]. n

☆ Supported by the 948 Project (2012-4-21) and the National Science Foundation of China under Grant 31370565. n Corresponding author. E-mail addresses: [email protected] (Q. Li), [email protected] (Y. Liu), [email protected] (L. Zhu).

0925-2312/$ - see front matter & 2013 Published by Elsevier B.V. http://dx.doi.org/10.1016/j.neucom.2013.10.008

Due to the real-time and hardware implementations, neural networks for solving optimization problems have been studied by many researchers and many meaningful results have been obtained. Among them, two classical recurrent neural network models for optimization problems are the Hopfield neural networks proposed for linear programming by Tank and Hopfield [2] and for nonlinear programming by Kennedy and Chua [3], which inspire many researchers to develop neural networks for optimization. A neural network with finite time convergence is proposed in [4] for linear programming problem. Projection is an effective and a simple method for solving the constraints, which has been used in neural networks for solving some kinds of constrained optimization problems [6,7]. However, it is difficult to deal with the more general constraints by the projection method. Then, Lagrangian and penalty methods are introduced into networks. Based on the Lagrangian function, Lagrangian networks were proposed for solving some general constrained optimization problems [8,9]. The dimension of Lagrangian network not only depends on the dimension of the variable in optimization problems, but also increases along with the number of the constraints. By utilizing a finite penalty parameter, Kennedy and Chua [3] developed a recurrent neural network whose equilibrium points correspond to approximate optimal solutions. In recent years, recurrent neural networks based on the penalty method were widely investigated for solving nonsmooth optimization problems. Generalized nonlinear programming circuit (G-NPC) proposed by Forti et al. [10] can be considered as a natural extension of nonlinear programming circuit (NPC) [3] for solving nonsmooth convex optimization

Q. Li et al. / Neurocomputing 131 (2014) 336–347

problems with inequality constraints. Nonempty interior of feasible region and large enough penalty parameters are needed for the network proposed by Forti et al. [10]. In order to overcome the nonempty interior condition on the feasible region, Xue and Bian [11] proposed a recurrent neural network for nonsmooth convex optimization based on the penalty method, whereafter Liu et al. [12] proposed a one-layer recurrent neural network with only one exact penalty parameter. Other interesting results for nonsmooth convex optimization using neural networks can be found in [13– 15]. Some dynamical properties of differential equation or differential inclusion networks make remarkable contributions to their applications in optimization [16–18]. Nonconvex quadratic programming is solved by Forti et al. [19], where the Lojasiewicz inequality is introduced to estimate the convergence rate. Some results on the neural networks for solving nonconvex optimization are given in [20,21], which are potentially useful for the applications of neural networks. In particular, pseudoconvex optimization is a special class of nonconvex optimization [22,23], which has many applications in fractional programming, computer vision, production planning, financial and corporate planning, etc. Hu and Wang [24] extended the projection neural network for the optimization problem, which is modeled by a differentiable and pseudoconvex objective function and a bounded convex constraint. In order to extend the applications of neural network for pseudoconvex optimization, Liu and Wang [23] proposed a one-layer recurrent neural network for solving a wider class of pseudoconvex optimization with linear equalities and box constraints. Exact penalty parameter is crucial for the effectiveness of the networks based on the penalty method. Some information on the objective and constraint functions are necessary to estimate the exact parameters, such as the work in [10–12,25]. In some worse cases, there is not an exact penalty function. Then, the Tikhonovlike regularization method is often used, which is also called as the “viscosity” regularization method and plays an important role in optimization and control. This method brings the system hierarchical structure, which also may let the system be global attractive. Based on the Tikhonov-like regularization method, some variants of nonautomatic systems for the minimization problems have been studied in [26–29]. Among them, Bolte [27] studied a continuous gradient project system to solve a class of optimization problems in a general Hilbert space H and got the strong convergence under some Lipschitzian conditions. Attouch and Czarnecki [28] considered the asymptotic behavior of differential inclusion _ þ∂ϕðxðtÞÞ þ ɛðtÞ∂ψ ðxðtÞÞ; 0 A xðtÞ where ϕ : H-R and ψ : H-R are closed convex functions.1 In order to avoid estimating the exact penalty parameters in [11] and ignore the feasible condition in [9–12], Bian and Xue [26] proposed a nonautonomous neural network for solving a class of constrained convex optimization problems. The contributions of this paper are as follows. First, a recurrent neural network is proposed for finding an optimal solution of a pseudoconvex optimization with general constraints. The pseudoconvex optimization is a class of nonsmooth or nonconvex optimization. Some classes of pseudoconvex functions in engineering and economics are given in [22,23]. The optimization problem in this paper has more general constraints than the problems in [23,24] and the networks in [23,24] are not capable for solving the pseudoconvex optimization considered in this paper. Second, by the general constraints in (1), the proposed network in this paper combines the projection method and the penalty function method, 1 We call function f : H-R a closed convex function, if f is convex and its epigraph is a closed set. A proper convex function is closed if and only if it is lower semi-continuous.

337

which brings the proposed network more simpler structure. We use the projection to handle the constraints, which can be solved by the project operator directly, while the others are dealt with by the penalty function method. By the projection operator, the proposed network in this paper has simpler structure and some better properties than the network in [26] even when the objective function in (1) is convex. Third, in order to avoid estimating, the exact penalty parameter in advance when it exists and solve the optimization problem where the exact penalty parameter does not exist, the Tikhonov-like regularization method is introduced into our network. Moreover, by the Tikhonov-like method, the proposed network ignores the feasible condition in [9–12,25]. Fourth, to the best of our knowledge, the proposed network in this paper is the first continuous system based on the Tikhonov-like regularization method for solving a class of nonconvex optimization problems, whereas the functions in the Tikhonov-like systems [26–29] are convex and the validity of these systems relies on the convexity of the functions. Owing to the nonconvexity of the objective function in (1), the convex inequalities cannot be used throughout the theoretical analysis. And we should state that the convex inequalities are used for proving the global existence, uniqueness and convergence of the proposed network in [26]. In this paper, by the Gronwall inequality, pseudoconvexity and categorized discussion method, we prove that any solution of the proposed network converges to the optimal solution set of (1). The remaining parts of this paper are as follows. In Section 2, some notations and necessary preliminary results are listed. In Section 3, the model of proposed network for solving (1) is discussed, which combines the projection, penalty and Tikhonov-like methods. Section 4 gives the main theoretical results, which indicate the effectiveness and good properties of the proposed network for solving (1). In Section 5, the proposed network is utilized for four numerical examples. Finally, Section 6 concludes this paper.

2. Some notations and preliminary results 2.1. Notations Given column vectors x ¼ ðx1 ; x2 ; …; xn ÞT and y ¼ ðy1 ; y2 ; …; yn ÞT , xi denotes the ith element of x, J x J denotes the Euclidean norm of x defined by J x J ¼ ð∑ni¼ 1 x2i Þ1=2 and 〈x; y〉 ¼ xT y ¼ ∑ni¼ 1 xi yi is the scalar product of x and y. For a closed convex subset Γ D Rn , mðΓ Þ is the element in Γ with the smallest Euclidean norm, distðx; Γ Þ is the distance from x to Γ defined by distðx; Γ Þ ¼ miny A Γ J x y J . X denotes the feasible region defined by X ¼ fx : g i ðxÞ r 0; i ¼ 1; 2; …; mg. Then, the feasible region of (1) is Ω \ X. The optimal solution set of (1) is denoted by M. Without loss of generality, we suppose that the optimal solution set of (1) is nonempty, i.e. M a ∅.

2.2. Preliminaries In this subsection, we present some necessary definitions and properties on set-valued map, nonsmooth analysis and projection operator, which are needed in this paper. We refer the readers to [30–33] for more thorough discussions. Definition 2.1. Suppose that to each point x of a set E D Rn there corresponds a nonempty set FðxÞ D Rn . Then x⊸FðxÞ is a set-valued map from E to Rn . A set-valued map F : E⊸Rn with nonempty values is said to be upper semicontinuous at x A Rn if for any open set V containing F(x), there exists a neighborhood U of x such that FðUÞ D V.

338

Q. Li et al. / Neurocomputing 131 (2014) 336–347

) 8 ζ ðx″Þ A ∂ϕðx″Þ : 〈ζ ðx″Þ; x″  x′〉 Z 0:

Consider the following differential inclusion: _ A FðxðtÞ; tÞ xðtÞ

ð2Þ

with a set-valued map F : Rn  ½0; TÞ⊸Rn . In order to present a good approximation for the solutions of the actual systems with very high slope nonlinearities, the solution of (2) is defined in the sense of Filippov's [32,34,35]. Definition 2.2. We call x : ½0; TÞ-Rn a solution of (2) with initial point xð0Þ ¼ x0 , if x is absolutely continuous and satisfies (2) almost everywhere (a.e.) on ½0; TÞ. Moreover, we call x : ½0; TÞ-Rn a “slow solution” of (2) with initial point xð0Þ ¼ x0 , if x is absolutely continuous and satisfies _ ¼ mðFðxðtÞ; tÞÞ xðtÞ

a:e: t A ½0; TÞ:

Definition 2.3. Function ϕ : Rn -R is said to be Lipschitz near x A Rn , if there exist ε; δ 4 0 such that for any x′; x″ A Rn such that J x  x′ J r δ and J x″  x J r δ, we have jϕðx″Þ  ϕðx′Þj r ε J x″  x′ J . If ϕ is Lipschitz near any point x A Rn , then it is said to be locally Lipschitz in Rn . When ϕ is Lipschitz near x, the generalized directional derivative of ϕ at x in the direction v A Rn is given by

ϕ0 ðx; vÞ ¼ lim sup y-x;t-0

ϕðy þ tvÞ  ϕðyÞ þ

t

:

A convex function is obviously pseudoconvex and it is shown in [22] that a regular function ϕðxÞ is pseudoconvex if and only if its generalized gradient ∂ϕ is pseudomonotone. Proposition 2.1. If V : Rn -R is a regular function at x(t), x : R-Rn is differentiable at t and Lipschitz near t, then d _ VðxðtÞÞ ¼ 〈ξ; xðtÞ〉; dt

8 ξ A ∂VðxðtÞÞ:

Projection operator is important in optimization, especially for the optimization problems with box or affine equality constraints. Since Ω in (1) is a nonempty, closed and convex set of Rn , the projection operator onto Ω is given by P Ω ðxÞ ¼ arg min J u  x J : uAΩ

Proposition 2.2. Let P Ω denote the projection operator onto Ω D Rn . Then (i) 〈v  P Ω ðvÞ; P Ω ðvÞ  u〉Z 0; 8 v A Rn ; u A Ω; (ii) J P Ω ðuÞ  P Ω ðvÞ J r J u  v J ; 8 u; v A Rn :

Then Clarke's generalized gradient of ϕ at x is defined as ∂ϕðxÞ ¼ fξ A Rn : ϕ ðx; vÞ Z 〈v; ξ〉; for all v A Rn g:

3. Model description

Definition 2.4. A function ϕ : Rn -R which is Lipschitz near x is said to be regular at x provided the following conditions hold.

This section gives a one-layer projection network for solving the nonsmooth, or even nonconvex optimization problem (1). We assume that Ω is a nonempty, closed and convex subset of Rn , onto which the projection operator, can be given explicitly. This condition will not restrict the applications of (1) by virtue of the convex constraints g i ðxÞ r 0, i¼ 1,…,m. Specially, if there are some box constraints in (1), we can let

0

(i) for all v A Rn , the usual one-sided directional derivative ϕðx þtvÞ  ϕðxÞ exists; ϕ′ðx; vÞ ¼ lim t t-0 þ (ii) for all v A Rn , ϕ′ðx; vÞ ¼ ϕ ðx; vÞ. 0

Property 2.1. If ϕ : Rn -R is a regular function, then the following properties hold (i) ∂ϕðxÞ is upper semicontinuous on Rn ; (ii) ∂ϕðxÞ is a nonempty, convex, compact set of Rn , and J ξ J r k;

8 ξ A ∂ϕðxÞ;

where k is a Lipschitz constant of ϕ at x; (iii) if ψ : Rn -R is convex, then ∂ðϕ þ ψ Þ ¼ ∂ϕ þ∂ψ :

Ω ¼ fx A Rn : u r x r vg with u; v A Rn ⋃f 7 1g. And P Ω ðxÞ ¼ ðp1 ; p2 ; …; pn ÞT is defined by 8 > < ui ; x i o ui

pi ¼

xi ; > :v ; i

ui r xi r vi vi o xi :

If there are some affine equality constraints in (1), then Ω can be defined by

Ω ¼ fx : Ax ¼ bg with a full row rank matrix A A Rrn and a vector b A Rr . And P Ω ðxÞ ¼ x AT ðAAT Þ  1 ðAx  bÞ:

Definition 2.5 (Penot and Quang [22]). Let E be a nonempty convex subset of Rn . A function ϕ is said to be pseudoconvex on E if for any x′; x″ A E, we have ( ζ ðx′Þ A ∂ϕðx′Þ : 〈ζ ðx′Þ; x″  x′〉Z 0 ) ϕðx″Þ Z ϕðx′Þ:

Pseudoconvexity of the objective function is an important property, which brings us the opportunity to find the optimal solutions of (1). Definition 2.6. Let E D Rn be a nonempty convex subset of Rn . A set-valued map F : E⊸Rn is said to be pseudomonotone on E if for any x′; x″ A E, we have ( ζ ðx′Þ A ∂ϕðx′Þ : 〈ζ ðx′Þ; x″  x′〉Z 0

Moreover, Ω can also be used to express a ball constraint, i.e.

Ω ¼ fx : J x  c J r ρg with c A Rn and ρ 4 0. Then 8 JxcJ rρ < x; ρðx cÞ P Ω ðxÞ ¼ ; J x  c J 4 ρ: : cþ JxcJ For the inequality constraints g i ðxÞ r 0, i ¼ 1; 2; …; m, the penalty method is introduced into the network for solving (1). The penalty function P(x) used in this paper can be given in any expression, which satisfies the following conditions: (i) PðxÞ ¼ 0 for x A X and PðxÞ 4 0 for x2 = X; (ii) P(x) is convex on Ω.

Q. Li et al. / Neurocomputing 131 (2014) 336–347

Then, we can define the penalty function for the constraints g i ðxÞ r0, i ¼ 1; 2; …; m, as follows: m

PðxÞ ¼ ∑ maxf0; g i ðxÞgα

ð3Þ

i¼1

with 1 r α r 2. By the penalty method, we often need to find the exact parameter s such that sPðxÞ is an exact penalty function. Different from some most recent used neural network models based on the penalty method, we introduce the following system modeled by a nonautonomous differential inclusion with projection to solve (1): _ A  xðtÞ þ P Ω ½xðtÞ  ∂PðxðtÞÞ  ɛðtÞ∂f ðxðtÞÞ; xðtÞ

ð4Þ

where ɛ : ½0; 1Þ-ð0; 1Þ is a decreasing function defined by

with ɛ 0 4 0. From Definition 2.2, if x : ½0; TÞ-Rn is a solution of (4), it is absolutely continuous on ½0; TÞ and there exist two functions ξ ðtÞ A ∂PðxðtÞÞ and η ðtÞ A ∂f ðxðtÞÞ such that a:e: t Z 0:

ð5Þ

Remark 3.1. Generally, if the constraints in (1) are u r x r v;

Ax ¼ b;

g i ðxÞ r 0;

i ¼ 1; 2; …; r;

ð6Þ

we can define P(x) and Ω by any one of the following three formats: ðiÞ Ω ¼ Rn ;

mathematical softwares. Next, the feasibility and optimality of (4) for solving (1) are also proved. Lyapunov method and nonsmooth analysis are employed to study the performance of (4). The following two Lyapunov energy functions are used throughout this paper. Define   Eðx; tÞ ¼ PðxÞ þ ɛðtÞ f ðxÞ  inf f ; Ω

  2 Gðx; tÞ ¼ 12 dist ðx; MÞ þ PðxÞ þ ɛðtÞ f ðxÞ  inf f : Ω

For readability, we give some preliminary analysis on the two Lyapunov functions at first. Lemma 4.1. When the solution x : ½0; TÞ-Rn of (4) satisfies xðtÞ A Ω for all t A ½0; TÞ with T 40, then for a.e. t A ½0; TÞ, the following estimations hold

ɛ0 ɛðtÞ ¼ t þ1

_ ¼ xðtÞ þP Ω ½xðtÞ  ξ ðtÞ  ɛðtÞη ðtÞ xðtÞ

339

(i) The derivative of Eðx; tÞ along this solution of (4) can be calculated by   d _ J 2 þ ɛ_ ðtÞ f ðxðtÞÞ  inf f r 0; EðxðtÞ; tÞ r  J xðtÞ dt Ω (ii) There exist ξ ðtÞ A ∂PðxðtÞÞ and η ðtÞ A ∂f ðxðtÞÞ such that   d _ J 2 þ ɛ_ ðtÞ f ðxðtÞÞ  inf f GðxðtÞ; tÞ r  J xðtÞ dt Ω  〈ξ ðtÞ þ ɛðtÞη ðtÞ; xðtÞ  P M ðxðtÞÞ〉:

r

PðxÞ ¼ ∑ maxf0; g i ðxÞgα1 þ J Ax  b J ρ i¼1

n

þ ∑ maxf0; ui  xi g i¼1

α2

n

α3

þ ∑ maxf0; xi  vi g ; i¼1

ðiiÞ Ω ¼ fx A Rn : Ax ¼ bg; PðxÞ ¼ ∑ri ¼ 1 maxf0; g i ðxÞgα1 þ ∑ni¼ 1 maxf0; ui xi gα2 þ ∑ni¼ 1 maxf0; xi  vi gα3 ; (iii) Ω ¼ fx A Rn : u r x r vg, PðxÞ ¼ ∑ri ¼ 1 maxf0; g i ðxÞgα1 þ J Ax  b J ρ , where 1 r α1 ; α2 ; α3 ; ρ r 2. The proposed network (4) uses the projection operator to solve the constraint x A Ω. (i) can be seen as the most direct version for Ω and P. However, it is well known that the projection operator is implemented more easily and cheaper than the penalty function for the box constraint. Then, formulations (ii) and (iii) are better than (i) in implementation. On the other hand, the projection operator brings the network some good properties, one of which is that any solution of (4) with initial point x0 A Ω stays in Ω always (see Proposition 4.1). Based on the penalty function for the affine equality constraints, most networks are sensitive and it is difficult to ensure in numerical experiments that the obtained optimal solution satisfies the affine equality constraints exactly. Thus, through the implementation cost for the affine equality constraints by the penalty function and the projection operator are almost the same, the projection operator performs better in numerical illustration. Thus, (ii) may be the best choice if there are some affine equalities in the constraints.

4. Theoretical analysis In this section, some necessary theoretical analysis on network (4) for solving (1) will be given. Throughout this paper, we assume that f is pseudoconvex and bounded from below on Ω. First, the global existence of the solution of (4) is necessary for the useability of it. With any initial point x0 A Ω, we can prove the global existence and uniqueness of the solution of (4) under some trivial conditions. Moreover, the solution of (4) is just its “slow solution”, which gives some hint on how to implement (4) by circuits and some

Proof. (i) From Proposition 2.1, differentiating EðxðtÞ; tÞ along the solutions of (4), for a.e. t A ½0; TÞ, we have   d _ EðxðtÞ; tÞ ¼ 〈ξðtÞ þ ɛðtÞηðtÞ; xðtÞ〉 þ ɛ_ ðtÞ f ðxðtÞÞ  inf f ; dt Ω ð7Þ 8 ξðtÞ A ∂PðxðtÞÞ; ηðtÞ A ∂f ðxðtÞÞ: Coming back to (4), there exist ξ ðtÞ A ∂PðxðtÞÞ and η ðtÞ A ∂f ðxðtÞÞ such that _ ¼  xðtÞ þ P Ω ½xðtÞ  ξ ðtÞ ɛðtÞη ðtÞ xðtÞ

a:e: t A ½0; TÞ;

which can be rewritten as _ þxðtÞ ¼ P Ω ½xðtÞ  ξ ðtÞ  ɛðtÞη ðtÞ xðtÞ

a:e: t A ½0; TÞ:

ð8Þ

Let v ¼ xðtÞ  ξ ðtÞ ɛðtÞη ðtÞ and u ¼ xðtÞ, using (4) and Proposition 2.2, for a.e. t A ½0; TÞ, we have _  xðtÞ; xðtÞ _ þ xðtÞ  xðtÞ〉 Z0; 〈xðtÞ  ξ ðtÞ  ɛðtÞη ðtÞ  xðtÞ which follows that _ _ J2 r  J xðtÞ 〈ξ ðtÞ þ ɛðtÞη ðtÞ; xðtÞ〉

a:e: t A ½0; TÞ:

ð9Þ

Combining (7) and (9), for a.e. t A ½0; TÞ, we get that   d _ J 2 þ ɛ_ ðtÞ f ðxðtÞ  inf f r 0: EðxðtÞ; tÞ r  J xðtÞ dt Ω (ii) Differentiating GðxðtÞ; tÞ along this solution of (4), we have d _ _ GðxðtÞ; tÞ ¼ 〈xðtÞ  P M ðxðtÞÞ; xðtÞ〉 þ〈ξðtÞ þ ɛðtÞηðtÞ; xðtÞ〉 dt   þ ɛ_ ðtÞ f ðxðtÞÞ  inf f ; 8 ξðtÞ A ∂PðxðtÞÞ; Ω

ηðtÞ A ∂f ðxðtÞÞ:

ð10Þ

Let v ¼ xðtÞ  ξ ðtÞ  ɛðtÞη ðtÞ and u ¼ P M ðxðtÞÞ, using Proposition 2.2 and (8), we have _ _ þxðtÞ P M ðxðtÞÞ〉r 0 xðtÞ 〈ξ ðtÞ þ ɛðtÞη ðtÞ þ xðtÞ;

a:e: t A ½0; TÞ;

340

Q. Li et al. / Neurocomputing 131 (2014) 336–347

which can be rewritten as

xi, then assumption (AP) holds. For example, when

_ _ 〈ξ ðtÞ þ ɛðtÞη ðtÞ; xðtÞ〉 þ 〈xðtÞ; xðtÞ P M ðxðtÞÞ〉

Ω ¼ fx A R3 : 0 r x1 ; x2 r 1g;

_ J 2: r 〈ξ ðtÞ þɛðtÞη ðtÞ; xðtÞ  P M ðxðtÞÞ〉  J xðtÞ

ð11Þ

Combining (10) and (11), we get d _ J 2 þ ɛ_ ðtÞðf ðxðtÞÞ  inf f Þ GðxðtÞ; tÞ r  J xðtÞ dt Ω  〈ξ ðtÞ þ ɛðtÞη ðtÞ; xðtÞ  P M ðxðtÞÞ〉:

□

ð12Þ

In order to prove the global existence and uniqueness of the solutions of (4), one basic property on the viability of the solutions of (4) is needed. Proposition 4.1. For any initial point x0 A Ω, there is a local solution of (4) defined on ½0; TÞ with T 4 0 and it satisfies xðtÞ A Ω;

for all t A ½0; TÞ:

Proof. Since the projection operator P Ω is Lipschitz continuous, ∂P and ∂f are upper semicontinuous with compact and convex values, then the right hand function of (4) is upper semicontinuous with compact and convex values. Thus, there is a local solution x : ½0; TÞ-Rn of (4), where ½0; TÞ is the maximal existence interval of this solution. From Definition 2.2, there exists θðtÞ A P Ω ½xðtÞ  ∂PðxðtÞÞ  ɛðtÞ∂f ðxðtÞÞ such that _ þ xðtÞ ¼ θðtÞ xðtÞ

a:e: t Z 0:

A simple integration procedure gives Z t xðtÞ ¼ e  t x0 þ e  t θðsÞes ds; 0

which can be rewritten as Z t es ds: xðtÞ ¼ e  t x0 þ ð1  e  t Þ θðsÞ t e 1 0 Rt Since 0 es =ðet  1Þds ¼ 1, θðsÞ A Ω, 8 0 r s r t and x0 A Ω, we obtain that xðtÞ A Ω, 8 t A ½0; TÞ. It is well-known that the boundedness is very important for the solution of a well proposed network, which is to ensure the convergence and efficiency of a network. Mostly, we cannot affirm the boundedness of the solutions of a network directly from the solved optimization problem, which brings the proposed network some uncertainty. Thus, in this paper, we give two sufficient conditions given below to ensure the boundedness of the solutions of (4). ðAP Þ P(x) is level-bounded on Ω2; ðAf Þ For an xn A M, fx A Ω : f ðxÞ r f ðxn Þg is bounded. It is easy to verify that either one of the following two conditions is a sufficient condition for ðAP Þ. (i) Ω is bounded; (ii) P is coercive.3

Remark 4.1. The condition (AP) is not overly restrictive. When P is defined with the format in (3), if there is an i A f1; 2; …; mg such that gi is coercive, then P is coercive. Moreover, even neither (i) nor (ii) holds, (AP) can be held in some cases. If for any index i A f1; 2; …; ng, either Ω is bounded about xi or P is coercive about 2 We call function P : Rn -R is level-bounded on Ω, if for any R 40, fx A Ω : PðxÞr Rg is bounded. 3 We call function P : Rn -R is coercive, if lim J x J - þ 1 PðxÞ ¼ þ 1.

PðxÞ ¼ maxf0; x23  1g;

Ω is unbounded and P is not coercive, however, P(x) is levelbounded on Ω. Moreover, if the constraints are modeled as in (6) with u; v A Rn , then assumption (AP) holds with any definition on Ω and P in Section 3 after (6). Remark 4.2. We notice that f is coercive on Ω is also a sufficient condition for (Af). Suppose that we have some foreknowledge on the optimal solution set of (1), that is there is ϱ 4 0 such that fx : J x J r ϱg⋂Ma ∅, then we can add J x J r ϱ to the constraints of (1), which let assumption (AP) holds. Specially, if the constraint in (1) is only x A Ω, assumption (Af) is equivalent to the condition that the optimal solution set M is bounded. Based on assumption ðAP Þ or ðAf Þ, we prove that the solutions of (4) exist globally and uniformly bounded. Moreover, from the analysis in Remarks 4.1 and 4.2, there are plenty of optimization problems that satisfy ðAP Þ or ðAf Þ. Theorem 4.1. Suppose that assumption ðAP Þ or ðAf Þ holds. Any solution of (4) with initial point x0 A Ω exists globally and satisfies (i) there is R 4 0 such that J xðtÞ J r R for all t A ½0; þ 1Þ; (ii) xðtÞ A Ω for all t A ½0; þ 1Þ; R þ1 _ J 2 dt o þ 1. (iii) 0 J xðtÞ Proof. From Proposition 4.1, denote x : ½0; TÞ a solution of (4) with initial point x0 A Ω. Suppose that T o þ 1 and ½0; TÞ is the maximal existence interval of this solution. (i) First, we prove that there is R 40 such that J xðtÞ J r R, 8 t A ½0; TÞ, when assumption (AP) holds. From Lemma 4.1 and xðtÞ A Ω, 8 t A ½0; TÞ, EðxðtÞ; tÞ is nonincreasing along the solution of (4), which derives   PðxðtÞÞ r PðxðtÞÞ þ ɛðtÞ f ðxðtÞÞ  inf f rEðx0 ; 0Þ; 8 t A ½0; TÞ: Ω

By the level-boundedness of P on Ω, x(t) is bounded on ½0; TÞ, then this solution of (4) can be extended, which leads a contradiction to the supposition that ½0; TÞ is the maximal existence interval of this solution. Therefore, this solution of (4) exists globally. Similar to the above analysis, we can also get PðxðtÞÞ r Eðx0 ; 0Þ;

8 t A ½0; þ 1Þ:

Thus, there is R 4 0 such that J x J r R;

8 t A ½0; þ 1Þ:

Next, we will prove the global existence and boundedness of the solution of (4) under assumption (Af). Denote   ~ tÞ ¼ 1 J x  xn J 2 þ PðxÞ þ ɛðtÞ f ðxÞ  inf f Gðx; 2 Ω

with xn A M. ~ tÞ along the solution of (4), similar to the Differentiating Gðx; analysis in Lemma 4.1, there exist ξ ðtÞ A ∂PðxðtÞÞ and η ðtÞ A ∂f ðxðtÞÞ such that   d ~ _ J 2 þ ɛ_ ðtÞ f ðxðtÞÞ inf f GðxðtÞ; tÞ r  J xðtÞ dt Ω  〈ξ ðtÞ þ ɛðtÞη ðtÞ; xðtÞ  xn 〉 r 〈ξ ðtÞ þɛðtÞη ðtÞ; xðtÞ  xn 〉:

ð13Þ

Q. Li et al. / Neurocomputing 131 (2014) 336–347

First, we will prove the uniqueness of the solution of (4) with initial point x0 A Ω. Suppose that there exist two solutions x : ½0; þ 1Þ-Rn and y : ½0; þ1Þ-Rn of (4) with initial point x0 A Ω. Then there exist ξ 1 ðtÞ A ∂PðxðtÞÞ, ξ 2 ðtÞ A ∂PðyðtÞÞ, η 1 ðtÞ A ∂f ðxðtÞÞ and η 2 ðtÞ A ∂f ðyðtÞÞ such that

Using the convex inequality of P, it gives 〈ξ ðtÞ; xðtÞ  x 〉 Z PðxðtÞÞ  Pðx Þ Z 0; n

n

8 t A ½0; TÞ:

Coming back to (13), we have d ~ GðxðtÞ; tÞ r ɛðtÞ〈η ðtÞ; xn  xðtÞ〉 dt

a:e: t A ½0; TÞ:

ð14Þ

Denote S ¼ fx A Ω : f ðxÞ  f ðxn Þ r 0g: From assumption (Af), S is bounded. From the pseudoconvexity of f on Ω, if 〈η ðtÞ; xn xðtÞ〉 Z0, then f ðxðtÞÞ r f ðxn Þ, which implies xðtÞ A S. Thus, we only need to consider the case that xðtÞ2 = S, which means 〈η ðtÞ; xn  xðtÞ〉 o 0. From (14), we obtain that

~ tÞ, there is G~ max 4 0 Since S is bounded, by the definition of Gðx; such that when xðtÞ A S, ~ GðxðtÞ; tÞ r G~ max :

~ 0 ; 0Þg; ~ GðxðtÞ; tÞ rmaxfG~ max ; Gðx

8 t A ½0; TÞ:

8 t A ½0; TÞ;

for all t A ½0; þ 1Þ:

Integrating the above inequality from 0 to t, we have _ J 2 ds r Eðx0 ; 0Þ  EðxðtÞÞ; tÞ; J xðsÞ

where Eðx0 ; 0Þ ¼ Pðx0 Þ þ ɛð0Þðf ðx0 Þ  inf Ω f Þ and EðxðtÞ; tÞ ¼ PðxðtÞÞ þɛðtÞðf ðxðtÞÞ  inf Ω f Þ Z 0, 8 t Z 0. Therefore, Z þ1 _ J 2 dt o þ1: J xðtÞ □ 0

Proposition 4.2. Suppose that Ω is defined by a class of affine equalities, i.e.

Ω ¼ fx A Rn : Ax ¼ bg

d1 _  yðtÞ〉 _ J xðtÞ  yðtÞ J 2 ¼ 〈xðtÞ  yðtÞ; xðtÞ dt 2 2 ¼  J xðtÞ  yðtÞ J þ 〈xðtÞ  yðtÞ; P Ω ½p1 ðtÞ  P Ω ½p2 ðtÞ〉:

〈p1 ðtÞ P Ω ½p1 ðtÞ; xðtÞ  yðtÞ〉 ¼ 0 〈p2 ðtÞ P Ω ½p2 ðtÞ; xðtÞ  yðtÞ〉 ¼ 0

ð16Þ

a:e: t A ½0; þ 1Þ; a:e: t A ½0; þ 1Þ:

ð17Þ

Using (17) into (16), for a.e. t A ½0; þ1Þ, we have

ð18Þ

ð15Þ

with a full row rank matrix A A Rrn and a column vector b A Rr . When assumption ðAP Þ or ðAf Þ holds, there is a unique solution of (4) with initial point x0 A Ω and it is just its “slow solution”. Proof. From Theorem 4.1, any solution of (4) with initial point x0 A Ω exists globally.

ð19Þ

By Theorem 4.1, there is R 4 0 such that J xðtÞ J r R; J yðtÞ J rR;

8 t A ½0; þ 1Þ:

Then, there is an L 4 0 such that f þ L J x J 2 is convex on fx : J x J rRg. Using the convex inequality again, we obtain 〈xðtÞ  yðtÞ; η 1 ðtÞ  η 2 ðtÞ〉 Z  2L J xðtÞ  yðtÞ J 2 ;

(ii) Similar to the proof of Proposition 4.1, we have xðtÞ A Ω, 8 t A ½0; þ 1Þ. (iii) Using Lemma 4.1 again, we have   d _ J 2 þ ɛ_ ðtÞ f ðxðtÞÞ  inf f r  J xðtÞ _ J 2: EðxðtÞ; tÞ r  J xðtÞ dt Ω

0

Denote p1 ðtÞ ¼ xðtÞ  ξ 1 ðtÞ  ɛðtÞη 1 ðtÞ and p2 ðtÞ ¼ yðtÞ  ξ 2 ðtÞ  ɛðtÞ η 2 ðtÞ. Differentiating ð1=2Þ J xðtÞ  yðtÞ J 2 along the two solutions of (4), we have

〈xðtÞ  yðtÞ; ξ 1 ðtÞ  ξ 2 ðtÞ〉Z 0; 8 t A ½0; þ 1Þ:

we obtain that x(t) is bounded on ½0; TÞ. Similar to the analysis for the case that ðAP Þ holds, we obtain that this solution of (4) exists globally and there is R 4 0 such that

t

a:e: t A ½0; þ1Þ:

Since P is convex, it follows that

By virtue of the fact that

Z

a:e: t A ½0; þ 1Þ;

_ ¼  yðtÞ þ P Ω ½yðtÞ  ξ 2 ðtÞ  ɛðtÞη 2 ðtÞ yðtÞ

d1 J xðtÞ  yðtÞ J 2 r  〈xðtÞ  yðtÞ; ξ 1 ðtÞ  ξ 2 ðtÞ〉 dt 2 ɛðtÞ〈xðtÞ  yðtÞ; η 1 ðtÞ  η 2 ðtÞ〉:

Therefore, we confirm that

J x J rR

_ ¼  xðtÞ þ P Ω ½xðtÞ  ξ 1 ðtÞ  ɛðtÞη 1 ðtÞ xðtÞ

Since Ω is defined by (15), we have

d ~ GðxðtÞ; tÞ o 0: dt

~ GðxðtÞ; tÞ Z 12 J xðtÞ  xn J 2 ;

341

8 t A ½0; þ 1Þ:

ð20Þ

Combining (18)–(20), we obtain d1 J xðtÞ  yðtÞ J 2 r 2LɛðtÞ J xðtÞ  yðtÞ J 2 dt 2

a:e: t A ½0; þ 1Þ:

Using Gronwall's inequality [30] into the above inequality gives xðtÞ ¼ yðtÞ, 8 t Z 0. Therefore, the solution of (4) with initial point x0 A Ω is unique. Denote x : ½0; þ 1Þ-Ω the unique solution of (4) with initial point x0 A Ω. _ þ xðtÞ A Ω, For any ξðtÞ A ∂PðxðtÞÞ and ηðtÞ A ∂f ðxðtÞÞ, since xðtÞ xðtÞ A Ω; a:e: t Z0, from the same property as in (17), we get _ 〈xðtÞ; P Ω ½xðtÞ  ξðtÞ  ɛðtÞηðtÞ xðtÞ þ ξðtÞ þ ɛðtÞηðtÞ〉 ¼ 0

a:e: t Z 0;

which can be written as _ 〈xðtÞ;  xðtÞ þP Ω ½xðtÞ  ξðtÞ  ɛðtÞηðtÞ〉 _ ¼ 〈xðtÞ;  ξðtÞ  ɛðtÞηðtÞ〉 a:e: t Z 0:

ð21Þ

From (4), (9) and (21), there exist ξ ðtÞ A ∂PðxðtÞÞ and η ðtÞ A ∂f ðxðtÞÞ such that for a.e. t A ½0; þ 1Þ, we have _ J 2 r 〈xðtÞ; _ J xðtÞ  ξ ðtÞ  ɛðtÞη ðtÞ〉 d d _  ξðtÞ  ɛðtÞηðtÞ〉: ¼  PðxðtÞÞ  ɛðtÞ f ðxðtÞÞ ¼ 〈xðtÞ; dt dt

ð22Þ

342

Q. Li et al. / Neurocomputing 131 (2014) 336–347

By PðxÞ Z 0 for all x A Rn , we have l 4 0, which follows that there is a T 1 Z 0 such that

Combining (21) and (22), we obtain _ J 2 r〈xðtÞ; _ J xðtÞ  xðtÞ þ P Ω ½xðtÞ  ξðtÞ  ɛðtÞηðtÞ〉 _ J  J  xðtÞ þ P Ω ½xðtÞ  ξðtÞ  ɛðtÞηðtÞ J : r J xðtÞ

ð23Þ

PðxðtÞÞ Z

l ; 2

8 t A ½T 1 ; þ 1Þ:

ð24Þ

From Lemma 4.1, differentiating GðxðtÞ; tÞ along the solution of (4), there are ξ ðtÞ A ∂PðxðtÞÞ and η ðtÞ A ∂f ðxðtÞÞ such that

_ J ¼ 0, On one hand, when J xðtÞ _ ¼ mð  xðtÞ þ P Ω ½xðtÞ ∂PðxðtÞÞ  ɛðtÞ∂f ðxðtÞÞÞ: xðtÞ _ J 4 0, owing to On the other hand, for a.e. t Z0 such that J xðtÞ (23), we verify that

d GðxðtÞ; tÞ r  〈ξ ðtÞ þ ɛðtÞη ðtÞ; xðtÞ P M ðxðtÞÞ〉 dt r  PðxðtÞÞ ɛðtÞ〈η ðtÞ; xðtÞ  P M ðxðtÞÞ〉:

ð25Þ

By the boundedness of x(t) on ½0; þ 1Þ,

_ J r J  xðtÞ þ P Ω ½xðtÞ  ξðtÞ  ɛðtÞηðtÞ J : J xðtÞ

lim ɛðtÞ〈η ðtÞ; xðtÞ  P M ðxðtÞÞ〉 ¼ 0;

t- þ 1

From the randomicity of ξðtÞ A ∂PðxðtÞÞ and ηðtÞ A ∂f ðxðtÞÞ, we obtain that _ ¼ mð  xðtÞ þ P Ω ½xðtÞ ∂PðxðtÞÞ  ɛðtÞ∂f ðxðtÞÞÞ: xðtÞ

□

Remark 4.3. If Ω ¼ Rn , then we can easily obtain the results in Proposition 4.2. In order to enlarge the application domain of network (4) without restrictions by the assumptions in Theorem 4.1, we show the efficiency of network (4) for solving (1) based on the boundedness of the solutions of (4), not assumption (AP) or (Af). First, it is a basic condition for the efficiency of the proposed network in optimization that any cluster point of the proposed network belongs to the feasible region X \ Ω of (1).

which implies that there is a T 2 Z T 1 such that   ɛðtÞ〈η ðtÞ; P M ðxðtÞÞ  xðtÞ〉 r l ; 4

8 t A ½T 2 ; þ 1Þ:

ð26Þ

From (24) to (26), we obtain d l GðxðtÞ; tÞ r  ; dt 4

8 t A ½T 2 ; þ 1Þ:

Integrating the above inequality from T2 to tð 4 T 2 Þ, we have GðxðtÞ; tÞ r GðxðT 2 Þ; T 2 Þ 

l t; 4

8 t A ½T 2 ; þ 1Þ:

Thus, lim GðxðtÞ; tÞ ¼  1;

t- þ 1

Theorem 4.2. For any initial point x0 A Ω, any solution x : ½0; þ 1Þ-Rn of (4) satisfies lim distðxðtÞ; Ω \ XÞ ¼ 0;

which leads a contradiction with the fact that GðxðtÞ; tÞ Z 0 for all t A ½0; þ 1Þ. Therefore, lim PðxðtÞÞ ¼ 0;

ð27Þ

t- þ 1

t- þ 1

if x(t) is bounded on ½0; þ 1Þ.

which guarantees that

Proof. Suppose that x : ½0; þ 1Þ-Rn is a global solution of (4) with initial point x0 A Ω and there is R 4 0 such that

t- þ 1

J xðtÞ J r R;

The next theorem indicates that any accumulation point of the solutions of (4) is not only a feasible solution but also an optimal solution of (4).

8 t A ½0; þ1Þ:

From xðtÞ A Ω, 8 t A ½0; þ1Þ and Lemma 4.1, EðxðtÞ; tÞ is nonincreasing along this solution of (4). Using EðxðtÞ; tÞ Z 0, 8 t A ½0; þ 1Þ, we confirm that lim EðxðtÞ; tÞ

t- þ 1

Theorem 4.3. For any initial point x0 A Ω, any solution x : ½0; þ 1Þ -Rn of (4) is globally convergent to the optimal solution set M of (1) if x(t) is bounded on ½0; þ 1Þ. Proof. Denote x : ½0; þ 1Þ-Rn a global solution of (4) with initial point x0 A Ω. From (25) and PðxÞ Z 0, 8 x A Rn , we have d GðxðtÞ; tÞ r  ɛðtÞ〈η ðtÞ; xðtÞ  P M ðxðtÞÞ〉: dt

ð28Þ

Ω

In order to analyze the asymptotic properties of the solution of (4), we consider three cases. Denote

which implies lim EðxðtÞ; tÞ ¼ lim PðxðtÞÞ:

t- þ 1

□

exists:

Coming back to the boundedness of x(t) on ½0; þ 1Þ and limt- þ 1 ɛðtÞ ¼ 0, we have   lim ɛðtÞ f ðxðtÞÞ  inf f ¼ 0; t- þ 1

lim distðxðtÞ; Ω \ XÞ ¼ 0:

t- þ 1

I ¼ ft A ½0; þ 1Þ : f ðxðtÞÞ rf ðxn Þg; J ¼ ft A ½0; þ 1Þ : f ðxðtÞÞ 4 f ðxn Þg;

Next, we will prove that limt- þ 1 PðxðtÞÞ ¼ 0. Arguing by contradiction, we assume that lim PðxðtÞÞ ¼ l a 0:

t- þ 1

with an xn A M. Owing to the continuity of f on Rn , I and J are closed and open in ½0; þ 1Þ, respectively. Case 1: In this case, we assume that there exists T Z 0 such that t A I, 8 t A ½T; þ 1Þ.

Q. Li et al. / Neurocomputing 131 (2014) 336–347

Coming back to Theorem 4.2, we have xnn A Ω \ X. Therefore, xnn A M, which means that there is a sequence ft nk g such that limk- þ 1 t nk ¼ þ 1 and

Combining the definition on I and Theorem 4.2, we have lim distðxðtÞ; MÞ ¼ 0:

t- þ 1

Case 2: In this case, we assume that there exists a T Z 0 such that t A J, 8 t A ½T; þ 1Þ, which means that f ðxðtÞÞ 4 f ðxn Þ ¼ f ðP M ðxðtÞÞÞ;

343

lim distðxðt nk Þ; MÞ ¼ 0:

k- þ 1

Combining the above result with (30), we confirm that

8 t A ½T; þ 1Þ:

lim distðxðtÞ; MÞ ¼ 0:

t- þ 1

From the pseudoconvexity of f on Ω, we have 〈ηðtÞ; xðtÞ  P M ðxðtÞÞ〉 4 0;

8 ηðtÞ A ∂f ðxðtÞÞ; t A ½T; þ 1Þ:

Case 3: In this case, we assume that both I and J are unbounded. For t A I, similar to the analysis in Case 1, we have lim

From (25), we have d GðxðtÞ; tÞ o 0; dt

t- þ 1;t A I

8 t A ½T; þ 1Þ;

ð29Þ

which implies lim GðxðtÞ; tÞ

t- þ 1

exists:

lim distðxðtÞ; MÞ

t- þ 1

lim

Suppose

GðxðtÞ; tÞ rGðxðτðtÞÞ; τðtÞÞ;

lim

Then, there is a T 1 Z T such that r ; 2

ð33Þ

8 t A J:

By τ ðtÞ A I, (32) and (33), we have

lim inf t- þ 1 〈η ðtÞ; xðtÞ  P M ðxðtÞÞ〉 ¼ r 4 0:

〈η ðtÞ; xðtÞ  P M ðxðtÞÞ〉 Z

τðtÞ ¼ þ 1:

From (29) and the continuity of GðxðtÞ; tÞ on ½0; þ 1Þ, we have ð30Þ

exists:

ð32Þ

For t A J, define τ ðtÞ ¼ sups r t;s A I s. Then τðtÞ A I and ðτðtÞ; t D J. By the unboundedness of I and J, we have t- þ 1;t A J

Since limt- þ 1 EðxðtÞ; tÞ exists, we obtain

distðxðtÞ; MÞ ¼ 0:

sup t- þ 1;t A J

GðxðtÞ; tÞ r

lim

t- þ 1;t A J

GðxðτðtÞÞ; τðtÞÞ ¼ 0;

which gives

8 t A ½T 1 ; þ 1Þ:

lim

t- þ 1;t A J

GðxðtÞ; tÞ ¼ 0:

The above inequality combines with (28) and gives d r GðxðtÞ; tÞ r  ɛðtÞ; dt 2

From the definition of Gðx; tÞ and the above result, we have

8 t A ½T 1 ; þ 1Þ;

lim

t- þ 1;t A J

Integrating the above inequality from T1 to tð 4 T 1 Þ, we have Z t d  ɛðsÞ ds: GðxðtÞ; tÞ r GðxðT 1 Þ; T 1 Þ þ 2 T1 Letting t- þ 1 in the above inequality, we have t- þ 1

which leads a contraction with the boundedness of GðxðtÞ; tÞ on ½0; þ 1Þ. Thus, lim inf 〈η ðtÞ; xðtÞ  P M ðxðtÞÞ〉 ¼ 0; t- þ 1

which follows that there is an increasing sequence tn such that t n - þ 1 and lim 〈η ðt n Þ; xðt n Þ  P M ðxðt n ÞÞ〉 ¼ 0:

ð31Þ

Since xðt n Þ is bounded, there is a cluster point xnn of xðt n Þ such that there is a subsequence ft nk g of ft n g such that limk- þ 1 t nk ¼ þ 1 and limk- þ 1 xðt nk Þ ¼ xnn . Owing to the upper semicontinuity of ∂f on Rn and η ðtÞ A ∂f ðxðtÞÞ, (31) follows that there is an η ðxnn Þ A ∂f ðxnn Þ such that 〈η ðxnn Þ; xnn P M ðxnn Þ〉 ¼ 0: From the pseudoconvexity of f on Ω and the above equation, we have f ðxnn Þ r f ðP M ðxnn ÞÞ:

ð34Þ

Therefore, combining (32) and (34), we obtain lim distðxðtÞ; MÞ ¼ 0:

t- þ 1

From Theorem 4.3, we can obtain the following corollary directly.

lim GðxðtÞ; tÞ ¼  1;

n- þ 1

distðxðtÞ; MÞ ¼ 0:

Corollary 4.1. Suppose that the points in optimal solution set M are isolated. Then, any bounded solution of (4) with initial point x0 A Ω is globally convergent to a particular element of M. Corollary 4.1 shows that when the points in the optimal solution set of (1) are isolated and the solutions of (4) with initial point x0 A Ω are bounded, the proposed network (4) is globally convergent, which indicates that the proposed network has only one limit point and it has some ability to withstand the disturbance.

5. Numerical examples In this section, four examples of constrained pseudoconvex optimization problems are given to illustrate the efficiency and good performance of the proposed network (4) for solving optimization problem (1). The numerical testings are carried out with the use of Matlab 7.4. Many functions in nature are pseudoconvex, such as Butterworth filter functions, fractional functions, and some density functions in probability theory. Examples 1 and 2 belong to a class

344

Q. Li et al. / Neurocomputing 131 (2014) 336–347

of constrained pseudoconvex optimization problems, which have many applications in economics and engineering [22,23,36]. From these two examples, we can find that the main superiority of network (4) comparing with some existing networks is that the network (4) need not estimate some values in advance. Example 3 is a general problem modeled by (1), which has not only affine equality constraints, but also convex inequality constraints. This kind of problem cannot be solved by any existing network proposed for pseudoconvex optimization given before. Example 4 focuses on the minimization of Gaussian function with more general constraints. Example 1. Consider the quadratic fractional programming problem f ðxÞ ¼

min

xT Qx þ aT x þa0 cT x þ c0

5 r xi r 10;

9 8

x (t)

5 B 1 B Q ¼B @ 2

i ¼ 1; 2; 3; 4; 1

1 5

2 1

1

3

0 3C C C; 0A

3

0

5

0

x 1 (t)

6 5

x 2 (t), x 4 (t)

4 3

ð35Þ

1 0

with 0

3

7

2

Ax ¼ b;

s:t:

10

0

1

1 B 2 C B C a¼B C; @ 2 A

0

0.1

0.2

0.3

CPU Time (second) Fig. 1. Solutions of (4) with 10 different random initial points in Ω.

a0 ¼ 2;

1 Denote

c ¼ ð2 1 1 0ÞT ;

c0 ¼ 4

Ω ¼ fx A R4 : Ax ¼ bg

and  A¼

1

1

0

0

0

0

1

1

 ;

and define

  1 b¼ : 2

4

4

i¼1

i¼1

PðxÞ ¼ ∑ maxf0; 2  xi g þ ∑ maxf0; xi 4g:

Denote

Ω ¼ fx A R4 : 5 r xi r10; i ¼ 1; 2; 3; 4g: Since Q is symmetric and positive definite, f is pseudoconvex on fx A R4 : cT x þc0 40g, which includes Ω as a subset. Since Ω is bounded, then network (4) is obviously capable of solving (35).

This example is similar as one example in [23]. Since Q is not semi-positive definite, f is not pseudoconvex on R4 . Similar to the analysis in [23], f is pseudoconvex on Ω. Moreover, P is coercive, then the network (4) is capable of solving the problem considered in this example.

This problem is given and solved in [23]. To solve (35), the network in [23] needs to estimate the upper bound of the Lipschitz constant of f over a compact set containing the feasible region. By network (4), we need not do this work and optimization problem (35) can be solved effectively. From the method, for constructing the penalty function in Section 3, we define

Let ɛ 0 ¼ 10. Fig. 2 presents the trajectories of (4) with 10 random initial points, which converge to the optimal solution xn ¼ ð3; 2; 2; 4ÞT of the problem considered in this example. Through xTQx is not semi-positive definite on fx : 2 r xi r 4; i ¼ 1; 2; 3; 4g, it is semi-positive definite on Ax ¼b. Then we can define Ω according to our demand that f is pseudoconvex on Ω.

PðxÞ ¼ J Ax  bJ 2 ;

Example 3. In this example, we consider a general problem modeled by (1)

which is differentiable. Then the network in (4) is reduced to a nonautonomous differential equation. Choose ɛ 0 ¼ 10. Fig. 1 shows that the trajectories of (4) are convergent to the optimal solution xn ¼ ð6; 5; 7; 5ÞT with 10 random initial points in Ω. Example 2. Consider the quadratic fractional programming problem in (35) with 0 1 0 1 1 0:5 1 0 1 B 0:5 C B C 5:5  1 0:5 C B B 1 C Q ¼B C; a ¼ B C; @ 1 @ 1 A 1 1 0 A  0:5

0 a0 ¼  2;  A¼

0

0

c ¼ ð1  1  1 1ÞT ;

1

1

1

0

0

2

0

1

 ;

and 2 r xi r 4, i ¼ 1; 2; 3; 4.

b¼

1 c0 ¼ 6;   3 0

f ðxÞ ¼

min s:t:

ejx1  1j þ jx1  x2 jþ x22  30 ðx1  x2 Þ2 þ 1

x1 þx21 þ ex1 þ x2

r 4;

J x J r 1; x1 þ x2 ¼ 1:

ð36Þ

Denote

Ω ¼ fx A R2 : J x J r 1g; wðxÞ ¼ ejx1  1j þ jx1 x2 j þ x22 30; vðxÞ ¼ ðx1  x2 Þ2 þ 1: Since w(x) and v(x) are convex on R2 , wðxÞ o0 and vðxÞ 4 0 for all x A Ω, from Theorem 7 in [23], we know that f is pseudoconvex on Ω. Moreover, by the boundedness of Ω, network (4) can be used for solving constrained optimization problem (36). By virtue of the

Q. Li et al. / Neurocomputing 131 (2014) 336–347

345

20

10

15

9

10 8

fea(x(t))

5 0

7

−5 6

−10 −15

5

−20

x (t) 4

4

f(x(t))

−25

x (t)

−30

1

3

0

0.06

2

2

0.6

CPU Time (second)

x (t)

Fig. 4. Objective and penalty functions along the solutions of (4) with 10 initial points used in Fig. 3.

x (t) 3

1

0

0

0.25

0.5

0.75

1

CPU Time (second) Fig. 2. Solutions of (4) with 10 different random initial points in Ω.

Example 4. Consider the following optimization with Gaussian objective function: ! 5 x2 min exp  ∑ i2 i ¼ 1 si s:t:

2

1 r xi r 1;

x21 þ x2 r1;

1.5

i ¼ 1; 2; …; 5; Ax ¼ b;

x23 þ jx4 j þ x5 r 1;

ð37Þ

where s ¼ ðs1 ; s2 ; …; s5 Þ ¼ ð1; 1=2; 1=4; 1=2; 1Þ, A A R35 and b A R3 are drawn from the uniform distribution with elements in ð0; 1Þ by MATLAB. Here, 0 1 0:8147 0:9134 0:2785 0:9649 0:9572 B C 0:4854 A; A ¼ @ 0:9058 0:6324 0:5469 0:1576 0:1270 0:0975 0:9575 0:9706  0:7873

1 0.5 0

x2(t)

−0.5

b ¼ ð0:7060; 0:0318; 0:2769ÞT :

x1(t)

−1 −1.5

0.06

0.6

CPU Time (second) Fig. 3. Solutions of (4) with 10 different random initial points in Ω.

constraints in (36), the networks in [23,37] cannot be used to solve (36) directly. Construct the penalty function as PðxÞ ¼ J x1 þ x2  1 J 2 þ maxf0; x1 þ x21 þex1 þ x2  4g: Then, we use network (4) to solve (36). Let ɛ0 ¼ 10. Fig. 3 depicts the solutions of (4) with 10 random initial points in Ω, which shows that any trajectory of (4) converges to the optimal solution xn ¼ ð0:50; 0:50ÞT with the optimal value f ðxn Þ ¼ 28:10, where the x-coordinate is labeled with operator log to see the movement clearly. The same scale operation is also used in Figs. 4 and 7. Define the feasible function as feaðxÞ ¼ PðxÞ þ maxf0; J x J  1g: Fig. 4 shows the objective function value f(x) and feasible function value fea ðxÞ along the trajectories of (4) with the same initial points used in Fig. 3.

Denote Ω ¼ fx :  1 rx r1g and let PðxÞ ¼ maxf0; x21 þ x2  1g þ maxf0; x23 þ jx4 jþ x5  1g þ 50 J Ax b J : ð38Þ Gaussian function is important in stochastic optimization. As studied in [37], Gaussian function is locally Lipschitz continuous and strictly pseudoconvex on Rn . This kind of optimization has been studied in [23,37]. Guo et al. [37] considered the pseudoconvex optimization only with affine equality constraints, while Liu et al. [23] considered it with affine equalities and box constraints. However, it is more likely that more constraints such as some general inequality constraints exist in this kind of optimization problems. Then, Example 4 focuses on the minimization of Gaussian function with more general constraints. With initial point x0 ¼ ð0; 0; 0; 0; 0ÞT , the solution of (4) is shown in Fig. 5, which converges to the optimal solution xn of (38), where xn ¼ ð  0:1203; 0:2300;  0:1331; 0:0153;  0:6728ÞT with optimal value f ðxn Þ ¼  0:3818. Also with this initial point, objective function value f(x) and penalty function value P(x) along this solution of (4) are described in Fig. 6, where P(x) is defined as in (38). Moreover, the tendency of J xðtÞ  xn J along the solutions of (4) with 10 random initial points in Ω is illustrated in Fig. 7.

346

Q. Li et al. / Neurocomputing 131 (2014) 336–347

0.3

3

x (t) 2

0.2 2.5

0.1

x5(t)

0

2

x (t)

−0.1

||x(t)_x*||

1

x (t) 3

−0.2 −0.3

1.5

1

−0.4 −0.5

0.5

−0.6 −0.7

x (t)

0

5

0

1

2

3

4

5

CPU Time (second)

0

0.05

0.5

5

CPU Time (second) Fig. 7. Tendency of J xðtÞ xn J along the solutions of (4) with 10 random initial points in Ω.

Fig. 5. Solution of (4) with initial point x0 ¼ ð0; 0; 0; 0; 0ÞT .

boundedness of the solutions of proposed network are given, while some other conditions to guarantee this thing will extend the application domains of the results obtained in this paper. The Tikhonov-like method plays an important rule in solving constrained optimization. Some existing results are for convex optimization. To the best of our knowledge, it is the first time that introducing the Tikhonov-like method into a class of constrained nonsmooth and nonconvex optimization, which extends some work in [26]. It is an interesting work for further study that whether this method can be used to solve some other classes of constrained nonconvex optimization. Moreover, the estimation on the convergence rate and convergence time is an important topic both in engineering and in mathematics, which inspires us to explore it in further work.

0.8

0.6

0.4

0.2

P(x(t)) 0

−0.2

f(x(t))

References

−0.4

−0.6

−0.8

−1

0

1

2

3

4

5

CPU Time (second) Fig. 6. f(x) and P(x) along the solution of (4) with initial point x0 ¼ ð0; 0; 0; 0; 0ÞT .

6. Conclusions This paper proposes a one-layer recurrent neural network for solving a class of constrained pseudoconvex optimization problems. The proposed network is modeled by a nonautonomous differential inclusion with a projection operator. Thanks to the Tikhonov-like method, the exact penalty parameter need not be evaluated in advance. Moreover, the proposed network is also capable of solving some constrained optimization problem, for which there is no exact penalty function. By virtue of the special properties of pseudoconvex function, when the solutions of the proposed network are bounded, every solution of the proposed network converges to the optimal solution set of considered optimization problem. Some sufficient conditions to ensure the

[1] J.D. Hopfield, D.W. Tank, Neural computation of decisions in optimization problems, Biol. Cybern. 52 (1985) 141–152. [2] D.W. Tank, J.J. Hopfield, Simple neural optimization network: an A/A converter, signal decision circuit, and a linear programming circuit, IEEE Trans. Circuit Syst. I 33 (1986) 533–541. [3] M.P. Kennedy, L.O. Chua, Neural networks for nonlinear programming, IEEE Trans. Circuit Syst. 35 (1988) 554–562. [4] E.K.P. Chong, S. Hui, S.H. Zak, An analysis of a class of neural networks for solving linear programming problems, IEEE Trans. Autom. Control 44 (1999) 1095–2006. [5] X.P. Xue, W. Bian, A project neural network for solving degenerate convex quadratic program, Neurocomputing 70 (2007) 2449–2459. [6] X. Hu, J. Wang, Design of general projection neural networks for solving monotone linear variational inequalities and linear and quadratic optimization problems, IEEE Trans. Syst. Man Cybern.: Part B—Cybern. 37 (2007) 1414–1421. [7] Q. Liu, J. Cao, A recurrent neural network based on projection operator for extended general variational inequalities, IEEE Trans. Syst. Man Cybern.: Part B —Cybern. 40 (2010) 928–938. [8] S. Zhang, A. Constantinides, Lagrange programming neural networks, IEEE Trans. Circuits Syst. II 39 (1992) 441–452. [9] L. Cheng, Z.G. Hou, Y.Z. Lin, M. Tan, W.J. Chris, F.X. Wu, Recurrent neural network for non-smooth convex optimization problems with application to identification of genetic regulatory networks, IEEE Trans. Neural Netw. 22 (2011) 714–726. [10] M. Forti, P. Nistri, M. Quincampoix, Generalized neural network for nonsmooth nonlinear programming problems, IEEE Trans. Circuits Syst. I 51 (2004) 1741–1754. [11] X.P. Xue, W. Bian, Subgradient-based neural networks for nonsmooth convex optimization problems, IEEE Trans. Circuits Syst. I 55 (2008) 2378–2391. [12] Q.S. Liu, J. Wang, A one-layer recurrent neural network for constrained nonsmooth optimization, IEEE Trans. Syst. Man Cybern.: Part B—Cybern. 41 (2011) 1323–1333. [13] G. Li, S. Song, C. Wu, Generalized gradient projection neural networks for nonsmooth optimization problems, Sci. China Inf. Sci. 53 (2010) 990–1005.

Q. Li et al. / Neurocomputing 131 (2014) 336–347

[14] Q.S. Liu, J. Wang, Finite-time convergent recurrent neural network with a hard-limiting activation function for constrained optimization with piecewiselinear objective functions, IEEE Trans. Neural Netw. 22 (2011) 601–613. [15] X.B. Gao, L.Z. Liao, A new projection-based neural network for constrained variational inequalities, IEEE Trans. Neural Netw. 20 (2009) 373–388. [16] C. Oh, S.H. Zak, Large-scale pattern storage and retrieval using generalized brain-state-in-a-box neural networks, IEEE Trans. Neural Netw. 21 (2010) 633–643. [17] M.D. Marco, M. Forti, M. Grazzini, P. Nistri, L. Pancioni, Lyapunov method and convergence of the full-range model of CNNs, IEEE Trans. Circuits Syst. I 55 (2008) 3528–3541. [18] M.D. Marco, M. Forti, M. Grazzini, L. Pancioni, On global exponential stability of standard and full-range CNNs, Int. J. Circuit Theory Appl. 36 (2008) 653–680. [19] M. Forti, P. Nistri, M. Quincampoix, Convergence of neural networks for programming problems via a nonsmooth Lojasiewicz inequality, IEEE Trans. Neural Netw. 17 (2006) 1471–1486. [20] W.L. Lu, J. Wang, Convergence analysis of a class of nonsmooth gradient systems, IEEE Trans. Circuits Syst. I 55 (2009) 3514–3527. [21] M. Forti, M. Grazzini, P. Nistri, L. Pancioni, Generalized Lyapunov approach for convergence of neural networks with discontinuous or non-Lipschitz activations, Physica D 214 (2006) 88–99. [22] J. Penot, P. Quang, Generalized convexity of functions and generalized monotonicity of set-valued maps, J. Optim. Theory Appl. 92 (1997) 343–356. [23] Q. Liu, Z. Guo, J. Wang, A one-layer recurrent neural network for constrained pseudoconvex optimization and its application for portfolio optimization, Neural Netw. 26 (2012) 99–109. [24] X. Hu, J. Wang, Solving pseudomonotone variational inequalities and pseudoconvex optimization problems using the projection neural network, IEEE Trans. Neural Netw. 17 (2006) 1487–1499. [25] W. Bian, X.P. Xue, Subgradient-based neural networks for nonsmooth nonconvex optimization problems, IEEE Trans. Neural Netw. 20 (2009) 1024–1038. [26] W. Bian, X.P. Xue, Neural network for solving constrained convex optimization problems with global attractivity, IEEE Trans. Circuits Syst. I 60 (2013) 710–723. [27] J. Bolte, Continuous gradient projection method in Hilbert spaces, J. Optim. Theory Appl. 119 (2003) 235–259. [28] H. Attouch, M.O. Czarnecki, Asymptotic behavior of coupled dynamical systems with multiscale aspects, J. Differ. Equ. 248 (2010) 1315–1344. [29] H. Attouch, R. Cominetti, A dynamical approach to convex minimization coupling approximation with the steepest descent method, J. Differ. Equ. 128 (1996) 519–540. [30] J.P. Aubin, A. Cellina, Differential Inclusion: Set-Valued Maps and Viability Theory, Springer-Verlag, Berlin, Germany, 1984. [31] F.H. Clarke, Optimization and Non-Smooth Analysis, Wiley, New York, 1969. [32] A.F. Filippov, Differential Equations With Discontinuous Right-Hand Side, Kluwer, Netherlands, 1988. [33] R.T. Rockafellar, R.J. Wets, Variational Analysis, Springer-Verlag, Berlin, Germany, 1988. [34] B.E. Paden, S.S. Sastry, Calculus for computing Filippov's differential inclusion with application to the variable structure control of robot manipulator, IEEE Trans. Circuits Syst. I 34 (1987) 73–82. [35] V.I. Utkin, Sliding Modes and Their Application in Variable Structure System, Mir, Moscow, 1987.

347

[36] W. Dinkelbach, On nonlinear fractional programming, Manag. Sci. 13 (1967) 492–498. [37] Z.S. Guo, Q.S. Liu, J. Wang, A one-layer recurrent neural network for pseudoconvex optimization subject to linear equality constraints, IEEE Trans. Neural Netw. 22 (2011) 1892–1990.

Qingfa Li was born in Inner Mongolia, China, in 1981. He received the Bachelor Degree in Mathematics from the university of Harbin Institute of Technology in 2006 and Master Degree in Software engineering from Harbin Institute of Technology in 2010. From 2012, he has been on the studying of the Doctoral Degree of Information and Computer Engineering in Northeast Forestry University. His main scientific interest is in the field of neural network theory, optimization and information sciences.

Yaqiu Liu was born in Jiamusi, Heilongjiang, China, in 1971. He joined the School of Information and Computer Engineering, Northeast Forestry University, Harbin, China, since 2004, where he is currently professor, Ph.D. supervisor and Vice-Dean. He received the degree of the B.Eng. (Electrical Engineering) from Harbin Institute of Electrical Technology, the M.Eng. (Control theory and engineering) from Northeast Forestry University and Ph.D. (Navigation, Guidance and control) from Harbin Institute of Technology in 1990, 1999 and 2004, respectively. His research interests include Forestry intelligent equipment, Information Control and Intelligent Computing.

Liangkuan Zhu was born in Liaoning Province, P.R. China, in 1978. He received the Bachelor Degree and Master Degree in Mathematics from Bohai University, Jinzhou, P.R. China, in 2001 and 2004, respectively, and the Ph.D. Degree in Control Science and Technology from Harbin Institute of Technology, Harbin, P.R. China, in 2008. Since 2008, he has been with the College of Electromechanical Engineering at Northeast Forestry University and is currently an Associate Professor. His research interests include network based control, robust control, intelligent control and application.