Pattern formation via mixed attractive and repulsive interactions for nonlinear Schrödinger systems

Accepted Manuscript Pattern formation via mixed attractive and repulsive interactions for nonlinear Schrödinger systems

Jaeyoung Byeon, Yohei Sato, Zhi-Qiang Wang

PII: DOI: Reference:

S0021-7824(16)30001-0 http://dx.doi.org/10.1016/j.matpur.2016.03.001 MATPUR 2818

To appear in:

Journal de Mathématiques Pures et Appliquées

Received date:

20 August 2015

Please cite this article in press as: J. Byeon et al., Pattern formation via mixed attractive and repulsive interactions for nonlinear Schrödinger systems, J. Math. Pures Appl. (2016), http://dx.doi.org/10.1016/j.matpur.2016.03.001

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Pattern formation via mixed attractive and repulsive interactions for nonlinear Schr¨odinger systems Jaeyoung Byeon Department of Mathematical Sciences, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon 305-701, Republic of Korea

Yohei Sato Department of Mathematics, Saitama University, Shimo-Okubo 255, Sakura-ku Saitama-shi, 338-8570, Japan

Zhi-Qiang Wang∗ Center for Applied Mathematics, Tianjin University, Tianjin, 300072, China, and Department of Mathematics and Statistics, Utah State University, Logan, UT 84322, USA

Abstract The paper is concerned with the asymptotic behavior of positive least energy vector solutions to nonlinear Schr¨ odinger systems with mixed couplings which arise from models in Bose Einstein condensates and nonlinear optics. We show that due to mixed attractive and repulsive interactions the least energy solutions exhibit new interesting component-wise pattern formations, including co-existence of partial synchronization and segregation. The novelty of our approach is the successful use of multiple scaling to carry out a refined asymptotic analysis of convergence to a multiply scaled limiting system. Resume Cet article est consacr´e a` l’´etude du comportement asymptotique des solutions vectorielles de moindres ´energies pour des syst`emes de Schr¨ odinger avec couplages mixtes qui proviennent de mod`eles de Bose Einstein et d’optique non lin´eaire. Nous montrons que, en cons´equence des attractions mixtes attrac∗ Corresponding

author Email address: [email protected], [email protected], [email protected] (Zhi-Qiang Wang)

Preprint submitted to Journal of JMPA Templates

March 15, 2016

tives et r´epulsives, les solutions de moindres ´energies pr´sentent des formations de motifs nouveaux et int´eressants, y compris la co-existence de synchronisation partielle et de s´egr´egation. La nouveaut´e de notre approche r´eside dans l’utilisation r´eussie de multiples changements d’´echelle afin de proc´eder a` une analyse asymptotique fine de la convergence vers un probl`eme limite multi´echelle. Keywords: Mixed interactions, nonlinear Schr¨ odinger systems, multiple scaling, partial synchronization and segregation MSC: 35J20, 35J50, 35J57, 35J60

1. Introduction In this paper, we are interested in the existence and asymptotic behavior of ground state vector solutions for the following coupled nonlinear Schr¨ odinger system −Δui + λi ui = μi u3i +

3 

βij ui u2j

inΩ,

ui ∈ H01 (Ω)

(i = 1, 2, 3).

(1)

j=i

Here Ω is a bounded domain with a smooth (C 2 ) boundary in Rn for n ≤ 3. This type of systems and more general versions of systems of N equations arise from many physical models such as in nonlinear optics and in Bose Einstein con5

densates for multi-species condensates (i.e., [8, 21, 26] and references therein). Physically, μj and βi,j are the intraspecies and interspecies scattering lengths respectively. The sign of the scattering length βi,j determines whether the interactions of states are repulsive or attractive. In the repulsive case the components tend to segregate from each other leading to phase separations and

10

multi-existence of solutions, while for the attractive case the components tend to synchronize with each other leading to a simpler structure of positive solutions. In this paper we will focus on the case of mixed couplings, i.e., the system being partially attractive and partially repulsive. For a system of three equations in general there are four cases in terms of the signs of the couplings:

15

1) purely attractive: β12 > 0, β13 > 0, β23 > 0; 2

2) purely repulsive: β12 < 0, β13 < 0, β23 < 0; 3) mixed coupling (i): β12 > 0, β13 < 0, β23 < 0; 4) mixed coupling (ii): β12 > 0, β13 > 0, β23 < 0. The purely attractive and purely repulsive cases have been studied exten20

sively in the last ten years, see [1, 2, 3, 4, 5, 7, 9, 10, 13, 14, 15, 17, 18, 19, 20, 22, 23, 24, 25, 28, 29, 32, 34, 35, 36, 37, 38, 39, 40, 41, 42] and references therein. In these studies one common theme is that by stretching out the attractive coupling constants the components of solutions tend to be synchronized in shape asymptotically, while with large repulsive coupling constants the com-

25

ponents of solutions tend to be segregated leading to phase separations. In this paper we investigate the effects of mixed attractive and repulsive interactions and establish co-existence of partial synchronization and partial segregation. There have not been much works for the cases of mixed couplings. The classical paper [14] of Lin and Wei contains an example for the case 4) above of a

30

system of three equations in which two attractive constants and one repulsive constant were considered and a bound state solution was given by perturbation arguments for small coupling constants, exhibiting segregation between the three components. Some general conditions were discussed in [33, 34] for the existence of least energy vector solutions of multi-components systems. In

35

[28, 29] the second and the third authors of the current paper started a study on the case 3) of mixed coupling by giving the existence of a positive least energy vector solution and some primitive asymptotic estimates for large attractive coupling. In this paper we continue our study on the case 3). Since there are two parameters being of opposite characteristics, it is interesting to

40

see their combined effect to the asymptotic behaviour of the least energy vector solution. In fact, our main result in this paper shows that the repulsion effect overwhelms the attraction effect. More precisely, as the attractive constant β12 goes to ∞, the first two components of a least energy vector solution vanish uniformly for β13 , β23 < 0 while the last component converges to a solution of

45

a single equation. As β12 → ∞ while β13 , β23 < 0 are fixed, we can show by stretching out the attractive coupling constants that the first two components 3

vanish uniformly (refer to [30],[31]). In this paper we are much interested in the asymptotic when both the attractive and the repulsive couplings go to ∞. We will identify the limiting equation and establish energy expansion in this pro50

cess and establish finer estimates as well as stronger convergence to a limiting system. Our asymptotic results give rise to interesting new pattern formations of co-existence of synchronization and segregation. Our analysis shows that the simple mechanism of taking large attractive and repulsive interactions creates new interesting component-wise patterns, which is quite different from the case

55

of scalar equations or from the cases of purely attractive and purely repulsive couplings for systems. Even more quite differently with the case of fixed repulsive couplings with the large attractive coupling, we will see that some fine pattern formation of first two components occurs when both the attractive and the repulsive couplings go to ∞. We believe that this would give rise to an ex-

60

istence of multiple solutions and a rich variety of pattern formation when both the attractive and the repulsive couplings are sufficiently large. The novelty of our analysis is that we make use of multiple scaling and establish multi-level convergence of positive least energy vector solutions to a limiting system whose components are defined in different domains resulting from multiple scales. For simplicity of presentations we rename the parameters: μj by βjj , β12 by αβ12 , β13 by −ββ13 and β23 by −ββ23 , and we rewrite the system as ⎧ ⎪ ⎪ Δu1 − λ1 u1 + β11 (u1 )3 + αβ12 u1 (u2 )2 − ββ13 u1 (u3 )2 = 0, u1 > 0 ⎪ ⎨ Δu2 − λ2 u2 + αβ21 (u1 )2 u2 + β22 (u2 )3 − ββ23 u2 (u3 )2 = 0, u2 > 0 ⎪ ⎪ ⎪ ⎩ Δu − λ u − ββ (u )2 u − ββ (u )2 u + β (u )3 = 0, u > 0 3

3 3

31

1

3

32

2

3

33

3

3

in Ω in Ω in Ω (2)

with uj = 0 on ∂Ω, j = 1, 2, 3, where βij = βji > 0 for 1 ≤ i, j ≤ 3, α, β > 0 and Ω is a bounded domain with C 2 - boundary in Rn for n ≤ 3. That is, we consider a case where two coupling constants are repulsive and one coupling constant is attractive. We will study the asymptotic behavior of a least energy vector solution for large α, β > 0. More precisely, we will study the asymptotic behaviors when α, β → ∞ with some relation α and β at ∞. For a, b ∈ R, we

4

use the following notation: αa #β b → ∞ ⇐⇒ α, β → ∞ with

αa = ∞. α,β→∞ β b lim

When we say αa #β b is large, it means that three numbers α, β,

αa βb

are all large.

To state our results we first have some preliminary and describe a limiting system. Define H0 (Ω) ≡ (H01 (Ω))3 . For p ≥ 1 and λ > 0, we use the following notation, 

 |u|pp,Ω 65

|u| dx, p

= Ω

||u||2λ,Ω

=

|∇u|22,Ω

+

λ|u|22,Ω ,

u =

3 

1/2 ||ui ||2λi ,Ω

.

i=1

We define an energy functional for our system (2) I(u)

=

3

 1 i=1

1 ||ui ||2λi ,Ω − βii |ui |44,Ω 2 4



1 − (αβ12 |u1 u2 |22,Ω − ββ13 |u1 u3 |22,Ω − ββ23 |u2 u3 |22,Ω ) 2 for u = (u1 , u2 , u3 ) ∈ H0 (Ω). Then critical points of I correspond to solutions of our system (2). A solution u = (u1 , u2 , u3 ) is called a vector solution if ui = 0 for each i = 1, 2, 3. By a positive least energy vector solution of (2) we mean a vector solution whose components are all positive and whose energy is the least among all vector solutions. It is well known that the following equation has a positive least energy solution Δu − λ3 u + β33 (u)3 = 0, u3 > 0 in Ω,

u = 0 on ∂Ω.

(3)

We denote the least energy for (3) by L and the set of positive least energy solutions for (3) by L. Given a positive least energy solution U3 of (3) we consider the following system of two equations defined on Rn+ ⎧ 2 2 ⎪ 3 ⎪ Δv1 + β12 v1 (v2 )2 − β13 ( ∂U in Rn+ , ⎪ ∂ν (p)) yn v1 = 0, v1 > 0 ⎨ 2 2 3 Δv2 + β21 (v1 )2 v2 − β23 ( ∂U in Rn+ , ∂ν (p)) yn v2 = 0, v2 > 0 ⎪ ⎪ ⎪ ⎩ v1 = v2 = 0 on ∂Rn+

(4)

where p ∈ ∂Ω and ν is the inward normal unit vector at p. We will show in Section 2 that this system has a positive least energy solution whose energy will 5

be denoted by M (p, U3 ). Set M = inf{M (p, U3 ) | p ∈ ∂Ω, U3 ∈ L}.

(5)

Since L is compact in C 1 (Ω), we see that M is attained by (˜ p, U˜3 ) satisfying

∂U3 ∂ U˜3 (˜ p) = inf (p) | p ∈ ∂Ω, U3 ∈ L . (6) ∂ν ∂ν Then, we define M = {(p, U3 ) ∈ ∂Ω × L | M (p, U3 ) = M }.

(7)

From now on, for any domain O ⊂ Rn , any function u ∈ H01 (O) will be also regarded as an element in H 1 (Rn ) by the zero extension u on Rn \ O. In this paper, our main result is the following theorem. Theorem 1. (a) There exists a constant α0 > 0, independent of β > 0 such that for any α ≥ α0 , β > 0, the system (2) has a least energy vector solution α,β α,β uα,β = (uα,β 1 , u2 , u3 ) of (2) satisfying that

lim I(uα,β ) = L uniformly for β > 0

α→∞

and that

  lim min uα,β − (0, 0, U3 ) | U3 ∈ L = 0

α→∞

uniformly for β > 0. (b) For any sequence α#β 1+δ → ∞ with δ ≥ 1/4 for n = 1, 70

δ > 1/4 for n = 2,

δ ≥ 1/12 for n = 3,

it holds that (i)

n

I(uα,β ) = L +

β 1− 4 (M + o(1)) as α#β 1+δ → ∞; α

(ii) there exists xα,β ∈ ∂Ω such that for some C, D, c > 0, independent of α, β > 0, 1

β4 D √ ≤ max uα,β i (x), x∈Ω α

1

β4 |x − xα,β | uα,β ), 1 i (x) ≤ C √ exp(−c α β4 6

i = 1, 2;

(iii) lim

α#β 1+δ →∞

  min |xα,β − p| + uα,β − U3 C 1 (Ω) | (p, U3 ) ∈ M = 0. 3

(c) For any sequence α#β 1+δ → ∞ with δ ≥ 1/4 for n = 1,

δ > 1/4 for n = 2,

δ ≥ 1/12 for n = 3,

there exist a subsequence αj #βj1+δ → ∞, a pair (p, U3 ) ∈ M and a positive least energy solution (U1 , U2 ) ∈ (H01 (Rn+ ))2 of (4) such that limj→∞ xαj ,βj = p ∈ ∂Ω, and that for A ∈ O(n) with A(0, · · · , 0, 1) = ν where ν is the inward normal unit vector at p ∈ Ω, 1

(

α2 β

1 4

−1

4 uα,β 1 (βj A(·) + xαj ,βj ),

1

α2 β

1 4

−1

4 uα,β 2 (βj A(·) + xαj ,βj )) → (U1 , U2 )

in (H 1 (Rn ))2 and (L∞ (Rn ))2 as αj #βj1+δ → ∞. Remark 2. 1) From Theorem 1, the limiting system for (2) contains equation (3) and system (4) which are defined on different domains but are still linked through U3 whose normal derivative on the boundary of Ω appears in the system 75

(4). This creates a new interesting pattern as the third component u3 converges to U3 , the first two components u1 and u2 behave as a synchronized small peak (on a different scale from U3 ) moving towards a point p of the boundary where ∂U3 ∂ν

takes a minimum value. 2) The limiting profile gives co-existence of synchronization and segregation

80

of the system and this is due to the mixed interactions. 3) The limiting location of the peak of the first two components is determined by the minimal point of the inner normal derivative of U3 , a nonlocal condition which depends on a nonlinear eigenvalue problem (3) on the domain Ω. Our strategy for the proof can be described intuitively as follows. We define 1

Ωβ = {y = β 4 x | x ∈ Ω}, u1 (x) =

1

1 β4 √ v (β 4 x), α 1

7

1

u2 (x) =

1 β4 √ v (β 4 x), α 2

and

u3 (x) = v3 (x). Then we see through some calculation that ⎧ 1 1 ⎪ ⎪ Δv1 − λ11 v1 + βα11 (v1 )3 + β12 v1 (v2 )2 − β 2 β13 v1 (v3 (β − 4 y))2 = 0 in Ωβ , ⎪ ⎪ 2 β ⎨ 1 1 Δv2 − λ21 v2 + β21 (v1 )2 v2 + βα22 (v2 )3 − β 2 β23 v2 (v3 (β − 4 y))2 = 0 in Ωβ , 2 ⎪ β ⎪  3  ⎪ 1 1 ⎪ 2 ⎩ Δv3 − λ3 v3 − β31 (v1 (β 4 x))2 + β32 (v2 (β 4 x))2 βα v3 + β33 (v3 )3 = 0 in Ω. Then by doing the above multiple scaling for components we observe that a 85

coefficient of v3 in the third equation tends to zero in some sense, leading to the limiting equation for u3 , and that some local expansions near the boundary would take the first two equations to the limiting system (4) . However to make this process rigorous in some appropriate spaces we need to develop some finer estimates of both energy type and point-wise local estimates.

90

The paper is organized as follows. Because our limiting systems are on different domains for the first two and the third components, in Section 2 we first consider the limiting system (4) for the first two components. We establish some existence results and decay estimates, which are of independent interest. In Section 3 we establish the existence of a least energy solution and prove

95

that all three components are non-trivial by taking α large. We also give some energy estimates which will be used in the proof of the main theorem. Section 4 is devoted to the proof of the main theorem, some local estimates are first given, which are combined with other estimates from previous sections to finish the proof of the main theorem.

100

2. A limit system The limiting systems are on different scales for the three components. While the limiting equation (3) for the third component is well understood the limiting system (4) for the first two components needs to be studied independently first in this section. In this section, we consider the following limit system ⎧ ⎪ ⎪ Δv1 − γ1 yn2 v1 + β12 v1 (v2 )2 = 0, v1 > 0 in Rn+ , ⎪ ⎨ ⎪ ⎪ ⎪ ⎩

Δv2 − γ2 yn2 v2 + β21 (v1 )2 v2 = 0, v2 > 0

v 1 = v2 = 0 8

in Rn+ , on ∂Rn+

(8)

where γ1 , γ2 , β1,2 > 0. We define a functional  1 |∇v1 |2 +|∇v2 |2 +yn2 (γ1 v12 +γ2 v22 )−β12 v12 v22 dy J(v1 , v2 ) = J(v1 , v2 ; γ1 , γ2 ) ≡ 2 Rn+     for (v1 , v2 ) ∈ H = (v1 , v2 ) ∈ (H01 (Rn+ ))2  Rn yn2 (γ1 v12 + γ2 v22 ) dy < ∞ . We +

also define





J (v1 , v2 ; γ1 , γ2 )(v1 , v2 ) ≡

Rn +

|∇v1 |2 + |∇v2 |2 + yn2 (γ1 v12 + γ2 v22 ) − 2β12 v12 v22 dy.

We have the following inequality. Lemma 3. For u ∈ H01 (Rn+ ),   2 2 |∇u| + u dy ≤ 2 Rn +

Rn +

|∇u|2 + yn2 u2 dy.

(9)

Proof. By an elementary estimation, we get   1 u2 dy ≤ |∇u|2 dy. 2 n n {y∈R | 0≤yn ≤1} {y∈R | 0≤yn ≤1} Thus we see that   2 u dy = Rn +

 2

≤

1 2 

≤

u2 dy

u dy + {y∈Rn | 0≤yn ≤1}

{y∈Rn | yn ≥1}





{y∈Rn | 0≤yn ≤1}

Rn +

|∇u|2 dy +

Rn +

yn2 u2 dy

|∇u|2 + yn2 u2 dy.

Therefore (9) follows.  Let U3 be a least energy solution of Δu − λ3 u + β33 (u)3 = 0, u > 0 in Ω,

u = 0 on ∂Ω.

(10)

For each p ∈ ∂Ω and the inward normal vector ν of ∂Ω at p, we denote γ1 = γ1 (p, U3 ) ≡ β13 |

∂U3 (p)|2 , ∂ν

γ2 = γ2 (p, U3 ) ≡ β23 |

∂U3 (p)|2 . ∂ν

We consider the following minimization problem M (p, U3 ) ≡ inf{J(v1 , v2 ) | J  (v1 , v2 ; γ1 , γ2 )(v1 , v2 ) = 0, (v1 , v2 ) ∈ H \ {(0, 0)}}. (11)

9

Proposition 4. (i) For each p ∈ ∂Ω and a least energy positive solution U3 of 105

(10), the minimum M (p, U3 ) is achieved by some (v1 , v2 ) ∈ H which is a least energy solution of (8). (ii) For any solution (v1 , v2 ) ∈ H, denoting y = (y  , yn ) ∈ Rn−1 × R+ and  s = (y1 )2 + · · · + (yn−1 )2 , vi (y) = vi (s, yn ) up to a translation parallel to ∂Rn+ for i = 1, 2 and

∂vi (s,yn ) ∂s

< 0 for yn > 0 and s > 0.

(iii) There exist constants c, C > 0 such that for any least energy solution (U1 , U2 ) of (8) 0 < Ui (y) ≤ Ce−

√

γi 2

2 yn −c|y  |

for y ∈ Rn+ , i = 1, 2.

(12)

Proof. For each R > 0, we consider a minimization problem M (p, U3 ; R) ≡ inf{J(v1 , v2 ) | J  (v1 , v2 ; γ1 , γ2 )(v1 , v2 ) = 0, (v1 , v2 ) ∈ HR \{(0, 0)}}, where HR ≡ {(v1 , v2 ) ∈ H | v1 (y) = v2 (y) = 0 for |y| ≥ R}. It is standard to show that there exists a minimizer (v1R , v2R ) of M (p, U3 ; R) which is a least energy solution of ⎧ ⎪ ⎪ Δv1 − γ1 yn2 v1 + β12 v1 (v2 )2 = 0, v1 ≥ 0 ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ 110

Δv2 − γ2 yn2 v2 + β21 (v1 )2 v2 = 0, v2 ≥ 0 v 1 = v2 = 0

in Rn+ ∩ B(0, R), in Rn+ ∩ B(0, R),

(13)

on ∂(Rn+ ∩ B(0, R)).

We see that if v1R ≡ 0, so is v2R . From the maximum principle, we see that v1R (y), v2R (y) > 0 for y ∈ Rn+ ∩ B(0, R). By the classical moving plane method,  denoting s = (y1 )2 + · · · + (yn−1 )2 , we see that for each i = 1, 2, vi depends only on s and yn and

115

∂viR (s,yn ) ∂s

< 0 for s > 0 and (s, yn ) ∈ Rn+ ∩ B(0, R). Now

we see that {(v1R , v2R )}R is a minimizing sequence of M (p, U3 ) as R → ∞. From   Lemma 3, we see that v1R H01 (Rn+ ∩B(0,R)) , v2R H01 (Rn+ ∩B(0,R) is bounded. Claim 1





v1R L∞ , v2R L∞ R

R

is bounded.

We give a proof just for n = 3 since the other cases n ≤ 2 are easier. We multiply the first equation of (13) by (v1R )2 and the second by (v2R )2 , integrate

10

by parts and add the equalities. Then, we get 2   8 |∇(viR )3/2 |2 dx + γi yn2 (viR )3 dy n ∩B(0,R) 9 R + i=1  = β12 (v1R )3 (v2R )2 + (v1R )2 (v2R )3 dy  ≤ β12

Rn + ∩B(0,R)

Rn + ∩B(0,R)

(v1R )6 + (v2R )4 + (v1R )4 + (v2R )6 dy.

  This estimate and (9) imply that (v1R )3/2 H01 (Rn+ ∩B(0,R)) , (v2R )3/2 H01 (Rn+ ∩B(0,R)) is bounded. Then, by the Sobolev inequality, we see that   |v1R |9,Rn+ ∩B(0,R) , |v2R |9,Rn+ ∩B(0,R)

R

is bounded. This implies that   |v1R (v2R )2 |2,Rn+ ∩B(0,R) , |(v1R )2 v2R |2,Rn+ ∩B(0,R) 120

R

is bounded. Then, applying the elliptic estimate [12, Theorem 8.25], we conclude   that v1R L∞ , v2R L∞ R is also bounded. Thus the claim follows.   Claim 2 v1R L∞ , v2R L∞ R is bounded away from 0. 

Note that Rn + ∩B(0,R)

|∇v1R |2 +|∇v2R |2 +yn2 (γ1 (v1R )2 +γ2 (v2R )2 ) dy = 2β12 v1R v2R 2L2 (Rn ∩B(0,R)) +

(14) By (9) and (14), there exists a constant C > 0 such that 2 

viR 2H 1 (Rn ∩B(0,R)) ≤ C

i=1

0

+

2 

viR 4L4 (Rn ∩B(0,R)) .

i=1

+

(15)

By the Sobolev’s inequality, from (15), there exists a δ > 0 independent of R > 0 such that

2 

viR 2H 1 (Rn ∩B(0,R)) ≥ δ. 0

i=1

(16)

+

We suppose that v1R L∞ → 0 as R → ∞. Then we see from (14) that 2   |∇viR |2 + γ2 yn2 (viR )2 dy = 2β12 v1R v2R 2L2 (Rn ∩B(0,R)) i=1

+

Rn + ∩B(0,R)

≤ β12 v1R 2L∞ v2R 2L2 (Rn ∩B(0,R)) → 0 as R → ∞. +

11

R

Then (9) implies that v1R 2H 1 (Rn ∩B(0,R)) + v2R 2H 1 (Rn ∩B(0,R)) → 0 as R → ∞. 0

0

+

+

This contradicts to (16). This proves the claim. 125

Claim 3 lim|y|→∞ v1R (y) = lim|x|→∞ v2R (y) = 0 uniformly for R > 0.  First we note that limt→∞ B((0,··· ,0,t),1)∩B(0,R) (viR )2 dy = 0 uniformly for  R > 0 and i = 1, 2 since { Rn ∩B(0,R) yn2 ((v1R )2 + (v2R )2 )dy}R is bounded. Then, +

from the monotone property of viR , we see that  lim2 2 (t1 ) +···(tn ) →∞,tn >0

B((t1 ,··· ,tn ),1)∩B(0,R)

(viR )2 dy = 0

uniformly for R > 0 and i = 1, 2. Then, from Claim 1 and the elliptic estimate [12, Theorem 8.25], we see that lim|y|→∞ viR (y) = 0 for j = 1, 2. This proves the claim. Claim 4 There exist constants c, C > 0 such that for any large R > 0, 0 < viR (y) ≤ Ce−

√

γi 2

2 yn −c|y  |

for y ∈ Rn+ ∩ B(0, R), i = 1, 2. 1

We show this estimate just for i = 1. For c ∈ (0, γ14 ), we set ϕ(y) =

e−

√

γi 2

2 yn −c|y  |

. Then we have 

n−2 √ c ϕ Δϕ = γ1 yn2 − γ1 + c2 − |y  |

in Rn+ .

(17)

Then, since lim|y|→∞ v2R (y) = 0 uniformly for R > 0, we see that for some 130

D > 0, independent of R > 0, −Δϕ + γ1 yn2 ϕ − β12 (v2R )2 ϕ

≥

c(n − 2) √ 2 − β12 (uR ( γ 1 − c2 + 2 ) )ϕ |y  | √ 2 ( γ1 − c2 − β12 (uR 2 ) )ϕ

≥

0

=

for |y| ≥ D.

Then, by the comparison principle, we get the estimate. Now we are ready to prove (i). We may assume that along a subsequence, (v1R , v2R ) converges weakly to some (U1 , U2 ) in H as R → ∞. Then, from Claim   4, we see that limR→∞ Rn ∩B(0,R) (v1R v2R )2 dy = Rn (U1 U2 )2 dy. We note that +

lim inf R→∞

2   i=1

Rn + ∩B(0,R)

|∇viR |2

+

+

γi yn2 (viR )2 dy

≥

2   i=1

12

Rn +

|∇Ui |2 + γi yn2 (Ui )2 dy.

If we have a strict inequality above, there exists a constant t ∈ (0, 1) such that J  (tU1 , tU2 ; γ1 , γ2 )(tU1 , tU2 ) = 0. Then we see that J(tU1 , tU2 ; γ1 , γ2 )

=

<

2  1 2 t |∇Ui |2 + γi yn2 (Ui )2 dy 4 i=1 Rn+ 2   1 lim inf |∇viR |2 + γi yn2 (viR )2 dy = M (p, U3 ). 4 R→∞ i=1 Rn+ ∩B(0,R)

This is a contradiction; thus it holds that lim inf R→∞

2   i=1

Rn + ∩B(0,R)

|∇viR |2

+

γi yn2 (viR )2 dy

=

2   i=1

Rn +

|∇Ui |2 + γi yn2 (Ui )2 dy.

This implies that along a subsequence, (v1R , v2R ) converges strongly to (U1 , U2 ) 135

in H and locally uniformly in C 2 (Rn+ ) as R → ∞. Then, we see that (U1 , U2 ) attains M (p, U3 ) and a least energy solution of (8). The symmetric property of 1 , v2R ). This prove (i). (U1 , U2 ) comes from the symmetry of (vR

The symmetry and monotone properties in (ii) come from the classical moving plane method(refer to [11]). 140

The decay estimate (iii) follows by the same argument with the proof of Claim 3. This completes the proof. 

3. Existence of a least energy vector solution In this section for α > 0 large we construct a least energy vector solution for (2). This will be done uniformly for β > 0, giving the proof of part (a) of 145

Theorem 1. We follow and refine the arguments in [30] where the existence of a least energy vector solution was proved for fixed β > 0. For u = (u1 , u2 , u3 ) ∈ H0 (Ω), we define l(u1 , u2 , u3 )

≡

 3

1 1 |∇ui |2 + λi (ui )2 − βii |ui |4 2 i=1 2

(18)

 1 αβ12 |u1 u2 |2 − ββ13 |u1 u3 |2 − ββ23 |u2 u3 |2 . 2  Recalling the energy functional I(u) = Ω l(u1 , u2 , u2 )dx for u = (u1 , u2 , u3 ) ∈ −

13

H0 (Ω), we define Dα,β ≡ {u ∈ H0 (Ω) |I  (u)(u1 , u2 , 0) = 0, I  (u)(0, 0, u3 ) = 0, (u1 , u2 ) = (0, 0), u3 = 0 } , α,β ≡ {u ∈ H0 (Ω) | I  (u)(u1 , u2 , 0) ≤ 0, I  (u)(0, 0, u3 ) ≤ 0, (u1 , u2 ) = (0, 0), u3 = 0 } . D α,β , there exists a unique (sα,β (u), tα,β (u)) ∈ (0, 1] × Lemma 5. For any u ∈ D (0, 1] such that (sα,β (u)u1 , sα,β (u)u2 , tα,β (u)u3 ) ∈ Dα,β , I(sα,β (u)u1 , sα,β (u)u2 , tα,β (u)u3 ) = max I(su1 , su2 , tu3 ). s,t>0

α,β , we consider a C 2 -function f (s, t) = I(su1 , su2 , tu3 ) : Proof. For u ∈ D R2+ → R. Then fs (s, t) = (||u1 ||2λ1 ,Ω + ||u2 ||2λ2 ,Ω )s − (β11 |u1 |44,Ω + β22 |u2 |44,Ω + 2αβ12 |u1 u2 |22,Ω )s3 + (ββ13 |u1 u3 |22,Ω + ββ23 |u2 u3 |22,Ω )st2 , ft (s, t) = ||u3 ||2λ3 ,Ω t − β33 |u3 |44,Ω t3 + (ββ13 |u1 u3 |22,Ω + ββ23 |u2 u3 |22,Ω )s2 t. Thus fs (s, t) = ft (s, t) = 0 implies ⎤ ⎡ ⎤ ⎡ ||u1 ||2λ1 ,Ω + ||u2 ||2λ2 ,Ω s2 ⎦. Aα,β, (u) ⎣ ⎦ = ⎣ t2 ||u3 ||2λ3 ,Ω Here the matrix Aα,β (u) is given by ⎡ β |u |4 + β22 |u2 |44,Ω + 2αβ12 |u1 u2 |22,Ω ⎣ 11 1 4,Ω −(ββ13 |u1 u3 |22,Ω + ββ23 |u2 u3 |22,Ω )

(19)

−(ββ13 |u1 u3 |22,Ω + ββ23 |u2 u3 |22,Ω ) β33 |u3 |44,Ω

⎤ ⎦.

(20) α,β implies det Aα,β (u) > 0 and all entries of Aα,β, (u)−1 are positive, Since u ∈ D α,β , we have (19) has a unique solution (sα,β (u), tα,β (u)) ∈ R2+ . For u ∈ D ⎤ ⎡ ⎤ ⎡ 1 ||u1 ||2λ1 ,Ω + ||u2 ||2λ2 ,Ω ⎦. Aα,β, (u) ⎣ ⎦ ≥ ⎣ ||u3 ||2λ3 ,Ω 1 Thus it follows that ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 2 2 2 || + ||u || ( u )) ||u (s 1 1 2 α,β λ1 ,Ω λ2 ,Ω ⎦ ⎦. ⎣ ⎦ ≥ Aα,β (u)−1 ⎣ =⎣ 2 ||u3 ||λ3 ,Ω (tα,β (u))2 1 14

Thus, we get sα,β (u), tα,β (u) ∈ (0, 1]. We note fss (s, t) = fs (s, t)/s − 2(β11 |u1 |44,Ω + β22 |u2 |44,Ω + 2αβ12 |u1 u2 |22,Ω )s2 , fst (s, t) = 2(ββ13 |u1 u3 |22,Ω + ββ23 |u2 u3 |22,Ω )st, ftt (s, t) = ft (s, t)/t − 2β33 |u3 |44,Ω t2 . Since the Hessian of f (s, t) at (s, t) = (sα,β (u), tα,β (u)) is negative definite, f (s, t) has a unique maximum at (s, t) = (sα,β (u), tα,β (u)).  Remark 6. For the matrix Aα,β (u) given in (20), we define α,β ≡ {u ∈ H0 (Ω) | det Aα,β (u) > 0 } . D α,β , there exists By the same argument with the proof of Lemma 5, for any u ∈ D a unique (sα,β (u), tα,β (u)) ∈ R2+ such that (sα,β (u)u1 , sα,β (u)u2 , tα,β (u)u3 ) ∈ Dα,β , I(sα,β (u)u1 , sα,β (u)u2 , tα,β (u)u3 ) = max I(su1 , su2 , tu3 ). s,t>0

Moreover, some direct calculations show I(sα,β (u)u1 , sα,β (u)u2 , tα,β (u)u3 ) =

H(u) , 4G(u)

(21)

where G(u) = (β11 |u1 |44,Ω + β22 |u2 |44,Ω + 2αβ12 |u1 u2 |22,Ω )β33 |u3 |44,Ω − (ββ13 |u1 u3 |22,Ω + ββ23 |u2 u3 |22,Ω )2 , H(u) = β33 |u3 |44,Ω (u1 2λ1 ,Ω + u2 2λ2 ,Ω )2 + 2(ββ13 |u1 u3 |22,Ω + ββ23 |u2 u3 |22,Ω )(u1 2λ1 ,Ω + u2 2λ2 ,Ω )u3 2λ3 ,Ω + (β11 |u1 |44,Ω + β22 |u2 |44,Ω + 2αβ12 |u1 u2 |22,Ω )u3 4λ3 ,Ω . Lemma 7. There exists δ > 0, independent of α ≥ 1, β > 0 such that |u1 |24,Ω + |u2 |24,Ω ≥

δ α,

|u3 |24,Ω ≥ δ

15

α,β . for all u ∈ D

(22)

Proof. Since I  (u)(u1 , u2 , 0) ≤ 0, we have u1 2λ1 ,Ω + u2 2λ2 ,Ω ≤ β11 |u1 |44,Ω + β22 |u2 |44,Ω + 2αβ12 |u1 u2 |22,Ω − ββ13 |u1 u3 |22,Ω − ββ23 |u2 u3 |22,Ω ≤ β11 |u1 |44,Ω + β22 |u2 |44,Ω + 2αβ12 |u1 u2 |22,Ω 2  ≤ max{β11 , β22 , αβ12 } |u1 |24,Ω + |u2 |24,Ω . Since I  (u)(0, 0, u3 ) ≤ 0, we have u3 2λ3 ,Ω ≤ β33 |u1 |44,Ω − ββ13 |u1 u3 |22,Ω − ββ23 |u2 u3 |22,Ω ≤ β33 |u3 |44,Ω . 150

By the Sobolev inequality, there exists a constant C > 0 such that |u|4,Ω ≤ Cuλi ,Ω for i = 1, 2, 3. Thus, by setting δ −1 = C 2 max{β11 , β22 , β33 , β12 }, (22) follows. We consider a minimization problem cα,β =

inf

 u∈Dα,β

I(u) =

inf

 u∈Dα,β

 1 ||u1 ||2λ1 ,Ω + ||u2 ||2λ2 ,Ω + ||u3 ||2λ3 ,Ω . 4

α,β α,β Proposition 8. cα,β is achieved by an element uα,β = (uα,β 1 , u2 , u3 ) ∈ Dα,β

which satisfies ⎧ ⎪ ⎪ Δu1 − λ1 u1 + β11 (u1 )3 + αβ12 u1 (u2 )2 − ββ13 u1 (u3 )2 = 0, u1 ≥ 0 ⎪ ⎨ Δu2 − λ2 u2 + αβ21 (u1 )2 u2 + β22 (u2 )3 − ββ23 u2 (u3 )2 = 0, u2 ≥ 0 ⎪ ⎪ ⎪ ⎩ Δu − λ u − ββ (u )2 u − ββ (u )2 u + β (u )3 = 0, u > 0 3 3 3 31 1 3 32 2 3 33 3 3

in Ω in Ω in Ω

k k k ∞ Proof. Let {uk }∞ k=1 = {(u1 , u2 , u3 )}k=1 ⊂ Dα,β be a minimizing sequence for

cα,β . Since {uki λi ,Ω }k is bounded for each i = 1, 2, 3, we may assume that, for some u0i ∈ H01,2 (Ω), uki → u0i strongly in L4 (Ω) and weakly in H01 (Ω) as k → ∞. Then, from Lemma 7, (u01 , u02 ) = (0, 0), u03 = 0. It is standard to see α,β . From Lemma 5, there exists sα,β , tα,β ∈ (0, 1] that u0 = (u01 , u02 , u03 ) ∈ D such that (sα,β u01 , sα,β u02 , tα,β u03 ) ∈ Dα,β . Then, it follows that  1 k 2 u1 λ1 ,Ω + uk2 2λ2 ,Ω + uk3 2λ3 ,Ω k→∞ 4  1 0 2 u1 λ1 ,Ω + u02 2λ2 ,Ω + u03 2λ3 ,Ω ≥ 4  1 ≥ sα,β u01 2λ1 ,Ω + sα,β u02 2λ2 ,Ω + tα,β u03 2λ3 ,Ω ≥ cα,β . 4

cα,β = lim inf

16

Thus we find that sα,β = tα,β = 1, u0 ∈ Dα,β and u0 is a minimizer for cα,β . If α,β α,β 0 0 0 u0 is a minimizer for cα,β , then uα,β = (uα,β 1 , u2 , u3 ) ≡ (|u1 |, |u2 |, |u3 |) is also

a minimizer for cα,β whose components are nonnegative. Therefore we conclude that cα,β is achieved by a uα,β ∈ Dα,β whose components are all nonnegative. Now there are two Lagrange multipliers m1 , m2 ∈ R such that ∇I(uα,β ) + m1 ∇I1 (uα,β ) + m2 ∇I2 (uα,β ) = 0, where I1 (u) = I  (u)(u1 , u2 , 0) and I2 (u) = I  (u)(0, 0, u3 ). Since uα,β ∈ Dα,β , we ⎤

⎡ ⎤ 0 Aα,β (uα,β ) ⎣ ⎦ = ⎣ ⎦ , m2 0 ⎡

deduce that

m1

where Aα,β (uα,β ) is given in (20). Since uα,β ∈ Dα,β , det(Aα,β (uα,β )) > 0. Thus the minimizer uα,β satisfies ⎧ ⎪ ⎪ Δu1 − λ1 u1 + β11 (u1 )3 + αβ12 u1 (u2 )2 − ββ13 u1 (u3 )2 = 0, u1 ≥ 0 ⎪ ⎨ Δu2 − λ2 u2 + αβ21 (u1 ) u2 + β22 (u2 ) − ββ23 u2 (u3 ) = 0, u2 ≥ 0 ⎪ ⎪ ⎪ ⎩ Δu − λ u − ββ (u )2 u − ββ (u )2 u + β (u )3 = 0, u ≥ 0 3 3 3 31 1 3 32 2 3 33 3 3 2

3

2

in Ω in Ω in Ω

The positivity of u3 comes from Lemma 7 and the strong maximum principle.  Proposition 9. The followings hold uniformly for β > 0: lim cα,β = L,

(23)

α,β 2 2 lim uα,β 1 λ1 ,Ω = lim u2 λ2 ,Ω = 0,

(24)

α→∞

α→∞

α→∞





lim β

α→∞

155

Ω

αβ 2 αβ 2 2 αβ 2 (uαβ 1 ) (u3 ) + (u2 ) (u3 ) dx = lim α α→∞

Ω

2 αβ 2 (uαβ 1 ) (u2 ) dx = 0,

where L is the least energy level of (10). Proof. For any small d > 0, we define (Ω)d ≡ {x ∈ Ω | dist(x, ∂Ω) > d}, (Ω)d ≡ {x ∈ Ω | dist(x, ∂Ω) < d}. 17

(25)

Then, let ud3 be a least energy solution of Δu3 − λ3 u3 + β33 (u3 )3 = 0, u3 > 0

in (Ω)d , and u3 = 0 on ∂(Ω)d .

We define 1 Ld = 2

 (Ω)d

|∇ud3 |2

+

λ3 (ud3 )2 dx

1 − 4

 (Ω)d

β33 (ud3 )4 dx.

Then, we easily see that limd→0 Ld = L. We define  α Nd ≡ inf

(Ω)d

 l(u1 , u2 , 0)dx | Jdα (u1 , u2 ) = 0, (u1 , u2 ) ∈ (H01 ((Ω)d ))2 \{(0, 0)} ,

where Jdα (u1 , u2 )

≡

2   i=1

 (|∇ui | + λi (ui ) − βii (ui ) )dx + 2 2

2

4

(Ω)d

αβ12 (u1 u2 )2 dx. (Ω)d

It is obvious that Ndα is independent of β > 0. It is well known in [1, 4] that α 1 2 if α > α0 (d) for some α0 (d) > 0, Ndα is attained by (uα 1 , u2 ) ∈ (H0 ((Ω)d ))

satisfying ⎧ ⎨ Δu − λ u + β (u )3 + αβ u (u )2 = 0, u > 0 1 1 1 11 1 12 1 2 1 ⎩ Δu − λ u + αβ (u )2 u + β (u )3 = 0, u > 0 2 2 2 21 1 2 22 2 2

in (Ω)d in (Ω)d .

By minimizing an energy functional on the Nehari manifold, it is standard to see that there exists a least energy solution (V1 , V2 ) of ⎧ ⎨ Δv − λ v + β v (v )2 = 0, v > 0 1 1 1 12 1 2 1 ⎩ Δv − λ v + β (v )2 v = 0, v > 0 2

2 2

21

1

2

2

in (Ω)d in (Ω)d .

Then, we see that 2  2   tV1 tV2 t4  |∇Vi |2 + λi (Vi )2 dx − βii (Vi )4 dx. αJd ( √ , √ ) = (t2 − t4 ) α α α (Ω) (Ω) d d i=1 i=1 α V1 t√ This implies that there exists tα > 0 such that limα→∞ tα = 1 and Jd ( t√ , α Vα2 ) = α

0. Then we deduce that for some C = C(d), independent of large α > 0,   tα V1 tα V2 C α l(uα , u )dx ≤ l( √ , √ )dx ≤ . Ndα = 1 2 α α α (Ω)d (Ω)d 18

α d Now we note that (uα 1 , u2 , u3 ) ∈ Dα,β and  C(d) α d α . I(uα , u , u ) = l(uα 1 2 3 1 , u2 , 0)dx + Ld ≤ Ld + α (Ω)d

Then, since limd→0 Ld = L, we see that lim sup cα,β ≤ L

uniformly for β > 0.

(26)

α→∞

For the reversed inequality, we see that for any α ≥ 1 and β > 0, 4   u4  uα,β 3 λ3 ,Ω λ3 ,Ω  1 4L = inf  u ∈ H0 (Ω) \ {0} ≤ 4 4 β33 |u|4,Ω β33 |uα,β 3 |4,Ω 4 uα,β 3 λ3 ,Ω

=

α,β α,β 2 α,β α,β 2 2 uα,β 3 λ3 ,Ω + ββ13 |u1 u3 |2,Ω + ββ23 |u2 u3 |2,Ω 4 uα,β 3 λ3 ,Ω

≤

2 = uα,β 3 λ3 ,Ω .

2 uα,β 3 λ3 ,Ω

1 4

Then, since cαβ = I(uα,β ) =

3 i=1

2 uα,β i λi ,Ω , it follows that cαβ ≥ L for any

α ≥ 1, β > 0. Thus combining this with (26), we get the following uniform convergence for β > 0 : lim cα,β = L,

α→∞

α,β 2 2 lim uα,β 1 λ1 ,Ω = lim u2 λ2 ,Ω = 0,

α→∞

α→∞

α,β 2 α,β α,β 2 lim (ββ13 |uα,β 1 u3 |2,Ω + ββ23 |u2 u3 |2,Ω ) = 0.

α→0

α d Then since (uα 1 , u2 , u3 ) ∈ Dα,β , it follows that α,β 2 lim α|uα,β 1 u2 |2,Ω = 0 uniformly for β > 0.

α→∞

This completes the proof.  α,β α,β α,β α,β For the minimizer uα,β = (uα,β > 1 , u2 , u3 ) of cα,β , we show that u1 , u2

0 for large α > 0 and any β > 0. 160

Proposition 10. It holds that  1 ||u2 ||2λ2 ,Ω + ||u3 ||2λ3 ,Ω 4  1 ||u1 ||2λ1 ,Ω + ||u3 ||2λ3 ,Ω c2 ≡ inf  u∈Dα,β ∩{u2 =0} 4 c1 ≡

inf

 u∈Dα,β ∩{u1 =0}

19

≥ L + L2 , ≥ L + L1 ,

where Li > 0 is a least energy level of Δu − λi u + βii (u)3 = 0, u > 0 in Ω, u = 0 on ∂Ω. α,β > 0 in Ω for large α > 0 and any β > 0. In particular, uα,β 1 , u2

Proof. We find Dα,β ∩ {u1 = 0} ⎧ ⎨ = (0, u2 , u3 ) ∈ Dα,β ⎩ ⎧ ⎨ ⊂ (0, u2 , u3 ) ∈ Dα,β ⎩

 ⎫   u2 2λ2 ,Ω − β22 |u2 |44,Ω + ββ23 |u2 u3 |22,Ω = 0, u2 = 0 ⎬    u3 2λ3 ,Ω − β33 |u3 |44,Ω + ββ23 |u2 u3 |22,Ω = 0, u3 = 0⎭  ⎫   u2 2λ2 ,Ω ≤ β22 |u2 |44,Ω , u2 = 0 ⎬  .   u 2 ≤ β |u |4 , u = 0 ⎭ 3 λ3 ,Ω

33

3 4,Ω

3

Thus we get   4c1 ≥ inf u2 2λ2 ,Ω | u2 2λ2 ,Ω ≤ β22 |u2 |44,Ω , u2 ∈ H01 (Ω) \ {0}   + inf u3 2λ3 ,Ω | u3 2λ3 ,Ω ≤ β33 |u3 |44,Ω , u3 ∈ H01 (Ω) \ {0} = 4L2 + 4L. This prove that c1 ≥ L2 + L. By the same argument, we can prove c2 ≥ L1 + L. Above two inequalities and the convergence in Proposition 9 imply that for large = 0 and uα,β = 0. Then from the strong α > 0, independent of β > 0, uα,β 1 2 165

α,β maximum principle, we get the positivity of uα,β in Ω. This completes the 1 , u2

proof. . 4. Convergence of the solutions as α → ∞ In the section, first, we prove a uniform boundedness of uα,β in L∞ (Ω). α,β α,β Proposition 11. For the least energy vector solution uα,β = (uα,β 1 , u2 , u3 ), 170

the set {|uα,β 3 |∞,Ω }α≥1,β>0 is bounded. Proof. We note that 3 −Δuα,β + λ3 uα,β ≤ β33 (uα,β 3 3 3 )

20

in

Ω.

Then, using the Moser iteration technique (refer to [6, Proposition 3.5] and [27]), we get the boundedness of {|uα,β 3 |∞,Ω }α≥1,β>0 from the boundedness of {|uα,β 3 |λ3 ,Ω }α≥1,β>0 .  Now we show the first level convergence of uα,β as α → ∞. Proposition 12. For the vector solution uα,β , it follows that   lim min uα,β − (0, 0, U3 ) | U3 ∈ L = 0

α→∞ 175

uniformly for β > 0. Proof. From the convergence (23) in Proposition 9, {uα,β 3 λ3 ,Ω }α≥1,β>0 is ∞ bounded. To the contrary, we assume that there exist sequences {αj }∞ j=1 , {βj }j=1

with limj→∞ αj = ∞, βj ∈ (0, ∞) such that   lim min uαj ,βj − (0, 0, U3 ) | U3 ∈ L > 0.

j→∞

(27)

From (24) in Proposition 9, we see that α ,βj 2 λ1 ,Ω

lim u1 j

j→∞

α ,βj 2 λ2 ,Ω

= lim u2 j j→∞

= 0.

Thus we see that  α ,β  lim min u3 j j − U3 2λ3 ,Ω | U3 ∈ L > 0.

j→∞

α ,βj

Taking a subsequence if it is necessary, we may assume that u3 j in

H01 (Ω)

(28) → U3 weakly

and strongly in L (Ω) as j → ∞. From Lemma 7, we see that U3 = 0. 4

Since uα,β ∈ Dα,β , it holds that  α,β 2 α,β 4 2 |∇uα,β 3 | + λ3 (u3 ) − β33 (u3 ) Ω  α,β 2 α,β α,β 2 + ββ13 (uα,β 1 u3 ) + ββ23 (u2 u3 ) dx = 0.

(29)

Ω

By (25) in Proposition 9,

 Ω

α,β 2 α,β α,β 2 ββ13 (uα,β 1 u3 ) + ββ23 (u2 u3 ) dx → 0 uni-

formly for β > 0 as α → ∞. Thus, it follows that  |∇U3 |2 + λ3 (U3 )2 − β33 (U3 )4 dx ≤ 0. Ω

21

From (23), we observe that α ,βj 2 λ3 ,Ω

U3 2λ3 ,Ω ≤ lim inf u3 j j→∞

α ,βj 2 λ3 ,Ω

≤ lim sup u3 j j→∞

≤ 4L.

If we have a strict inequality  |∇U3 |2 + λ3 (U3 )2 − β33 (U3 )4 dx < 0, Ω

there exists t ∈ (0, 1) such that

 Ω

|∇tU3 |2 + λ3 (tU3 )2 − β33 (tU3 )4 dx = 0. Then,

we see that  1 |∇tU3 | + λ3 (tU3 ) dx − β33 (tU3 )4 dx 4 Ω Ω   1 1 |∇tU3 |2 + λ3 (tU3 )2 dx < |∇U3 |2 + λ3 (U3 )2 dx ≤ L. = 4 Ω 4 Ω 1 2



2

2

This is a contradiction. Thus it holds that  |∇U3 |2 + λ3 (U3 )2 − β33 (U3 )4 dx = 0. Ω

This means that α ,βj 2 λ3 ,Ω

lim inf u3 j j→∞

α ,βj

thus u3 j

= U3 2λ3 ,Ω ;

→ U3 strongly in H01 (Ω) as j → ∞. Since

1 2 4 ||U3 λ3 ,Ω

≤ L, we see

that U3 is a least energy solution of (3). This contradicts (28) and completes 180

the proof.  5. A refined convergence by a renormalization α,β α,β We recall uα,β = (uα,β 1 , u2 , u3 ) as a minimizer of cα,β . In this section, α,β we renormalize (uα,β 1 , u2 ) so that the renormalized solution converges to a

least energy solution of the elliptic system studied in section 2. After these convergence analysis we will prove Theorem 1 at the end. We define 1

Ωβ = {y = β 4 x | x ∈ Ω}, 1

β 4 α,β 1 4 uα,β 1 (x) = √ v1 (β x), α

1

β 4 α,β 1 4 uα,β 2 (x) = √ v2 (β x), α

α,β uα,β 3 (x) = v3 (x).

(30) 22

Then we see that v α,β ≡ (v1α,β , v2α,β , v3α,β ) satisfies ⎧ 1 1 ⎪ ⎪ Δv1 − λ11 v1 + βα11 (v1 )3 + β12 v1 (v2 )2 − β 2 β13 v1 (v3 (β − 4 y))2 = 0 ⎪ ⎪ 2 β ⎨ 1 1 Δv2 − λ21 v2 + β21 (v1 )2 v2 + βα22 (v2 )3 − β 2 β23 v2 (v3 (β − 4 y))2 = 0 2 ⎪ β ⎪   3 ⎪ 1 ⎪ 2 β2 3 ⎩ 4 x))2 Δv3 − λ3 v3 − β (v (β 3i i i=1 α v3 + β33 (v3 ) = 0

in Ωβ , in Ωβ , in Ω. (31)

Proposition 13. cα,β has the following upper estimate n

cα,β ≤ L +

β 1− 4 (M + o(1)) as α#β 1−n/4 → ∞. α

(32)

Here L is a least energy level of (10) and M = inf{M (p, U3 ) | p ∈ ∂Ω, U3 is a least energy solution of (10)}. for M (p, U3 ) defined in (11). Proof. Since ∂Ω is smooth, for any p ∈ ∂Ω, there exists a C 2 -diffeomorphism 185

Φ : B(0, 1) → B(p, 1) such that (i) Φ(0) = p, DΦ(0) = A ∈ O(n) with det A = 1; (ii) Φ(Rn+ ∩ B(0, 1)) ⊂ Ω; (iii) Φ(∂Rn+ ∩ B(0, 1)) ⊂ ∂Ω. We choose a cut-off function χβ ∈ C 2 (Rn+ , [0, 1]) such that 1

1

(i) χβ (y) = 1 for y ∈ Rn+ ∩B(0, β 12 ),

(ii) χβ (y) = 0 for y ∈ Rn+ \B(0, 2β 12 ).

We take a least energy solution U3 for (10) and a least energy solution (U1 , U2 )     for (8) with γ1 = β13  ∂U3 (p) and γ2 = β23  ∂U3 (p) such that for y = (y  , yn ) ∈ ∂ν

R

n−1

∂ν



× R+ , Ui (y) = Ui (|y |, yn ), i = 1, 2. We note that 4L = U3 2λ3 ,Ω = β33 |U3 |44,Ω ,

4M (p, U3 ) = |∇U1 |22,Rn+ + |∇U2 |22,Rn+ +

 Rn +

= 2β12 |U1 U2 |22,Rn+ .

(33) 2 2   ∂U   3 (p) yn β13 (U1 )2 + β23 (U2 )2 dy ∂ν

(34) 23

Now, for x ∈ Ω, we define good approximate solutions 1

1

1 1 β4 β4 w1α,β (x) = √ (χβ U1 )(β 4 Φ−1 (x)), w2α,β (x) = √ (χβ U2 )(β 4 Φ−1 (x)), w3 (x) = U3 (x). α α

Then (w1α,β , w2α,β , w3 ) ∈ H0 (Ω) and, from Proposition 4, by a change of variables 1

x = Φ(β − 4 y), it holds that for each i = 1, 2, 

 |∇Ui |22,Rn+ + oβ (1) ,   |Ui |44,Rn+ + oβ (1) ,   β12 |U1 U2 |22,Rn+ + oβ (1) ,     2 2 3  yn (Ui )2 dy + oβ (1) , βi3 Rn  ∂U (p) ∂ν

n

β 1− 4 α n β 1− 4 = α2 1− n β 4 = α n β 1− 4 = α

wiα,β 2λi ,Ω = |wiα,β |44,Ω αβ12 |w1α,β w2α,β |22,Ω ββi3 |wiα,β w3 |22,Ω

(35) (36) (37) (38)

+

where oβ (1) → 0 as β → ∞. We prove (38) since (35)–(37) can be proved by a similar way. First we note that 3

ββi3 |wiα,β w3 |22,Ω Setting x = Φ(β

− 14

β2 βi3 = α



1

((χβ U1 )(β 4 Φ−1 (x)))2 U3 (x)2 dx. Ω 1

y), we see that dx = β − 4 |DΦ(β − 4 y)|dy. Thus it follows n

that 3

ββi3 |wiα,β w3 |22,Ω =

β 2− 4 βi3 α n



1

Rn +

1

((χβ U1 )(y))2 |U3 (Φ(β − 4 y))|2 |DΦ(β − 4 y)| dy. (39)

If y ∈ B(0, 2β = U3 (Φ(β

− 14

1 12

), then β

− 14

y ∈ B(0, 2β

− 16

). For any y ∈ B(0, 2β

1 12

), setting g(t)

ty)), we see from the mean-value theorem that for some t0 ∈ (0, 1),

1

1

1

1

U3 (Φ(β − 4 y)) = g(1) − g(0) = g  (t0 ) = ∇U3 (Φ(β − 4 t0 y))DΦ(β − 4 t0 y)(β − 4 y). 

Since

∂  ∂yi 

1

y=0

U3 (Φ(β − 4 y)) = 0 for i = 1, · · · , n − 1, it follows that 1

1

β 4 U3 (Φ(β − 4 y)) → 1

∂U3 (p)yn ∂ν

uniformly on B(0, 2β 12 ) as β → ∞. From the exponential decay of U1 , U2 in 190

Proposition 4, we see that (39) implies (38).

24

From (34), (35), (37), and (38), it follows that 

n

wiα,β 2λi ,Ω + ββi3 |wiα,β w3 |22,Ω =

i=1,2

β 1− 4 (4M (p, U3 ) + oβ (1)), α

(40)

n

2αβ12 |w1α,β w2α,β |22,Ω

β 1− 4 (4M (p, U3 ) + oβ (1)). = α

(41)

α,β for sufficiently large From (33), (40), (41), (w  α,β ) ≡ (w1α,β , w2α,β , w3 ) ∈ D α#β 1−n/4 . Thus from Remark 6, there exists a unique (s, t) ∈ R2+ such that (sw1α,β , sw2α,β , tw3 ) ∈ Dα,β . By the expression (21) for I(sw1α,β , sw2α,β , tw3 ), we deduce that I(sw1α,β , sw2α,β , tw3 ) =

w3 4λ3 ,Ω K(w  α,β ) , + 4 4 4β33 |w3 |4,Ω 4β33 |w3 |4,Ω G(w  α,β )

(42)

where 2 |w3 |84,Ω (w1 2λ1 ,Ω + w2 2λ2 ,Ω )2 K(w)  = β33

+ 2β33 |w3 |44,Ω (ββ13 |w1 w3 |22,Ω + ββ23 |w2 w3 |22,Ω )(w1 2λ1 ,Ω + w2 2λ2 ,Ω )w3 2λ3 ,Ω + (ββ13 |w1 w3 |22,Ω + ββ23 |w2 w3 |22,Ω )2 w3 4λ3 ,Ω and G(w  α,β ) =(β11 |w1 |44,Ω + β22 |w2 |44,Ω + 2αβ12 |w1 w2 |22,Ω )β33 |w3 |44,Ω − (ββ13 |w1 w3 |22,Ω + ββ23 |w2 w3 |22,Ω )2 . Now, by using (33), (42) is simplified to I(sw1α,β , sw2α,β , tw3 ) L(w1α,β 2λ1 ,Ω + w2α,β 2λ2 ,Ω + ββ13 |w1α,β w3 |22,Ω + ββ23 |w2α,β w3 |22,Ω )2 . =L+ G(uα,β ) From (36), (40), and (41), n

G(w  α,β ) =

β 1− 4 (16M (p, U3 )L + o(1)) α

as α#β 1−n/4 → ∞.

(43)

Thus, from (40) and (43), it follows that as α#β 1−n/4 → ∞, n

cα,β ≤ I(sw1α,β , sw2α,β , tw3 ) = L +

β 1− 4 (M (p, U3 ) + o(1)). α

Since p ∈ Ω and a least energy solution U3 of (10) are arbitrary, we obtain the upper estimate (32). 25

Proposition 14. It holds that as α#β 1−n/4 → ∞, α,β 4 α,β α,β 2 4 β11 |uα,β 1 |4,Ω + β22 |u2 |4,Ω + 2αβ12 |u1 u2 |2,Ω α,β 2 α,β α,β 2 α,β α,β 2 2 = uα,β 1 λ1 ,Ω + u2 λ2 ,Ω + ββ13 |u1 u3 |2,Ω + ββ23 |u2 u3 |2,Ω n

≤

β 1− 4 (4M + o(1)). α

With respect to viα,β , i = 1, 2, 3, it also holds that β11 α,β 4 β22 α,β 4 |v1 |4,Ωβ + |v | + 2β12 |v1α,β v2α,β |22,Ωβ α α 2 4,Ωβ  2  1 1 λi = (|∇viα,β |22,Ωβ + √ |viα,β |22,Ωβ ) + β13 β 2 (v3α,β (β − 4 y))2 (v1α,β )2 dy β Ωβ i=1  1 1 + β23 β 2 (v3α,β (β − 4 y))2 (v2α,β )2 dy Ωβ

≤ 4M + o(1) as α#β 1−n/4 → ∞. Proof. The first equality follows from note that 4L = inf u∈H01 (Ω)\{0} 0=

u 4λ3 ,Ω β33 |u|44,Ω

∂I uα,β )uα,β 1 ∂u1 (

+

∂I uα,β )uα,β 2 ∂u2 (

= 0. We

. Since

∂I α,β α,β α,β 4 α,β α,β 2 α,β α,β 2 2 (u )u3 = uα,β 3 λ3 ,Ω −β33 |u3 |4,Ω +ββ13 |u1 u3 |2,Ω +ββ23 |u2 u3 |2,Ω , ∂u3

we see that 4L ≤

4 uα,β 3 λ3 ,Ω 4 β33 |uα,β 3 |4,Ω

=

4 uα,β 3 λ3 ,Ω α,β α,β 2 α,β α,β 2 2 uα,β 3 λ3 ,Ω + ββ13 |u1 u3 |2,Ω + ββ23 |u2 u3 |2,Ω

.

This implies α,β 2 4 uα,β 3 λ3 ,Ω − 4Lu3 λ3 ,Ω − 4La ≥ 0, α,β 2 α,β α,β 2 where we set a = ββ13 |uα,β 1 u3 |2,Ω + ββ23 |u2 u3 |2,Ω . From the quadratic

formula , we have 2 uα,β 3 λ3 ,Ω ≥ 2L +

 4L2 + 4La.

(44)

On the other hand, from the upper estimate (32) in Proposition 9, it follows that 2L +



n

2 4L2 + 4La ≤ uα,β 3 λ3 ,Ω ≤ 4L +

26

β 1− 4 (4M + o(1)). α

From this inequality, it follows that as α#β 1−n/4 → ∞, n

α,β 2 α,β α,β 2 a = ββ13 |uα,β 1 u3 |2,Ω + ββ23 |u2 u3 |2,Ω ≤

Next, by using an inequality that

√

β 1− 4 (4M + o(1)). α

(45)

1 + x ≥ 1 + 12 x − 12 x2 for x ≥ 0, we see from (44)

#  a2 a ≥ 4L + a − . ≥ 2L 1 + 1 + L L

2 uα,β 3 λ3 ,Ω

Combining this with the upper estimate (32) in Proposition 9, we see that as α#β 1−n/4 → ∞, n

α,β 2 2 uα,β 1 λ1 ,Ω + u2 λ2 ,Ω ≤ −a +

β 1− 4 a2 + (4M + o(1)). L α

Thus, it follows from (45) that as α#β 1−n/4 → ∞, n

α,β 2 α,β α,β 2 α,β α,β 2 2 uα,β 1 λ1 ,Ω +u2 λ2 ,Ω +ββ13 |u1 u3 |2,Ω +ββ23 |u2 u3 |2,Ω ≤

195

β 1− 4 (4M +o(1)). α

The second estimate comes from the first estimate by taking a change of variables.  Proposition 15. There exists a constant C > 0 such ⎧ 3 β8 ⎪ ⎪ C√ , ⎪ α ⎨ 1 α,β α,β 2 β |u1 |∞,Ωβ , |u2 |∞,Ωβ ≤ C √α , ⎪ ⎪ 1 ⎪ β4 ⎩ C√ , α

that for large α#β 1−n/4 , n = 1, n = 2, n = 3.

Proof. For n = 1, there exists C1 > 0, independent of β > 0, such that for α,β i = 1, 2, |uα,β i |∞,Ω ≤ C1 ui λi ,Ω . Then, Proposition 14 implies that there

exists some C > 0, independent of β > 0, such that for large α#β 1−n/4 , |uα,β i |∞,Ω

3  β 1− 14 1/2 β8 ≤C = C√ . α α

For n = 2, we note that α,β α,β 2 3 + λ1 uα,β ≤ β11 (uα,β −Δuα,β 1 1 1 ) + αβ12 u1 (u2 ) ≡ f

in Ω.

From [12, Theorem 8.16], for any p > 1, there exists a C1 > 0 such that |uα,β 1 |∞,Ω ≤ C1 |f |p,Ω . We take p ∈ (1, 2). Then, we see from Proposition 14 that 27

for some C2 , · · · , C6 , independent of large α#β 1/2 , |f |p,Ω

≤

α,β α,β 2 3 C2 |uα,β 1 |3p,Ω + C2 α|u1 (u2 ) |p,Ω

≤

α,β α,β α,β 3 C3 uα,β 1 λ1 ,Ω + C2 α|u1 u2 |2,Ω |u2 |2p/(2−p),Ω

β 1/2 1/2 α,β ) u2 λ2 ,Ω α2  β 1/2   β 1/2 1/2  β 1/2 1/2  3/2 C5 +α α α2 α  β 1/2 C6 α 3 C3 uα,β 1 λ1 ,Ω + C4 α(

≤ ≤ ≤ 200

if α#β 1/2 is large. This proves the case n = 2. For n = 3, it suffices to prove that |v1α,β |∞,Ω and |v2α,β |∞,Ω is bounded for large α#β 1/4 . Since (v1α,β , v2α,β ) satisfies Δv1α,β − Δv2α,β −

1

v1α,β +

1

v2α,β + β21 v2α,β (v1α,β )2 +

λ1

β2 λ2

β2

α,β 3 β11 α (v1 )

1

+ β12 v1α,β (v2α,β )2 − β 2 β13 v1α,β (uα,β 3 ( α,β 3 β22 α (v2 )

−

y

1

β4 1 y β 2 β23 v2α,β (uα,β ( 1 3 β4

))2 = 0 in Ωβ , ))2 = 0 in Ωβ .

we have −Δv1α,β

≤

−Δv2α,β

≤

β11 α,β 2 (v ) + β12 (v2α,β )2 )v1α,β α 1 β22 α,β 2 α,β (v ) )v2 (β21 (v1α,β )2 + α 2 (

in Ωβ ,

(46)

in Ωβ .

(47)

Then, for each l ≥ 0, we multiply both sides of (46) by (v1α,β )2l+1 , (47) by 205

(v2α,β )2l+1 and integrate parts and add two inequalities. Then for some C > 0, independent of α, β > 1, l ≥ 0,  |∇(v1α,β )l+1 |2 + |∇(v2α,β )l+1 |2 dy Ωβ



≤ C(l + 1) Ωβ

(v1α,β )2l+4 + (v1α,β )2l+2 (v2α,β )2 + (v1α,β )2 (v2α,β )2l+2 + (v2α,β )2l+4 dy.

Then, using the Sobolev inequality and H¨older’s inequality, we see that for some C  > 0, independent of l ≥ 0,  1/3  α,β 6(l+1) α,β 6(l+1)  (v1 ) + (v2 ) dy ≤ C (l + 1) Ωβ

Since

2

q ≥ 6,



i=1

Ωβ

Ωβ

(v1α,β )2l+4 + (v2α,β )2l+4 dy.

|∇viα,β |22,Ωβ is bounded for large α#β 1/4 , this implies that for any

(v1α,β )q + (v2α,β )q dx is bounded for large α#β 1/4 . . Then, applying 28

Theorem 9.20 and Theorem 9.26 in [12], we get the boundedness of |v1α,β |∞,Ωβ 210

and |v2α,β |∞,Ωβ for large α#β 1/4 . This proves the claim.  From now on, we assume that α#β 1+δn → ∞ for δ1 = 1/4 if n = 1, δ2 > 1/4 if n = 2 and δ3 > 1/12 if n = 3. We note that α#β 1−n/4 → ∞ if α#β 1+δn → ∞. Now, from Proposition 12, taking a subsequence if it is necessary, we may → U3 in H01 (Ω) as assume that for a least energy solution U3 of (10), uα,β 3

215

to U3 . α#β 1+δn → ∞. Then, we get the following strong convergence of uα,β 3 Proposition 16. For δ1 = 14 , any δ2 >

1 4

and any δ3 >

1 12 ,

uα,β → U3 up to a 3

subsequence in C 1 (Ω) as α#β 1+δn → ∞. and U3 satisfy Proof. Note that uα,β 3  α,β 2 2 uα,β βj β13 (uα,β 1 ) + ββ23 (u2 ) 3



3 − λ3 uα,β + β33 (uα,β Δuα,β 3 3 3 )

=

ΔU3 − λ3 U3 + β33 (U3 )3

=

0

in

in Ω,

Ω.

− U3 , we see Then, defining wα,β ≡ uα,β 3   α,β 2 α,β 2 3 3 uα,β −Δw3α,β + λ3 w3α,β = β33 (uα,β 3 ) − β33 (U3 ) + ββ13 (u1 ) + ββ23 (u2 ) 3   α,β 2 α,β 2 3 3 We define f ≡ β33 (uα,β ) − β (U ) , g ≡ ββ (u ) + ββ (u ) uα,β 33 3 13 23 3 1 2 3 . We proved in Proposition 11 that {|uα,β 3 |∞,Ω }α≥1,β>0 is bounded. Thus, for any p > 1, there exists a constant C > 0, independent of large α, β > 0 such that |f |p,Ω ≤ C|wα,β |p,Ω ,

α,β 2 α,β 2 α,β |g|p,Ω ≤ Cβ|(uα,β 1 ) u3 |p,Ω + Cβ|(u2 ) u3 |p,Ω .

For n = 1, we take p = 2. Then, we see from Propositions 14, 15 that for some C  > 0, independent of large α, β > 0, |g|p,Ω ≤ Cβ

2 

α,β α,β  |uα,β i |∞,Ω |ui u3 |2,Ω ≤ C β

i=1

1+1/4 β 3/8 1 β . = C α α1/2 α1/2 β 1/8

Thus, |f |2,Ω and |g|2,Ω converge to 0 as α#β 1+1/4 → ∞. Then it follows from the 220

W 2,2 − estimate [12, Theorem 8.12] that wα,β → 0 in W 2,2 (Ω) as α#β 1+1/4 → ∞. This implies that wα,β → 0 in C 1 (Ω) as α#β 1+1/4 → ∞.

29

For n = 2, we take any p > 2. Then, we see from Propositions 11, 14, 15 that for some C  > 0, independent of large α, β > 0, |g|p,Ω ≤ Cβ

2 

p−2 2 α,β α,β p (|u p (|uα,β i |∞,Ω ) i u3 |2,Ω )



≤Cβ

i=1

β

2p−2 2p

1

α

2p−2 2p

α p β 2p

1

1

=C

β

3 2− 2p

α

.

Thus, it follows from Lemma 9.17 and Theorem 9.19 in [12] that wα,β → 0 in 3

W 2,p (Ω) as α#β 2− 2p → ∞. Since W 2,p (Ω) → C 1 (Ω) for p > 2, it follows that 3

wα,β → 0 in C 1 (Ω) as α#β 2− 2p → ∞. Since the C 1 convergence holds for any 225

p > 2 and 2 −

3 2p

= 1 + 1/4 for p = 2, we conclude that for any δ2 > 1/4,

wα,β → 0 in C 1 (Ω) as α#β 1+δ2 → ∞. For n = 3, we take any p > 3. Then, we see from Propositions 11, 14, 15 that for some C  > 0, independent of large α, β > 0, |g|p,Ω ≤ Cβ

2 

p−2 2 α,β α,β p (|u p (|uα,β i |∞,Ω ) i u3 |2,Ω )



≤Cβ

i=1

β

2p−2 4p

1

α

2p−2 2p

α p β 4p

1

3

=C

β

1+ 2p−5 4p

α

Thus, it follows from Lemma 9.17 and Theorem 9.19 in [12] that wα,β → 0

W 2,p (Ω)

in

as

α#β 1+

2p−5 4p

→ ∞.

Since W 2,p (Ω) → C 1 (Ω) for p > 3, it follows that wα,β → 0 in C 1 (Ω) as α#β 1+

2p−5 4p

→ ∞.

Since the C 1 convergence holds for any p > 3 and 1 + we conclude that for any δ3 >

1 12 ,

2p−5 4p

=1+

1 12

for p = 3,

wα,β → 0 in C 1 (Ω) as α#β 1+δ3 → ∞. This

completes the proof.  230

By Proposition 16, for any α#β 1+δn → ∞, there exist sequences {αj }j , α ,βj

{βj }j with limj→∞ αj #βj1+δn = ∞ such that u3 j

→ U3 in C 1 (Ω) as j → ∞.

Proposition 17. The following holds : α ,βj ∞ }j=1

(i) {v1 j

α ,βj ∞ }j=1

and {v2 j

are bounded in H 1 (Rn ).

(ii) There exists a constant δ  > 0 such that for j = 1, 2, · · · , α ,βj 4 |4,Ωβ j

|v1 j

α ,βj 4 |4,Ωβ j

+ |v2 j 30

≥ δ .

.

Proof. We show that for some C > 0, independent of j ≥ 1, α ,βj 2 |2,Ω

|ui j

  1 1 − α ,β α ,β α ,β ≤ C βj 2 |∇ui j j |22,Ω + βj2 βi3 |u3 j j ui j j |22,Ω

(i = 1, 2). (48)

−1

To show (48), we define (Ω)bj = {x ∈ Ω | dist(x, ∂Ω) ≤ βj 4 } and (Ω)cj = Ω \ Ωbj . α ,βj

Note that u3 j

→ U3 in C 1 (Ω), and that by the Hopf lemma, supx∈∂Ω

∂U3 (x) ∂n

< 0 for the outnormal n of ∂Ω at x ∈ ∂Ω. Thus there exists a constant D > 0, independent of j ≥ 1 such that α ,βj

u3 j

(x) ≥ Ddist(x, ∂Ω) in Ω.

1/2

Then, since βj (dist(·, ∂Ω))2 ≥ 1 in (Ω)cj , it follows that for each i = 1, 2, 1/2

α ,β |ui j j |22,(Ω)cj

βj α ,β α ,β ≤ |u j j ui j j |22,Ω . D2 3

(49)

On the other hand, by the Poincar´e inequality, there exists a constant E > 0, independent of j ≥ 1, such that for i = 1, 2, α ,βj 2 |2,(Ω)b j

|ui j 235

≤

E 1/2 βj

α ,βj 2 |2,(Ω)b j

|∇ui j

≤

E 1/2 βj

α ,βj 2 |2,Ω .

|∇ui j

(50)

Combining (49) and (50), we get (48). The inequality (48) is equivalent to   αj ,βj 2 αj ,βj 2 |vi |2,Ωβ ≤ C |∇vi |2,Ωβ + βi3 j

j

1 2

Ωβ j

−1 α ,β α ,β βj v3 j j (βj 4 y)2 (vi j j )2

dy . (51)

Then, Proposition 14 implies the boundedness of

α ,β α ,β {v1 j j , v2 j j }∞ j=1

in H 1 (Rn ).

This proves (i). α ,βj

To show (ii), we multiply the first equation of (31) by v1 j equation by

α ,β v2 j j ,

α ,βj 2 |2,Ωβ j

|∇v1 j



1 2

+ β13 Ωβj

=

, the second

integrate over Ωβj and add two equalities. Then we get α ,βj 2 |2,Ωβ j

+ |∇v2 j

+

λ1 1 2

βj

α ,βj 2 |2,Ωβ j

|v1 j

−1 α ,β α ,β βj v3 j j (βj 4 y)2 (v1 j j )2

+



λ2 1 2

βj

1

dy + β23 Ωβj

α ,βj 2 |2,Ωβ j

|v1 j

α ,βj

βj2 v3 j

−1

β22 αj ,βj 4 β11 αj ,βj 4 α ,β α ,β |v |4,Ωβ + |v |4,Ωβ + 2β12 |v1 j j v2 j j |22,Ωβ . j j j αj 1 αj 2 31

α ,βj 2

(βj 4 y)2 (v2 j

) dy (52)

Combining (51) and (52), we deduce that for some c > 0, independent of j ≥ 1, α ,βj 2 H 1 (Ωβ ) 0 j

c(v1 j

α ,βj 2 H 1 (Ωβ ) ) 0 j

+ v2 j

α ,βj 2 |4,Ωβ j

≤ β12 (|v1 j

α ,βj 2 |4,Ωβ )2 . j

+ |v2 j

(53)

Therefore, combining the Sobolev inequality and (53), we get (ii).  α ,βj

Proposition 18. The set {|v1 j 240

α ,βj

|∞,Ωβj , |v2 j

|∞,Ωβj }j is bounded and bounded

away from 0. Proof. For n = 3, the boundedness in L∞ (RN ) was proved in Proposition 15 For n = 2, we note from (31) that α ,βj

−Δv1 j

α ,βj

−Δv2 j

+ +

λ1

1

βj2 λ2

1

βj2

α ,βj

≤

α ,βj

≤ β21 (v1 j

v1 j v2 j

αj ,βj 3 β11 ) αj (v1

α ,βj

+ β12 v1 j

α ,βj 2 αj ,βj ) v2

+

α ,βj 2

(v2 j

) ≡ h1

αj ,βj 3 β22 ) αj (v2

≡ h2

in Ωβj , in Ωβj . (54)

By Proposition 17 and the Sobolev inequality, we see that for any q ≥ 1, the set {|h1 |q,Ωβj , |h2 |q,Ωβj } is bounded. Applying [12, Theorem 9.26], we see that the set {|v1 |∞,Ωβj , |v2 |∞,Ωβj } is bounded. For n = 1, it follows from Proposition 17 since for any u ∈ H01 (Ωβj ),  |u |2 + u2 dy. |u|2L∞ (Ωβ ) ≤ j

α ,βj

If lim inf j→∞ |v1 j

Ωβj

α ,βj

|∞,Ωβj = 0 or lim inf j→∞ |v2 j

|∞,Ωβj = 0, we see from

(51) and (52) that α ,βj 2 H 1 (Ωβ ) 0 j

lim inf (v1 j j→∞

245

α ,βj 2 H 1 (Ωβ ) ) 0 j

+ v2 j

=0

α ,βj 2 α ,β H 1 (Ωβ ) +v2 j j 2H 1 (Ωβ ) }j . 0 0 j j

since Proposition 17 implies the boundedness of {v1 j This contradicts (53). This completes the proof.  Proposition 19. It holds that lim dist(x,∂Ωβj )→∞



α ,βj

v1 j

α ,βj

(x) + v2 j

 (x) = 0 uniformly for j = 1, 2, · · · .

Moreover, there exist constants C, c > 0 such for j ≥ 1 and i = 1, 2, α ,βj

vi j

(y) ≤ C exp(−c(d(y, ∂Ωβj ))2 ),

where d(y, ∂Ωβj ) ≡ min{|y − z| | z ∈ ∂Ωβj }. 32

1

−1

α ,βj

(βj 4 y))2 → ∞ as d(y, ∂Ωβj ) → ∞, it follows from  α ,β α ,β Proposition 15 that liml→∞ {y∈Ωβ | d(y,∂Ωβ )≥l} (v1 j j (y))2 + (v2 j j (y))2 dy =

Proof. Since (βj ) 2 (v3 j

j

j

0 uniformly for j ≥ 1. Note from Proposition 18 that for some C > 0, independent of large j ≥ 1, α ,βj

−Δv1 j

α ,βj

−Δv2 j

+ +

λ1

1

βj2 λ2

1 βj2

α ,βj

≤ C(v1 j

α ,βj

≤ C(v1 j

v1 j v2 j

α ,βj

+ v2 j

α ,βj

)

in Ωβj ,

α ,βj

+ v2 j

α ,βj

)

in Ωβj .

(55)

Using an elliptic estimate [12, Theorem 9.26] and the property that  α ,β α ,β (v1 j j (y))2 + (v2 j j (y))2 dx = 0 lim l→∞

{x∈Ωβj | d(x,∂Ωβj )≥l}

uniformly for j ≥ 1, we see that 

lim dist(y,∂Ωβj )→∞

α ,βj

v1 j

α ,βj

(y) + v2 j

 (y) = 0 uniformly for j = 1, 2, · · · . α ,βj

For the exponential decay property of v1 j

α ,βj

and v2 j

, we take the first

eigenfunction ϕ > 0 in Ω of −Δ with ϕ = 0 on ∂Ω and maxx∈Ω ϕ(x) = 1. Let μ1 > 0 be the corresponding first eigenvalue. For a > 0, which will be determined later, we define 1

1

1

Φj (y) ≡ exp(−a(βj4 ϕ((βj )− 4 y))2 ), y = (βj ) 4 x ∈ Ωβj . We note that α ,βj

−Δv1 j

  λ 1 β11 αj ,βj 2 −1 α ,β α ,β α ,β 1 +  +βj2 (v3 j j (βj 4 y))2 − (v1 ) −β12 (v2 j j )2 v1 j j = 0 in Ωβj , α βj j

and from Proposition 16 that for some e1 > 0, independent of j ≥ 1, 1

α ,βj

βj2 (v3 j

−1

(βj 4 y))2 ≥ e1 (d(y, Ωβj ))2 , y ∈ Ωβj .

We also note that 1

−ΔΦj (y) = (−2aμ1 (ϕ(x))2 + 2a|∇ϕ(x)|2 − 4a2 (βj ) 2 (ϕ(x))2 |∇ϕ(x)|2 )Φj (y), and that for some e2 > 0, independent of j ≥ 1, 1

1

1

(βj ) 2 (ϕ(x))2 = (βj ) 2 (ϕ((βj )− 4 )y)2 ≤ e2 (d(y, Ωβj ))2 , y ∈ Ωβj . 33

Then, for e3 ≡ maxx∈Ω |∇ϕ(x)|,  λ  1 β11 αj ,βj 2 −1 α ,β α ,β 1 −ΔΦj +  + βj2 (v3 j j (βj 4 y))2 − (v1 ) − β12 (v2 j j )2 Φj αj βj   λ 1 1 β11 αj ,βj 2 − α ,β α ,β 1 =  + βj2 (v3 j j (βj 4 y))2 − (v1 ) − β12 (v2 j j )2 Φj αj βj   1 1 − −1 −1 −1 − 2aμ1 (ϕ(βj 4 y))2 − 2a|∇ϕ(βj 4 y)|2 + 4a2 (βj ) 2 (ϕ(βj 4 y))2 |∇ϕ(βj 4 y)|2 Φj   ≥ e1 (d(y, Ωβj ))2 − 4a2 e3 (d(y, Ωβj ))2 Φj  β −1 α ,β α ,β 11 − (v1 j j )2 + β12 (v2 j j )2 + 2aμ1 (ϕ(βj 4 y))2 Φj αj e  1 (d(y, Ωβj ))2 − 4a2 e2 e3 (d(y, Ωβj ))2 Φj = 2  e β11 αj ,βj 2 −1 α ,β 1 (d(y, Ωβj ))2 − + (v1 ) − β12 (v2 j j )2 − 2aμ1 (ϕ(βj 4 y))2 Φj . 2 αj α ,βj

Since {v1 j

α ,βj

, v2 j

}j is bounded in L∞ (RN ), for large m > 0, we see that for

d(y, ∂Ωβj ) ≥ m, β11 αj ,βj 2 e1 −1 α ,β (d(y, Ωβj ))2 − (v1 ) − β12 (v2 j j )2 − 2aμ1 (ϕ(βj 4 y))2 > 0. 2 αj We take small a > 0 so that

e1 2

− 4a2 e2 e3 ≥

e1 4 .

Then for d(y, ∂Ωβj ) ≥ m,

  λ 1 1 β11 αj ,βj 2 α ,β α ,β 1 (v1 ) − β12 (v2 j j )2 Φj > 0. −ΔΦj +  + βj2 (v3 j j (β − 4 y))2 − αj βj Thus, it follows from the comparison principle that for some large m > 0, independent of j, there exists a constant D > 0 such that α ,βj

v1 j

(y) ≤ DΦj (x),

d(y, ∂Ωβj ) ≥ m. α ,βj

Then, since for some e4 > 0, e4 d(x, ∂Ω) ≤ ϕ(x), x ∈ Ω and {v1 j ∞

bounded in L (R ), there exist constant C, c > 0 such for j ≥ 1, n

α ,βj

v1 j

(y) ≤ C exp(−c(d(y, ∂Ωβj )2 )), α ,βj

By the same argument with v1 j α ,βj

v2 j

x ∈ Ωβj .

, we get the same estimate

(y) ≤ C exp(−c(d(y, ∂Ωβj )2 )),

This completes the proof.  34

x ∈ Ωβj .

α ,βj

, v2 j

}j is

For each m > 0, we define (Ωβj )m ≡ {x ∈ Ωβj | dist(y, ∂Ωβj ) < m}. 250

α ,βj

Proposition 20. For each i = 1, 2 and m > 0, the set {||vi j

||C 1,γ ((Ωβj )m ) }j

is bounded for γ ∈ (0, 1). Proof. From the results of Proposition 19 and the first equation of (31), we see α ,βj

that for each m > 0, {|Δv1 j 255

α ,βj

|∞,(Ωβj )m , |Δv2 j

|∞,(Ωβj )m } is bounded. Then

applying the W 2,p estimates [12, Theorem 9.11], [12, Theorem 9.13] and the embedding W 2,p → C 1,γ for large p > 1, we conclude that for each i = 1, 2 and α ,βj

m > 0, the set {||vi j

||C 1,γ ((Ωβj )m ) }j is bounded for γ ∈ (0, 1). 

Proposition 21. It holds that α ,βj αj ,βj 2 v2 |2,Ωβ j

lim 2β12 |v1 j 

j→∞

= lim

j→∞

α ,βj 2 |2,Ωβ j

|∇v1 j

α ,βj 2 |2,Ωβ j

+ |∇v2 j



1

+ β13 Ωβj



α ,βj

βj2 v3 j

−1

Ωβj

) dy



1 2

+β23

α ,βj 2

(βj 4 y)2 (v1 j

−1 α ,β α ,β βj v3 j j (βj 4 y)2 (v2 j j )2

dy

= 4M.

Moreover, there exists a sequence {yj }∞ j=1 ⊂ ∂Ωβj such that lim

α ,βj

|y−yj |→∞

vi j

(y) = 0 uniformly for i = 1, 2, j ≥ 1.

Proof. In the proof of Proposition 17, we derived the inequality (53), α ,βj 2 H 1 (Ωβ ) 0 j

c(v1 j

α ,βj 2 H 1 (Ωβ ) ) 0 j

+ v2 j

α ,βj 2 |4,Ωβ j

≤ β12 (|v1 j

α ,βj 2 |4,Ωβ )2 , j

+ |v2 j

where c > 0 is independent of j ≥ 1. Since α ,βj 2 α ,β 4,Ωβ +v2 j j 24,Ωβ j j

v1 j

α ,βj

≤ |v1 j

α ,βj

|∞,Ωβj |v1 j

α ,βj

|2,Ωβj +|v2 j

we see from (i) and (ii) of Proposition 17 that

 αj ,βj αj ,βj lim inf |v1 |∞,Ωβj + |v2 |∞,Ωβj > 0. j→∞

35

α ,βj

|∞,Ωβj |v2 j

|2,Ωβj ,

This and Proposition 19 imply that there exists a sequence {zj }j ⊂ Ωβj such that {dist(zj , ∂Ωβj )}j is bounded and 

αj ,βj αj ,βj (zj ) + v2 (zj ) > 0. lim inf v1 j→∞

We find yj ∈ ∂Ωβj such that |yj − zj | = dsit(zj , ∂Ωβj ). For the sake of convenience, we may assume that yj = 0 and zj = (0, · · · , 0, tj ) for some tj > 0. α ,βj

From Proposition 17, we may assume further that (v1 j

α ,βj

, v2 j

) converges

weakly to (v1 , v2 ) in H. Then, we see from Proposition 16 and Proposition 20 α ,βj

that (v1 j

α ,βj

, v2 j

) converges locally uniformly in Rn , up to a subsequence of

j ≥ 1, to a nontrivial solution (v1 , v2 ) ∈ H of ⎧ 3 (0) 2 2 2 ⎪ ⎪ Δv1 − β13 ( ∂U ⎪ ∂xn ) yn v1 + β12 v1 (v2 ) = 0, v1 ≥ 0 ⎨ 3 (0) 2 2 2 Δv2 − β23 ( ∂U ∂xn ) yn v2 + β21 (v1 ) v2 = 0, v2 ≥ 0 ⎪ ⎪ ⎪ ⎩ v 1 = v2 = 0

in Rn+ , in Rn+ ,

(56)

on ∂Rn+

If one of v1 , v2 is identically zero, both should be identically zero. Thus by the maximum principle, it follows that v1 , v2 > 0 in Rn+ . From Proposition 14 and (i) of Proposition 17 we have α ,βj αj ,βj 2 v2 |2,Ωβ j

lim sup 2β12 |v1 j j→∞



α ,βj 2 |2,Ωβ j

= lim sup |∇v1 j j→∞

α ,βj 2 |2,Ωβ j

+ |∇v2 j



 Ωβ j 1 2

+β23 Ωβj

1

+ β13

α ,βj

βj2 v3 j

−1

α ,βj 2

(βj 4 y)2 (v1 j

−1 α ,β α ,β βj v3 j j (βj 4 y)2 (v2 j j )2

) dy

dy

≤ 4M. (57)

From (56), we see that 2β12 |v1 v2 |22,Rn+ = |∇v1 |22,Rn+ + |∇v2 |22,Rn+ +

 Rn +

β13 (

∂U3 (0) 2 2 ∂U3 (0) 2 2 ) yn (v1 )2 + β23 ( ) yn (v2 )2 dy ∂xn ∂xn

≥ 4M.

(58)

36

Note that for each R > 0,  1 1 −1 −1 α ,β α ,β α ,β α ,β β13 βj2 v3 j j (βj 4 y)2 (v1 j j )2 + β23 βj2 v3 j j (βj 4 y)2 (v2 j j )2 dy lim j→∞



Ωβj ∩B(0,R)

= Rn + ∩B(0,R)

β13 (

∂U3 (0) 2 2 ∂U3 (0) 2 2 ) yn (v1 )2 + β23 ( ) yn (v2 )2 dy, ∂xn ∂xn

(59)

and that α ,βj 2 |2,Ωβ j

α ,βj 2 |2,Ωβ j

lim inf |∇v1 j j→∞

+ |∇v2 j

≥ |∇v1 |22,Rn+ + |∇v2 |22,Rn+ .

(60)

Then we deduce from (57), (58), (59), (60) that α ,βj αj ,βj 2 v2 |2,Ωβ j

lim 2β12 |v1 j 

j→∞

= lim

j→∞

α ,β |∇v1 j j |22,Ωβ j

+

α ,β |∇v2 j j |22,Ωβ j



1

+ β13 Ωβj



α ,βj

βj2 v3 j

−1

α ,βj 2

(βj 4 y)2 (v1 j



1 2

+β23 Ωβj

) dy

−1 α ,β α ,β βj v3 j j (βj 4 y)2 (v2 j j )2

dy

= 4M, (61)

and that uniformly for j ≥ 1,  1 −1 α ,β α ,β α ,β βj2 v3 j j (βj 4 y)2 ((v1 j j )2 + (v2 j j )2 ) dy = 0. lim R→∞

(62)

Ωβj \B(0,R)

1

α ,βj

Since βj2 v3 j

−1

(βj 4 y)2 ≥ c dist(y, ∂Ωβj ) in Ωβj for some c > 0, (62) implies

that uniformly for j ≥ 1,  α ,β α ,β (dist(x, ∂Ωβj ))2 ((v1 j j )2 + (v2 j j )2 ) dy = 0. lim R→∞

(63)

Ωβj \B(0,R)

It follows from the Poincar´e inequality that uniformly for j ≥ 1,  lim 

s→0+

{x∈Ωβj | dist(x,∂Ωβj )≤s}

{x∈Ωβj | dist(x,∂Ωβj )≤s}

α ,βj 2

((v1 j

α ,βj 2 |

|∇v1 j

α ,βj 2

) + (v2 j

) ) dy

α ,βj 2 |

+ |∇v2 j

= 0. dy

Thus it follows from (i) of Proposition 17 that uniformly for j ≥ 1,  α ,β α ,β ((v1 j j )2 + (v2 j j )2 ) dy = 0. lim s→0+

{x∈Ωβj | dist(x,∂Ωβj )≤s}

37

(64)

Then, combining (63) and 64, we get that uniformly for j ≥ 1,  α ,β α ,β lim (v1 j j )2 + (v2 j j )2 dy = 0. R→∞

Ωβj \B(0,R)

Then applying the elliptic estimate [12, Theorem 9.26], we see that lim

|y−yj |→∞

α ,βj

v1 j

α ,βj

(y) + v2 j

(y) = 0.

This completes the proof.  Proposition 22. There exist constants C, c > 0 such that for each j ≥ 1 and i = 1, 2, α ,βj

vi j

(y) ≤ C exp(−c|y − yj |), y ∈ Ωβj .

Proof. We note that for any small δ > 0 and x ∈ (Ω)δ ≡ {x ∈ Ω | dist(x, ∂Ω) < δ}, there exists a unique z = z(x) ∈ ∂Ω with s(x) ≡ |x − z| = minz ∈∂Ω |x − z  | such that the functions s and z are C 2 . We define (Ω)δ,βj ≡ {y ∈ Ωβj | dist(y, ∂Ωβj ) < δ(βj )1/4 }. Then, for any y ∈ (Ω)δ,βj , it holds that tj (y) ≡  min |y − y  | = (βj )1/4 s(y/(βj )1/4 ) = dist(y, ∂Ωβj ). y ∈∂Ωβj

For any l > 0, we define  1 1 1 (Ω)lδ,βj ≡ {y ∈ (Ω)δ,βj  βj 4 dist(z(βj − 4 y), βj − 4 yj ) > l}. We can find a function ψj ∈ C 2 (∂Ωβj \ {yj }) ∩ C(∂Ωβj ) such that there exists a constant b1 ∈ (0, 1) satisfying b1 dist(y, yj ) ≤ ψj (y) ≤ (b1 )−1 dist(y, yj )

for any j = 1, 2, · · · and y ∈ Ωβj ,

and that for any l > 0, there exists b2 > 0 satisfying |∇ψj (x)| + |∇2 ψj (y)| ≤ b2

for any j = 1, 2, · · · and dist(y, yj ) ≥ l.

Now for γ > 0, which will be determined later, we define 1

−1

Ψj (y) ≡ exp(−γ(tj (y))2 − γ 2 ψj (βj4 z(βj 4 y))), 38

y ∈ (Ω)lδ,βj .

Denoting Ωβj (tj ) ≡ {y ∈ Ωβj | tj (y) = tj }, we see that Δy =

∂2 ∂ − H(tj ; y) + ΔΩβj (tj ) , ∂tj ∂tj 2

where H(t; y) is the mean curvature of Ωβj (tj ) at y ∈ Ωβj (tj ). Then it follows that ΔΨj (y) ≡ (−2γ + 4γ 2 (tj )2 + 2γtj H(tj ; y))Ψj + ΔΩβj (tj ) Ψj

in

(Ω)lδ,βj .

It is easy to see that for some a1 > 0, independent of j ≥ 1 and γ > 0, 1

|H(tj ; y)| ≤ a1 (βj )− 4 , α ,βj

Since v3 j

ΔΩβj (tj ) Ψj ≤ a1 γ 4 Ψj

in

(Ω)lδ,βj .

(65)

→ U3 in C 1 (Ω), there exists a constant a2 > 0, independent of l,

such that 1

α ,βj

β13 βj2 (v3 j

−1

(βj 4 y))2 ≥ a2 (d(y, Ωβj ))2 ,

y ∈ Ωβj .

(66)

Then, it follows from (65) and (66) that for y ∈ (Ω)lδ,βj ,   λ 1 β11 αj ,βj 2 −1 α ,β α ,β 1 (v1 ) − β12 (v2 j j )2 Ψj −ΔΨj +  + β13 βj2 (v3 j j (βj 4 y))2 − αj βj   β11 αj ,βj 2 α ,β ≥ a2 (d(y, Ωβj ))2 − (v1 ) − β12 (v2 j j )2 Ψj αj   1 + 2γ − 4γ 2 (tj )2 − 2γa1 (βj )− 4 tj − a1 γ 4 Ψj . 260

1

1

Since tj (y) = d(y, ∂Ωβj ) and 2γa1 (βj )− 4 tj ≤ (a1 )2 (βj )− 2 + γ 2 (tj )2 , we see that for y ∈ (Ω)lδ,βj ,  λ  1 β11 αj ,βj 2 −1 α ,β α ,β 1 −ΔΨj +  + β13 βj2 (v3 j j (βj 4 y))2 − (v1 ) − β12 (v2 j j )2 Ψj αj βj   2 (a1 ) β11 αj ,βj 2 αj ,βj 2 4 ≥ (a2 − 5γ 2 )(d(y, Ωβj ))2 + 2γ − − a γ − (v ) − β (v ) Ψj . 1 12 1 2 αj 1 (βj ) 2 We take a small γ > 0 such that a2 − 5γ 2 ≥ a2 /2, 2γ − a1 γ 4 ≥ γ,

2γ ≤ c, α ,βj

where c > 0 is the number given in Proposition 19. Then, since vi j

(y) → 0

uniformly for i = 1, 2 and j = 1, 2, · · · as |y − yj | → ∞, there exist a large l > 0 39

and j0 > 0 such that for j ≥ j0 ,  λ  1 y β11 αj ,βj 2 α ,β α ,β 1 (v1 ) −β12 (v2 j j )2 Ψj > 0 in (Ω)lδ,βj . −ΔΨj +  +β13 βj2 (v3 j j ( 1 ))2 − αj βj β4 j

We claim that for some E > 0, α ,βj

v1 j

(y) ≤ EΨj (y)

for y ∈ ∂(Ω)lδ,βj . 1

In fact, we see that if tj (y) = dist(y, ∂Ωβj ) = δβj4 and j > 1 is large, Ψj (y)

=

1

−1

exp(−γ(tj (y))2 − γ 2 ψj (βj4 z(βj 4 y)))

≥ 1

γb2

1

1

≥ exp(−γδ 2 (βj ) 2 − γ 2 βj4 (b2 )−1 ) = exp(−(1 + 1 2

(βj )

1

1 4

exp(−2γδ (βj ) ) ≥ exp(−c(tj (y)) ). 2

1

2

)γδ 2 (βj ) 2 ) (67)

1

If βj 4 dist(z(βj − 4 y), βj − 4 yj ) = l for y ∈ (Ω)δ,βj , Ψj (y) = exp(−γ(tj (y))2 ) exp(−γ 2 l).

(68)

Since v1 (y) = 0 for tj (y) = 0, that is, y ∈ ∂Ωβj , we see from (67) and (68) that for some large E > 0, α ,βj

v1 j

(y) ≤ EΨj (y)

for y ∈ ∂(Ω)lδ,βj .

Since for y ∈ (Ω)lδ,βj ,  λ  1 β11 αj ,βj 2 −1 α ,β α ,β α ,β α ,β 1 −Δv1 j j +  +β13 βj2 (v3 j j (βj 4 y))2 − (v1 ) −β12 (v2 j j )2 v1 j j = 0, αj βj it follows from the comparison principle that α ,βj

v1 j

(y) ≤ EΨj (y)

for y ∈ (Ω)lδ,βj .

Then, since ψj (y) ≥ b1 dist(y, yj ) for y ∈ ∂Ω, for some A, a > 0, we get the estimate α ,βj

v1 j

(y) ≤ A exp(−a|y − yj |), y ∈ Ωβj . α ,βj

By the same argument for v2 j

, we also get the same decay estimate. Thus,

for some A, a > 0, independent of j ≥ 1, α ,βj

v1 j

α ,βj

(y), v2 j

(y) ≤ A exp(−a|y − yj |), y ∈ Ωβj .

This completes the proof.  40

Proposition 23. There exist a point p ∈ ∂Ω and a least energy solution (U1 , U2 ) ∈ H of (4) such that for A ∈ O(n) with A(0, · · · , 0, 1) = ν, the inward normal unit vector at p ∈ Ω, α ,βj

(v1 j

α ,βj

(A · +yj ), v2 j

(A · +yj )) → (U1 , U2 ) strongly in H 1 (Rn )2 ∩ (L∞ (Rn ))2 ,

−1

βj 4 yj → p ∈ ∂Ω with M (p, U3 ) = M as j → ∞. Proof. By the exponential decay result of Proposition 22, we see that  α ,β α ,β lim (v1 j j )2 + (v2 j j )2 dy = 0 uniformly for j = 1, 2, · · · .  l→∞ {y∈Ω  |y−yj |≥l} βj (69) 265

For each l > 0 and j ≥ 1, we take a nonnegative function χlj ∈ C01 (Rn ) such that χlj (y) = 1 for |y −yj | ≤ l, χlj (y) = 0 for |y −yj | ≥ 2l, χlj (y) ≤ 1, |∇χlj | ≤ 2/l α ,βj

for y ∈ Rn . Multiplying v1 j

(χlj )2 to the first equation of (31) and integrating

by parts, we see that  α ,β |∇v1 j j χlj |2 dx Ωβj



α ,βj 2

≤ Ωβj

) |∇χlj |2 +

(v1 j

β11 αj ,βj 4 l 2 α ,β α ,β (v ) (χj ) + β12 (v1 j j )2 (v2 j j )2 (χlj )2 dx. αj 1

Combining this with (69), we see that liml→∞

 {y∈Ωβj

 

|y−yj |≥l}

α ,βj 2

|∇v1 j

| dx =

0 uniformly for j = 1, 2, · · · . By the same argument, we see the same decay for α ,βj

v2 j

. Therefore,  lim

l→∞

{y∈Ωβj

 

α ,βj 2

|y−yj |≥l}

|∇vi j

α ,βj 2

| + (vi j

) dx = 0

uniformly for i = 1, 2 and j = 1, 2, · · · . α ,βj

Since v3 j

→ U3 in C 1 (Ω) as j → ∞, for each l > 0, sup

sup

j

{y∈Ωβj | |y−yj |≤l}

1

α ,βj

βj2 (v3 j

−1

(βj 4 y))2 < ∞.

Then, we see from the first and second equations of (31) that for each l ≥ 0, there α ,βj

exists Kl > 0, independent of i = 1, 2 and j = 1, 2, · · · , such that |Δvi j 41

(y)| ≤

Kl for |y − yj | ≤ l. Then, applying the W 2,p estimates [12, Theorem 9.11], [12, Theorem 9.13] and the embedding W 2,p → C 1,γ for large p > 1, we conclude α ,β  that for for some γ ∈ (0, 1), {vi j j  1,γ }i,j is bounded. C ({y∈Ωβ  |y−yj |≤l}) j

Then, taking a subsequence of j → ∞, if it is necessary, we deduce that as j → ∞, α ,βj

(v1 j

α ,βj

(A · +yj ), v2 j

(A · +yj )) → (U1 , U2 ) strongly in H 1 (Rn )2 ∩ (L∞ (Rn ))2 ,

−1

βj 4 yj → p ∈ ∂Ω, where (U1 , U2 ) ∈ H satisfies ⎧ 3 (p) 2 2 2 ⎪ ⎪ ΔU1 − β13 ( ∂U ⎪ ∂xn ) yn U1 + β12 U1 (U2 ) = 0, U1 ≥ 0 ⎨ 3 (p) 2 2 2 ΔU2 − β23 ( ∂U ∂xn ) yn U2 + β21 (U1 ) U2 = 0, U2 ≥ 0 ⎪ ⎪ ⎪ ⎩ U1 = U 2 = 0

in Rn+ , in Rn+ ,

(70)

on ∂Rn+

By (ii) of Proposition 17, we see that U1 + U2 > 0 in Rn+ . If U1 ≡ 0 or U2 ≡ 0 in Rn+ , we see from equation (70) that U1 = U2 = 0; this contradicts U1 + U2 > 0 in Rn+ . Thus by the maximum principle, we see that U1 > 0, U2 > 0 in Rn+ . α ,βj

Now, since u3 j

→ U3 in C 1 (Ω) as j → ∞, we deduce from Propositions 21

and 22 that 2β12 |U1 U2 |22,Rn+ = |∇U1 |22,Rn+ + |∇U2 |22,Rn+ +

 Rn +

β13 (

∂U3 (0) 2 2 ∂U3 (0) 2 2 ) yn (U1 )2 + β23 ( ) yn (U2 )2 dy ∂xn ∂xn

= 4M. 270

Thus, we get M (p, U3 ) = M. This completes the proof.  Completion of the proof of Theorem 1. We have proved (a) of Theorem 1 in Propositions 9, 10, 12. The first claim (i) of (b) follows from Proposition 23 since for any α#β 1+δ → ∞, there exists a subsequence {αj #βj1+δ }j such that αj #βj1+δ → ∞ as j → ∞,

275

and that Proposition (23) holds. α,β  To show (ii) of (b), we take xα,β ∈ Ω satisfying uα,β 1 (xα,β ) = maxx∈Ω u1 (x).

From Proposition 18 and Proposition 22, we see that limα#β 1+δ →∞ dist(xα,β , ∂Ω) = 42

0. Then, there exists a unique point xα,β ∈ ∂Ω satisfying |xα,β − xα,β | = min dist(xα,β , x). x∈∂Ω

1 4

1 4

We define yα,β = β xα,β ∈ ∂(β Ω). Then, we claim as in Proposition 21 that lim|y−yα,β |→∞ viα,β (y) = 0 uniformly for i = 1, 2 and large α#β 1+β > 0. If this is 1

not true, from Proposition 21, there exist a sequence αj #βj1+δ → ∞, zj ∈ βj4 Ω 1

α ,βj

with limj→∞ |zj −yαj ,βj | = ∞ and yj ∈ ∂(βj4 Ω) such that limj→∞ vi j 280

for some i ∈ {1, 2} and

α ,β lim|x−yj |→∞ vi j j (y)

(zj ) > 0

= 0 uniformly for i = 1, 2, j ≥ 1.

This implies that {|zj − yj |}j is bounded; thus limj→∞ |yj − yαj ,βj | = ∞. This α ,βj

implies that limj→∞ |vi j

|

proves the claim.

1

∞,βj4 Ω

= 0; this contradicts Proposition 18 and

Now by the same argument with the proof of Proposition 22, we get that for large α#β 1+δ > 0 and i = 1, 2, viα,β (y) ≤ C exp(−c|y − yα,β |), y ∈ Ωβ . This proves (ii) of (b). 285

Proposition 23 implies that for any α#β 1+δ → ∞, there exists a subsequence α ,βj

αj #βj1+δ → ∞ such that xαj ,βj → p and u3 j

→ U3 for some (p, U3 ) ∈ M as

j → ∞. This implies (iii) of (b). The last claim (c) follows from Proposition 23. This completes the proof of Theorem 1.  290

Acknowledgments. The first author was supported by Mid-career Researcher Program through the National Research Foundation of Korea funded by the Ministry of Science, ICT and Future Planning (NRF-2013R1A2A2A05006371), and the third author by NSFC-11271201 and BCMIIS. References

295

[1] A. Ambrosetti, E. Colorado,

Standing waves of some coupled nonlinear Schr¨ odinger

equations. J. Lond. Math. Soc. 75 (2007), 67–82. [2] A. Ambrosetti, E. Colorado, D. Ruiz, Multi-bump solitons to linearly coupled systems of nonlinear Schr¨ odinger equations. Calc. Var. Partial Differential Equations 30 (2007) p85–112.

43

300

[3] T. Bartsch, E. N. Dancer, Z.-Q. Wang, A Liouville theorem, a-priori bounds, and bifurcating branches of positive solutions for a nonlinear elliptic system. Calc. Var. Partial Differential Equations 37 (2010) 345–361. [4] T. Bartsch, Z.-Q. Wang, Note on ground states of nonlinear Schr¨ odinger systems. J. Part. Diff. Equ. 19 (2006), 200–207.

305

[5] T. Bartsch, Z.-Q. Wang, J. Wei, Bound states for a coupled Schr¨ odinger system. J. Fixed Point Theory Appl. 2 (2007) 353–367. [6] J. Byeon, Existence of large positive solutions of some nonlinear elliptic equations on singularly perturbed domains, Comm. in P. D. E. 22 (1997), 1731 – 1769. [7] J. Byeon, Semi-classical standing waves for nonlinear Schr¨ odinger systems, Calc. Var.

310

Partial Differential Equations 54 (2015), 2287 - 2340. [8] S. Chang, C.S. Lin, T.C. Lin, W. Lin, Segregated nodal domains of two-dimensional multispecies Bose-Einstein condensates. Phys. D, 196 (2004), 341–361. [9] M. Conti, S. Terracini, G. Verzini, Nehari’s problem and competing species system. Ann. Inst. H. Poincar´e Anal. Non Lin´ eaire 19 (2002), 871–888.

315

[10] E. N. Dancer, J. Wei, T. Weth, A priori bounds versus multiple existence of positive solutions for a nonlinear Schr¨ odinger system. Ann. Inst. H. Poincar´e Anal. Non Lin´ eaire 27 (2010), 953–969. [11] B. Gidas, W.-M. Ni, L. Nirenberg, Symmetry and related properties via the maximum principle. Comm. Math. Phys. 68 (1979) no. 3, 209–243.

320

[12] D. Gilbarg, N. S. Trudinger, Elliptic partial differential equations of second order. Second edition. 224. Springer-Verlag, Berlin, (1983). [13] N. Ikoma, K. Tanaka, A local mountain pass type result for a system of nonlinear Schr¨ odinger equations. Calc. Var. Partial Differential Equations 40 (2011) 449–480 [14] T.-C. Lin, J. Wei, Ground state of N coupled nonlinear Schr¨ odinger equations in Rn , n ≤

325

3. Comm. Math. Phys. 255 (2005), 629–653. [15] T.-C. Lin, J. Wei, Spikes in two coupled nonlinear Schr¨ odinger equations. Ann. Inst. H. Poincar´e Anal. Non Lin´ eaire 22 (2005), 403–439. [16] P. L. Lions, The concentration-compactness principle in the calculus of variations. The locally compact case, part II. Ann. Inst. H. Poincar´e Anal. Non Lin´ eaire 1 (1984), 223–

330

283.

44

[17] J. Liu, X. Liu, Z.-Q. Wang, Multiple mixed states of nodal solutions for nonlinear Schr¨ odinger systems. Calc. Var. Partial Differential Equations 52 (2015), 565 - 586. [18] Z. L. Liu, Z.-Q. Wang, Multiple bound states of nonlinear Schr¨ odinger systems. Comm. Math. Phys. 282 (2008) 721 - 731. 335

[19] Z. L. Liu, Z.-Q. Wang, Ground states and bound states of a nonlinear Schr¨ odinger system. Advanced Nonlinear Studies 10 (2010) 175–193 [20] L.A. Maia, E. Montefusco, B. Pellacci, Positive solutions for a weakly coupled nonlinear Schr¨ odinger system. J. Diff. Equ. 299 (2006), 743–767. [21] M. Mitchell, M. Segev, Self-trapping of inconherent white light. Nature, 387 (1997),

340

880–882. [22] E. Montefusco, B. Pellacci, M. Squassina, Semiclassical states for weakly coupled nonlinear Schr¨ odinger systems. J. Eur. Math. Soc. 10 (2008), 41–71. [23] B. Noris, M. Ramos, Existence and bounds of positive solutions for a nonlinear Schr¨ odinger system. Proceedings of the AMS 138 (2010) 1681–1692.

345

[24] B. Noris, H. Tavares, S. Terracini, G. Verzini,

Uniform H¨ older bounds for nonlinear

Schr¨ odinger systems with strong competition. Comm. Pure and Appl. Math. 63 (2010), 267–302. [25] S. Peng, Z.-Q. Wang, Segregated and Synchronized Vector Solutions for Nonlinear Schr¨ odinger Systems, Arch. Ration. Mech. Anal. 208 (2013), 305–339. 350

[26] Ch. R¨ uegg et al, Bose-Einstein condensation of the triple states in the magnetic insulator TlCuCl3 . Nature, 423 (2003), 62–65. [27] Y. Sato, Multi-peak solutions for nonlinear Schr¨ odinger equations. Thesis, Waseda University, (2007). [28] Y. Sato, Z.-Q. Wang, On the multiple existence of semi-positive solutions for a nonlinear

355

Schr¨ odinger system. Ann. Inst. H. Poincar´ e Anal. Non Lin´ eaire 30 (2013), 1–22. [29] Y. Sato, Z.-Q. Wang, On the least energy sign-changing solutions for a nonlinear elliptic system. Discrete Contin. Dyn. Syst. 35 (2015), 2151–2164. [30] Y. Sato, Z.-Q. Wang, Least energy solutions for nonlinear Schr¨ odinger systems with mixed attractive and repulsive couplings. Advanced Nonlinear Studies 15 (2015), 1–22

360

[31] Y. Sato, Z.-Q. Wang, Multiple positive solutions for Schr¨ odinger systems with mixed couplings. Calc. Var. Partial Differential Equations 54 (2015), 1373 - 1392.

45

[32] B. Sirakov, Least energy solitary waves for a system of nonlinear Schr¨ odinger equations in Rn . Comm. Math. Phys. 271 (2007), 199–221. [33] N. Soave, On existence and phase separation of solitary waves for nonlinear Schr¨ odinger 365

systems modelling simultaneous cooperation and competition. Calc. Var. Partial Differential Equations 53 (2015), 689 - 718. [34] N. Soave and H. Tavares New existence and symmetry results for least energy positive solutions of Schr¨ odinger systems with mixed competition and cooperation terms. arXiv:1412.4336.

370

[35] H. Tavares and S. Terracini, Sign-changing solutions of competition-diffusion elliptic systems and optimal partition problems. Ann. Inst. H. Poincar´e Anal. Non Lin´eaire 29 (2012), 279–300. [36] H. Tavares, S. Terracini, G. Verzini, T. Weth, Existence and nonexistence of entire solutions for non-cooperative cubic elliptic systems. Comm. Partial Differential Equations 36

375

(2011), 1988–2010. [37] S. Terracini, G. Verzini, Multipulse Phase in k-mixtures of Bose-Einstein condensates. Arch. Rat. Mech. Anal. 194 (2009), 717–741. [38] R. Tian, Z.-Q. Wang, Multiple solitary wave solutions of nonlinear Schr¨ odinger systems. Topo. Meth. Non. Anal. 37 (2011), 203–223.

380

[39] Z.-Q. Wang, M. Willem, Partial symmetry of vector solutions for elliptic systems. Journald’ Analyse Mathematique, 122 (2014), 69–85. [40] J. Wei, T. Weth, T. Nonradial symmetric bound states for a system of two coupled Schr¨ odinger equations. Rend. Lincei Mat. Appl. 18 (2007), 279–293. [41] J. Wei, T. Weth, T. Radial solutions and phase separation in a system of two coupled

385

Schr¨ odinger equations. Arch. Rat. Mech. Anal. 190 (2008), 83–106. [42] J. Wei, W. Yao, Uniqueness of positive solutions to some coupled nonlinear Schr¨ odinger equations. Commun. Pure Appl. Anal. 11 (2012), 1003-1011.

46