Fractional Schrödinger–Poisson–Kirchhoff type systems involving critical nonlinearities

Nonlinear Analysis 164 (2017) 1–26

Contents lists available at ScienceDirect

Nonlinear Analysis www.elsevier.com/locate/na

Fractional Schrödinger–Poisson–Kirchhoff type systems involving critical nonlinearities Mingqi Xiang*, Fuliang Wang College of Science, Civil Aviation University of China, Tianjin, 300300, PR China

article

info

Article history: Received 1 May 2017 Accepted 30 July 2017 Communicated by S. Carl

abstract The paper is concerned with existence, multiplicity and asymptotic behavior of nonnegative solutions for a fractional Schrödinger–Poisson–Kirchhoff type system. As a consequence, the results can be applied to the special case

( MSC 2010: 35R11 35A15 47G20 Keywords: Fractional Schrödinger–Poisson–Kirchhoff system Variational methods Critical nonlinearity

a + b∥u∥2

)[

∗

(−∆)s u + V (x)u + ϕk(x)|u|p−2 u = λh(x)|u|q−2 u + |u|2s −2 u in R3 ,

]

(−∆)t ϕ = k(x)|u|p in R3 ,

(∫

∫

∥u∥ = R3

R3

|u(x) − u(y)|2 |x − y|3+2s

∫

2

)1/2

V (x)|u| dx

dxdy +

,

R3

where a, b ≥ 0 are two numbers, with a + b > 0, 1 < p < 2∗s,t =

3+2t , 3−2s

λ > 0 is a

6 3+2t−p(3−2s)

(−∆)s

parameter, s, t ∈ (0, 1), is the fractional Laplacian, k ∈ L (R3 ) may change sign, V : R3 → [0, ∞) is a potential function, 2∗s = 6/(3 − 2s) is 2∗ s ∗

the critical Sobolev exponent, 1 < q < 2∗s and h ∈ L 2s −q (R3 ). First, when θ < p < 2∗s /2, 2p ≤ q < 2∗s and λ is large enough, existence of nonnegative solutions is obtained by the mountain pass theorem. Moreover, we obtain that limλ→∞ ∥uλ ∥ = 0. Then, via the Ekeland variational principle, existence of nonnegative solutions is investigated when θ < p < 2∗s /2, 1 < q < 2 and λ is small enough, and we obtain that limλ→0 ∥uλ ∥ = 0. Finally, we consider the system with double critical exponents, that is, p = 2∗s,t , and obtain two nontrivial and nonnegative solutions in which one is least energy solution and another is mountain pass solution. The paper covers a novel feature of Kirchhoff problems, which is the fact that the parameter a can be zero. Hence the results of the paper are new even for the standard stationary Schrödinger–Poisson–Kirchhoff system. © 2017 Published by Elsevier Ltd.

1. Introduction and main results In this paper, we study the following fractional Schr¨odinger–Poisson–Kirchhoff type system: { p−2 q−2 2∗ −2 M (∥u∥2 )[(−∆)s u + V (x)u] + ϕk(x)|u| u = λh(x)|u| u + |u| s u in R3 , p (−∆)t ϕ = k(x)|u| in R3 ,

*

Corresponding author. E-mail addresses: xiangmingqi [email protected], [email protected] (M. Xiang), [email protected] (F. Wang).

http://dx.doi.org/10.1016/j.na.2017.07.012 0362-546X/© 2017 Published by Elsevier Ltd.

(1.1)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

2

where ( ∥u∥ =

[u]2s

∫ +

)1/2 V (x)|u| dx , 2

(∫

∫

[u]s =

R3

R3

|u(x) − u(y)|

R3

3+2s

|x − y|

)1/2

2

dxdy

,

(1.2) 2∗ s ∗

3+2t , 1 < q < 2∗s , λ > 0, h ∈ L 2s −q (R3 ), s, t ∈ (0, 1), M : [0, ∞) → [0, ∞) is a Kirchhoff function, 1 < p < 3−2s 6 , V : R3 → [0, ∞) is a continuous function, 2∗s = 6/(3 − 2s) is the critical k ∈ Lν (R3 ), ν = 3+2t−p(3−2s) fractional Sobolev exponent and (−∆)s is the fractional Laplace operator which, up to a normalization constant, is defined as ∫ φ(x) − φ(y) s x ∈ R3 , (−∆) φ(x) = 2 lim 3+2s dy, + ε→0 R3 \Bε (x) |x − y|

along functions φ ∈ C0∞ (R3 ). Henceforward Bε (x) denotes the ball of R3 centered at x ∈ R3 and radius ε > 0. For details on fractional Laplace operator we refer the readers to [21] and the references therein. A typical example of M is given by M (t) = a + b tθ−1 for t ≥ 0, where a, b ≥ 0 and a + b > 0, if θ > 1, and M (t) = a > 0 if θ = 1. For θ > 1, when M is of this type, problem (1.1) is said to be non-degenerate if a > 0, while it is called degenerate if a = 0. When M ≡ 1 and k ≡ 0, system (1.1) reduces to the following fractional Schr¨odinger equation (−∆)s u + V (x)u = λh(x)|u|

q−2

2∗ s −2

u + |u|

u in R3 ,

(1.3)

which is a fundamental equation in fractional quantum mechanics in the study of particles on stochastic fields modeled by L´evy processes [29,30]. See also [20] for a detailed mathematical description of fractional Schr¨ odinger equation. In recent years, nonlocal operators and related equations have been receiving a great attention. Actually, nonlocal operators can be seen as the infinitesimal generators of L´evy stable diffusion processes [3]. Moreover, they allow us to develop a generalization of quantum mechanics and also to describe the motion of a chain or an array of particles that are connected by elastic springs as well as unusual diffusion processes in turbulent fluid motions and material transports in fractured media, for more details see [3,13,14] and the references therein. Indeed, the literature on nonlocal fractional operators and on their applications is quite large, see for example the recent monograph [43], the extensive paper [22] and the references cited therein. When s, t → 1, M ≡ 1, K ≡ 1 and p = 2, system (1.1) becomes the Schr¨odinger–Poisson type system { −∆u + V (x)u + ϕu = f (x, u) in R3 , (1.4) −∆ϕ = u2 in R3 , which has been first introduced by Benci and Fortunato [8] as a physical model describing solitary waves for nonlinear Schr¨ odinger type equations interacting with an unknown electrostatic field. The first equation of (1.4) is coupled with a Poisson equation, which means that the potential is determined by the charge of the wave function. The term ϕu is nonlocal and concerns the interaction with the electric field. For more details about the physical background of the system (1.4), we refer the readers to [9,35,40] and the references cited there. In the last decades, many papers have been devoted to the existence and multiplicity of solutions for system like (1.4) via variational methods and critical point theory under various assumptions on the potential V and the nonlinearity, see for example [26,27,31,38,51]. In [36], Liu considered the following generalized Schr¨ odinger–Poisson system { 3 −∆u + V (x)u − k(x)ϕ|u| u = f (x, u) in R3 , (1.5) 5 3 −∆ϕ = k(x)|u| in R ,

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

3

where V, k, f are continuous functions. By the reduction method proposed by Benci and Fortunato in [8], system (1.5) reduces to the well known Choquard type equation ( ) 5 3 − ∆u + V (x)u − Iµ ∗ (k|u| ) k|u| u = f (x, u) in R3 , (1.6) which was elaborated by Pekar [45] in the framework of quantum mechanics. Subsequently, it was adopted as an approximation of the Hartree–Fock theory, see [10]. Recently, Penrose [46] settled it as a model of self-gravitational collapse of a quantum mechanical wave function. Here Iµ : R3 \ {0} → R denotes the Riesz potential. The first investigations for existence and symmetry of the solutions to (1.6) go back to the works of Lieb [32] and Lions [34]. Equations of type (1.6) have been extensively studied, see e.g. [1,6,7,44,58]. For the critical case in the sense of Hardy–Littlewood–Sobolev inequality, we refer the interested reader to [25] for recent existence results in a smooth bounded domain of RN . In the setting of the fractional Laplacian, Wu in [55] investigated existence and stability of solutions for the equation q

q−2

(−∆)s u + ωu = (Iµ ∗ |u| )|u|

u + λf (x, u)

in RN ,

(1.7)

where q = 2, λ = 0 and µ ∈ (N − 2s, N ). Subsequently, D’Avenia and Squassina in [19] studied (1.7) with λ = 0, such as existence, nonexistence, and regularity, decays properties of solutions. In particular, they claimed the nonexistence of solutions as q ∈ (2 − µ/N, 2∗µ,s ). The existence of ground states of fractional Choquard equations like (1.7) was investigated by [52] and [16]. On the other hand, Zhao, Zhu and Li in [60] studied the following Kirchhoff–Schr¨odinger–Poisson system ⎧[ ] ∫ ⎨ 2 2 a+b (|∇u| + V (x)|u| dx) (−∆u + V (x)u) + λl(x)ϕu = f (x, u), x ∈ R3 , (1.8) 3 R ⎩ −∆ϕ = λl(x)u2 , x ∈ R3 , where constants a > 0, b ≥ 0 and λ ≥ 0. The authors obtained infinitely many solutions of (1.8) by using the symmetric mountain pass theorem. L¨ u in [39] studied the following Kirchhoff-type equation ( ) ∫ 2 q−2 − a+b |∇u| dx ∆u + Vλ (x)u = (Iµ ∗ uq )|u| u in R3 , (1.9) R3

where a > 0, b ≥ 0 are given numbers, Vλ (x) = 1 + λg(x), λ ∈ R+ is a parameter and g(x) is a continuous potential function on R3 , q ∈ (2, 6 − µ). By using the Nehari manifold and the concentration compactness principle, L¨ u obtained the existence of ground state solutions for (1.9) if the parameter λ is large enough. Actually, the study of Kirchhoff-type problems, which arise in various models of physical and biological systems, have received more and more attention in recent years. More precisely, Kirchhoff [28] established a model given by ( ∫ L ⏐ ⏐2 ) 2 ⏐ ∂u ⏐ ∂2u p0 E ⏐ ⏐ dx ∂ u = 0, ρ 2 − + (1.10) ∂t h 2L 0 ⏐ ∂x ⏐ ∂x2 where ρ, p0 , h, E, L are constants which represent some physical meanings respectively. (1.10) extends the classical D’Alembert wave equation by considering the effects of the changes in the length of the strings during the vibrations. Recently, Fiscella and Valdinoci in [24] first proposed a stationary Kirchhoff model involving the fractional Laplacian by taking into account the nonlocal aspect of the tension arising from nonlocal measurements of the fractional length of the string, see [24, Appendix A] for more details. Since then many papers have been dedicated to investigating existence of solutions for the fractional Kirchhoff problems, see [5,15,42,50] and the references therein for the degenerate case of Kirchhoff-type problems. We also collect some recent existence results for fractional non-degenerate Kirchhoff problems, see [47,49,56,57].

4

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

To the best of our knowledge, there are few results which studied the system (1.1) in the literature. In [59], Zhang and Squassina considered the following fractional Schr¨odinger–Poisson system { (−∆)s u + λϕu = g(u) in R3 , (−∆)t ϕ = λu2 in R3 , where λ > 0 and g satisfies subcritical or critical growth conditions. By using a perturbation approach, the authors obtained the existence of positive solutions for small λ and studied the asymptotic of solutions for λ → 0+ . In [53], Teng studied the following fractional Schr¨odinger–Poisson system { q−1 2∗ −2 (−∆)s u + V (x)u + ϕu = µ|u| u + |u| s u in R3 , (1.11) (−∆)t ϕ = u2 in R3 , where µ > 0 is a parameter, 1 < q < 2∗s − 1 and 4s + 2t > 3. When λ is large enough, the existence of a nontrivial ground state solution was obtained in [53] by using the method of Pohozaev–Nehari manifold and the arguments of Brezis–Nirenberg, the monotonic trick and global compactness lemma. Motivated by the above works, we investigate the existence and asymptotic behavior of solutions of (1.1) and overcome the lack of compactness due to the presence of critical terms as well as the possibly degenerate nature of the Kirchhoff coefficient. To the best of our knowledge, there are no results for system (1.1). Throughout the paper, we assume without further mention that (V1 ) V : R3 → [0, ∞) is a continuous function satisfying inf x∈R3 V (x) ≥ 0. 2∗ s ∗

(H1 ) h : R3 → [0, ∞), h ̸≡ 0 and h ∈ L 2s −q (R3 ) with 1 < q < 2∗s . (H2 ) There exists a non-empty open set Ω ⊂ R3 such that inf x∈Ω h(x) > 0. 6 (K1 ) k ∈ Lν (R3 ), where ν = 3+2t−p(3−2s) . p

q

Remark 1.1. From assumptions (H1 ) and (K1 ), we can deduce that the functions ϕk|u| and h|u| 6 6 must belong to L1 provided u ∈ L 3−2s , ϕ ∈ L 3−2t , so the definition of weak solution makes sense (see Definition 2.1). For the function M , we assume that M : [0, ∞) → [0, ∞) is a continuous function verifying (M1 ) For any τ > 0 there exists κ = κ(τ ) > 0 such that M (ξ) ≥ κ for all ξ ≥ τ . ∫ξ (M2 ) There exists θ ∈ [1, 2∗s /2) such that ξM (ξ) ≤ θM (ξ) for all ξ ≥ 0, where M (ξ) = 0 M (τ )dτ . (M3 ) There exists m0 > 0 such that M (ξ) ≥ m0 ξ θ−1 for all ξ ∈ [0, 1]. Remark 1.2. Clearly, assumptions (M1 )–(M3 ) cover the degenerate case and (M2 )–(M3 ) are automatic in the non-degenerate case. It is worth mentioning that the degenerate case is rather interesting and is treated in well-known papers in Kirchhoff theory, see for example [18,50]. In [50], condition (M3 ) was also applied to investigate the existence of entire solutions for the stationary Kirchhoff type equations driven by the fractional p-Laplacian operator in RN . In the large literature on degenerate Kirchhoff problems, the transverse oscillations of a stretched string, with nonlocal flexural rigidity, depend continuously on the Sobolev deflection norm of u via M (∥u∥2 ). From a physical point of view, the fact that M (0) = 0 means that the base tension of the string is zero, a very realistic model. Before stating our main results, we introduce some notations. Let { } ∫ 2 L2 (R3 , V ) = u : R3 → R is measurable : V (x)|u| dx < ∞ , R3

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

5

endowed with the norm (∫ ∥u∥2,V =

2

)1/2

V (x)|u| dx

.

R3

The natural solution space for (1.1) is HVs,2 (R3 ), which is defined as { } HVs,2 (R3 ) = u ∈ L2 (R3 , V ) : [u]s < ∞ , endowed with the norm )1/2 ( 2 ∥u∥ = ∥u∥2,V + [u]2s , where [u]s is given by (1.2). When λ is large enough, we get our first result as follows. Theorem 1.1 (λ Large). Assume that V verifies (V1 ), M fulfills (M1 )–(M3 ), h satisfies (H1 ) with 2θ < q < 2∗s , and k satisfies (K1 ) with θ < p < 2∗s /2. Then there exists λ∗ ≥ 0 such that for all λ > λ∗ problem (1.1) admits a nontrivial nonnegative mountain pass solution uλ in HVs,2 (R3 ). Moreover, lim ∥uλ ∥ = 0.

λ→∞

(1.12)

Remark 1.3. Note that the function k in (1.1) can change sign. Thus, we need the condition θ < p. If k is nonnegative, then the condition θ < p is not necessary and we can replace it with more general condition 1 < p. Let us simply describe the main approach to obtain Theorem 1.1. To show the existence of at least one critical point of the energy functional, we shall use the mountain pass theorem of Ambrosetti and Rabinowitz [2]. However, since system (1.1) contains a critical nonlinearity, it is difficult to get the global Palais–Smale condition. To overcome the lack of compactness, we follow some techniques from [5], where a critical Kirchhoff problem involving the fractional Laplacian has been studied. See also [23] for a critical fractional p-Kirchhoff problem. We first show that the energy functional associated with system (1.1) satisfies the Palais–Smale condition at suitable levels cλ . In this process, the key point is to study the asymptotical behavior of cλ as λ → ∞ (see Lemma 3.3). Then by the mountain pass theorem, we get the existence of nontrivial nonnegative solution and obtain the asymptotic of (1.12) by considering the asymptotic of cλ as λ → ∞. When λ is small enough, we obtain the following result. Theorem 1.2 (λ Small). Assume that V satisfies (V1 ), M fulfills (M1 )–(M3 ), h satisfies (H1 ) with 1 < q < 2 and (H2 ), and k satisfies (K1 ) with θ < p < 2∗s /2. Then there exists λ∗∗ > 0 such that for all λ ∈ (0, λ∗∗ ] problem (1.1) admits a nontrivial nonnegative solution uλ in HVs,2 (R3 ). Moreover, lim ∥uλ ∥ = 0.

λ→0+

The proof of Theorem 1.2 is mainly based on a minimization argument and takes inspiration from Theorem 1.4 of [23] and Theorem 1.2 of [48]. Finally, we study the following system with double critical exponents ( )[ ] 2∗ −2 q−2 2∗ −2 a + b∥u∥2 (−∆)s u + V (x)u − ϕ|u| s,t u = λh(x)|u| u + |u| s u in R3 , (−∆)t ϕ = |u|

2∗ s,t

in R3 ,

(1.13)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

6

3+2t where a, b ≥ 0 are two numbers, s, t ∈ (0, 1), λ > 0, 2∗s,t = 3−2s is the critical exponent in the sense of 3 Hardy–Littlewood–Sobolev inequality, h : R → [0, ∞) satisfies (H1 ), V : R3 → [0, ∞) is a potential function satisfying (V1 ). Our third result reads as follow.

Theorem 1.3. Let 2 < q < 2∗s and (H1 ) hold. Assume that s, t ∈ (0, 1) and 4s + 2t ≤ 3. If 4s + 2t = 3, −2 b > SH,L and 2∗ s

a>

(4 −

− ∗ 2∗s )S 4−2s

(

−2 2(b − SH,L )

2∗s − 2

2

)− 2∗s −2∗ 4−2s

,

(1.14)

or 4s + 2t < 3, a, b > 0 and a>

S−

2∗ s 2

∗

∗

2s,t )− 2s,t −1 ( )− 2∗s −2∗ ( − 4−2s 2−2∗ 4−22∗ (4 − 2∗s ) b b s,t ∗ + (2 − 2s,t ) SH,L s,t , 2 2∗s − 2 22∗s,t − 2

(1.15)

then there exists λ∗∗∗ > 0 such that system (1.13) at least has two nontrivial and nonnegative solutions in which one is least energy solution and another is mountain pass solution for all λ > λ∗∗∗ . In the proof Theorem 1.3, some tricks are taken from [37] and [54]. By the assumption 4s + 2t ≤ 3, we can deduce the fact that 2∗s,t ≤ 2. From this and the assumptions (1.14) and (1.15), we can prove the energy functional associated with system (1.13) satisfies the Palais–Smale condition, which will be crucial to proving Theorem 1.3. To the best of our knowledge, this paper is the first time to deal with fractional Schr¨ odinger–Poisson–Kirchhoff type systems with double critical exponents. Our result is new even in the Laplacian setting. However, multiplicity of solutions for the degenerate case of system (1.13) is still an open problem. The paper is organized as follows. In Section 2, we recall some necessary definitions and properties for the functional setting. In Section 3, we obtain the existence of nontrivial nonnegative solutions for system (1.1) by using the mountain pass theorem. In Section 4, we get the existence of nonnegative solutions for (1.1) by using the Ekeland variational principle. In Section 5, we obtain two nontrivial solutions for system (1.13) with double critical exponents by using variational methods, where a least energy solution and a mountain pass solution will be obtained. 2. Preliminaries We first provide some basic functional setting that will be used in the next sections. Let s ∈ (0, 1). The critical exponent 2∗s is defined as 6/(3−2s). We define the fractional Sobolev space Ds (R3 ) as the completion of C0∞ (R3 ) with respect to the norm (∫ ∫ )1/2 2 |u(x) − u(y)| [u]s = . 3+2s dxdy R3 R3 |x − y| ∗

The embeddings HVs,2 (R3 ) ↪→ Ds (R3 ) ↪→ L2s (R3 ) are continuous by [21, Theorem 6.7]. Set S=

inf

s,2

u∈HV (R3 )\{0}

[u]2s 2

∥u∥2∗s

.

Clearly, S > 0. By assumption (V1 ), HVs,2 (R3 ) can be equipped with the inner product ∫ ∫ ∫ (u(x) − u(y))(v(x) − v(y)) ⟨u, v⟩s = dxdy + V (x)uvdx. 3+2s |x − y| R3 R3 R3

(2.1)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

7

Let us now recall the Hardy–Littlewood–Sobolev inequality, see [33, Theorem 4.3]. Hereafter we denote by ∥ · ∥q the norm of Lebesgue space Lq (R3 ). Theorem 2.1. Assume that 1 < r1 , r2 < ∞, 0 < µ < 3 and 1 1 µ + + = 2. r1 r2 3 Then there exists C(µ, r1 , r2 ) > 0 such that ∫ ∫ |u(x)| |v(y)| µ dxdy ≤ C(µ, r1 , r2 )∥u∥r1 ∥v∥r2 |x − y| 3 3 R R for all u ∈ Lr1 (R3 ) and v ∈ Lr2 (R3 ). Note that, by the Hardy–Littlewood–Sobolev inequality, the integral ∫ ∫ q q |u(x)| |u(y)| dxdy µ |x − y| R3 R3 q

is finite, whenever |u| ∈ Lr (R3 ) for some r > 1 satisfying 2 µ 6 + = 2, that is r = . r 3 6−µ Hence, by the fractional Sobolev embedding theorem, if u ∈ H s (R3 ) this occurs provided that rq ∈ [2, 2∗s ]. Thus, q has to satisfy 6−µ 6−µ ˜ 2µ,s = ≤q≤ = 2∗µ,s . 3 3 − 2s Hence, ˜ 2µ,s is called the lower critical exponent and 2∗µ,s is said to be the upper critical exponent in the sense of the Hardy–Littlewood–Sobolev inequality. ˜ Let t ∈ (0, 1) and µ = 3 − 2t. By the Hardy–Littlewood–Sobolev inequality, there exists C(t) > 0 such that ∫ ∫ 2∗ 2∗ ∗ 22∗ |u(x)| s,t |u(y)| s,t s,t ˜ dxdy ≤ C(t)∥u∥ ≤ C(t)∥u∥22s,t , 2∗ 3−2t s 3 3 |x − y| R R ∗

−2s,t ˜ for all u ∈ HVs,2 (R3 ), where C(t) = C(t)S and 2∗s,t =

SH,L :=

3+2t 3−2s .

Let us define

∥u∥2

inf

( s,2 u∈HV (R3 )\{0} ∫

∗

R3

∫

)1/2∗s,t .

∗

2 2 |u(x)| s,t |u(y)| s,t 3 3−2t R |x−y|

dxdy

Clearly, SH,L > 0. Let 1 < p < 2∗s,t and fix u ∈ HVs,2 (R3 ), we define a linear functional Lu : Dt (R3 ) → R as ∫ p Lu (v) = k(x)|u| vdx, R3

t

3

for all v ∈ D (R ). By H¨ older’s inequality and the fractional Sobolev inequality, one has ∗

2∗ t p 2∗ −1 t

(∫ |Lu (v)| ≤

(k(x)|u| )

) 2t2−1 ∗ (∫ t

|v| dx

dx

R3

≤ ∥k(x)∥

R3 2∗ t −1 2∗ t

≤ C∥k(x)∥

p

∥u∥2∗s ∥v∥2∗t

6 (3+2t)−p(3−2s) ∗ 2t −1 2∗ t 6 (3+2t)−p(3−2s)

2∗ t

∥u∥p [v]t ,

)

1 2∗ t

(2.2)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

8

which implies that Lu is a continuous linear functional. By the Lax–Milgram theorem, the exists a unique ϕu ∈ Dt (R3 ) such that ∫ ∫ ∫ (ϕu (x) − ϕu (y))(v(x) − v(y)) p dx = k(x)|u| vdx, ∀v ∈ Dt (R3 ), 3+2t |x − y| R3 R3 R3 which means that ϕu is a weak solution of p

(−∆)t ϕu = k(x)|u| , x ∈ R3 . Moreover, the representation formula holds ∫ ϕu = Ct

k(y)|u(y)|

R3

|x − y|

p

3−2t

dy, ∀ x ∈ R3 ,

(2.3)

1 where Ct > 0 is a constant. Let Kt (x) = |x|3−2t for x ∈ R3 \ {0}. Then the convolution Kt ∗ (·) is called p t-Riesz potential and ϕu = Ct Kt ∗ (k|u| ). In the sequel, we omit the constant Ct for convenience in (2.3). Substituting ϕu in (1.1), it reduces to the following fractional Schr¨odinger–Kirchhoff type equation p

p−2

M (∥u∥2 )[(−∆)s u + V (x)u] + (Kt ∗ (k|u| )) k(x)|u|

q−2

u = λh(x)|u|

u + |u|

2∗ s −2

u,

(2.4)

for all x ∈ R3 . Definition 2.1. We say that (u, ϕ) ∈ HVs,2 (R3 ) × Dt (R3 ) is a (weak) solution of system (1.1), if u is a (weak) solution of Eq. (2.4), namely, ∫ ∫ ∫ p p−2 q−2 2∗ −2 M (∥u∥2 )⟨u, φ⟩s + (Kt ∗ (k|u| )) k(x)|u| uφdx = λ h(x)|u| uφdx + |u| s uφdx, R3

for all φ ∈

R3

R3

HVs,2 (R3 ).

To study the nonnegative solutions of Eq. (2.4), we define the functional Iλ : HVs,2 (R3 ) → R as ∫ ( 1 1 p ) p 2 Kt ∗ (k|u+ | ) k(x)|u+ | dx Iλ (u) = M (∥u∥ ) + 2 2p R3 ∫ ∫ λ 1 q 2∗ − h(x)|u+ | dx − ∗ |u+ | s dx, q R3 2s R3 where u+ = max{u, 0}. By H¨ older’s inequality, one can deduce that ∗

∫

(

p ) p Kt ∗ (k|u+ | ) k|u+ | dx ≤

R3

(∫

2∗ t −1 + p 2∗ t

(k|u | )

) 2t2−1 ∗ (∫ t

dx

R3

≤ C∥k(x)∥

R3 2∗ t −1 2∗ t 6 (3+2t)−p(3−2s)

) 1∗ ⏐ ⏐∗ 2 ⏐Kt ∗ (k|u+ |p )⏐2t dx t

[ p ] ∥u∥p Kt ∗ (k|u| ) t ,

which together with assumptions (H1 ) and (K1 ) implies that Iλ is well-defined and of class C 1 (HVs,2 (R3 ), R). Moreover, ∫ ( p ) p−1 ⟨Iλ′ (u), v⟩ = M (∥u∥2 )⟨u, v⟩s + Kt ∗ (k|u+ | ) k(x)|u+ | vdx N R ∫ ∫ 2∗ −1 + q−1 −λ h(x)|u | vdx − |u+ | s vdx R3

R3

( )∗ for all u, v ∈ HVs,2 (R3 ). From now on, ⟨·, ·⟩ denotes the duality pairing between HVs,2 (R3 ) and HVs,2 (R3 ). ( s,2 3 )∗ Here HV (R ) is the dual space of HVs,2 (R3 ).

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

9

Lemma 2.1. Assume that V verifies (V1 ), M fulfills (M1 ), h satisfies (H1 ) and k satisfies (K1 ). Then, for all λ > 0 any nontrivial critical point of functional Iλ is nonnegative. Proof . Fix λ > 0 and let uλ ∈ HVs,2 (R3 )\{0} be a critical point of functional Iλ . Clearly, u− λ = max{−u, 0} ∈ s,2 − 3 ′ HV (R ). Then ⟨Iλ (uλ ), −uλ ⟩ = 0, a.e. ∫ ( p ) + p−1 M (∥uλ ∥2 )⟨uλ , −u− ⟩ + Kt ∗ (k|u+ (−u− λ s λ | ) k(x)|uλ | λ )dx 3 R ∫ ∫ q−1 2∗ s −1 =λ h(x)|u+ (−u− |u+ (−u− λ| λ )dx + λ| λ )dx. R3

R3

Observe that for a.e. x, y ∈ R3 , − − − − − + + 2 (uλ (x) − uλ (y))(−u− λ (x) + uλ (y)) = uλ (x)uλ (y) + uλ (x)uλ (y) + [uλ (x) − uλ (y)] 2

− ≥ |u− λ − uλ (y)| ,

∫ R3

V uλ (−u− λ )dx =

∫ R3

2

V |u− λ | dx.

and ∫ R3

Moreover, h(x)|u+ λ|

q−1

(

p ) + p−1 Kt ∗ (k|u+ (−u− λ | ) k(x)|uλ | λ )dx = 0.

+ (−u− λ ) = 0 and |uλ |

2∗ s −1 − uλ

= 0 a.e. in R3 . Hence,

2 M (∥uλ ∥2 )∥u− λ ∥ ≤ 0. 3 This, together with ∥uλ ∥ > 0 and (M1 ), implies that u− λ ≡ 0. Hence uλ ≥ 0 a.e. in R . The proof is complete. □

3. Proof of Theorem 1.1 In this section, we always assume that V verifies (V1 ), M fulfills (M1 )–(M3 ), h satisfies (H1 ) with 2p ≤ q < 2∗s and k satisfies (K1 ) with θ < p < 2∗s /2. Let us recall that Iλ satisfies the (P S)c condition in HVs,2 (R3 ), if any (P S)c sequence {un }n ⊂ HVs,2 (R3 ), namely a sequence such that Iλ (un ) → c and Iλ′ (un ) → 0 as n → ∞, admits a strongly convergent subsequence in HVs,2 (R3 ). In the sequel, we shall make use of the following general mountain pass theorem (see [2,4]). Theorem 3.1. Let E be a real Banach space and J ∈ C 1 (E, R) with J(0) = 0. Suppose that (i) there exist ρ, α > 0 such that J(u) ≥ α for all u ∈ E, with ∥u∥E = ρ; (ii) there exists e ∈ E satisfying ∥e∥E > ρ such that J(e) < 0. Define Γ = {γ ∈ C 1 ([0, 1]; E) : γ(0) = 1, γ(1) = e}. Then c = inf max J(γ(t)) ≥ α γ∈Γ 0≤t≤1

and there exists a (P S)c sequence {un }n ⊂ E. To use Theorem 3.1, let us first show that the functional Iλ possesses the mountain pass geometry.

10

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

Lemma 3.1 (Mountain Pass Geometry I). For any parameter λ ∈ R+ , there exist α > 0 and ρ ∈ (0, 1) such that Iλ (u) ≥ α > 0 for any u ∈ HVs,2 (R3 ), with ∥u∥ = ρ. Proof . By (M2 ), one can deduce that M (ξ) ≥ M (1)ξ θ

for all ξ ∈ [0, 1].

(3.1)

Thus, by using (3.1), (2.2), H¨ older’s inequality and Theorem 2.1 with µ = 3 − 2t, we obtain for all u ∈ HVs,2 (R3 ), with ∥u∥ ≤ 1, −2∗ s

q

∗ S− 2 S 2 1 2ν ∥u∥2s , ∥h∥ 2∗s ∥u∥q − Iλ (u) ≥ M (1)∥u∥2θ − C∥k∥ν ∥u∥2p − λ ∗ 2 q 2 ∗ s 2s −q

where S is given by (2.1), ν =

6 3+2t−p(3−2s)

(3.2)

and C > 0. Choose ρ ∈ (0, 1] so small that ηλ (ρ) > 0, where q

2∗ s

S − 2 2∗s M (1) 2θ S− 2 2ν ηλ (τ ) = τ − C∥k∥ν τ 2p − λ ∥h∥ 2∗s τ q − τ , ∀τ ≥ 0. 2 q 2∗s 2∗ s −q This can be done since 2θ < q < 2∗s and 2θ < 2p < 2∗s . Thus, Iλ (u) ≥ α = ηλ (ρ) for all u ∈ HVs,2 (R3 ), with ∥u∥ = ρ. □ Remark 3.1. Note that if k ≥ 0 a.e. in R3 , the condition θ < p can be dropped and replaced with 1 < p. Lemma 3.2 (Mountain Pass Geometry II). There exists a nonnegative function e ∈ C0∞ (R3 ), independent of λ, such that Iλ (e) < 0 and ∥e∥ ≥ 2 for all λ ∈ R+ . Proof . It follows from (M2 ) that M (ξ) ≤ M (1)ξ θ

for all ξ ≥ 1. (3.3) ∫ 2∗ Let u0 ∈ C0∞ (R3 ), with u0 ≥ 0 a.e. in R3 such that ∥u0 ∥ = 1 and R3 u0s dx > 0. Then for all τ ≥ 1, we deduce from Theorem 2.1 with µ = 3 − 2t that ∫ ∗ ∫ τ 2p τ 2s 1 2∗ (Kt ∗ (kup0 )) kup0 dx − ∗ u0s dx Iλ (τ u0 ) ≤ M (1)τ 2θ ∥u0 ∥θp + 2 2p R3 2s R3 ∫ 2p 2∗ s 1 t τ 2∗ 2ν ≤ M (1)τ 2θ ∥u0 ∥2θ + C∥k∥ν ∥u0 ∥2p − ∗ u0s dx 2 2p 2s R3 ∫ ∗ ∗ 1 C 1 2 = M (1)τ 2θ + τ 2p − ∗ τ 2s u0s dx. (3.4) p 2p 2s R3 Hence, Iλ (τ u0 ) → −∞ as τ → ∞, since θ, p < 2∗s /2. The lemma is proved by taking e = T u0 , with T > 0 large enough such that ∥e∥ ≥ 2 and Iλ (e) < 0. □ Now we discuss the compactness property of the functional Iλ , given by the (P S)c condition at a suitable level. Fix λ > 0 and define cλ = inf max Iλ (γ(τ )), γ∈Γ τ ∈[0,1]

(3.5)

{ } where Γ = γ ∈ C([0, 1]; HVs,2 (R3 )) : γ(0) = 0, γ(1) = e . Obviously, cλ > 0 by Lemma 3.1. Moreover, the following asymptotic property holds true.

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

11

Lemma 3.3. lim cλ = 0,

λ→∞

where cλ is given by (3.5). Proof . For e given by Lemma 3.2, we have limτ →∞ Iλ (τ e) = −∞, then there exists τλ > 0 such that Iλ (τλ e) = maxτ ≥0 Iλ (τ e). Hence, by Iλ′ (τλ e) = 0, we have ∫ ∫ ∫ ∗ 2∗ −1 τλ M (∥τλ e∥2 )∥e∥2 + τλ2p−1 (Kt ∗ (kep )) kep dx = λτλq−1 heq dx + τλ s e2s dx. (3.6) R2

R3

R3

Let us first claim that {τλ }λ is bounded. From (5.6) we immediately get ∫ ∫ 2∗ 2p−1 2 2 p p s −1 τλ M (∥τλ e∥ )∥e∥ + τλ (Kt ∗ (ke )) ke dx ≥ τλ R3

∗

e2s dx,

R3

thanks to λ > 0 and h ≥ 0. Without loss of generality, we assume that τλ ≥ 1 for all λ > 0. Then, by (M2 ), (3.3) and ∥e∥ ≥ 2, we have ∫ ∫ ∗ 2∗ −1 θM (1)τλ2θ−1 + τλ2p−1 (Kt ∗ (kep )) kep dx ≥ τλ s e2s dx. R3

R3

The fact that θ, p < 2∗s /2 implies the claim. Fix any sequences {λn }n ⊂ R+ such that λn → ∞ as n → ∞. Obviously, {τλn }n is bounded. Hence there exist τ0 ≥ 0 and a subsequence, still denoted by {λn }n , such that τλn → τ0 as n → ∞. Thus there exists C > 0 such that ∫ (Kt ∗ (kep )) kep dx ≤ C ∀n. (3.7) τλn M (∥τλn e∥2 )∥e∥2 + τλ2p−1 n R3

We claim that τ0 = 0. Arguing by contradiction, we assume that τ0 > 0. Then we have ∫ ∫ ∗ 2∗ q s −1 e2s dx → ∞ as n → ∞, h(x)e dx + τ λn τλq−1 λn n R3

R3

which contradicts (5.6) and (3.7) and proves the claim. Consequently τ0 = 0. Thus we obtain τλ → 0 as λ → ∞. Put γ(τ ) = τ e. Clearly γ ∈ Γ . Since e independent of λ, by the continuity of M and the nonnegativity of h we have 0 < cλ ≤ max Iλ (γ(τ )) = Iλ (τλ e) τ ≥0

≤

τ 2p 1 M (∥τλ e∥2 ) + λ p 2p

∫ R3

∫ R3

p

k(x)k(y)|e(x)| |e(y)| 3−2t

|x − y|

p

dxdy → 0

as λ → ∞. The proof is complete. □ Lemma 3.4 (The (P S)cλ Condition). There exists λ∗ ≥ 0 such that Iλ satisfies the (P S)cλ condition on HVs,2 (R3 ) for all λ > λ∗ . Proof . Fix any sequence {un }n ⊂ HVs,2 (R3 ) such that Iλ (un ) → cλ and Iλ′ (un ) → 0 as n → ∞. Then, there exists C > 0 such that |⟨Iλ′ (un ), un ⟩| ≤ C∥un ∥ and |Iλ (un )| ≤ C. As in Lemma 4.5 of [15], see also [17], we divide the proof into two parts. We begin with the case inf n∈N ∥un ∥ = d > 0.

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

12

First, we show that {un }n is bounded. By (M1 ), (H1 ) and the assumptions that θ < p < 2∗s /2 and 2p ≤ q < 2∗s , we get 1 ′ ⟨I (un ), un ⟩ 2p λ 1 1 = M (∥un ∥2 ) − M (∥un ∥2 )∥un ∥2 2 2p )∫ ) ( ∫ ( 1 1 1 1 2∗ + q s |u+ − h|un | dx + − ∗ +λ n | dx 2p q 2p 2s R3 R3 ( ) ( ) 1 1 1 1 2 2 ≥ − M (∥un ∥ )∥un ∥ ≥ − κ∥un ∥2 . 2θ 2p 2θ 2p

C + C∥un ∥ ≥ Iλ (un ) −

(3.8)

This yields at once that {un }n is bounded in HVs,2 (R3 ). It is easy to verify that ∫ ∫ ⏐ ⏐ 2∗s 2∗ ∗ s ⏐ + 2∗s −1 ⏐ 2∗s −1 2∗ dx = |un | s dx ≤ S − 2 ∥un ∥2s ≤ C. ⏐ |un | ⏐ R3

R3

HVs,2 (R3 ),

Therefore, since {un }n is bounded in an application of Theorem 4.9 of [11] gives the existence of some uλ ∈ HVs,2 (R3 ) and αλ , δλ ≥ 0 such that, up to a subsequence, still denoted by {un }n , s,2 3 u ∫n ⇀ uλ weakly in HV (R ), ∗ 2 |un − uλ | s dx → δλ ,

∥un ∥ → αλ ,

R3

(3.9)

un → uλ a.e. in R3 , 2∗ s −1

|u+ n| Let 2∗s,t =

3+2t 3−2s .

⇀ |u+ λ|

2∗ s −1

weakly in L

2∗ s 2∗ s −1

(R3 ).

Since k ∈ Lν (R3 ), for any ε > 0 there exists Rε > 0 such that ∫ (k(x))ν dx ≤ ε. R3 \BRε

Also, for any measurable subset U ⊂ BRε , we have (∫ )1−p 3−2s (∫ )p 3−2s ∫ 2∗ s 3+2t 3+2t ⏐ p ⏐⏐ 2∗ 2∗ ν s ⏐k(x)|u+ s,t dx ≤ k (x)dx |un | dx n| U

U

U

)1−p 3−2s 3+2t ν , k (x)dx

(∫ ≤C U

which implies that 2∗ s ∗ + p 2s,t

|k(x)|un | |

{

2∗ s ∗ + p 2s,t

|k(x)|un | |

n

2∗ s + p 2∗ s,t

→ |k(x)|uλ | |

}

is equi-integrable in BRε . By un → uλ a.e. in R3 , we have

a.e. in R3 . Then Vitali convergence theorem yields ∫ 2∗ ⏐ ⏐ s + p ⏐ 2∗ + p ⏐ lim k(x)|un | − k(x)|uλ | s,t dx = 0.

n→∞

B Rε

Note that ∫ R3

2∗

⏐ ⏐ s p + p ⏐ 2∗ ⏐k(x)|u+ s,t dx ≤ n | − k(x)|u | λ

2∗

⏐ ⏐ s p + p ⏐ 2∗ ⏐k(x)|u+ s,t dx n | − k(x)|u |

∫

λ

B Rε

2∗

∫ + R3 \BRε

∫ ≤ B Rε

⏐ ⏐ s p + p ⏐ 2∗ ⏐k(x)|u+ s,t dx | − k(x)|u | n λ

2∗

⏐ ⏐ s 3−2s p + p ⏐ 2∗ ⏐k(x)|u+ s,t dx + Cε1−p 3+2t . n | − k(x)|u | λ

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

13

Letting n → ∞ and using the arbitrary of ε, one has p k(x)|u+ n|

→

p k(x)|u+ λ|

strongly in L

2∗ s 2∗ s,t

(R3 ).

(3.10)

By the Hardy–Littlewood–Sobolev inequality and the fact that 2∗s /2∗s,t = 6/(3 + 2t), the Riesz potential 2∗ s 2∗ s,t

6

defines a linear continuous map Kt (·) : L (R3 ) → L 3+2t (R3 ). It is easy to verify that k(x)|u+ n| + p 2∗ 3 k(x)|uλ | in L t (R ). Thus, it follows from (3.10) that ∫ ∫ p p + p + p (Kt ∗ (k|u+ | ))k|u | dx = (Kt ∗ (k|u+ lim n n λ | ))k|uλ | dx. n→∞

R3

p−1

uλ →

(3.11)

R3

Similarly, ∫ (Kt ∗

lim

n→∞

Since h ∈ L

2∗ s 2∗ s −q

R3

p + p−2 + (k|u+ uλ un dx λ | ))|uλ |

∫ = R3

p

p

+ (Kt ∗ (k|u+ λ | ))|uλ | dx.

(R3 ), by the Vitali convergence theorem one can deduce that ∫ ∫ q q lim h(x)|u+ | dx = h(x)|u+ n λ | dx. n→∞

R3

(3.13)

R3

This together with the Brezis–Lieb lemma [12] yields that ∫ + q h(x)|u+ lim n − uλ | dx = 0. n→∞

(3.12)

(3.14)

R3

Let us now introduce, for simplicity, the bi-linear functional L(·, ·) on HVs,2 (R3 ) × HVs,2 (R3 ) defined by L(v, w) = ⟨v, w⟩s for all v, w ∈ HVs,2 (R3 ). The H¨ older inequality gives |L(v, w)| ≤ [v]s [w]s + ∥v∥2,V ∥w∥2,V ≤ ∥v∥∥w∥. Thus, the bi-linear functional L(·, ·) is continuous on HVs,2 (R3 ) × HVs,2 (R3 ). Hence, the weak convergence of {un }n in HVs,2 (R3 ) gives that lim L(uλ , un − uλ ) = 0.

n→∞

(3.15)

Since {un }n in bounded in HVs,2 (R3 ), we have lim L(un , v) = L(uλ , v)

n→∞

(3.16)

for any v ∈ HVs,2 (R3 ). Since {un }n is a (P S) sequence, we deduce from Lemma 2.1 and (3.10)–(3.16) that o(1) = ⟨Iλ′ (un ) − Iλ′ (uλ ), un − uλ ⟩ = M (∥un ∥2 )∥un ∥2 − M (∥un ∥2 )⟨un , uλ ⟩s − M (∥u∥2 )⟨uλ , un − uλ ⟩s ∫ [ ] p p + p−2 + p−2 + (Kt ∗ (k|u+ un − (Kt ∗ (k|u+ | ))k|u | u (un − uλ )dx λ n | ))k|un | λ λ R∫3 p−1 p−1 −λ h(x)[|u+ − |u+ ](un − uλ )dx n| λ| 3 ∫ R 2∗ 2∗ s −1 s −1 − (|u+ − |u+ )(un − uλ )dx n| λ| 3 R ∫ ∗ + 2s = M (∥un ∥2 )⟨un − uλ , un − uλ ⟩s − |u+ n − uλ | dx + o(1). R3

(3.17)

14

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

Here we have used the following fact: ∫ ∫ ∗ ∗ + 2s + 2s + lim (|un | − |un − uλ | )dx = n→∞

R3

R3

2∗

s |u+ λ | dx,

being the Brezis–Lieb lemma (see [12]). Hence (3.17) yields at once ∫ 2∗ lim M (∥un ∥2 )∥un − uλ ∥2 = lim |un − uλ | s dx. n→∞

n→∞

(3.18)

R3

By the definition of S, we get 2

∥un − uλ ∥2∗s ≤ S −1 ∥un − uλ ∥2 .

(3.19)

Combining this with (3.18), we have SM (αλ2 )

)2/2∗s ∫ |un − uλ | dx ≤ lim

(∫

2∗ s

lim

n→∞

n→∞

R3

2∗

|un − uλ | s dx.

R3

Hence, it follows from (3.9) that 2/2∗ s

SM (αλ2 )δλ

≤ δλ .

(3.20)

Now we claim that lim αλ = 0.

λ→∞

(3.21)

Otherwise, there exists sequence λk , with λk → ∞ as k → ∞, such that αλk → α0 > 0 as k → ∞. Note that ) ( 1 ′ cλk = lim Iλ (un ) − ⟨Iλ (un ), un ⟩ . n→∞ 2p A similar discussion as in (3.8) gives that ( cλk ≥

1 1 − 2θ 2p

)

M (αλ2 k )αλ2 k .

Letting k → ∞ in above inequality and using Lemma 3.3, we get ) ( 1 1 M (α02 )α02 > 0, 0≥ − 2θ 2p which is impossible. Thus, (3.21) holds true. By un ⇀ uλ weakly in HVs,2 (R3 ) and the weak lower semi-continuity of the norm, we get ∥uλ ∥ ≤ limn→∞ ∥un ∥, this together with (3.21) gives that lim ∥uλ ∥ = 0.

λ→∞

(3.22)

Define ∗

λ =

{

sup{λ > 0 : δλ > 0}, if δλ ̸≡ 0, 0, if δλ ≡ 0.

(3.23)

Now we claim that λ∗ < ∞. Arguing by contradiction, we assume λ∗ = ∞. Then, there exists a sequence {λk }k , with λk → ∞ as k → ∞, such that δλk > 0 for all k. Without loss of generality, we can assume that αλk ∈ (0, 1) by (3.21).

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

15

Using (3.17) again, we obtain M (αλ2 k )αλ2 k = δλk .

(3.24)

This equality and (3.20) imply that αλ2 k

≥S

2∗ s 2∗ s −2

≥S

2∗ s 2∗ s −2

2 ∗

(M (αλ2 k )) 2s −2 .

It follows from (M3 ) that αλ2 k

2 2∗ −2

m0 s

4(θ−1) 2∗ −2

αλks

,

that is 22∗ s −4θ 2∗ s −2

αλk

≥S

2∗ s 2∗ s −2

2 2∗ −2

m0 s

,

which contradicts αλk → 0 as k → ∞, since θ < 2∗s /2. The claim is so proved and λ∗ is finite. Thus, for all λ > λ∗ ∫ 2∗ |un − uλ | s dx = 0, lim n→∞

(3.25)

R3

which, together with the main formula (3.18), yields that lim M (αλ2 )∥un − uλ ∥2 = 0.

n→∞

Furthermore, by αλ > 0 and (M1 ), we get un → uλ strongly in HVs.2 (R3 ) as n → ∞. Finally, we consider the case inf n∈N ∥un ∥ = 0. Then 0 is an isolated point of sequence {∥un ∥}n or an accumulation point of the sequence {∥un ∥}n . In first case, there is a subsequence {unk }k such that inf k∈N ∥unk ∥ = d > 0, and we can proceed as before. In latter case, there exists a subsequence {unk }k of {un }n such that unk → 0 strongly in HVs,2 (R3 ) as n → ∞. In conclusion, Iλ verifies the (P S)cλ condition in HVs,2 (R3 ) at the level cλ . □ Proof of Theorem 1.1. It follows from Lemmas 3.1 and 3.2 that Iλ satisfies all the conditions of Theorem 3.1. Hence there exists a (P S)cλ sequence. Further, by Lemma 3.4, there exists a λ∗ ≥ 0 such that for all λ > λ∗ the functional Iλ admits a nontrivial critical point uλ ∈ HVs,2 (R3 ). The critical point uλ is a nonnegative and nontrivial mountain pass solution of system (1.1). Moreover, (1.12) follows from (3.22). □

4. Proof of Theorem 1.2 In this section, we always assume that V verifies (V1 ), M fulfills (M1 )–(M3 ), h satisfies (H1 )–(H2 ) with 3+2t 1 < q < 2 and k satisfies (K1 ) with θ < p < 2∗s /2 < 2∗s,t = 3−2s . Before going to prove Theorem 1.2, we first give some auxiliary lemmas. Lemma 4.1. There exist ρ1 ∈ (0, 1] independent of λ, λ0 > 0 and α1 > 0, such that Iλ (u) ≥ α1 > 0 for any u ∈ HVs,2 (R3 ), with ∥u∥ = ρ1 , and for all λ ∈ (0, λ0 ].

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

16

Proof . For all u ∈ HVs,2 (R3 ), with ∥u∥ ≤ 1, we obtain by (3.1) that ∫ M (1) 1 p p (Kt ∗ (k|u+ | ))k|u+ | dx Iλ (u) ≥ ∥u∥2θ + 2 2p RN 2∗ ∗ 1 s q − λ∥h∥ 2∗s ∥u∥2∗s − ∗ S − 2 ∥u∥2s 2 ∗ s 2s −q ≥

2∗ q ∗ M (1) 1 s 2ν ∥u∥2θ − C∥k∥ν ∥u∥2p − λ∥h∥ 2∗s S − 2 ∥u∥q − ∗ S − 2 ∥u∥2s . 2 2 s 2∗ s −q

(4.1)

From Young’s inequality, for any ε > 0 we have λS

q

−2

∥h∥

2∗ s 2∗ s −q

q

∥u∥ ≤ ε∥u∥

2θ

q

+ε

(

− 2θ−q

λS

q

−2

) ∥h∥

2θ 2θ−q

,

2∗ s 2∗ s −q

since q < 2θ. Thus, for ε = M (1)/4, M (1) Iλ (u) ≥ ∥u∥2θ − 4

(

2ν

4 M (1)

− C∥k∥ν ∥u∥2p − Define F (t) = M4(1) t2θ − C∥k∥ν t2p − enough such that F (ρ1 ) > 0. Set 2ν

)

q 2θ−q

( λ∥h∥

2∗ s 2∗ s −q

S

q −2

)

2θ 2θ−q

∗ 1 − 2∗s S 2 ∥u∥2s . ∗ 2s

∗

2 1 − 2s 2∗ ts S 2∗ s

2θ−q 1 λ0 = F (ρ1 ) 2θ 2

(

for all t ≥ 0. Since 2θ < 2p, there exists ρ1 ∈ (0, 1] small

M (1) 2

)q/2θ / ∥h∥

q

2∗ s 2∗ s −q

S− 2 .

Then, for all u ∈ HVs,2 (R3 ), with ∥u∥ = ρ1 ≤ 1, and for all λ ≤ λ0 , we get ) 2θ ( ) q ( 2θ−q 2θ−q 2θ q 4 Iλ (u) ≥ F (ρ1 ) − ∥h∥ 2∗s S − 2 λ 2θ−q M (1) 2∗ s −q 1 ≥ F (ρ1 ) := α1 > 0. 2 This completes the proof. □ Lemma 4.2. Set ˜ cλ = inf{Iλ (u) : u ∈ Bρ1 }, where Bρ1 = {u ∈ λ ∈ (0, λ0 ].

HVs,2 (R3 )

: ∥u∥ < ρ1 } and ρ1 ∈ (0, 1] is given by Lemma 4.1. Then ˜ cλ < 0 for all

Proof . Let x0 ∈ Ω and R ∈ (0, 1) be so small that B2R (x0∫) ⊂ Ω , where Ω is given in (H2 ). Choose a function φ ∈ C0∞ (B2R (x0 )) such that 0 ≤ φ ≤ 1, ∥φ∥ = 1 and B (x ) h(x)φq dx > 0. Fix λ ∈ (0, λ0 ]. Then, 2R 0 by (H1 ) and (3.3), for all τ , with 0 < τ < 1, we have ∫ ∫ ∗ ∫ ∗ τ 2p λ q τ 2s 1 2 2 p p q (Kt ∗ (kφ ))kφ dx − τ h(x)φ dx − ∗ φ2s dx Iλ (τ φ) = M (∥φ∥ τ ) + 2 2p R3 q 2 R3 R3 s ( ) ∫ 1 λ q 2ν q 2 2 2p 2p ≤ sup M (τ ) ∥φ∥ τ + τ C∥k∥ν ∥φ∥ − τ h(x)|φ| dx 2 0≤τ ≤1 q B2R (x0 ) (∫ ) ( ) 1 λ 2ν 2p q 2 ≤ sup M (τ ) τ + (C∥k∥ν )τ − h(x)|φ| dx τ q . 2 0≤τ ≤1 q B2R (x0 )

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

17

Since 1 < q < 2, fixing τ > 0 even smaller so that we have that τ φ ∈ Bρ1 and Iλ (τ φ) < 0. This gives that ˜ cλ < 0 for all λ ∈ (0, λ0 ], as desired. □ By Lemmas 4.1 and 4.2 and the Ekeland variational principle (see [4]), applied in Bρ1 , there exists a sequence {un }n such that ˜ cλ ≤ Iλ (un ) ≤ ˜ cλ + 1/n,

(4.2)

/ Iλ (v) ≥ Iλ (un ) − ∥un − v∥ n

(4.3)

and

for all v ∈ Bρ1 . We show first that ∥un ∥ < ρ1 for n sufficiently large. Arguing by contradiction, we assume that ∥un ∥ = ρ1 for infinitely many n. Without loss of generality, we may assume that ∥un ∥ = ρ1 for any n ∈ N. From Lemma 4.1, we deduce Iλ (un ) ≥ α1 > 0. This, together with (4.2), implies that ˜ cλ ≥ α1 > 0, which contradicts Lemma 4.2. Next we show that s,2 ′ 3 ∗ Iλ (un ) → 0 in (HV (R )) . Set wn = un + τ v, ∀v ∈ B1 := {v ∈ HVs,2 (R3 ) : ∥v∥ = 1}, where τ > 0 small enough such that 2τ ρ1 + τ 2 ≤ ρ21 − ∥un ∥2 for fixed n large. Then ∥wn ∥2 = ∥un ∥2 + 2τ ρ1 ⟨un , v⟩s + τ 2 ≤ ∥un ∥2 + 2ρ1 τ + τ 2 ≤ ρ21 , which means that wn ∈ Bρ1 . Thus, from (4.3), we obtain Iλ (wn ) ≥ Iλ (un ) −

τ ∥un − wn ∥, n

that is, Iλ (un + τ v) − Iλ (un ) 1 ≥− . τ n By letting τ → 0+ , we get ⟨Iλ′ (un ), v⟩ ≥ − n1 for any fixed n large. Similarly, choosing τ < 0 and |τ | small enough and proceeding the process as before, one can obtain ⟨Iλ′ (un ), v⟩ ≤

1 for any fixed n large. n

Thus, we conclude lim sup |⟨Iλ (un ), v⟩| = 0,

n→∞ v∈B

1

which yields that Iλ (un ) → 0 in (HVs,2 (R3 ))∗ as n → ∞. Therefore, {un } is a (P S)c sequence for the functional Iλ . Moreover, we have the following compactness result. Lemma 4.3. There exists λ∗∗ > 0 such that, up to a subsequence, {un }n is strongly convergent to some function uλ in HVs,2 (R3 ) for all λ ∈ (0, λ∗∗ ].

18

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

Proof . Since {un }n ⊂ Bρ1 , there exists a subsequence of {un }n , still denoted by {un }n , such that un ⇀ uλ in HVs,2 (R3 ) and un → uλ a.e. in R3 . As justified to get (3.9), there exist uλ ∈ HVs,2 (R3 ), αλ ≥ 0 and δλ ≥ 0 such that, up to a subsequence, still denoted by {un }n , un ⇀ uλ weakly in HVs,2 (R3 ), un → uλ a.e. in R3 , ∥un ∥ → αλ , ∫ ∗ + 2s |u+ n − uλ | dx → δλ .

(4.4)

R3

Let us first discuss the case inf n≥1 ∥un ∥ > 0. Then αλ > 0. Similarly, also in this section, we can obtain the validity of (3.13) and (3.14). By the Hardy–Littlewood– Sobolev inequality and the argument proceeded in the proof of Lemma 3.4, also in this context we are able to prove that (3.11) and (3.12) hold true. Let L be defined as in the proof of Lemma 3.4. Then, (3.15) and (3.16) continue to hold and since {un }n is a (P S)cλ sequence, we have as before (3.17) and (3.18). Define ∫ ∫ ∫ 1 λ 1 1 p p q 2∗ (Kt ∗ (k|u+ | ))k|u+ | dx − h|u+ | dx − ∗ |u+ | s dx. Iαλ (u) = M (αλ2 )∥u∥2 + 2 2p R3 q R3 2 s R3 Clearly, Iα′ λ (uλ ) = 0. Let vn = un − uλ . Then o(1) = ⟨Iλ′ (un ), un ⟩ = ⟨Iλ′ (un ), un ⟩ − ⟨Iα′ λ (uλ ), uλ ⟩ ∫ ∗ + 2s = M (αλ2 )∥vn ∥2 − |u+ n − uλ | dx + o(1).

(4.5)

R3

Also, we have by (M2 ), (3.13), (4.4) and (3.19) that 1 ′ ⟨I (uλ ), uλ ⟩ 2p αλ ∫ ∫ 1 λ λ 1 q 2 2 2 + q h|un | dx + h|u+ = M (∥un ∥ ) − M (αλ )∥uλ ∥ − λ | dx 2 2p q R3 2p R3 ∫ ∫ 1 1 2∗ 2∗ s s − ∗ |u+ | |u+ dx + n λ | dx 2s R3 2p R3 ∫ ∫ 1 λ 1 λ q 2 2 2 2 + q ≥ M (∥un ∥ )∥un ∥ − M (αλ )∥uλ ∥ − h|un | dx + h|u+ λ | dx 2p 2p q R3 2p R3 ∫ ∫ 1 1 2∗ 2∗ s s − ∗ |u+ | dx + |u+ n λ | dx ∗ 2s R3 2 s R3 ( )∫ ∫ ∗ 1 λ λ 1 + q + 2s 2 2 = M (αλ )∥vn ∥ − − h|uλ | dx − ∗ |u+ n − uλ | dx + o(1), 2p q 2p 2s R3 R3

˜ cλ + o(1) = Iλ (un ) −

thanks to θ < p < 2∗s /2. It follows from (4.5) ( )∫ ∫ ∗ 1 λ λ 1 q + 2s ˜ cλ + o(1) ≥ M (αλ2 )∥vn ∥2 − − h|u+ | dx − |u+ n − uλ | dx + o(1) λ ∗ 2p q 2p 2 3 R3 )∫ )∫ s R ( ( ∗ 1 1 λ λ q 2 + s − ∗ |u+ − h|u+ = n − uλ | dx − λ | dx + o(1), 2p 2s q 2p R3 R3 which together with q < 2p and ˜ cλ < 0 by Lemma 4.2 yields ( )∫ ( )∫ ∗ 1 1 λ λ q + 2s − ∗ |u+ − u | dx ≤ − h|u+ n λ λ | dx + o(1) 2p 2s q 2p 3 3 R R ≤ Cλ + o(1),

(4.6)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

19

where C > 0 independent of λ, since ∥uλ ∥ < ρ1 and ρ1 does not depend on λ. This, together with Lemma 4.2, implies that ∫ ∗ + 2s lim δλ = lim lim |u+ (4.7) n − uλ | dx = 0. λ→0 n→∞

λ→0

R3

Then by (4.5) we get 2 2∗

δλ ≥ M (αλ )Sδλ s .

(4.8)

Set λ

∗∗

{ =

inf{λ ∈ (0, λ0 ] : δλ > 0}, if δλ ̸≡ 0, λ0 , if δλ ≡ 0.

If δλ ≡ ̸ 0, then λ∗∗ = inf{λ ∈ (0, λ0 ] : δλ > 0} > 0. Otherwise, there exists a sequence {λk }k , with δλk > 0, such that λk → 0 as k → ∞. Thus, (4.8) implies that 1− 2∗

δλk

2s

≥ SM (αλ2 k ),

which means that αλk → 0 as k → ∞. Without loss of generality, we assume that αλk ∈ (0, 1] for all k ≥ 1. Then, from (3.24) we have [ ]1/(1−2/2∗s ) M (αλ2 k )αλ2 k = δλk ≥ SM (αλ2 k ) . Hence, (M3 ) gives 2(2∗ s −2θ) 2∗ s −2

αλk

2 ∗

≥ (Sm0 ) 2s −2 .

This is impossible, since θ < 2∗s /2 and αλk → 0 as k → ∞. Therefore, δλ = 0 for all λ ∈ (0, λ∗∗ ], that is, ∫ ∗ + 2s lim |u+ n − uλ | dx = 0 n→∞

R3

for all λ ∈ (0, λ∗∗ ]. This, together with (4.5) yields that un → uλ strongly in HVs,2 (R3 ) as n → ∞. If inf n≥1 ∥un ∥ = 0, the discussion is the last case treated in the proof of Lemma 3.4. In conclusion, the proof is complete. □ Proof of Theorem 1.2. It follows from Lemmas 4.1 and 4.2 that there exists a (P S)˜c sequence {un }n λ at the level ˜ cλ < 0 defined in Lemma 4.2. Further, by Lemma 4.3, there exists λ∗∗ > 0 such that for all λ ∈ (0, λ∗∗ ], up to a subsequence, {un }n strongly converges to uλ . Moreover, cλ = Iλ (uλ ) < 0 and Iλ′ (uλ ) = 0. Combining this with Lemma 2.1, we conclude that uλ is a nontrivial and nonnegative solution of system (1.1). Next we show that limλ→0+ ∥uλ ∥ = 0. By (M1 ), (4.9) and (4.5), we have ∫ ∫ ∫ 1 1 λ 1 p 2∗ + p + q s ˜ cλ ≥ M (∥un ∥2 )∥un ∥2 + (Kt ∗ (k|u+ | ))k|u | dx − h|u | dx − |u+ n n n n | dx + o(1) ∗ 2θ 2p R3 q R3 2 s R3 ∫ 1 1 1 p + p = M (∥un ∥2 )∥un − uλ ∥2 + M (∥un ∥2 )∥uλ ∥2 + (Kt ∗ (k|u+ λ | ))k|uλ | dx 2θ 2θ 2p R3 ∫ ∫ ∫ ∗ ∗ 1 1 λ q + 2s + 2s − ∗ |un − uλ | dx − ∗ |uλ | dx − h|u+ λ | dx + o(1) 2s R3 2s R3 q R3 )∫ ( ∫ ∗ 1 1 1 1 p + 2s + p 2 2 + = M (αλ )∥uλ ∥ + − ∗ |un − uλ | dx + (Kt ∗ (k|u+ λ | ))k|uλ | dx 2θ 2θ 2s 2p 3 3 R R ∫ ∫ ∗ 1 λ + 2s + q − ∗ |u | dx − h|uλ | dx + o(1). (4.9) 2s R3 λ q R3

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

20

Note that ⟨Iα′ λ (uλ ), uλ ⟩ = 0, we obtain ∫ ∫ + p + p 2 2 M (αλ )∥uλ ∥ + (Kt ∗ (k|uλ | ))k|uλ | dx = λ R3

R3

q h|u+ λ | dx

∫

2∗

+ R3

s |u+ λ | dx.

Combining this with (4.9), we get )∫ ) ( ( ∗ 1 1 1 1 1 1 + 2s |u+ − M (αλ2 )∥uλ ∥2 + − ∗+ − ∗ ˜ cλ ≥ n − uλ | dx 2θ 2p 2θ 2s 2p 2s R3 ( )∫ 1 1 q h|u+ −λ − λ | dx + o(1) q 2p R3 ( ) ( )∫ 1 1 1 1 q 2 2 ≥ − M (αλ )∥uλ ∥ − λ − h|u+ λ | dx + o(1). 2θ 2p q 2p 3 R It follows from ∥uλ ∥ < ρ1 and ρ1 does not depend on λ that there exists positive constant C independent of λ such that ) ( 1 1 − M (αλ2 )∥uλ ∥2 − Cλ + o(1). ˜ cλ ≥ 2θ 2p By Lemma 4.3, we know that αλ = ∥uλ ∥, this together with ˜ cλ < 0 yields ( ) 1 1 − M (∥uλ ∥2 )∥uλ ∥2 ≤ Cλ. 2θ 2p Further, from 0 < ρ1 ≤ 1 and (M2 ), we deduce ( ) 1 1 − m0 ∥uλ ∥2θ ≤ Cλ, 2θ 2p which together with θ < p implies that limλ→0+ ∥uλ ∥ = 0. The proof is complete.

□

5. Proof of Theorem 1.3 In this section, we consider system (1.13). Using the same discussion as that in Section 2, system (1.13) becomes the following equation ( )[ ] 2∗ 2∗ −2 q−2 2∗ −2 a + b∥u∥2 (−∆)s u + V (x)u − (Kt ∗ (|u| s,t ))|u| s,t u = λh(x)|u| u + |u| s u in R3 . (5.1) Define ∫ a b λ q 2 4 I(u) = ∥u∥ + ∥u∥ − h|u+ | dx 2 4 q R3 ∫ ∫ ∫ 2∗ 2∗ + 1 |u (x)| s,t |u+ (y)| s,t 1 2∗ − dxdy − |u+ | s dx. 3−2t ∗ ∗ 22s,t R3 R3 2 s R3 |x − y| for all u ∈ HVs,2 (R3 ). By the Hardy–Littlewood–Sobolev inequality and the fractional Sobolev inequality, one can show that I is well-defined, of class C 1 and ∫ q−2 ′ 2 ⟨I (u), v⟩ = (a + b∥u∥ )⟨u, v⟩s − λ h|u+ | u+ vdx ∫

∫

− R3

R3

+

2∗ s,t

|u (y)|

R3 +

2∗ s,t −2 +

|u (x)|

u (x)v(x)

3−2t

|x − y|

∫ dxdy −

2∗ s −2 +

|u+ |

u vdx,

R3

for all u, v ∈ HVs,2 (R3 ). Hence a critical point of I is a nonnegative (weak) solution of (5.1).

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

21

In the following, we always assume that s, t ∈ (0, 1), 4s + 2t ≤ 3 and h satisfies (H1 ) with 2 < q < 2∗s . It follows the assumption that 2∗s,t ≤ 2 < 2∗s . The following lemma shows that I verifies the global (P S)c condition, which is a key lemma for the proof of main result. −2 Lemma 5.1. Assume that s, t ∈ (0, 1) and 4s + 2t ≤ 3. If 4s + 2t = 3, b > SH,L and a, b satisfy (1.14) or 4s + 2t < 3, a, b > 0 and a, b satisfy (1.15). Then functional I satisfies the (P S)c condition in HVs,2 (R3 ) for all λ > 0.

Proof . Let {un }n ⊂ HVs,2 (R3 ) be a any (P S)c sequence of functional I, i.e. I(un ) → c, I ′ (un ) → 0 as n → ∞. It follows from (2.1) and (2.2) that I(u) ≥

2∗ q ∗ ∗ −2∗ b 1 a 1 s ∥u∥2 + ∥u∥4 − λS − 2 ∥h∥2∗s −q ∥u∥q − ∗ SH,Ls,t ∥u∥22s,t − ∗ S − 2 ∥u∥2s , 2 4 22s,t 2s

(5.2)

−2 for all u ∈ HVs,2 (R3 ). When 4s + 2t = 3, since b > SH,L , 2∗s,t = 2, q < 2∗s and 2 < 2∗s < 4, it follows that I is coercive and bounded from below on HVs,2 (R3 ). When 4s + 2t < 3, since a > 0, b > 0, q < 2∗s , and 2∗s,t < 2 < 2∗s < 4, it follows that I is coercive and bounded from below on HVs,2 (R3 ). Hence, {un }n is bounded in HVs,2 (R3 ). Then there exist a subsequence of {un }n still denoted by {un }n and u ∈ HVs,2 (R3 ) such that ∗

un ⇀ u in HVs,2 (R3 ) and in L2s (R3 ), un → u a.e. in R3 and in Lqloc (R3 ), 1 ≤ q < 2∗s , 2∗ s −2

|un |

2∗ s −2

un ⇀ |u|

u weakly in L

2∗ s 2∗ s −1

(R3 ),

as n → ∞. First, by using a similar discussion as above section, we can obtain ∫ ∫ ∫ + q + q−2 + lim h(x)|u+ − u | dx = 0, lim h(x)|u | u udx = n n n n→∞ n→∞ R3 ∫ R3 ∫ q q lim h(x)|u+ h(x)|u+ | dx. n | dx = n→

R3

(5.3)

q

h(x)|u+ | dx R3

(5.4)

R3

Then by un ⇀ u in HVs,2 (R3 ) we easily get ∥un ∥2 = ∥un − u∥2 + ∥u∥2 + o(1). By using a Brezis–Lieb type result [25, Lemma 2.2], one has ∫ ∫ ∗ ∗ 2∗ 2∗ + + 2s,t + + 2s,t s,t s,t (Kt ∗ (|un − u | ))|un − u | dx = (Kt ∗ (|u+ ))|u+ dx n| n| R3 ∫ R3 2∗ 2∗ − (Kt ∗ (|u+ | s,t ))|u+ | s,t dx + o(1).

(5.5)

(5.6)

R3

Also, from the Brezis–Lieb lemma (see [12]), we have ∫ ∫ ∫ ∗ ∗ + 2s + 2s |u+ − u | dx = |u | dx − n n R3

R3

R3

2∗

|u+ | s dx + o(1).

(5.7)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

22

By the weak convergence of {un } in HVs,2 (R3 ), one can deduce that lim L(u, un − u) = 0,

(5.8)

n→∞

where L is given by Lemma 3.4. Without loss of generality, we assume that limn→∞ ∥wn ∥ = η. Since {un }n is a (P S)c sequence, by the boundedness of {un }n , (5.4), (5.6), (5.7) and (5.8), we deduce o(1) = ⟨I ′ (un ) − I ′ (u), un − u⟩ = (a + b∥un ∥2 )L(un , un − u) − (a + b∥u∥2 )L(u, un − u) ∫ ( ) q−1 q−1 −λ h(x) |u+ (un − u)dx − |u+ | n| ∫ R(3 ) 2∗ 2∗ −1 s −1 − |u+ − |u+ | s (un − u)dx n| ∫R3 [ ] 2∗ 2∗ 2∗ 2∗ −1 s,t s,t −1 − (Kt ∗ (|u+ ))|u+ − (Kt ∗ (|u+ | s,t ))|u+ | s,t (un − u)dx n| n| R3

= (a + b∥un ∥2 )L(un − u, un − u) ∫ ∫ ∗ ∗ + + 2s,t + + 2s,t − (Kt ∗ |un − u | )|un − u | dx − R3

2∗

R3

+ s |u+ n − u | dx + o(1).

Here we use the following facts: ∫ ∫ ∗ ∗ 2∗ 2∗ + 2s,t + 2s,t −1 (Kt ∗ (|un | ))|un | lim udx = (Kt ∗ (|u+ | s,t ))|u+ | s,t dx, n→∞

R3

(5.9)

(5.10)

R3

and ∫

2∗ s −1

lim

n→∞

R3

|u+ n|

∫

2∗

|u+ | s dx.

udx =

(5.11)

R3

Indeed, by the Hardy–Littlewood–Sobolev inequality, the Riesz potential defines a linear continuous map 6 6 from L 3+2t (R3 ) to L 3−2t (R3 ). Then Kt ∗ (|u+ n|

2∗ s,t

) ⇀ Kt ∗ (|u+ |

2∗ s,t

6

) in L 3−2t (R3 ),

(5.12)

as n → ∞. Note that for any measurable subset U ⊂ R3 , we have ∗

2s,t −1 1 ∫ ⏐ ⏐ 2∗s 2∗ 2∗ ⏐ + 2∗s,t −1 ⏐ 2∗s,t s,t s,t dx ≤ ∥u ∥ u ∥u∥ |u | ⏐ n ⏐ n 2∗ 2∗ s s

L

U

(U )

1 2∗ s,t ∗ L2s (U )

≤ C∥u∥

,

2∗

⏐ ⏐ ∗s 2∗ 2∗ 2∗ s,t −1 ⏐ 2s,t s,t −1 which implies that {⏐|u+ | u } is equi-integrable in R3 . Observe that |u+ u → |u| s,t a.e. in R3 , n n| then the Vitali convergence theorem yields that ∗

2s,t −1 u |u+ n|

2∗ s,t

→ |u|

in L

2∗ s 2∗ s,t

(R3 ).

(5.13)

2∗

6 Combining (5.12) with (5.13) and 2∗s = 3+2t , we get the desired result (5.10). s,t It follows from (5.5), (5.6) and (5.9) that

(a + b∥un − u∥2 + b∥u∥2 )L(un − u, un − u) ∫ ∫ ∗ ∗ + 2s,t + + 2s,t − (Kt ∗ (|u+ − u | ))|u − u | dx − n n R3

R3

2∗

+ s |u+ n − u | dx = o(1).

(5.14)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

23

In view of the definitions of SH,L and S, we get ∫ ∗ ∗ ∗ −2∗ + 2s,t + 2s,t (Kt ∗ (|u+ ))|u+ dx ≤ SH,Ls,t ∥un − u∥22s,t , n −u | n −u | R3

and ∫

2∗

+ s − |u+ n − u | dx ≤ S

R3

2∗ s 2

∗

∥un − u∥2s .

Putting these two inequalities in (5.14), we arrive at the inequality −2∗

∗

(a + b∥un − u∥2 + b∥u∥2 )∥un − u∥2 ≤ SH,Ls,t ∥un − u∥22s,t + S −

2∗ s 2

∗

∥un − u∥2s + o(1).

Letting n → ∞, we have −2∗

∗

aη 2 + bη 4 ≤ SH,Ls,t η 22s,t + S −

2∗ s 2

∗

η 2s .

(5.15)

−2 When 4s + 2t = 3 and SH,L < b, it follows from (5.15) that −2 aη 2 + (b − SH,L )η 4 ≤ S −

2∗ s 2

∗

η 2s .

(5.16)

Note that for any ε > 0, by Young’s inequality we have ∗

S Taking ε =

2 (b 2∗ s −2

2 − 2s

η

2∗ s

≤

ε

4

2 2∗ s −2

η +

(S −

2∗ s 2

2

∗

2 4−2∗ s

∗

∗

η 4−2s ) 4−2s

ε

2 −2 − s ∗ 4−2

.

(5.17)

η2 .

(5.18)

s

−2 − SH,L ) in (5.17) and inserting it in (5.16), we obtain 2∗ s

aη 2 ≤

(4 −

− ∗ 2∗s )S 4−2s

(

−2 2(b − SH,L )

2∗s

2

)− 2∗s −2∗ 4−2s

−2

Since 2∗ s

a>

(4 −

− ∗ 2∗s )S 4−2s

(

2(b −

−2 SH,L )

)− 2∗s −2∗ 4−2s

2∗s − 2

2

by assumption (1.14), we deduce from (5.18) that −2 (b − SH,L )

2

η 4 ≤ 0,

which means that η = 0. Hence, un → u in HVs,2 (R3 ). Finally, we consider the case 4s + 2t < 3, that is, 2∗s < 22∗s,t < 4. Taking ε = 2∗b−2 in (5.17) and using (5.15), we have s

∗ −2∗ b S− aη + η 4 ≤ SH,Ls,t η 22s,t + 2

2

2∗ s 2

( )− 2∗s −2∗ 4−2s (4 − 2∗s ) b η2 , ∗ 2 2s − 2

that is, ⎡ ⎣a −

S−

2∗ s 2

⎤ ( )− 2∗s −2∗ ∗ 4−2s ∗ (4 − 2∗s ) b ⎦ η 2 + b η 4 ≤ S −2s,t η 22s,t . H,L 2 2∗s − 2 2

(5.19)

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

24

For any ξ > 0, we deduce from the Young’s inequality that ∗ −2∗ SH,Ls,t η 22s,t

Choosing ξ =

b 2(2∗ s,t −1)

⎡

≤ ξ(2∗s,t − 1)(η

1 2∗ −1 42∗ s,t s,t −4

S ⎢ ⎣a −

+ξ

2∗ s,t −1 2−2∗ s,t

(2 − 2∗s,t )(η

4−22∗ s,t

)

1 2−2∗ s,t

−

2∗ s,t 2−2∗ s,t

SH,L

.

in above inequality and putting it in (5.19), we conclude

∗

2 − 2s

)

−

(4 − 2

2∗s )

( 2∗s

b −2

)− 2∗s −2∗ 4−2s

∗

− (2 − 2∗s,t )

(

)− 2s,t −1 ∗

b 22∗s,t

2−2

−2

s,t

−

2∗ s,t 2−2∗ s,t

SH,L

It follows from the assumption (1.15) that η = 0. Therefore un → u in HVs,2 (R3 ).

⎤ ⎥ 2 ⎦ η ≤ 0.

□

Proof of Theorem 1.3. First, we show that Eq. (1.13) has a nontrivial least energy solution. By (5.2), we know m := inf u∈H s,2 (R3 ) I(u) is well-defined. Now we claim that there exists λ∗∗∗ > 0 such that m < 0 for V ∫ q all λ > λ∗∗∗ . Indeed, we can choose a function v0 ∈ HVs,2 (R3 ) with ∥v0 ∥ = 1 and R3 h(x)|v0 | dx > 0, then ∫ a b λ q I(v0 ) ≤ + − h(x)|v0 | dx < 0, 2 4 q R3 for all λ > ∫

b q( a 2+4)

h(x)|v0 |q dx R3

. Hence our claim holds true. Further, by Lemma 5.1 and [41, Theorem 4.4], there

exists u1 ∈ HVs,2 (R3 ) such that I(u1 ) = m. Therefore, u1 is a nontrivial least energy solution of (1.13) with I(u1 ) < 0. In view of Lemma 2.1, we know that u1 is nonnegative. Now we prove that Eq. (1.1) has a mountain pass solution. We deduce from (H1 ) and (2.2) that [ ] 2∗ q ∗ ∗ −2∗ a b 1 1 1 s I(u) ≥ + ∥u∥2 − S − 2 λ∥h∥ 2∗s ∥u∥q−2 − ∗ SH,Ls,t ∥u∥22s,t −2 − ∗ S − 2 ∥u∥2s −2 ∥u∥2 , 2 4 q 22s,t 2s 2∗ s −q for all u ∈ HVs,2 (R3 ). Since q ∈ (2, 2∗s ), there exist ρ2 > 0 small enough and α2 > 0 such that I(u) > α2 for all u ∈ HVs,2 (R3 ) with ∥u∥ = ρ2 . Define c = inf max I(γ(τ )), γ∈Γ τ ∈[0,1]

where Γ = {γ ∈ C([0, 1], HVs,2 (R3 )) : γ(0) = 0, γ(1) = u1 }. Then c > 0. By Lemma 5.1 and the mountain pass theorem (see Theorem 3.1), we know that there exists u2 ∈ HVs,2 (R3 ) such that I(u2 ) = c > 0 and I ′ (u2 ) = 0. Thus, u2 is a nontrivial and nonnegative solution of Eq. (1.1). □ Acknowledgments The authors would like to thank the referees for their useful suggestions and comments. Mingqi Xiang was supported by the National Nature Science Foundation of China (No. 11601515) and the Fundamental Research Funds for the Central Universities (No. 3122017080). References

[1] C.O. Alves, M.A. Souto, Existence of least energy nodal solution for a Schr¨ odinger-Poisson system in bounded domains, Z. Angew. Math. Phys. 65 (2014) 1153–1166. [2] A. Ambrosetti, P.H. Rabinowitz, Dual variational methods in critical point theorey and applications, J. Funct. Anal. 14 (1973) 349–381.

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

25

[3] D. Applebaum, L´ evy processes–From probability to finance quantum groups, Notices Amer. Math. Soc. 51 (2004) 1336–1347. [4] J.P. Aubin, I. Ekeland, Applied Nonlinear Analysis, Wiley, New York, 1984. [5] G. Autuori, A. Fiscella, P. Pucci, Stationary Kirchhoff problems involving a fractional operator and a critical nonlinearity, Nonlinear Anal. 125 (2015) 699–714. [6] A. Azzollini, P. d’Avenia, On a system involving a critical nonlinearity, J. Math. Anal. Appl. 387 (2012) 433–438. [7] A. Azzollini, P. d’Avenia, V. Luisi, Generalized Schr¨ odinger-Poisson type systems, Commun. Pure Appl. Anal. 12 (2013) 867–879. [8] V. Benci, D. Fortunato, An eigenvalue problem for the Schr¨ odinger-Maxwell equations, Topol. Methods Nonlinear Anal. 11 (1998) 283–293. [9] V. Benci, D. Fortunato, Solitary waves of the nonlinear Klein-Gordon equation coupled with the Maxwell equations, Rev. Math. Phys. 14 (2002) 409–420. [10] A. Bongers, Existenzaussagen f¨ ur die Choquard-Gleichung: Ein nichtlineares eigenwertproblem der plasma-physik, ZAMM Z. Angew. Math. Mech. 60 (1980) 240–242. [11] H. Br´ ezis, Functional analysis, in: Sobolev Spaces and Partial Differential Equations, in: Universitext, Springer, New York, 2011. [12] H. Br´ ezis, E. Lieb, A relation between pointwise convergence of functions and convergence of functionals, Proc. Amer. Math. Soc. 88 (1983) 486–490. [13] L. Caffarelli, Nonlocal equations, drifts and games, in: Nonlinear Partial Differential Equations, in: Abel Symposia, vol. 7, 2012, pp. 37–52. [14] L. Caffarelli, L. Silvestre, An extension problem related to the fractional Laplacian, Comm. Partial Differential Equations 32 (2007) 1245–1260. [15] M. Caponi, P. Pucci, Existence theorems for entire solutions of stationary Kirchhoff fractional p–Laplacian equations, Ann. Mat. Pura Appl. 195 (2016) 2099–2129. [16] Y.H. Chen, C.G. Liu, Ground state solutions for non-autonomous fractional Choquard equations, Nonlinearity 29 (2016) 1827–1842. [17] F. Colasuonno, P. Pucci, Multiplicity of solutions for p(x)–polyharmonic Kirchhoff equations, Nonlinear Anal. 74 (2011) 5962–5974. [18] P. D’Ancona, S. Spagnolo, Global solvability for the degenerate Kirchhoff equation with real analytic data, Invent. Math. 108 (1992) 247–262. [19] P. D’Avenia, M. Squassina, On fractional Choquard equations, Math. Models Methods Appl. Sci. 25 (2015) 1447–1476. [20] J. Davila, M. del Pino, S. Dipierro, E. Valdinoci, Concentration phenomena for the nonlocal Schr¨ odinger equation with Dirichlet datum, Anal. PDE 8 (2015) 1165–1235. [21] E. Di Nezza, G. Palatucci, E. Valdinoci, Hitchhiker guide to the fractional Sobolev spaces, Bull. Sci. Math. 136 (2012) 521–573. [22] S. Dipierro, M. Medina, E. Valdinoci, Fractional elliptic problems with critical growth in the whole of Rn , arXiv:1506. 01748, p. 122. [23] A. Fiscella, P. Pucci, p-fractional Kirchhoff equations involving critical nonlinearities, Nonlinear Anal. RWA 35 (2017) 350–378. [24] A. Fiscella, E. Valdinoci, A critical Kirchhoff type problem involving a nonlocal operator, Nonlinear Anal. 94 (2014) 156–170. [25] F. Gao, M. Yang, On the Br´ ezis–Nirenberg type critical problem for nonlinear Choquard equation, Sci. China Math. (2017). http://dx.doi.org/10.1007/s11425-000-0000-0. [26] X.M. He, W.M. Zou, Existence and concetration of ground states for Schr¨ odinger-Poisson equations with critical growth, J. Math. Phys. 53 (2012) 023702. [27] Y.S. Jiang, H.S. Zhou, Schr¨ odinger-Poisson system with steep potential well, J. Differential Equations 251 (2011) 582–608. ¨ [28] G. Kirchhoff, Vorlesungen Uber Mathematische Physik, BG Teubner, 1876. [29] N. Laskin, Fractional quantum mechanics and L´ evy path integrals, Phys. Lett. A 268 (2000) 298–305. [30] N. Laskin, Fractional Schr¨ odinger equation, Phys. Rev. 66 (2002) 56–108. [31] F.Y. Li, Y.H. Li, J.P. Shi, Existence of positive solutions to Schr¨ odinger-Poisson type systems with critical exponent, Commun. Contemp. Math. 16 (2014) 1450036. [32] E.H. Lieb, Existence and uniqueness of the minimizing solution of Choquard’s nonlinear equation, Stud. Appl. Math. 57 (1977) 93–105. [33] E. Lieb, M. Loss, Analysis, second ed., in: Graduate Studies in Mathematics, vol. 14, AMS, Providence, Rhode island, 2001. [34] P.-L. Lions, The Choquard equation and related questions, Nonlinear Anal. 4 (1980) 1063–1072. [35] P.-L. Lions, Solutions of Hartree–Fock equations for coulomb systems, Comm. Math. Phys. 109 (1987) 33–97. [36] H.L. Liu, Positive slutions of an asymptotically periodic Schr¨ odinger-Poisson system with critical exponent, Nonlinear Anal. RWA 32 (2016) 198–212. [37] J. Liu, J.F. Liao, C.L. Tang, Positive solutions for Kirchhoff-type equations with critical exponent in RN , J. Math. Anal. Appl. 429 (2015) 1153–1172. [38] Z.L. Liu, Z.Q. Wang, J.J. Zhang, Infinitely many sign-changing solutions for the nonlinear Schr¨ odinger-Poisson system, Ann. Mat. Pura Appl. 195 (2016) 775–794. [39] D.F. L¨ u, A note on Kirchhoff-type equations with Hartree–type nonlinearities, Nonlinear Anal. 99 (2014) 35–48. [40] P.A. Markowich, C.A. Ringhofer, C. Schmeiser, Semiconductor Equations, Springer-Verlag, Vienna, 1990.

26

M. Xiang, F. Wang / Nonlinear Analysis 164 (2017) 1–26

[41] J. Mawhin, M. Willem, Critical Point Theory and Hamiltonian Systems, Springer-Verlag, New York/Berlin, 1989. [42] X. Mingqi, G. Molica Bisci, G. Tian, B. Zhang, Infinitely many solutions for the stationary Kirchhoff problems involving the fractional p–Laplacian, Nonlinearity 29 (2016) 357–374. [43] G. Molica Bisci, V. R˘ adulescu, R. Servadei, Variational Methods for Nonlocal Fractional Equations, in: Encyclopedia of Mathematics and its Applications, vol. 162, Cambridge University Press, Cambridge, 2016. [44] V. Moroz, J.V. Schaftingen, Groundstates of nonlinear Choquard equations: Hardy-Littlewood-Sobolev critical exponent, Commun. Contemp. Math. 17 (2015) 1550005 12pp. [45] S. Pekar, Untersuchung Uber Die Elektronentheorie Der Kristalle, Akademie Verlag, 1954. [46] R. Penrose, Quantum computation, entanglement and state reduction, Philos. Trans. Roy. Soc. 356 (1998) 1–13. [47] P. Pucci, S. Saldi, Critical stationary Kirchhoff equations in RN involving nonlocal operators, Rev. Mat. Iberoam. 32 (2016) 1–22. [48] P. Pucci, M. Xiang, B. Zhang, Existence results for Schr¨ odinger-Choquard-Kirchhoff equations involving the fractional p-Laplacian, Adv. Calc. Var. (2017). http://dx.doi.org/10.1515/acv-2016-0049. [49] P. Pucci, M. Xiang, B. Zhang, Multiple solutions for nonhomogeneous Schr¨ odinger–Kirchhoff type equations involving the fractional p–Laplacian in RN , Calc. Var. Partial Differential Equations 54 (2015) 2785–2806. [50] P. Pucci, M. Xiang, B. Zhang, Existence and multiplicity of entire solutions for fractional p–Kirchhoff equations, Adv. Nonlinear Anal. 5 (2016) 27–55. [51] D. Ruiz, The Schr¨ odinger-Poisson equation under the effect of a nonlinear local term, J. Funct. Anal. 237 (2006) 655–674. [52] Z.F. Shen, F.S. Gao, M.B. Yang, Ground states for nonlinear fractional Choquard equations with general nonlinearities, Math. Methods Appl. Sci. 39 (2016) 4082–4098. [53] K.M. Teng, Existence of ground state solutions for the nonlinear fractional Schr¨ odinger-Poisson system with critical Sobolev exponent, J. Differential Equations 261 (2016) 3061–3106. [54] F.L. Wang, M.Q. Xiang, Multiplicity of solutions to a nonlocal Choquard equation involving fractional magnetic operators and critical exponent, Electron. J. Differential Equations 2016 (2016) 1–11. [55] D. Wu, Existence and stability of standing waves for nonlinear fractional Schr¨ odinger equations with Hartree type nonlinearity, J. Math. Anal. Appl. 411 (2014) 530–542. [56] M. Xiang, B. Zhang, M. Ferrara, Existence of solutions for Kirchhoff type problem involving the non-local fractional p–Laplacian, J. Math. Anal. Appl. 424 (2015) 1021–1041. [57] M. Xiang, B. Zhang, X. Guo, Infinitely many solutions for a fractional Kirchhoff type problem via Fountain Theorem, Nonlinear Anal. 120 (2015) 299–313. [58] H.Y. Ye, The existence of least energy nodal solutions for some class of Kirchhoff equations and Choquard equations in RN , J. Math. Anal. Appl. 431 (2015) 936–954. ´ M. Squassina, Fractional Schr¨ [59] J.J. Zhang, J.M. do O, odinger-Poisson system with a general subcritical or critical nonlinearity, Adv. Nonlinear Stud. 16 (2016) 15–30. [60] G.L. Zhao, X.L. Zhu, Y.H. Li, Existence of infinitely many solutions to a class of Kirchhoff- Schr¨ odinger-Poisson system, Appl. Math. Comput. 256 (2015) 572–581.