Existence of Positive Ground State Solutions for Fractional Choquard Systems in Subcritical and Critical Cases

Abstract

We investigate a class of fractional linearly coupled Choquard systems. For the subcritical case and all critical cases, we prove the existence, nonexistence and symmetry of positive ground state solutions of systems, by using the Nehari manifold method, the Pohožaev identity and the Schwartz symmetrization rearrangements. In particular, we overcome the lack of compactness of the critical nonlinearities by using the behaviour of sufficiently small Nehari energy levels.

1. Introduction

In this paper, we are interested in establishing existence and nonexistence results for the following coupled fractional, nonlinear Schrödinger systems with Choquard nonlinear terms:

$$\left\{\begin{array}{cccc}\hfill {(-\mathsf{\Delta })}^{s}u+{\lambda }_{1}u& =(I_{\alpha }\ast {|u|}^p{)|u|}^{p-2}u+\beta v\hfill & \hfill & \mathrm{in}{\mathbb{R}}^N,\hfill \\ \hfill {(-\mathsf{\Delta })}^{s}v+{\lambda }_{2}v& =(I_{\alpha }\ast {|v|}^q{)|v|}^{q-2}v+\beta u\hfill & \hfill & \mathrm{in}{\mathbb{R}}^N,\hfill \end{array}\right.$$

(1)

where $N>2s,s\in (0,1),\alpha \in (0,N),p,q>1$ and ${\lambda }_1,{\lambda }_2>0$ are constants, $\beta >0$ is a parameter and $I_{\alpha }(x)$ is the Riesz potential, defined by

$$I_{\alpha }(x)=\frac{\mathsf{\Gamma }(\frac{N-\alpha }{2})}{\mathsf{\Gamma }(\frac{\alpha }{2}){\pi }^{\frac{N}{2}}{2}^{\alpha }{|x|}^{N-\alpha }},x\in {\mathbb{R}}^N\setminus \left\{0\right\},$$

where $\mathsf{\Gamma }$ is the gamma function.

Here, for any function $u:{\mathbb{R}}^N\to \mathbb{R}$ in the Schwartz class, the fractional Laplacian ${(-\mathsf{\Delta })}^s$ is a nonlocal operator with $s\in (0,1)$ defined by

$${(-\mathsf{\Delta })}^{s}u(x)=C(N,s)\mathrm{P}.\mathrm{V}.{\int }_{{\mathbb{R}}^N}\frac{u(x)-u(y)}{{|x-y|}^{N+2s}}dy,$$

where P.V. represents the Cauchy principal value on the integral and $C(N,s)$ is some positive normalization constant (see [1] for details).

Our main goal here is to prove the existence of positive ground state solutions for the subcritical case and all critical cases. In the critical cases, the existence of positive ground state solutions will be related with the parameter $\alpha$. For the inferior or superior supercritical case, when $p,q\le 1+\frac{\alpha }{N}$ or $p,q\ge \frac{\alpha +N}{N-2s}$, we make use of a Pohožaev-type identity to prove that the system (1) does not admit a nontrivial solution.

Note that when $N=3$, ${\lambda }_1={\lambda }_2=1$, $\alpha =2$, $s=1$, $\beta =0$ and $p=2$, system (1) is known as the Choquard–Pekar equation:

$$-\mathsf{\Delta }u+u=(I_2\ast {|u|}^2)u,\mathrm{in}{\mathbb{R}}^3.$$

(2)

In 1954, Pekar [2] studied the quantum theory of a polaron at rest for Equation (2). In 1976, Choquard [3] used this equation as an approximation to the Hartree–Fock theory of one-component plasmas. In particular, it was studied as a self-gravitational collapse model for quantum mechanical wave functions by Penrose [4] in 1996 (see also [5]). Furthermore, in the case of $\alpha \in (0,N)$, where $N\ge 3$, the existence and qualitative results of power-type nonlinearities ${|u|}^{p-2}u$ within [6,7,8,9,10] were studied by using the variational method. In the spirit of H. Berestycki and P.L. Lions [11], under the almost necessary assumption of nonlinearity F, Moroz and Schaftingen [12] argued for the existence of a ground state solution $u\in H^1({\mathbb{R}}^N)$ for the nonlinear Choquard equation:

$$-\mathsf{\Delta }u+u=(I_{\alpha }\ast F(u))F^{\prime }(u),\mathrm{in}{\mathbb{R}}^N.$$

The scalar problem can, of course, be extended to the systems. Note that the following system

$$\left\{\begin{array}{cccc}\hfill {(-\mathsf{\Delta })}^{s}u+{\lambda }_{1}u& =(I_{\alpha }\ast {|u|}^p{)|u|}^{p-2}u+\beta v\hfill & \hfill & \mathrm{in}{\mathbb{R}}^N,\hfill \\ \hfill {(-\mathsf{\Delta })}^{s}v+{\lambda }_{2}v& =(I_{\alpha }\ast {|v|}^p{)|v|}^{p-2}v+\beta u\hfill & \hfill & \mathrm{in}{\mathbb{R}}^N\hfill \end{array}\right.$$

(3)

is the case of the system (1) with $p=q$. When $s\in (0,1)$, the fractional Laplace operator ${(-\mathsf{\Delta })}^s$ introduced by Laskin [13] as an extension of the classical local Laplace operator in the study of the nonlinear Schrödinger equation, replaces the path integral over Brownian motion with Lévy flights [14]. The operator has specific applications in a wide range of fields (see [1,10] and the references therein). In recent years, many researchers have begun to use the well-known Hardy–Littlewood–Sobolev inequality to study Choquard-type equations. Usually, $1+\frac{\alpha }{N}$ is called the lower critical exponent, $\frac{N+\alpha }{N-2s}$ is the upper critical exponent in the sense of the Hardy–Littlewood–Sobolev inequality, and $2_s^{\ast }:=\frac{2N}{N-2s}$ is the critical Sobolev exponent. When $\beta$ reaches zero, the systems (1) and (3) are reduced to the same single equation below:

$${(-\mathsf{\Delta })}^{s}u+\lambda u=(I_{\alpha }\ast {|u|}^p{)|u|}^{p-2}u.$$

In [15], we studied the existence of positive ground state solutions for a Choquard-type system (3) when $p\in (1+\frac{\alpha }{N},\frac{N+\alpha }{N-2s})$ and the asymptotic behavior of solutions when $\beta$ reaches zero (see [15], namely Theorem 1.1). Nonexistence results for the nontrivial solutions of system (3) were also obtained in the ranges $p\le 1+\frac{\alpha }{N}$ and $p\ge \frac{N+\alpha }{N-2s}$. Thus, a natural question is the following: What happens if $p\ne q$ in the subcritical case and in all critical cases? In order to answer this question, in this paper, we consider the existence result for the solutions of system (1) in the subcritical case and in all critical cases. We establish the main results of this paper in the following cases:

Hypothesis 1 (H1).

“Subcritical” case, where $1+\frac{\alpha }{N}<p<\frac{\alpha +N}{N-2s}\mathrm{and}1+\frac{\alpha }{N}<q<\frac{\alpha +N}{N-2s}$;

Hypothesis 2 (H2).

“Half-critical” case 1, where $1+\frac{\alpha }{N}<min\{p,q\}<\frac{\alpha +N}{N-2s}\mathrm{and}max\{p,q\}=\frac{\alpha +N}{N-2s}$;

Hypothesis 3 (H3).

“Half-critical” case 2, where $min\{p,q\}=1+\frac{\alpha }{N}\mathrm{and}1+\frac{\alpha }{N}<max\{p,q\}<\frac{\alpha +N}{N-2s}$;

Hypothesis 4 (H4).

“Doubly critical” case, where $min\{p,q\}=1+\frac{\alpha }{N}\mathrm{and}max\{p,q\}=\frac{\alpha +N}{N-2s}$;

Hypothesis 5 (H5).

“Inferior or superior supercritical” case, where $p,q\le 1+\frac{\alpha }{N}\mathrm{or}p,q\ge \frac{\alpha +N}{N-2s}$.

To the best of our knowledge, many studies on systems with Choquard terms have considered the case where $p=q$ in subcritical, critical or supercritical cases, whereas we study the case where p and q are not equal in a subcritical case as well as in all critical cases.

Outline of the Paper

The rest of this paper is organized as follows. In Section 2, we introduce some functional spaces and notations. The main results of this paper are given in Section 3. Some auxiliary results are given in Section 4, along with some basic properties of the Nehari manifold ${\mathcal{N}}_{\alpha }$ and energy functional $E_{\alpha }$, which play an important role in this paper. In Section 5, we obtain the existence of ground state solutions of the system (1) in the subcritical case and in all critical cases. In Section 6, we work on proving Theorem 3 using the Pohožaev identity type.

2. Functional Spaces and Notation

Throughout this paper, for any $1\le t<\infty$, we denote the norm of $L^t({\mathbb{R}}^N)$ as ${∥·∥}_t$. The Hilbert space $H^s({\mathbb{R}}^N)$ is defined as

$$H^s({\mathbb{R}}^N):=\left\{u\in L^2({\mathbb{R}}^N):{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{{|u(x)-u(y)|}^2}{{|x-y|}^{N+2s}}dxdy<+\infty \right\}$$

with the scalar product and norm given by

$$\begin{array}{cc}\hfill ⟨u,v⟩& :={\int }_{{\mathbb{R}}^N}{(-\mathsf{\Delta })}^{\frac{s}{2}}u{(-\mathsf{\Delta })}^{\frac{s}{2}}vdx+{\int }_{{\mathbb{R}}^N}uvdx,\hfill \\ \hfill ∥u∥& :={(\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^2+{\parallel u\parallel }_2^2{)}^{\frac{1}{2}},\hfill \end{array}$$

where

$${\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^2:=\frac{C(N,s)}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{{|u(x)-u(y)|}^2}{{|x-y|}^{N+2s}}dxdy.$$

$H_r^s({\mathbb{R}}^N)$ is the radial space of $H^s({\mathbb{R}}^N)$, defined by

$$H_r^s({\mathbb{R}}^N):=\{u\in H^s({\mathbb{R}}^N)|u(x)=u(|x|)\}$$

with the $H^s({\mathbb{R}}^N)$ norm.

Let

$${\parallel u\parallel }_{{\lambda }_i}^2{:=\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^2+{\lambda }_i{\parallel u\parallel }_2^{2}i=1,2,\forall u\in H^s({\mathbb{R}}^N).$$

It is easy to obtain that ${\parallel ·\parallel }_{{\lambda }_i}$ and $\parallel ·\parallel$ are equivalent norms in $H^s({\mathbb{R}}^N)$. We denote $H:=H^s({\mathbb{R}}^N)\times H^s({\mathbb{R}}^N)$ and $H_r:=H_r^s({\mathbb{R}}^N)\times H_r^s({\mathbb{R}}^N)$. The norm of H is given by

$${\parallel (u,v)\parallel }_H^2={\parallel u\parallel }_{{\lambda }_1}^2+{\parallel v\parallel }_{{\lambda }_2}^2\mathrm{for}\mathrm{all}(u,v)\in H.$$

The energy functional $E_{\alpha }$ of (1) is

$$\begin{array}{cc}\hfill E_{\alpha }(u,v)=& \frac{1}{2}{\int }_{{\mathbb{R}}^N}\left[{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u|}^2{+|(-\mathsf{\Delta })}^{\frac{s}{2}}{v|}^2+{\lambda }_1{|u|}^2+{\lambda }_2{|v|}^2\right]dx-\beta {\int }_{{\mathbb{R}}^N}uvdx\hfill \\ \hfill & -\frac{1}{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast {|u|}^p{)|u|}^{p}dx-\frac{1}{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast {|v|}^q{)|v|}^{q}dx\hfill \end{array}$$

(4)

for all $(u,v)\in H$. Under the Hardy–Littlewood–Sobolev inequality, it is easy to obtain that $E_{\alpha }\in C^1(H,\mathbb{R})$ and

$$\begin{array}{cc}\hfill ⟨E_{\alpha }^{\prime }(u,v),(\phi ,\psi )⟩=& {\int }_{{\mathbb{R}}^N}\left[{(-\mathsf{\Delta })}^{\frac{s}{2}}u{(-\mathsf{\Delta })}^{\frac{s}{2}}\phi +{(-\mathsf{\Delta })}^{\frac{s}{2}}v{(-\mathsf{\Delta })}^{\frac{s}{2}}\psi +{\lambda }_{1}u\phi +{\lambda }_{2}v\psi \right]dx\hfill \\ \hfill & -{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p-2}u\phi dx-{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q-2}v\psi dx\hfill \\ \hfill & -\beta {\int }_{{\mathbb{R}}^N}(v\phi +u\psi )dx\hfill \end{array}$$

for any $(\phi ,\psi )\in H$.

In addition, $(u_0,v_0)$ is called a nontrivial solution of system (1) if $u_0,v_0\in H^s({\mathbb{R}}^N)\setminus \left\{0\right\}$ and $(u_0,v_0)$ satisfies $⟨E_{\alpha }^{\prime }(u_0,v_0),(\phi ,\psi )⟩=0$ for any $(\phi ,\psi )\in H$. A nontrivial solution $(u_0,v_0)$ of system (1) is called a positive ground state solution if $u_0>0$, $v_0>0$ and

$$E_{\alpha }(u_0,v_0)=inf\{E_{\alpha }(u,v):(u,v)\in H\setminus \left\{(0,0)\right\}\mathrm{and}{E}_{\alpha }^{\prime }(u,v)=0\}.$$

In order to find the positive ground state solutions of system (1), we introduce the Nehari manifold:

$${\mathcal{N}}_{\alpha }=\{(u,v)\in H\setminus \left\{(0,0)\right\}:⟨E_{\alpha }^{\prime }(u,v),(u,v)⟩=0\}.$$

Under the Nehari manifold constraint, we investigate the existence of critical points of $E_{\alpha }$, which are defined in Equation (4). We define

$$m_{\alpha }=inf\{E_{\alpha }(u,v):(u,v)\in {\mathcal{N}}_{\alpha }\}.$$

3. Main Results

We now give the results for the existence of the subcritical case:

Theorem 1.

Suppose that $N>2s,s\in (0,1),\alpha \in (0,N)$ and p, q satisfy ($H_1$). Then, system (1) possesses a positive radial ground state solution $(u_0,v_0)\in {\mathcal{N}}_{\alpha }$ with $E_{\alpha }(u_0,v_0)=m_{\alpha }>0$ for any $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$.

Our second main result concerns the existence of positive ground state solutions for all critical cases:

Theorem 2.

Suppose that $N>2s,s\in (0,1),\alpha \in (0,N)$ and p, q satisfy ($H_2$), ($H_3$) or ($H_4$). Then, there exists $a_0>0$ such that system (1) has a positive radial ground state solution that holds for any ${\alpha }_0<\alpha <N$ and $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$.

By using the Pohožaev identity, we can obtain the following nonexistence result:

Theorem 3.

If p, q satisfies ($H_5$), then the systems in system (1) has no nontrivial solution for any $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$.

4. Preliminary Results

First of all, we recall the fractional Sobolev embedding lemma below:

Lemma 1

(see [10]). The space $H^s({\mathbb{R}}^N)$ is continuously embedded into $L^t({\mathbb{R}}^N)$ for every $t\in [2,2_s^{\ast }]$. The space $H_r^s({\mathbb{R}}^N)$ is compactly embedded in $L^t({\mathbb{R}}^N)$ for every $t\in (2,2_s^{\ast })$.

Remark 1.

In particular, if $1+\frac{\alpha }{N}<p<\frac{\alpha +N}{N-2s}$, then we have that $2<\frac{2Np}{N+\alpha }<2_s^{\ast }$, with the space $H_r^s({\mathbb{R}}^N)$ compactly embedded into $L^{\frac{2Np}{N+\alpha }}({\mathbb{R}}^N)$.

The following Hardy–Littlewood–Sobolev inequality will be used frequently in this paper:

Lemma 2

(see [16]). Let $r,q>1$, $0<\alpha <N$ and $1\le s<t<\infty$ be such that

$$\frac{1}{r}+\frac{1}{q}=1+\frac{\alpha }{N},\frac{1}{s}-\frac{1}{t}=\frac{\alpha }{N}.$$

For any $u\in L^r({\mathbb{R}}^N)$ and $v\in L^q({\mathbb{R}}^N)$, we have

$$\left|{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast u)v\right|\le {C(N,\alpha ,q)\parallel u\parallel }_r{\parallel v\parallel }_q,$$

(5)

where the sharp constant $C(N,\alpha ,p)$ is

$$C(N,\alpha ,p)=C_{\alpha }(N)={\pi }^{\frac{N-\alpha }{2}}\frac{\mathsf{\Gamma }(\frac{\alpha }{2})}{\mathsf{\Gamma }(\frac{N+\alpha }{2})}{\left\{\frac{\mathsf{\Gamma }(\frac{N}{2})}{\mathsf{\Gamma }(N)}\right\}}^{-\frac{\alpha }{N}}.$$

Remark 2.

If $p\in [1+\frac{\alpha }{N},\frac{\alpha +N}{N-2s}]$ and $r=q=\frac{2N}{N+\alpha }$, then

$$\left|{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^p\right|\le C(N,\alpha ,p){\parallel u\parallel }_{\frac{2Np}{N+\alpha }}^{2p}.$$

(6)

Next, we present the results that are crucial in the proof of this paper:

Lemma 3

(see [17]). Assume that $N\in \mathbb{N}$, $0<\alpha <N$ and $p\in (1+\frac{\alpha }{N},\frac{\alpha +N}{N-2s})$. Let $\left\{u_n\right\}\subset H_r^s({\mathbb{R}}^N)$ be a sequence satisfying that $u_n\rightharpoonup u$ weakly in $H_r^s({\mathbb{R}}^N)$ as $n\to \infty$. Then, we have

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^p={\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^p.$$

(7)

Lemma 4

(see [18]). Let f, g and h be three non-negative Lebesgue measurable functions on ${\mathbb{R}}^N$. Let

$$W(f,g,h):={\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}f(x)g(y)h(x-y)dxdy,$$

Then, we obtain

$$W(f^{\ast },g^{\ast },h^{\ast })\ge W(f,g,h),$$

where $f^{\ast }$, $g^{\ast }$ and $h^{\ast }$ denote the symmetric radial decreasing rearrangement of f, g and h.

Next, we introduce the basic properties of the Nehari manifold ${\mathcal{N}}_{\alpha }$ and $E_{\alpha }$:

Lemma 5.

For all fixed values where $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$, let $p,q\in [1+\frac{\alpha }{N},\frac{N+\alpha }{N-2s}]$. The following assertions hold:

(1) 

There exists $c>0$ such that ${\parallel (u,v)\parallel }_H\ge c$ for any $(u,v)\in {\mathcal{N}}_{\alpha }$;

(2) 

$m_{\alpha }={inf}_{(u,v)\in {\mathcal{N}}_{\alpha }}{E}_{\alpha }(u,v)>0$;

(3) 

${\mathcal{N}}_{\alpha }$ is a $C^1$ manifold;

(4) 

If $(u_0,v_0)$ is a critical point of $E_{\alpha }$ on ${\mathcal{N}}_{\alpha }$, then $(u_0,v_0)\in H\setminus \left\{(0,0)\right\}$, and $(u_0,v_0)$ is a critical point of $E_{\alpha }$.

Proof. 

(1)

According to the definition of ${\mathcal{N}}_{\alpha }$, using the Hardy–Littlewood–Sobolev inequality in Equation (6) and Lemma 1, for any $(u,v)\in {\mathcal{N}}_{\alpha }$, we have

$$\begin{array}{cc}\hfill {\parallel (u,v)\parallel }_H^2& ={\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx+{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx+2\beta {\int }_{{\mathbb{R}}^N}uvdx\hfill \\ \hfill & \le {C(N,\alpha ,p,q)(\parallel u\parallel }_{\frac{2Np}{N+\alpha }}^{2p}+{\parallel v\parallel }_{\frac{2Nq}{N+\alpha }}^{2q})+\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\left(2\sqrt{{\lambda }_1{\lambda }_2}{\int }_{{\mathbb{R}}^N}uvdx\right)\hfill \\ \hfill & \le C_1{C(N,\alpha ,p,q)(\parallel u\parallel }_{{\lambda }_1}^{2p}+{\parallel v\parallel }_{{\lambda }_2}^{2q})+\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\left[{\int }_{{\mathbb{R}}^N}({\lambda }_1{u}^2+{\lambda }_2{v}^2)dx\right]\hfill \\ \hfill & \le C_1{C(N,\alpha ,p,q)(\parallel (u,v)\parallel }_H^{2p}+{\parallel (u,v)\parallel }_H^{2q})+\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}{(\parallel u\parallel }_{{\lambda }_1}^2+{\parallel v\parallel }_{{\lambda }_2}^2),\hfill \end{array}$$

where $C_1>0$ denotes the fractional Sobolev embedding constant and $C_1$ does not depend on u or v. Hence, we can deduce that

$$\frac{1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}}{C_{1}C(N,\alpha ,p,q)}\le {\parallel (u,v)\parallel }_H^{2p-2}+{\parallel (u,v)\parallel }_H^{2q-2}.$$

Since $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$, then $\frac{1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}}{C_{1}C(N,\alpha ,p,q)}>0$, and we obtain (1).

(2)

For any $(u,v)\in {\mathcal{N}}_{\alpha }$, we have that

$$E_{\alpha }(u,v)=\left(\frac{1}{2}-\frac{1}{2p}\right)\left({\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uv\right)+\left(\frac{1}{2p}-\frac{1}{2q}\right){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^q.$$

Without loss of generality, we may assume that $1+\frac{\alpha }{N}\le p\le q$, and the case where $p>q$ can be proven similarly. Since $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$, it follows from (1) that

$$\begin{array}{cc}\hfill E_{\alpha }(u,v)\ge & \left(\frac{1}{2}-\frac{1}{2p}\right)\left({\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uv\right)\hfill \\ \hfill \ge & \left(\frac{1}{2}-\frac{1}{2p}\right)\left(1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\right){\parallel (u,v)\parallel }_H^2\hfill \\ \hfill \ge & \left(\frac{1}{2}-\frac{1}{2p}\right)(1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}})c^2>0,\hfill \end{array}$$

(8)

which implies $m_{\alpha }>0$ and $E_{\alpha }$ is coercive on ${\mathcal{N}}_{\alpha }$.

(3)

Consider the $C^1$ functional $J_{\alpha }:H\setminus \left\{(0,0)\right\}\to \mathbb{R}$ defined by

$$\begin{array}{cc}\hfill J_{\alpha }(u,v)=& ⟨E_{\alpha }^{\prime }(u,v),(u,v)⟩\hfill \\ \hfill =& {\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx-{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx-{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx.\hfill \end{array}$$

Obviously, ${\mathcal{N}}_{\alpha }=J_{\alpha }^{-1}(0)$. If $(u,v)\in {\mathcal{N}}_{\alpha }$, then we have

$$\begin{array}{cc}\hfill & ⟨J_{\alpha }^{\prime }(u,v),(u,v)⟩\hfill \\ \hfill =& 2\left({\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx\right)-2p{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx-2q{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx\hfill \\ \hfill =& (2-2p)\left({\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx\right)+(2p-2q){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx.\hfill \end{array}$$

Here, we also assume that $p\le q$, and the case where $p>q$ can be proven similarly. By combining (1) with $1+\frac{\alpha }{N}\le p\le q$, we see that

$$\begin{array}{cc}\hfill ⟨J_{\alpha }^{\prime }(u,v),(u,v)⟩& \le (2-2p)\left(1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\right){\parallel (u,v)\parallel }_H^2\hfill \\ \hfill & \le (2-2p)\left(1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\right)c^2<0.\hfill \end{array}$$

(9)

Hence, ${\mathcal{N}}_{\alpha }$ is a $C^1$ manifold.

(4)

If $(u_0,v_0)$ is a critical point of $E_{\alpha }$ on ${\mathcal{N}}_{\alpha }$, then $(u_0,v_0)$ is a solution to the optimization problem

$$\underset{J_{\alpha }(u,v)=0}{min}{E}_{\alpha }(u,v).$$

Under Lagrange multiplier theory, we know that there exists a constant $\mu$ such that $E_{\alpha }^{\prime }(u_0,v_0)=\mu J_{\alpha }^{\prime }(u_0,v_0)$. Thus, we find

$$⟨E_{\alpha }^{\prime }(u_0,v_0),(u_0,v_0)⟩=\mu ⟨J_{\alpha }^{\prime }(u_0,v_0),(u_0,v_0)⟩.$$

(10)

Since $(u_0,v_0)\in {\mathcal{N}}_{\alpha }$, using Equations (9) and (10), we have that $\mu =0$. Therefore, $E_{\alpha }^{\prime }(u_0,v_0)=0$.

□

Lemma 6.

Let $p,q\in [1+\frac{\alpha }{N},\frac{N+\alpha }{N-2s}]$ and $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$. For any $(u,v)\in H\setminus \{(0,0)\}$, there exists a unique $t_{\alpha }>0$ such that

$$(t_{\alpha }u,t_{\alpha }v)\in {\mathcal{N}}_{\alpha }\mathrm{and}{E}_{\alpha }(t_{\alpha }u,t_{\alpha }v)=\underset{t\ge 0}{max}{E}_{\alpha }(tu,tv).$$

Proof. 

For a fixed $(u,v)\in H\setminus \{(0,0)\}$, let the function $\gamma :[0,\infty )\to \mathbb{R}$ be defined as $\gamma (t)=E_{\alpha }(tu,tv)$. It is easy to see that $⟨E_{\alpha }^{\prime }(tu,tv),(tu,tv)⟩=t{\gamma }^{\prime }(t)$. Notice that $(t_{\alpha }u,t_{\alpha }v)\in {\mathcal{N}}_{\alpha }$ if and only if $t_{\alpha }$ is a positive critical point of $\gamma$. Since $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$, we have

$${\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uv\ge \left(1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\right){\parallel (u,v)\parallel }_H^2>0$$

for all $(u,v)\in H$. Moreover, $p,q\ge 1+\frac{\alpha }{N}$ and

$$\begin{array}{cc}\hfill \gamma (t)=& \frac{t^2}{2}\left({\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx\right)-\frac{t^{2p}}{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast {|u|}^p{)|u|}^{p}dx\hfill \\ \hfill & -\frac{t^{2q}}{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast {|v|}^q{)|v|}^{q}dx,\hfill \end{array}$$

Thus, when $t>0$ is sufficiently large, we conclude that $\gamma (t)<0$. On the other hand, through Lemma 1, Equation (6) and $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$, we obtain that

$$\gamma (t)\ge \left(1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\right)\frac{t^2}{2}{\parallel (u,v)\parallel }_H^2-C_1\frac{t^{2p}}{2p}{\parallel u\parallel }_{{\lambda }_1}^{2p}-C_2\frac{t^{2q}}{2q}{\parallel v\parallel }_{{\lambda }_2}^{2q}>0$$

provided that $t>0$ is sufficiently small, where $C_1$ and $C_2$ denote the fractional Sobolev embedding constants. Thus, by combining $\gamma (0)=0$, we find that $\gamma$ has its maximum points in $(0,\infty )$. Suppose the maximum points of $\gamma$ are not unique (i.e., there exists $t_1$, $t_2>0$ with $t_1<t_2$ such that ${\gamma }^{\prime }(t_1)={\gamma }^{\prime }(t_2)=0$), since every critical point of $\gamma$ satisfies

$${\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uv=t^{2p-2}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx+t^{2q-2}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx,$$

which implies that

$$(t_1^{2p-2}-t_2^{2p-2}){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx+(t_1^{2q-2}-t_2^{2q-2}){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx=0.$$

Thus, $u=v=0$, and we find a contradiction. □

Lemma 7.

If $u\in H^s({\mathbb{R}}^N)\setminus \left\{0\right\}$, then ${\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx\to \infty$ as $\alpha \to N$.

Proof. 

Since $\mathsf{\Gamma }(\frac{N-\alpha }{2})\to \infty$ as $\alpha \to N$, we deduce that $I_{\alpha }\to \infty$ as $\alpha \to N$. □

Lemma 8.

Let $p,q\in [1+\frac{\alpha }{N},\frac{N+\alpha }{N-2s}]$. For a given $\epsilon >0$, there exists ${\alpha }_0>0$ such that $m_{\alpha }<\epsilon$, which holds for all ${\alpha }_0<\alpha <N$.

Proof. 

Under Lemma 6, if $u,v\in H^s({\mathbb{R}}^N)\setminus \left\{0\right\}$, then there exists $t_{\alpha }>0$ such that $(t_{\alpha }u,t_{\alpha }v)\in {\mathcal{N}}_{\alpha }$. Hence, we have

$${\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uv=t_{\alpha }^{2p-2}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx+t_{\alpha }^{2q-2}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx.$$

(11)

By combining Lemma 7 and Equation (11), we have that $t_{\alpha }\to 0$ as $\alpha \to N$. Thus, there exists ${\alpha }_0>0$ such that

$$0<m_{\alpha }\le E_{\alpha }(t_{\alpha }u,t_{\alpha }v)\le \frac{t_{\alpha }^2}{2}{\parallel (u,v)\parallel }_H^2<\epsilon$$

for all ${\alpha }_0<\alpha <N$. □

Lemma 9

(see [17]). If we let $r\in [2,2_s^{\ast }]$, then

$${\parallel u\parallel }_r{\le C(\alpha ,N,s)\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^{\theta }{\parallel u\parallel }_2^{(1-\theta )},$$

where θ, satisfying $\frac{1}{r}=\frac{\theta }{2_s^{\ast }}+\frac{1-\theta }{2}$ and $C(\alpha ,N,s)>0$, denotes the optimal constant depending only on α, N and s.

Lemma 10.

Let ${\mathcal{S}}_1$ be the constant defined by

$${\mathcal{S}}_1=\underset{u\in H^s({\mathbb{R}}^N)\setminus \left\{0\right\}}{inf}\frac{{\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^2}{{\left[{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast {|u|}^{2_{\alpha ,s}^{\ast }}{)|u|}^{2_{\alpha ,s}^{\ast }}dx\right]}^{\frac{1}{2_{\alpha ,s}^{\ast }}}},$$

(12)

where $2_{\alpha ,s}^{\ast }:=\frac{N+\alpha }{N-2s}$ is the critical index in the sense of the Hardy–Littlewood–Sobolev inequality, in which ${\mathcal{S}}_1>0$.

Proof. 

Notice that if we let $f\in L^r({\mathbb{R}}^N)$, then under the Hardy–Littlewood–Sobolev inequality in Equation (5), we obtain

$${\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast f)fdx\le C(N,\alpha ,r){\parallel f\parallel }_r^2,$$

where $\frac{2}{r}=1+\frac{\alpha }{N}$ and $r=\frac{2N}{N+\alpha }$. If we let $f=u^{2_{\alpha ,s}^{\ast }}$, then we have that $u\in L^{2_s^{\ast }}({\mathbb{R}}^N)$ and

$${\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^{2_{\alpha ,s}^{\ast }}{)|u|}^{2_{\alpha ,s}^{\ast }}dx\le C(N,\alpha )\parallel u^{2_{\alpha ,s}^{\ast }}{\parallel }_{\frac{2N}{N+\alpha }}^2=C(N,\alpha ){\parallel u\parallel }_{2_s^{\ast }}^{2·2_{\alpha ,s}^{\ast }},$$

where

$${\left({\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^{2_{\alpha ,s}^{\ast }}{)|u|}^{2_{\alpha ,s}^{\ast }}dx\right)}^{\frac{1}{2_{\alpha ,s}^{\ast }}}\le C{(N,\alpha )}^{\frac{1}{2_{\alpha ,s}^{\ast }}}{\parallel u\parallel }_{2_s^{\ast }}^2.$$

(13)

In light of Lemma 9, we have

$${\parallel u\parallel }_{2_s^{\ast }}\le C(\alpha ,N,s){\parallel {(-\mathsf{\Delta })}^{\frac{s}{2}}u\parallel }_2.$$

Hence, under the Hardy–Littlewood–Sobolev inequality in Equation (13), we can see that

$${\mathcal{S}}_1\ge \frac{1}{C{(N,\alpha )}^{\frac{1}{2_{\alpha ,s}^{\ast }}}}\underset{u\in H^s({\mathbb{R}}^N)\setminus \left\{0\right\}}{inf}\frac{{\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^2}{{\parallel u\parallel }_{2_s^{\ast }}^2}=\frac{1}{C{(\alpha ,N,s)}^{2}C{(N,\alpha )}^{\frac{1}{2_{\alpha ,s}^{\ast }}}}>0.$$

The proof is thereby complete. □

Lemma 11.

Let ${\mathcal{S}}_2$ be the constant defined by

$${\mathcal{S}}_2=\underset{u\in H^s({\mathbb{R}}^N)\setminus \left\{0\right\}}{inf}\frac{{\parallel u\parallel }_2^2}{{\left[{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast {|u|}^{\frac{N+\alpha }{N}}{)|u|}^{\frac{N+\alpha }{N}}dx\right]}^{\frac{N}{N+\alpha }}},$$

(14)

Then, ${\mathcal{S}}_2>0$.

Proof. 

The proof is similar to that for Lemma 10, and thus we omitted it here. □

5. Ground State Solutions

First, we investigate the existence of positive ground state solutions in the subcritical case.

Proof of Theorem 1.

The proof of Theorem 1 can be given by modifying the proof of Theorem 1.1 in [15]. Since we consider the case where $p\ne q$ in this paper, our proof for the existence of a positive radial ground state solution in the subcritical case differs in two places from the proof of Theorem 1.1 in [15]. One is that if $\left\{(u_n,v_n)\right\}\subseteq {\mathcal{N}}_{\alpha }$ is a minimizing sequence for $E_{\alpha }$, specifically such that $E_{\alpha }(u_n,v_n)\to m_{\alpha }$, then we claim that there exists $0<t_n\le 1$ such that $(t_n{u}_n,t_n{v}_n)\in {\mathcal{N}}_{\alpha }$. Here, we consider only the case where $p\le q$. In fact, the case where $p>q$ can be proven in a similar fashion. By contradiction, assume that $t_n>1$, and we find that

$$\begin{array}{cc}\hfill t_n^2\left(\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_n\right)=& t_n^{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^p+t_n^{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^q\hfill \\ \hfill \ge & t_n^{2p}\left({\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast |u_n{|}^p)|u_n{|}^p+{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast |v_n{|}^q)|v_n{|}^q\right).\hfill \end{array}$$

Hence, we have

$$t_n^{2p-2}\le \frac{\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_n}{{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^p+{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^q}=1.$$

Since $1+\frac{\alpha }{N}<p$, we have that $2p-2>2+\frac{2\alpha }{N}-2=\frac{2\alpha }{N}>0$. This is a contradiction, and thus $0<t_n\le 1$.

Next, similar to the proof in [15], we can also prove that the minimizing sequence $\left\{(u_n,v_n)\right\}$ is bounded, non-negative and radial and that there exists $(u_0,v_0)\in H$ with $u_0\ge 0$ and $v_0\ge 0$ such that up to the subsequences, $(u_n,v_n)\rightharpoonup (u_0,v_0)$ weakly in H. In light of Lemma 3, we have that $u_0\not\equiv 0$ or $v_0\not\equiv 0$. By combining Lemma 3 and the Fatou lemma, we obtain

$$\parallel u_0{\parallel }_{{\lambda }_1}^2+\parallel v_0{\parallel }_{{\lambda }_2}^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_0{v}_0\le {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_0{|}^p)|u_0{|}^p+{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_0{|}^q)|v_0{|}^q.$$

Under Lemma 6, there is $t>0$ such that $(t{u}_0,t{v}_0)\in {\mathcal{N}}_{\alpha }$. We claim that $0<t\le 1$. The proof about the claim here is different from that in [15]. By contradiction, assume that $t>1$. Due to $(t{u}_0,t{v}_0)\in {\mathcal{N}}_{\alpha }$, we obtain

$$\begin{array}{cc}\hfill t^2\left(\parallel (u_0,v_0){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_0{v}_0\right)=& t^{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_0{|}^p)|u_0{|}^p+t^{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_0{|}^q)|v_0{|}^q\hfill \\ \hfill \ge & t^{2p}\left({\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast |u_0{|}^p)|u_0{|}^p+{\int }_{{\mathbb{R}}^N}(I_{\alpha }\ast |v_0{|}^q)|v_0{|}^q\right).\hfill \end{array}$$

Thus, we have

$$t^{2p-2}\le \frac{\parallel (u_0,v_0){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_0{v}_0}{{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_0{|}^p)|u_0{|}^p+{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_0{|}^q)|v_0{|}^q}=1,$$

which is a contradiction. Similar to the proof of Theorem 1.1 in [15], we can obtain that $t=1$ and $(u_0,v_0)$ is a positive and radial ground state solution of system (1). □

Next, we consider the existence of positive ground state solutions in the all critical cases.

Proof of Theorem 2.

By using Ekeland’s variational principle (see [19]), we can obtain a minimizing sequence $\left\{(u_n,v_n)\right\}\subseteq {\mathcal{N}}_{\alpha }$ of $E_{\alpha }$ satisfying

$$E_{\alpha }(u_n,v_n)\to m_{\alpha }\mathrm{and}{E}_{\alpha }^{\prime }(u_n,v_n)\to 0.$$

(15)

Since $E_{\alpha }$ is coercive on ${\mathcal{N}}_{\alpha }$, we have that ${\left\{(u_n,v_n)\right\}}_{n\in \mathbb{N}}$ is bounded in H. In light of Lemma 6, there exists $t_n>0$ such that $(t_n|u_n|,t_n|v_n|)\in {\mathcal{N}}_{\alpha }$. Thus, we have

$$\begin{array}{cc}\hfill m_{\alpha }& \le E_{\alpha }(t_n|u_n|,t_n|v_n|)\hfill \\ \hfill & =\left(\frac{1}{2}-\frac{1}{2p}\right)t_n^{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^p+\left(\frac{1}{2}-\frac{1}{2q}\right)t_n^{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^q\hfill \\ \hfill & =E_{\alpha }(t_n{u}_n,t_n{v}_n)\le \underset{t\ge 0}{max}{E}_{\alpha }(t{u}_n,t{v}_n)=E_{\alpha }(u_n,v_n)=m_{\alpha }+o_n(1).\hfill \end{array}$$

Therefore, we can assume that $u_n\ge 0$ and $v_n\ge 0$. Let $u_n^{\ast }$ and $v_n^{\ast }$ denote the symmetric decreasing rearrangement of $u_n$ and $v_n$, respectively. Under Lemma 4 with $f(x)=|u_n{(x)|}^p$, $g(y)=|u_n{(y)|}^p$ and $h(x-y)={|x-y|}^{\alpha -N}$, we have

$${\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n^{\ast }{|}^p)|u_n^{\ast }{|}^p\ge {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^p.$$

(16)

Similarly, we obtain

$${\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n^{\ast }{|}^q)|v_n^{\ast }{|}^q\ge {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^q.$$

(17)

In addition, the following is well known (see Theorem 3 in [20]):

$${\int }_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u}_n^{\ast }{|}^2\le {\int }_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u}_n{|}^2\mathrm{and}{\int }_{{\mathbb{R}}^N}|u_n^{\ast }{|}^2={\int }_{{\mathbb{R}}^N}{|u_n|}^2$$

(18)

Under the Hardy–Littlewood inequality and Riesz rearrangement inequality (see [20]), we have

$${\int }_{{\mathbb{R}}^N}{u}_n^{\ast }{v}_n^{\ast }\ge {\int }_{{\mathbb{R}}^N}{u}_n{v}_n.$$

(19)

Under Equations (16)–(19), we have

$$\begin{array}{cc}\hfill & E_{\alpha }(u_n^{\ast },v_n^{\ast })\hfill \\ \hfill =& \frac{1}{2}(\parallel u_n^{\ast }{\parallel }_{{\lambda }_1}^2+\parallel v_n^{\ast }{\parallel }_{{\lambda }_2}^2)-\frac{1}{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n^{\ast }{|}^p)|u_n^{\ast }{|}^p-\frac{1}{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n^{\ast }{|}^q)|v_n^{\ast }{|}^q-\beta {\int }_{{\mathbb{R}}^N}{u}_n^{\ast }{v}_n^{\ast }\hfill \\ \hfill \le & \frac{1}{2}(\parallel u_n{\parallel }_{{\lambda }_1}^2+\parallel v_n{\parallel }_{{\lambda }_2}^2)-\frac{1}{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^p-\frac{1}{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^q-\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_n\hfill \\ \hfill =& E_{\alpha }(u_n,v_n).\hfill \end{array}$$

Therefore, we can further assume that $(u_n,v_n)\in H^r$. Since $\left\{(u_n,v_n)\right\}$ is bounded in H, there exists $(u_0,v_0)\in H$ where $u_0\ge 0$ and $v_0\ge 0$ such that up to the subsequences, $(u_n,v_n)\rightharpoonup (u_0,v_0)$ weakly in H. Moreover, we can also assume that $u_n\to u_0$, $v_n\to v_0$ almost everywhere in ${\mathbb{R}}^N$ and the minimizing sequence $(u_n,v_n)$ is bounded in $H^r$. In fact, since max $\{\frac{1}{2p},\frac{1}{2q}\}<\frac{1}{2}$, by taking $\eta$ with max $\{\frac{1}{2p},\frac{1}{2q}\}<\eta <\frac{1}{2}$, we deduce that

$$m_{\alpha }+o_n(1)=E_{\alpha }(u_n,v_n)-\eta ⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩\ge (\frac{1}{2}-\eta )\left(1-\frac{\beta }{\sqrt{{\lambda }_1{\lambda }_2}}\right){\parallel (u_n,v_n)\parallel }_H^2.$$

Thus, up to a subsequence, $(u_n,v_n)\rightharpoonup (u_0,v_0)$ weakly in $H^r$.

Through the standard density arguments, $E_{\alpha }^{\prime }(u_0,v_0)=0$. If $(u_0,v_0)\ne (0,0)$, then $(u_0,v_0)\in {\mathcal{N}}_{\alpha }$ and $m_{\alpha }\le E_{\alpha }(u_0,v_0)$. Furthermore, according to Fatou’s lemma, we can obtain

$$\begin{array}{cc}\hfill & m_{\alpha }+o_n(1)\hfill \\ \hfill =& E_{\alpha }(u_n,v_n)-\frac{1}{2}⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩\hfill \\ \hfill =& \left(\frac{1}{2}-\frac{1}{2p}\right){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^{p}dx+\left(\frac{1}{2}-\frac{1}{2q}\right){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^{q}dx\hfill \\ \hfill \ge & \left(\frac{1}{2}-\frac{1}{2p}\right){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_0{|}^p)|u_0{|}^{p}dx+\left(\frac{1}{2}-\frac{1}{2q}\right){\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_0{|}^q)|v_0{|}^{q}dx+o_n(1)\hfill \\ \hfill =& E_{\alpha }(u_0,v_0)+o_n(1).\hfill \end{array}$$

This implies that $(u_0,v_0)$ is a non-negative radial ground state solution to the system (1). Since system (1) has no semi-trivial solution, $(u_0,0)$ and $(0,v_0)$ in particular are not solutions to system (1). We deduce that $(u_0,v_0)$ is a positive and radial ground state solution of system (1).

Next, we prove that $(u_0,v_0)\ne (0,0)$ when $\alpha$ is sufficiently close to N. We divide the remainder of the proof into three cases:

Case 1.

Suppose that p, q satisfies ($H_2$). We can further assume that $p<q$, and the case of $p>q$ is similar. In light of Lemmas 7 and 10, there exists ${\alpha }_0>0$ such that

$$m_{\alpha }<\frac{\alpha +2s}{2(N+\alpha )}{\mathcal{S}}_1^{\frac{N+\alpha }{\alpha +2s}},\mathrm{for}\mathrm{all}{\alpha }_0<\alpha <N.$$

(20)

We claim that $(u_0,v_0)\ne (0,0)$. Next we argue by contradiction while assuming that $(u_0,v_0)=(0,0)$. It is clear from Lemma 3 that

$${\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^{p}dx\to 0,\mathrm{as}n\to \infty .$$

(21)

Notice that

$$\begin{array}{cc}\hfill E_{\alpha }(u_n,v_n)-\frac{1}{2}⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩=& \frac{p-1}{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^{p}dx\hfill \\ \hfill & +\frac{\alpha +2s}{2(N+\alpha )}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^{\frac{N+\alpha }{N-2s}})|v_n{|}^{\frac{N+\alpha }{N-2s}}dx,\hfill \end{array}$$

which, together with Equations (21) and (15), means that

$$\begin{array}{cc}\hfill & \frac{2(N+\alpha )}{\alpha +2s}{m}_{\alpha }+o_n(1)\hfill \\ \hfill =& \frac{2(N+\alpha )}{\alpha +2s}\left(E_{\alpha }(u_n,v_n)-\frac{1}{2}⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩-\frac{p-1}{2p}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^{p}dx\right)\hfill \\ \hfill =& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^{\frac{N+\alpha }{N-2s}})|v_n{|}^{\frac{N+\alpha }{N-2s}}dx.\hfill \end{array}$$

(22)

With Equation (22), we can obtain

$$\begin{array}{cc}\hfill & \frac{2(N+\alpha )}{\alpha +2s}{m}_{\alpha }\hfill \\ \hfill =& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^{\frac{N+\alpha }{N-2s}})|v_n{|}^{\frac{N+\alpha }{N-2s}}dx+o_n(1)\hfill \\ \hfill =& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^{\frac{N+\alpha }{N-2s}})|v_n{|}^{\frac{N+\alpha }{N-2s}}dx+{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^p)|u_n{|}^{p}dx+⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩\hfill \\ \hfill =& \parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx.\hfill \end{array}$$

(23)

Since $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$, we deduce that

$${\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u}_n{\parallel }_2^2+{\int }_{{\mathbb{R}}^N}({\lambda }_1{u}_n^2+{\lambda }_2{v}_n^2-2\beta u_n{v}_n)\ge 0.$$

(24)

Thus, by combining Equations (12) and (22)–(24), we deduce that

$$\begin{array}{cc}\hfill \frac{2(N+\alpha )}{\alpha +2s}{m}_{\alpha }+o_n(1)=& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^{\frac{N+\alpha }{N-2s}})|v_n{|}^{\frac{N+\alpha }{N-2s}}dx\hfill \\ \hfill \le & {\mathcal{S}}_1^{-\frac{N+\alpha }{N-2s}}{\parallel {(-\mathsf{\Delta })}^{\frac{s}{2}}{v}_n\parallel }_2^{2\frac{N+\alpha }{N-2s}}\hfill \\ \hfill \le & {\mathcal{S}}_1^{-\frac{N+\alpha }{N-2s}}{\left(\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx\right)}^{\frac{N+\alpha }{N-2s}}\hfill \\ \hfill =& {\mathcal{S}}_1^{-\frac{N+\alpha }{N-2s}}{\left(\frac{2(N+\alpha )}{\alpha +2s}{m}_{\alpha }+o_n(1)\right)}^{\frac{N+\alpha }{N-2s}}.\hfill \end{array}$$

Hence, we conclude that

$$\frac{2(N+\alpha )}{\alpha +2s}{m}_{\alpha }+o_n(1)\le {\left[\frac{2(N+\alpha )}{(\alpha +2s){\mathcal{S}}_1}{m}_{\alpha }\right]}^{\frac{N+\alpha }{N-2s}}+o_n(1).$$

Thus, $m_{\alpha }\ge \frac{\alpha +2s}{2(N+\alpha )}{\mathcal{S}}_1^{\frac{N+\alpha }{\alpha +2s}}$, which is a contradiction with Equation (20). Therefore, in this case, we conclude that $(u_0,v_0)\ne (0,0)$.

Case 2.

Suppose that p, q satisfies ($H_3$). Similar to case 1, in case 2, we still assume that $p<q$. Notice that

$${\lambda }_1{u}^2+{\lambda }_2{v}^2-2\beta uv={\lambda }_1{u}^2+{\left(\sqrt{{\lambda }_2}v-\frac{\beta }{\sqrt{{\lambda }_2}}u\right)}^2-\frac{{\beta }^2}{{\lambda }_2}{u}^2\ge \left({\lambda }_1-\frac{{\beta }^2}{{\lambda }_2}\right)u^2$$

holds for all $(u,v)\in H$. Hence, we have

$$\begin{array}{cc}\hfill \left({\lambda }_1-\frac{{\beta }^2}{{\lambda }_2}\right){\parallel u\parallel }_2^2& \le {\lambda }_1{\parallel u\parallel }_2^2+{\lambda }_2{\parallel v\parallel }_2^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx\hfill \\ \hfill & \le {\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx,\mathrm{for}\mathrm{all}(u,v)\in H.\hfill \end{array}$$

(25)

It follows from Lemma 8, Equation (14) and $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$ that there exists ${\alpha }_0>0$ such that

$$m_{\alpha }<\frac{\alpha }{2(N+\alpha )}{\left({\lambda }_1-\frac{{\beta }^2}{{\lambda }_2}\right)}^{\frac{N+\alpha }{\alpha }}{\mathcal{S}}_2^{\frac{N+\alpha }{\alpha }},\mathrm{for}\mathrm{all}{\alpha }_0<\alpha <N.$$

(26)

For this case, assume by contradiction that $(u_0,v_0)=(0,0)$. In light of Lemma 3, we have

$${\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^{q}dx\to 0,\mathrm{as}n\to \infty .$$

(27)

Due to $(u_n,v_n)\in {\mathcal{N}}_{\alpha }$, we have

$$\begin{array}{cc}\hfill E_{\alpha }(u_n,v_n)-\frac{1}{2}⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩=& \frac{\alpha }{2(N+\alpha )}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^{\frac{N+\alpha }{N}})|u_n{|}^{\frac{N+\alpha }{N}}dx\hfill \\ \hfill & +\frac{q-1}{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^{q}dx.\hfill \end{array}$$

By combining the above identity with Equations (15) and (27), we can obtain

$$\begin{array}{cc}\hfill & \frac{2(N+\alpha )}{\alpha }{m}_{\alpha }+o_n(1)\hfill \\ \hfill =& \frac{2(N+\alpha )}{\alpha }\left(E_{\alpha }(u_n,v_n)-\frac{1}{2}⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩-\frac{q-1}{2q}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^{q}dx\right)\hfill \\ \hfill =& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^{\frac{N+\alpha }{N}})|u_n{|}^{\frac{N+\alpha }{N}}dx.\hfill \end{array}$$

(28)

Thus, we find that

$$\begin{array}{cc}\hfill & \frac{2(N+\alpha )}{\alpha }{m}_{\alpha }\hfill \\ \hfill =& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^{\frac{N+\alpha }{N}})|u_n{|}^{\frac{N+\alpha }{N}}dx+o_n(1)\hfill \\ \hfill =& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^{\frac{N+\alpha }{N}})|u_n{|}^{\frac{N+\alpha }{N}}dx+{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^q)|v_n{|}^{q}dx+⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩\hfill \\ \hfill =& \parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx.\hfill \end{array}$$

(29)

By using Equations (14), (25), (28) and (29), we deduce that

$$\begin{array}{cc}\hfill \frac{2(N+\alpha )}{\alpha }{m}_{\alpha }+o_n(1)=& {\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^{\frac{N+\alpha }{N}})|u_n{|}^{\frac{N+\alpha }{N}}dx\hfill \\ \hfill \le & {\mathcal{S}}_2^{-\frac{N+\alpha }{N}}{\parallel u_n\parallel }_2^{2\frac{N+\alpha }{N}}\hfill \\ \hfill \le & {\mathcal{S}}_2^{-\frac{N+\alpha }{N}}{\left(\frac{1}{{\lambda }_1-\frac{{\beta }^2}{{\lambda }_2}}\right)}^{\frac{N+\alpha }{N}}{\left(\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx\right)}^{\frac{N+\alpha }{N}}\hfill \\ \hfill =& {\mathcal{S}}_2^{-\frac{N+\alpha }{N}}{\left(\frac{{\lambda }_2}{{\lambda }_1{\lambda }_2-{\beta }^2}\right)}^{\frac{N+\alpha }{N}}{\left[\frac{2(N+\alpha )}{\alpha }{m}_{\alpha }+o_n(1)\right]}^{\frac{N+\alpha }{N}}.\hfill \end{array}$$

Based on the inequality above, we find that

$$\frac{2(N+\alpha )}{\alpha }{m}_{\alpha }+o_n(1)\le {\left[\frac{{\lambda }_2}{({\lambda }_1{\lambda }_2-{\beta }^2){\mathcal{S}}_2}\right]}^{\frac{N+\alpha }{N}}{\left[\frac{2(N+\alpha )}{\alpha }{m}_{\alpha }\right]}^{\frac{N+\alpha }{N}}+o_n(1).$$

Hence, we have $m_{\alpha }\ge \frac{\alpha }{2(N+\alpha )}{\left({\lambda }_1-\frac{{\beta }^2}{{\lambda }_2}\right)}^{\frac{N+\alpha }{\alpha }}{\mathcal{S}}_2^{\frac{N+\alpha }{\alpha }}$, which is a contradiction with Equation (26). Therefore, we conclude that $(u_0,v_0)\ne (0,0)$.

Case 3.

Suppose that p, q satisfies ($H_4$). Without loss of generality, we still assume that $p<q$. It follows from Lemma 8 that there exists ${\alpha }_0>0$ such that

$$m_{\alpha }<min\left\{\frac{1}{2^{\frac{N-2s}{\alpha +2s}}}\frac{\alpha +2s}{2(N+\alpha )}{\mathcal{S}}_1^{\frac{N+\alpha }{\alpha +2s}},\frac{1}{2^{\frac{N}{\alpha }}}\frac{\alpha }{2(N+\alpha )}{\left({\lambda }_1-\frac{{\beta }^2}{{\lambda }_2}\right)}^{\frac{N+\alpha }{\alpha }}{\mathcal{S}}_2^{\frac{N+\alpha }{\alpha }}\right\},$$

(30)

for all ${\alpha }_0<\alpha <N$. By combining Equations (12), (14) and (15), we have that

$$\begin{array}{cc}\hfill & m_{\alpha }+o_n(1)\hfill \\ \hfill =& E_{\alpha }(u_n,v_n)-\frac{1}{2}⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩\hfill \\ \hfill =& \frac{\alpha }{2(N+\alpha )}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |u_n{|}^{\frac{N+\alpha }{N}})|u_n{|}^{\frac{N+\alpha }{N}}dx+\frac{\alpha +2s}{2(N+\alpha )}{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^{\frac{N+\alpha }{N-2s}})|v_n{|}^{\frac{N+\alpha }{N-2s}}dx\hfill \\ \hfill \le & \frac{\alpha }{2(N+\alpha )}{\mathcal{S}}_2^{-\frac{N+\alpha }{N}}\parallel u_n{\parallel }_2^{2\frac{N+\alpha }{N}}+\frac{\alpha +2s}{2(N+\alpha )}{\mathcal{S}}_1^{-\frac{N+\alpha }{N-2s}}{\parallel {(-\mathsf{\Delta })}^{\frac{s}{2}}{u}_n\parallel }_2^{2\frac{N+\alpha }{N-2s}},\hfill \end{array}$$

which, together with Equations (24) and (25), implies that

$$\begin{array}{cc}\hfill m_{\alpha }+o_n(1)\le & \frac{\alpha }{2(N+\alpha )}{\left[\frac{{\lambda }_2}{({\lambda }_1{\lambda }_2-{\beta }^2){\mathcal{S}}_2}\right]}^{\frac{N+\alpha }{N}}{\left(\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx\right)}^{\frac{N+\alpha }{N}}\hfill \\ \hfill & +\frac{\alpha +2s}{2(N+\alpha )}{\mathcal{S}}_1^{-\frac{N+\alpha }{N-2s}}{\left(\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx\right)}^{\frac{N+\alpha }{N-2s}}.\hfill \end{array}$$

(31)

Furthermore, we observe that

$$\begin{array}{c}{E}_{\alpha }(u_n,v_n)-\frac{N}{2(N+\alpha )}⟨E_{\alpha }^{\prime }(u_n,v_n),(u_n,v_n)⟩\hfill \\ =\frac{\alpha }{2(N+\alpha )}\left(\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx\right)\hfill \\ +\frac{s}{N+\alpha }{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast |v_n{|}^{\frac{N+\alpha }{N-2s}})|v_n{|}^{\frac{N+\alpha }{N-2s}}dx,\hfill \end{array}$$

which means that

$$\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx\le \frac{2(N+\alpha )}{\alpha }{m}_{\alpha }+o_n(1).$$

(32)

We can similarly deduce that

$$\parallel (u_n,v_n){\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}{u}_n{v}_{n}dx\le \frac{2(N+\alpha )}{\alpha +2s}{m}_{\alpha }+o_n(1).$$

(33)

By using Equations (30)–(33), we obtain the following contradiction:

$$\begin{array}{c}1+o_n(1)\hfill \\ \le {\left[\frac{{\lambda }_2}{({\lambda }_1{\lambda }_2-{\beta }^2){\mathcal{S}}_2}\right]}^{\frac{N+\alpha }{N}}{\left[\frac{2(N+\alpha )}{\alpha }{m}_{\alpha }\right]}^{\frac{\alpha }{N}}+{\mathcal{S}}_1^{-\frac{N+\alpha }{N-2s}}{\left[\frac{2(N+\alpha )}{\alpha +2s}{m}_{\alpha }\right]}^{\frac{\alpha +2s}{N-2s}}+o_n(1)\hfill \\ <\frac{1}{2}+\frac{1}{2}+o_n(1)=1+o_n(1).\hfill \end{array}$$

□

6. Nonexistence

In this section, we obtain a nonexistence result based on the following Pohožaev identity type:

Lemma 12.

Let $N>2s$ and $(u,v)\in H$ be any solution of system (1). Then, $(u,v)$ satisfies the following Pohožaev identity:

$$\begin{array}{cc}\hfill & \frac{N}{2}{\parallel (u,v)\parallel }_H^2-{s(\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^2{+\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}v{\parallel }_2^2)\hfill \\ \hfill =& N\beta \int uvdx+\frac{N+\alpha }{2p}\int (I_{\alpha }\ast {|u|}^p{)|u|}^{p}dx+\frac{N+\alpha }{2q}\int (I_{\alpha }\ast {|v|}^q{)|v|}^{q}dx.\hfill \end{array}$$

(34)

Proof. 

The proof is similar to the argument of Theorem 1.13 in [17], and thus we omitted it here. □

Proof of Theorem 3.

We divide the proof into two cases:

Case 1.

The system (1) does not have any nontrivial solution as long as $p,q\le \frac{N+\alpha }{N}$. Since $p,q\le \frac{N+\alpha }{N}$, we have that

$$\frac{N}{2}\le \frac{N+\alpha }{2p}\mathrm{and}\frac{N}{2}\le \frac{N+\alpha }{2q}.$$

(35)

Let $(u,v)\in H$ be a nontrivial solution of system (1). Then we obtain

$${\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uv={\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|u|}^p{)|u|}^p+{\int }_{{\mathbb{R}}^N}(I_{\alpha } \ast {|v|}^q{)|v|}^q.$$

(36)

Next we can write Equation (34) as

$$\begin{array}{cc}\hfill & \frac{N}{2}\left({\parallel (u,v)\parallel }_H^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx\right)\hfill \\ \hfill =& {s(\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{u\parallel }_2^2{+\parallel (-\mathsf{\Delta })}^{\frac{s}{2}}{v\parallel }_2^2)+\frac{N+\alpha }{2p}\int (I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx\hfill \\ \hfill & +\frac{N+\alpha }{2q}\int (I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx.\hfill \end{array}$$

(37)

By combining Equations (36) and (37), we have

$$\begin{array}{cc}\hfill & s(\parallel {(-\mathsf{\Delta })}^{\frac{s}{2}}u{\parallel }_2^2+\parallel {(-\mathsf{\Delta })}^{\frac{s}{2}}v{\parallel }_2^2)\hfill \\ \hfill =& \left(\frac{N}{2}-\frac{N+\alpha }{2p}\right)\int (I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx+\left(\frac{N}{2}-\frac{N+\alpha }{2q}\right)\int (I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx.\hfill \end{array}$$

Together with Equation (35) and $(u,v)\in H$, which is a nontrivial solution of system (1), we find that

$$s(\parallel {(-\mathsf{\Delta })}^{\frac{s}{2}}u{\parallel }_2^2+\parallel {(-\mathsf{\Delta })}^{\frac{s}{2}}v{\parallel }_2^2)>0,$$

$$\left(\frac{N}{2}-\frac{N+\alpha }{2p}\right)\int (I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx+\left(\frac{N}{2}-\frac{N+\alpha }{2q}\right)\int (I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx\le 0,$$

This leads to a contradictory result.

Case 2.

When $p,q\ge \frac{N+\alpha }{N-2s}$, the following holds:

$$\frac{N+\alpha }{2p}\le \frac{N-2s}{2}\mathrm{and}\frac{N+\alpha }{2q}\le \frac{N-2s}{2}.$$

(38)

Let $(u,v)$ be a nontrivial solution of system (1). Then, Equation (36) can be multiplied by $-\frac{N-2s}{2}$ and summed with Equation (34), and with Equation (38), we obtain

$$\begin{array}{cc}\hfill & s\left({\lambda }_1{\parallel u\parallel }_2^2+{\lambda }_2{\parallel v\parallel }_2^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx\right)\hfill \\ \hfill =& \left(\frac{N+\alpha }{2p}-\frac{N-2s}{2}\right)\int (I_{\alpha } \ast {|u|}^p{)|u|}^{p}dx+\left(\frac{N+\alpha }{2q}-\frac{N-2s}{2}\right)\int (I_{\alpha } \ast {|v|}^q{)|v|}^{q}dx\hfill \\ \hfill \le & 0.\hfill \end{array}$$

(39)

On the other hand, since $0<\beta <\sqrt{{\lambda }_1{\lambda }_2}$ the following holds:

$${\lambda }_1{\parallel u\parallel }_2^2+{\lambda }_2{\parallel v\parallel }_2^2-2\beta {\int }_{{\mathbb{R}}^N}uvdx>0.$$

(40)

Thus, combining Equations (39) and (40) and $(u,v)\in H$ is a nontrivial solution of Equation (1), and we obtain a contradiction.

□