Existence and Multiplicity of Solutions for Fractional p-Laplacian Systems Involving Critical Homogeneous Nonlinearities

Abstract

This paper is concerned with a class of fractional p-Laplacian systems with critical homogeneous nonlinearities. Under proper conditions, the existence and multiplicity results of nontrivial solutions are obtained by variational methods. To some extent, our results improve and supplement some existing relevant results.

1. Introduction

In this article, we investigate the existence and multiplicity of nontrivial solutions for the following fractional p-Laplacian system:

$$\left\{\begin{array}{cc}{(-{\Delta }_p)}^{s}u=\lambda f(x){|u|}^{q-2}u+{\displaystyle \frac{1}{p_s^{*}}}{F}_u(u,v),\hfill & \hfill x\in {\mathbb{R}}^N,\\ {(-{\Delta }_p)}^{s}v=\mu f(x){|v|}^{q-2}v+{\displaystyle \frac{1}{p_s^{*}}}{F}_v(u,v),\hfill & \hfill x\in {\mathbb{R}}^N,\end{array}\right.$$

(1)

where $N>ps$ with $s\in (0,1)$, $\lambda ,\mu >0$ are two parameters, $1<q<p$ and $p_s^{*}={\displaystyle \frac{Np}{N-ps}}$ is the fractional critical Sobolev exponent, and ${(-{\Delta }_p)}^s$ is the fractional p-Laplacian operator, which is defined as

$$\begin{array}{c}\hfill {(-{\Delta }_p)}^{s}u(x)=2\underset{\epsilon \to 0}{lim}{\int }_{{\mathbb{R}}^N\setminus B_{\epsilon }(x)}{\displaystyle \frac{{|u(x)-u(y)|}^{p-2}(u(x)-u(y))}{{|x-y|}^{N+ps}}}dy,x\in {\mathbb{R}}^N,\end{array}$$

where $B_{\epsilon }(x):=\{y\in {\mathbb{R}}^N:|x-y|<\epsilon \}$. The weight function f is positive continuous function on ${\mathbb{R}}^N$ satisfying the following condition:

$$(H_1)f\in L^{q^{*}}({\mathbb{R}}^N),\mathrm{where}{q}^{*}={\displaystyle \frac{p_s^{*}}{p_s^{*}-q}}\mathrm{and}f(x)\ge \kappa >0.$$

The function $F\in C^1({\mathbb{R}}^{+}\times {\mathbb{R}}^{+},{\mathbb{R}}^{+})$ satisfies the following assumptions:

$$(H_2)\left\{\begin{array}{c}F(tu,tv)=t^{p_s^{*}}F(u,v)\mathrm{for}\mathrm{all}t>0,u,v\ge 0,(p_s^{*}-\mathrm{homogeneity}),\hfill \\ F_u(u,0)=F_u(0,v)=F_v(u,0)=F_v(0,v)=0,\mathrm{for}u,v\ge 0.\hfill \end{array}\right.$$

In recent years, a great deal of attention has been paid to the study of fractional problems, from the view of pure mathematics and concrete applications, since this type of operator arises in many different applications, such as the thin obstacle problem, stratified materials, anomalous diffusion, finance, phase transitions, water waves, and many others. For more details, we refer to [1,2,3].

On the one hand, the fractional elliptic problems for $p=2$ have been studied by many researchers, see [4,5,6,7,8,9] for the subcritical case, [10,11,12,13] for the critical case, and [14,15] for the fractional Kirchhoff-type problem. On the other hand, for the case $p\ne 2$, many interesting results have also been obtained; see [16,17,18,19] and references therein. In particular, Chen and Deng [20] considered the following system with a subcritical concave–convex nonlinearity:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{cc}{(-{\Delta }_p)}^{s}u={\lambda |u|}^{q-2}u+{\displaystyle \frac{2\alpha }{\alpha +\beta }}{|u|}^{\alpha -2}u{|v|}^{\beta },\hfill & \mathrm{in}\Omega ,\hfill \\ {(-{\Delta }_p)}^{s}v={\mu |v|}^{q-2}v+{\displaystyle \frac{2\beta }{\alpha +\beta }}{|u|}^{\alpha }{|v|}^{\beta -2}v,\hfill & \mathrm{in}\Omega ,\hfill \\ u=v=0,\hfill & \mathrm{in}{\mathbb{R}}^N\setminus \Omega ,\hfill \end{array}\right.\end{array}\end{array}$$

(2)

where $\Omega$ is a smooth bounded set in ${\mathbb{R}}^N$. They obtained the existence of at least two nontrivial solutions for (2) by variational methods. Furthermore, Chen and Squassina [21] obtained the multiplicity of solutions for the critical fractional p-Laplacian system (2). In [22], Zhen and Zhang discussed the following system involving concave–convex nonlinearities and sign-changing weight functions:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{cc}{(-{\Delta }_p)}^{s}u=\lambda f(x){|u|}^{q-2}u+{\displaystyle \frac{2\alpha }{\alpha +\beta }}h(x){|u|}^{\alpha -2}u{|v|}^{\beta },\hfill & \mathrm{in}\Omega ,\hfill \\ {(-{\Delta }_p)}^{s}v=\mu g(x){|v|}^{q-2}v+{\displaystyle \frac{2\beta }{\alpha +\beta }}h(x){|u|}^{\alpha }{|v|}^{\beta -2}v,\hfill & \mathrm{in}\Omega ,\hfill \\ u=v=0,\hfill & \mathrm{in}{\mathbb{R}}^N\setminus \Omega .\hfill \end{array}\right.\end{array}\end{array}$$

(3)

where $\Omega$ is a smooth bounded domain in ${\mathbb{R}}^N$. Under suitable assumptions, they proved that (3) has at least two nontrivial solutions by using the Nehari manifold together with Ekeland’s variational principle.

In a recent paper, the authors of [23] considered the following fractional p-Laplacian elliptic system:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{cc}{(-{\Delta }_p)}^{s}u+{|u|}^{p-2}u=\lambda g(x){|u|}^{q-2}u+{\displaystyle \frac{\alpha }{\alpha +\beta }}f(x){|u|}^{\alpha -2}u{|v|}^{\beta },\hfill & \mathrm{in}\Omega ,\hfill \\ {(-{\Delta }_p)}^{s}v+{|v|}^{p-2}v=\mu h(x){|v|}^{q-2}v+{\displaystyle \frac{\beta }{\alpha +\beta }}f(x){|u|}^{\alpha }{|v|}^{\beta -2}v,\hfill & \mathrm{in}\Omega ,\hfill \\ u=v=0,\hfill & \mathrm{in}{\mathbb{R}}^N\setminus \Omega ,\hfill \end{array}\right.\end{array}\end{array}$$

(4)

where $\Omega$ is a smooth bounded set in ${\mathbb{R}}^N$. The existence of multiple solutions were obtained by the Nehari manifold and Lusternik–Schnirelmann category.

Recently, some researchers have focused on fractional p-Laplacian systems with homogeneous nonlinearities of critical growth; for example, Lu and Shen [24] considered the following system with homogeneous nonlinearities:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{cc}{(-{\Delta }_p)}^{s}u=Q_u(u,v)+H_u(u,v),\hfill & \mathrm{in}\Omega ,\hfill \\ {(-{\Delta }_p)}^{s}v=Q_v(u,v)+H_v(u,v),\hfill & \mathrm{in}\Omega ,\hfill \\ u=v=0,\hfill & \mathrm{in}{\mathbb{R}}^N\setminus \Omega ,\hfill \\ u,v\ge 0,u,v\ne 0,\hfill & \mathrm{in}\Omega \hfill \end{array}\right.\end{array}\end{array}$$

(5)

where $\Omega$ is a bounded domain in ${\mathbb{R}}^N$ with Lipschitz boundary, and Q and H are homogeneous functions. Under some proper conditions, they concluded that (5) provides a nontrivial solution by variational methods.

In [25], Shen studied the following type of fractional p-Laplacian elliptic systems:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}{(-{\Delta }_p)}^{s}u={\displaystyle \frac{{\eta }_1}{p_s^{*}}}{H}_u(u,v)+{\displaystyle \frac{{\eta }_2}{p_s^{*}(\alpha )}}{\displaystyle \frac{Q_u(u,v)}{{|x|}^{\alpha }}},\hfill \\ {(-{\Delta }_p)}^{s}v={\displaystyle \frac{{\eta }_1}{p_s^{*}}}{H}_v(u,v)+{\displaystyle \frac{{\eta }_2}{p_s^{*}(\alpha )}}{\displaystyle \frac{Q_v(u,v)}{{|x|}^{\alpha }}},\hfill \\ u,v>0,(u,v)\in {\tilde{W}}^{s,p}({\mathbb{R}}^N)\times {\tilde{W}}^{s,p}({\mathbb{R}}^N),\hfill \end{array}\right.\end{array}\end{array}$$

(6)

where $H_u,H_v,Q_u$ and $Q_v$ are the partial derivatives of the 2-variable $C^1$-functions $H(u,v)$ and $Q(u,v)$, ${\tilde{W}}^{s,p}({\mathbb{R}}^N)$ denotes the completion of $C_0^{\infty }({\mathbb{R}}^N)$. Under some suitable assumptions, the author proved that system (6) has a weak solution by variational methods.

However, as far as we know, there are few results on fractional p-Laplacian elliptic systems with homogeneous nonlinearities in ${\mathbb{R}}^N$. Motivated by the aforementioned work, in this paper, we discuss the existence and multiplicity of nontrivial solutions for system (1) in ${\mathbb{R}}^N$ by variational methods. The first nontrivial solution can be obtained by using the same argument as that in the subcritical case. In order to obtain the second nontrivial solution, we have to add restrictions on the functions $f,F$ to obtain the compactness of the extraction of the Palais–Smale sequences in the Nehari manifold.

The main results of this paper are as follows.

Theorem 1. 

Let $1<q<p$ for $N>sp$. Assume that $(H_1)$ and $(H_2)$ hold. Then, system (1) has at least one nontrivial solution for all ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$, where ${\mathsf{\Lambda }}_1={\left[{\displaystyle \frac{p-q}{K(p_s^{*}-q)}}{S}^{{\displaystyle \frac{p_s^{*}}{p}}}\right]}^{{\displaystyle \frac{p}{p_s^{*}-p}}}{\left[{\displaystyle \frac{p_s^{*}-p}{p_s^{*}-q}}{S}^{{\displaystyle \frac{q}{p}}}\right]}^{{\displaystyle \frac{p}{p-q}}}$.

Theorem 2. 

Let $1<{\displaystyle \frac{N(p-1)}{N-ps}}\le q<p$ for $N>s{p}^2$. Assume that $(H_1)$ and $(H_2)$ hold. Then, system (1) has at least two nontrivial solutions for all ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_{*}$, where ${\mathsf{\Lambda }}_{*}$ will be given in Section 5.

In order to obtain the results of Theorems 1 and 2, we face two main difficulties: how to determine the range of dual parameters $\lambda ,\mu$ and ensure the existence of solutions for the problem (1). To overcome this difficulty, we will adopt the Nehari manifold and Fibering maps, which are widely used to deal with problems with concave–convex terms [26,27,28]. On the other hand, for $p\ne 2$, the minimizers of S (see (7)) are not yet known, so this remains an open question currently. As in [29], we can overcome this difficulty through the optimal asymptotic behavior of minimizers, which was obtained in [30]. We give some useful estimates on auxiliary functions $u_{\epsilon ,\delta }$ (see Section 2 for their definition), which help us to deal with the study in the critical case.

Remark 1. 

(i) Compared with the fractional p-Laplacian problem (2)–(4) with a bounded domain, system (1) is in ${\mathbb{R}}^N$ and the nonlinearity is more general. Theorems 1 and 2 extend the results in [27] to a class of fractional p-Laplacian elliptic systems with homogeneous nonlinearities.

(ii) In [20,22], the authors study the subcritical growth, and we further discuss the existence of solutions with the critical growth. We note that if $f(x)=1,F(u,v)={2|u|}^{\alpha }{|v|}^{\beta },\alpha >1,\beta >1$ and $\alpha +\beta =p_s^{*}$, system (1) reduces to system (2) in bounded domains. Consequently, our results extend and complement the existing relevant results in the literature.

The structure of this paper is as follows. In Section 2, we introduce some preliminaries. In Section 3, we give some properties of the Nehari manifold and fibering maps. In Section 4 and Section 5, we prove Theorem 1 and Theorem 2, respectively.

2. Preliminaries

Firstly, we recall some facts about the fractional Sobolev space. For 0 $<s<1$, the fractional Sobolev space ${\dot{W}}^{s,p}({\mathbb{R}}^N)$ is defined by

$${\dot{W}}^{s,p}({\mathbb{R}}^N)=\{u\in L^{p_s^{*}}({\mathbb{R}}^N):{\left[u\right]}_{s,p}<\infty \},$$

where the term

$${\left[u\right]}_{s,p}^p=\int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{{|u(x)-u(y)|}^p}{{|x-y|}^{N+ps}}}dxdy$$

is the Gagliardo norm. With the induced norm ${||u||}_{{\dot{W}}^{s,p}({\mathbb{R}}^N)}={\left[u\right]}_{s,p}$, the space ${\dot{W}}^{s,p}({\mathbb{R}}^N)$ is a uniformly convex Banach space. Let $L^p({\mathbb{R}}^N),1\le p\le \infty$ denote Lebesque space with norm ${|·|}_p$. Let

$$\begin{array}{c}\hfill S:=\underset{u\in {\dot{W}}^{s,p}({\mathbb{R}}^N)\setminus \{0\}}{inf}{\displaystyle \frac{{\left[u\right]}_{s,p}^p}{({\int }_{{\mathbb{R}}^N}{|u|}^{p_s^{*}}{dx)}^{\frac{p}{p_s^{*}}}}}\end{array}$$

(7)

be the best fractional Sobolev constant. $C,C_i(i=0,1,\dots )$ denote various positive constants.

For system (1), we work in the product space $W={\dot{W}}^{s,p}({\mathbb{R}}^N)\times {\dot{W}}^{s,p}({\mathbb{R}}^N)$ with the norm

$$\begin{array}{c}\hfill {||(u,v)||}^p={\left[u\right]}_{s,p}^p+{\left[v\right]}_{s,p}^p.\end{array}$$

For the convenience of the reader, we list the following homogeneous properties; see [25,31]. Let $F(s,t)$ be a $\gamma$-homogeneous differential function with $\gamma \ge 1$.

(i) There exists $K_F>0$, such that

$$\begin{array}{c}\hfill |F(\sigma ,\tau )|\le K_F{(|\sigma |}^{\gamma }+{|\tau |}^{\gamma })\mathrm{for}\mathrm{all}\sigma ,\tau \in \mathbb{R}.\end{array}$$

(ii) $K_F$ is attained at some $({\sigma }_0,{\tau }_0)\in {\mathbb{R}}^2$, where

$$\begin{array}{c}\hfill K_F=max\{F(\sigma ,\tau )|\sigma ,\tau \in {\mathbb{R},|\sigma |}^{\gamma }+{|\tau |}^{\gamma }=1\}.\end{array}$$

(iii) For $\sigma ,\tau \in \mathbb{R}$,

$$\begin{array}{c}\hfill \sigma F_{\sigma }(\sigma ,\tau )+\tau F_{\tau }(\sigma ,\tau )=\gamma F(\sigma ,\tau ).\end{array}$$

(iv) $F_{\sigma }(\sigma ,\tau )$ and $F_{\tau }(\sigma ,\tau )$ are ($\gamma$-1)-homogeneous.

Based on the above homogeneous properties and assumption $(H_2)$, we have the so-called Euler identity:

$$\begin{array}{c}\hfill (\sigma ,\tau )·\nabla F(\sigma ,\tau )=p_s^{*}F(\sigma ,\tau )\end{array}$$

and

$$\begin{array}{c}\hfill F(\sigma ,\tau )\le {K(|\sigma |}^p+{|\tau |}^p{)}^{{\displaystyle \frac{p_s^{*}}{p}}}\mathrm{for}\mathrm{all}(\sigma ,\tau )\in (\mathbb{R},\mathbb{R}),\end{array}$$

(8)

where $K=max\{F(\sigma ,\tau )|\sigma ,\tau \in \mathbb{R},|\sigma {|}^p+|\tau {|}^p=1\}$.

Definition 1. 

A pair of functions $(u,v)\in W$ is said to be a weak solution of system (1), if for all $(\varphi ,\phi )\in W$, there holds

$$\begin{array}{cc}\hfill & \int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{{|u(x)-u(y)|}^{p-2}(u(x)-u(y))(\varphi (x)-\varphi (y))}{{|x-y|}^{N+ps}}}dxdy\hfill \\ \hfill & +\int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{{|v(x)-v(y)|}^{p-2}(v(x)-v(y))(\phi (x)-\phi (y))}{{|x-y|}^{N+ps}}}dxdy\hfill \\ \hfill & ={\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^{q-2}u\varphi +{\mu f(x)|v|}^{q-2}v\phi )dx+{\displaystyle \frac{1}{p_s^{*}}}{\int }_{{\mathbb{R}}^N}(F_u(u,v)\varphi +F_v(u,v)\phi )dx.\hfill \end{array}$$

The corresponding energy functional of system (1) is defined by

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)& ={\displaystyle \frac{1}{p}}\int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{{|u(x)-u(y)|}^p}{{|x-y|}^{N+ps}}}dxdy+{\displaystyle \frac{1}{p}}\int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{{|v(x)-v(y)|}^p}{{|x-y|}^{N+ps}}}dxdy\hfill \\ \hfill & -{\displaystyle \frac{1}{q}}{\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx-{\displaystyle \frac{1}{p_s^{*}}}{\int }_{{\mathbb{R}}^N}F(u,v)dx.\hfill \end{array}$$

(9)

It is easy to see that ${\mathsf{\Phi }}_{\lambda ,\mu }\in C^1(W,\mathbb{R})$ and the critical points of the functional ${\mathsf{\Phi }}_{\lambda ,\mu }$ are equivalent to the weak solutions of system (1).

In the following, we fix a radially symmetric non-negative decreasing minimizer $U=U(r)$ for S. Multiplying U by a positive constant if necessary, we may assume that

$$\begin{array}{c}\hfill {(-{\Delta }_p)}^{s}U=U^{p_s^{*}-1}\mathrm{in}{\mathbb{R}}^N.\end{array}$$

(10)

For any $\epsilon >0$, we know that

$$\begin{array}{c}\hfill U_{\epsilon }(x)={\displaystyle \frac{1}{{\epsilon }^{\frac{N-sp}{p}}}}U({\displaystyle \frac{|x|}{\epsilon }})\end{array}$$

is also a minimizer for S satisfying (10).

Lemma 1 

([30]). There exist constants $C_1,C_2>0$ and $\theta >1$ such that for all $r\ge 1$,

$$\begin{array}{c}\hfill {\displaystyle \frac{C_1}{r^{\frac{N-sp}{p-1}}}}\le U(r)\le {\displaystyle \frac{C_2}{r^{\frac{N-sp}{p-1}}}}\mathrm{and}{\displaystyle \frac{U(\theta r)}{U(r)}}\le {\displaystyle \frac{1}{2}}.\end{array}$$

In what follows, if $\theta =\theta (N,s,p)$ is the above constant, then for $\epsilon ,\delta >0$, as in [18], set

$$\begin{array}{c}\hfill \begin{array}{c}\hfill m_{\epsilon ,\delta }={\displaystyle \frac{U_{\epsilon }(\delta )}{U_{\epsilon }(\delta )-U_{\epsilon }(\theta \delta )}},g_{\epsilon ,\delta }(t)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}0\le t\le U_{\epsilon }(\theta \delta ),\hfill \\ m_{\epsilon ,\delta }^p(t-U_{\epsilon }(\theta \delta ),\hfill & \mathrm{if}{U}_{\epsilon }(\theta \delta )\le t\le U_{\epsilon }(\delta ),\hfill \\ t+U_{\epsilon }(\delta )(m_{\epsilon ,\delta }^{p-1}-1),\hfill & \mathrm{if}t\ge U_{\epsilon }(\delta ),\hfill \end{array}\right.\end{array}\end{array}$$

and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill G_{\epsilon ,\delta }(t)={\int }_0^t{g}_{\epsilon ,\delta }^{\prime }{(\tau )}^{{\displaystyle \frac{1}{p}}}d\tau =\left\{\begin{array}{cc}0,\hfill & \mathrm{if}0\le t\le U_{\epsilon }(\theta \delta ),\hfill \\ m_{\epsilon ,\delta }^p(t-U_{\epsilon }(\theta \delta ),\hfill & \mathrm{if}{U}_{\epsilon }(\theta \delta )\le t\le U_{\epsilon }(\delta ),\hfill \\ t,\hfill & \mathrm{if}t\ge U_{\epsilon }(\delta ).\hfill \end{array}\right.\end{array}\end{array}$$

The functions $g_{\epsilon ,\delta }$ and $G_{\epsilon ,\delta }$ are nondecreasing and absolutely continuous. Consider the radially symmetric non-increasing function

$$\begin{array}{c}\hfill u_{\epsilon ,\delta }(r)=G_{\epsilon ,\delta }(U_{\epsilon }(r)),\end{array}$$

(11)

which satisfies

$$\begin{array}{c}\hfill \begin{array}{c}\hfill u_{\epsilon ,\delta }(r)=\left\{\begin{array}{cc}{U}_{\epsilon }(r),\hfill & \mathrm{if}r\le \delta ,\hfill \\ 0,\hfill & \mathrm{if}r\ge \theta \delta .\hfill \end{array}\right.\end{array}\end{array}$$

Then, we have the following estimates for $u_{\epsilon ,\delta }$.

Lemma 2 

([29,32]). There exists $C=C(N,p,s)>0$ such that for any $0<\epsilon <{\displaystyle \frac{\delta }{2}}$, there holds

$$\begin{array}{c}\hfill \int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{|u_{\epsilon ,\delta }(x)-u_{\epsilon ,\delta }{(y)|}^p}{{|x-y|}^{N+ps}}}dxdy\le S^{{\displaystyle \frac{N}{ps}}}+C{({\displaystyle \frac{\epsilon }{\delta }})}^{{\displaystyle \frac{N-ps}{p-1}}},{\int }_{{\mathbb{R}}^N}{|u_{\epsilon ,\delta }(x)|}^{p_s^{*}}dx\ge S^{{\displaystyle \frac{N}{ps}}}-C{({\displaystyle \frac{\epsilon }{\delta }})}^{{\displaystyle \frac{N}{p-1}}}.\end{array}$$

(12)

Moreover, there exists $C=C(N,q,s)>0$, such that

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}{|u_{\epsilon ,\delta }(x)|}^{q}dx\ge C\left\{\begin{array}{ccc}\hfill & {\epsilon }^{N-{\displaystyle \frac{N-ps}{p}}q},\hfill & \hfill \mathrm{if}q>{\displaystyle \frac{N(p-1)}{N-ps}},\\ \hfill & {\epsilon }^{N-{\displaystyle \frac{N-ps}{p}}q}|log\epsilon |,\hfill & \hfill \mathrm{if}q={\displaystyle \frac{N(p-1)}{N-ps}},\\ \hfill & {\epsilon }^{{\displaystyle \frac{(N-ps)q}{p(p-1)}}},\hfill & \hfill \mathrm{if}q<{\displaystyle \frac{N(p-1)}{N-ps}}.\end{array}\right.\end{array}$$

(13)

3. Nehari Manifold and Fibering Maps

We consider the system (1) on the Nehari manifold. Define the Nehari manifold as follows:

$$\begin{array}{c}\hfill N_{\lambda ,\mu }=\{(u,v)\in W\setminus \{(0,0)\}|⟨{\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u,v),(u,v)⟩=0\}.\end{array}$$

Note that $(u,v)\in N_{\lambda ,\mu }$ if and only if

$$\begin{array}{c}\hfill {||(u,v)||}^p={\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx+{\int }_{{\mathbb{R}}^N}F(u,v)dx.\end{array}$$

(14)

The Nehari manifold $N_{\lambda ,\mu }$ is closely linked to the behavior of fibering maps ${\mathsf{\Psi }}_{u,v}:t\mapsto {\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv)$ for $t>0$, defined by

$$\begin{array}{c}\hfill {\mathsf{\Psi }}_{u,v}(t):={\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv)={\displaystyle \frac{t^p}{p}}{||(u,v)||}^p-{\displaystyle \frac{t^q}{q}}{\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx-{\displaystyle \frac{t^{p_s^{*}}}{p_s^{*}}}{\int }_{{\mathbb{R}}^N}F(u,v)dx.\end{array}$$

Such maps were first introduced by Drabek and Pohozaev in [33] and were discussed by Brown and Wu in [34].

For $(u,v)\in W$, we note that

$$\begin{array}{cc}\hfill {\mathsf{\Psi }}_{u,v}^{\prime }(t)& =t^{p-1}{||(u,v)||}^p-t^{q-1}{\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx-t^{p_s^{*}-1}{\int }_{{\mathbb{R}}^N}F(u,v)dx\hfill \end{array}$$

(15)

and

$$\begin{array}{c}\hfill {\mathsf{\Psi }}_{u,v}^{″}(t)=(p-1)t^{p-2}{||(u,v)||}^p-(q-1)t^{q-2}{\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx-(p_s^{*}-1)t^{p_s^{*}-2}{\int }_{{\mathbb{R}}^N}F(u,v)dx,\end{array}$$

which implies that $(u,v)\in N_{\lambda ,\mu }$ if and only if ${\mathsf{\Psi }}_{u,v}^{\prime }(1)=0$. Hence, for $(u,v)\in N_{\lambda ,\mu }$, by (15), we obtain

$$\begin{array}{cc}\hfill {\mathsf{\Psi }}_{u,v}^{″}(1)& ={(p-1)||(u,v)||}^p-(q-1){\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx-(p_s^{*}-1){\int }_{{\mathbb{R}}^N}F(u,v)dx\hfill \\ \hfill & =(p-p_s^{*})||(u,v){||}^p-(q-p_s^{*}){\int }_{{\mathbb{R}}^N}(\lambda f(x){|u|}^q+\mu f(x){|v|}^q)dx\hfill \\ \hfill & ={(p-q)||(u,v)||}^p-(p_s^{*}-q){\int }_{{\mathbb{R}}^N}F(u,v)dx.\hfill \end{array}$$

(16)

Now, we split $N_{\lambda ,\mu }$ into three parts.

$$\begin{array}{cc}\hfill N_{\lambda ,\mu }^{+}& =\{(u,v)\in N_{\lambda ,\mu }:{\mathsf{\Psi }}_{u,v}^{″}(1)>0\};\hfill \\ \hfill N_{\lambda ,\mu }^0& =\{(u,v)\in N_{\lambda ,\mu }:{\mathsf{\Psi }}_{u,v}^{″}(1)=0\};\hfill \\ \hfill N_{\lambda ,\mu }^{-}& =\{(u,v)\in N_{\lambda ,\mu }:{\mathsf{\Psi }}_{u,v}^{″}(1)<0\}.\hfill \end{array}$$

Lemma 3. 

Assume that $(H_1)$ holds. Then, there exists a constant ${\mathsf{\Lambda }}_1>0$ such that if

$$\begin{array}{c}\hfill {0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1,\end{array}$$

we have $N_{\lambda ,\mu }^0=\varnothing$, where ${\mathsf{\Lambda }}_1={\left[{\displaystyle \frac{p-q}{K(p_s^{*}-q)}}{S}^{{\displaystyle \frac{p_s^{*}}{p}}}\right]}^{{\displaystyle \frac{p}{p_s^{*}-p}}}{\left[{\displaystyle \frac{p_s^{*}-p}{p_s^{*}-q}}{S}^{{\displaystyle \frac{q}{p}}}\right]}^{{\displaystyle \frac{p}{p-q}}}$.

Proof. 

We argue by contradiction. Assume that there exist $\lambda ,\mu >0$ with

$$\begin{array}{c}\hfill {0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1\end{array}$$

such that $N_{\lambda ,\mu }^0\ne \varnothing$. Then, for $(u,v)\in N_{\lambda ,\mu }^0$, by (16), we have

$$\begin{array}{cc}\hfill {||(u,v)||}^p& ={\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}{\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx,\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {||(u,v)||}^p& ={\displaystyle \frac{p_s^{*}-q}{p-q}}{\int }_{{\mathbb{R}}^N}F(u,v)dx.\hfill \end{array}$$

(18)

Using $(H_1)$, (7) and the Hölder inequality, we obtain

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx& \le \lambda {\left({\int }_{{\mathbb{R}}^N}{|f(x)|}^{q^{*}}dx\right)}^{{\displaystyle \frac{1}{q^{*}}}}{\left({\int }_{{\mathbb{R}}^N}{|u|}^{p_s^{*}}dx\right)}^{{\displaystyle \frac{q}{p_s^{*}}}}\hfill \\ \hfill & +\mu {\left({\int }_{{\mathbb{R}}^N}{|f(x)|}^{q^{*}}dx\right)}^{{\displaystyle \frac{1}{q^{*}}}}{\left({\int }_{{\mathbb{R}}^N}{|v|}^{p_s^{*}}dx\right)}^{{\displaystyle \frac{q}{p_s^{*}}}}\hfill \\ \hfill & \le {\lambda |f|}_{q^{*}}{S}^{-{\displaystyle \frac{q}{p}}}{\left[u\right]}_{s,p}^q+\mu {|f|}_{q^{*}}{S}^{-{\displaystyle \frac{q}{p}}}{\left[v\right]}_{s,p}^q\hfill \\ \hfill & \le S^{-{\displaystyle \frac{q}{p}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-q}{p}}}{||(u,v)||}^q.\hfill \end{array}$$

(19)

Utilizing (17) and (19), we have

$$\begin{array}{c}\hfill ||(u,v)||\le {\left[{\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}{S}^{-{\displaystyle \frac{q}{p}}}\right]}^{{\displaystyle \frac{1}{p-q}}}{\left[{(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right]}^{{\displaystyle \frac{1}{p}}}.\end{array}$$

(20)

By (7), (8) and the Minkowski inequality, we obtain

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{R}}^N}F(u,v)dx& \le K{\left[{\int }_{{\mathbb{R}}^N}{\left({|u|}^p+{|v|}^p\right)}^{{\displaystyle \frac{p_s^{*}}{p}}}dx\right]}^{{\displaystyle \frac{p}{p_s^{*}}}{\displaystyle \frac{p_s^{*}}{p}}}\hfill \\ \hfill & \le K{\left[{\left({\int }_{{\mathbb{R}}^N}{|u|}^{p_s^{*}}dx\right)}^{{\displaystyle \frac{p}{p_s^{*}}}}+{\left({\int }_{{\mathbb{R}}^N}{|v|}^{p_s^{*}}dx\right)}^{{\displaystyle \frac{p}{p_s^{*}}}}\right]}^{{\displaystyle \frac{p_s^{*}}{p}}}\hfill \\ \hfill & \le K{S}^{-{\displaystyle \frac{p_s^{*}}{p}}}{\left({\left[u\right]}_{s,p}^p+{\left[v\right]}_{s,p}^p\right)}^{{\displaystyle \frac{p_s^{*}}{p}}}\hfill \\ \hfill & =K{S}^{-{\displaystyle \frac{p_s^{*}}{p}}}{||(u,v)||}^{p_s^{*}}.\hfill \end{array}$$

(21)

From (18) and (21), we have

$$\begin{array}{c}\hfill ||(u,v)||\ge {\left[{\displaystyle \frac{p-q}{K(p_s^{*}-q)}}{S}^{{\displaystyle \frac{p_s^{*}}{p}}}\right]}^{{\displaystyle \frac{1}{p_s^{*}-p}}}.\end{array}$$

(22)

By (20) and (22), we obtain

$$\begin{array}{c}\hfill {(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\ge {\left[{\displaystyle \frac{p-q}{K(p_s^{*}-q)}}{S}^{{\displaystyle \frac{p_s^{*}}{p}}}\right]}^{{\displaystyle \frac{p}{p_s^{*}-p}}}{\left[{\displaystyle \frac{p_s^{*}-p}{p_s^{*}-q}}{S}^{{\displaystyle \frac{q}{p}}}\right]}^{{\displaystyle \frac{p}{p-q}}}:={\mathsf{\Lambda }}_1,\end{array}$$

which is a contradiction with the assumption. Therefore, if ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$, then we have $N_{\lambda ,\mu }^0=\varnothing$. □

Lemma 4. 

The energy functional ${\mathsf{\Phi }}_{\lambda ,\mu }$ is coercive and bounded below on $N_{\lambda ,\mu }$ for ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$.

Proof. 

If $(u,v)\in N_{\lambda ,\mu }$, then by (9) and (14), we have

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)=({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x){|u|}^q+\mu f(x){|v|}^q)dx.\end{array}$$

(23)

Combining (19) with (23), we have

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)\ge ({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}})S^{-{\displaystyle \frac{q}{p}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-q}{p}}}||(u,v){||}^q.\end{array}$$

(24)

Due to $1<q<p$, we see that ${\mathsf{\Phi }}_{\lambda ,\mu }$ is coercive and bounded from below on $N_{\lambda ,\mu }$. □

By Lemmas 3 and 4, we know that $N_{\lambda ,\mu }=N_{\lambda ,\mu }^{+}\bigcup N_{\lambda ,\mu }^{-}$ for any ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$. Therefore, we may define

$$\begin{array}{c}\hfill c_{\lambda ,\mu }=\underset{(u,v)\in N_{\lambda ,\mu }}{inf}{\mathsf{\Phi }}_{\lambda ,\mu }(u,v),c_{\lambda ,\mu }^{+}=\underset{(u,v)\in N_{\lambda ,\mu }^{+}}{inf}{\mathsf{\Phi }}_{\lambda ,\mu }(u,v)\mathrm{and}{c}_{\lambda ,\mu }^{-}=\underset{(u,v)\in N_{\lambda ,\mu }^{-}}{inf}{\mathsf{\Phi }}_{\lambda ,\mu }(u,v).\end{array}$$

Lemma 5. 

Assume that $(u_0,v_0)$ is a local minimizer of ${\mathsf{\Phi }}_{\lambda ,\mu }$ on $N_{\lambda ,\mu }$ and $(u_0,v_0)\notin N_{\lambda ,\mu }^0$, then $(u_0,v_0)$ is a critical point of ${\mathsf{\Phi }}_{\lambda ,\mu }$.

Proof. 

The proof is almost the same as [22,35], so we omit it here. □

Lemma 6. 

The following facts hold:

(i) Suppose ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$, then $c_{\lambda ,\mu }\le c_{\lambda ,\mu }^{+}<0$.

(ii) Suppose ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{({\displaystyle \frac{q}{p}})}^{{\displaystyle \frac{p}{p-q}}}{\mathsf{\Lambda }}_1$, then there exists a $d_0$ such that $c_{\lambda ,\mu }^{-}>d_0$, where $d_0$ is a positive constant depending on $p,q,N,s,\lambda ,\mu ,S,K$.

Proof. 

(i) Let $(u,v)\in N_{\lambda ,\mu }^{+}$, and by (16), we have

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx\ge {\displaystyle \frac{p_s^{*}-p}{p_s^{*}-q}}{||(u,v)||}^p.\end{array}$$

Thus,

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)& =({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x){|u|}^q+\mu f(x){|v|}^q)dx\hfill \\ \hfill & \le ({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\displaystyle \frac{p_s^{*}-p}{p_s^{*}-q}}||(u,v){||}^p\hfill \\ \hfill & \le -{\displaystyle \frac{p_s^{*}(p-q)}{pq{p}_s^{*}}}{||(u,v)||}^p<0.\hfill \end{array}$$

(25)

Therefore, by the definition of $c_{\lambda ,\mu },c_{\lambda ,\mu }^{+}$, we can obtain $c_{\lambda ,\mu }\le c_{\lambda ,\mu }^{+}<0$.

(ii) Let $(u,v)\in N_{\lambda ,\mu }^{-}$, and we have ${\mathsf{\Psi }}^{″}(1)<0$. By (16), we have

$$\begin{array}{c}\hfill {\displaystyle \frac{p-q}{p_s^{*}-q}}{||(u,v)||}^p<{\int }_{{\mathbb{R}}^N}F(u,v)dx.\end{array}$$

(26)

From (21), we obtain

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}F(u,v)dx\le K{S}^{-{\displaystyle \frac{p_s^{*}}{p}}}{||(u,v)||}^{p_s^{*}}.\end{array}$$

(27)

By (26) and (27), we have

$$\begin{array}{c}\hfill ||(u,v)||\ge {\left[{\displaystyle \frac{p-q}{K(p_s^{*}-q)}}{S}^{{\displaystyle \frac{p_s^{*}}{p}}}\right]}^{{\displaystyle \frac{1}{p_s^{*}-p}}}.\end{array}$$

(28)

Taking (28) into (24), we deduce that

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)& \ge ({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}})S^{-{\displaystyle \frac{q}{p}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-q}{p}}}||(u,v){||}^q\hfill \\ \hfill & ={||(u,v)||}^q\left[({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^{p-q}-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}})S^{-{\displaystyle \frac{q}{p}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-q}{p}}}\right]\hfill \\ \hfill & \ge {\left[{\displaystyle \frac{p-q}{K(p_s^{*}-q)}}{S}^{{\displaystyle \frac{p_s^{*}}{p}}}\right]}^{{\displaystyle \frac{q}{p_s^{*}-p}}}[({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}}){\left[{\displaystyle \frac{p-q}{K(p_s^{*}-q)}}{S}^{{\displaystyle \frac{p_s^{*}}{p}}}\right]}^{{\displaystyle \frac{p-q}{p_s^{*}-p}}}\hfill \\ \hfill & -({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}})S^{-{\displaystyle \frac{q}{p}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-q}{p}}}]\ge d_0>0,\hfill \end{array}$$

where $d_0=d_0(p,q,N,s,\lambda ,\mu ,S,K)$. We complete the proof. □

Next, we consider the function $J_{u,v}:{\mathbb{R}}^{+}\to \mathbb{R}$ defined by

$$\begin{array}{c}\hfill J_{u,v}(t)=t^{p-p_s^{*}}{||(u,v)||}^p-t^{q-p_s^{*}}{\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx.\end{array}$$

By simple computations, we have the following results.

Lemma 7. 

Suppose $(u,v)\in W\setminus \{(0,0)\}$, then the function $J_{u,v}$ satisfies the following properties:

(i) $J_{u,v}(t)$ has a unique critical point at

$$\begin{array}{c}\hfill t=t_{max}(u,v)={\left({\displaystyle \frac{(p_s^{*}-q){\int }_{{\mathbb{R}}^N}(\lambda f(x){|u|}^q+\mu f(x){|v|}^q)dx}{(p_s^{*}-p)||(u,v){||}^p}}\right)}^{{\displaystyle \frac{1}{p-q}}}>0.\end{array}$$

(ii) $J_{u,v}(t)$ is strictly increasing on $(0,t_{max}(u,v))$ and strictly decreasing on $(t_{max}(u,v),+\infty )$.

(iii) ${lim}_{t\to +0}{J}_{u,v}(t)=-\infty ,{lim}_{t\to +\infty }{J}_{u,v}(t)=0$.

Lemma 8. 

$(tu,tv)\in N_{\lambda ,\mu }^{\pm }$ if and only if $\pm J_{u,v}^{\prime }(t)>0$.

Proof. 

It is easy to see that for $t>0,(tu,tv)\in N_{\lambda ,\mu }$ if and only if

$$\begin{array}{c}\hfill J_{u,v}(t)={\int }_{{\mathbb{R}}^N}F(u,v)dx.\end{array}$$

(29)

Moreover,

$$\begin{array}{c}\hfill J_{u,v}^{\prime }(t)=(p-p_s^{*})t^{p-p_s^{*}-1}||(u,v){||}^p-(q-p_s^{*})t^{q-p_s^{*}-1}{\int }_{{\mathbb{R}}^N}(\lambda f(x){|u|}^q+\mu f(x){|v|}^q)dx,\end{array}$$

(30)

and if $(tu,tv)\in N_{\lambda ,\mu }$, then

$$\begin{array}{c}\hfill t^{p_s^{*}-1}{J}_{u,v}^{\prime }(t)={\mathsf{\Psi }}_{u,v}^{″}(t).\end{array}$$

(31)

Thus, $(tu,tv)\in N_{\lambda ,\mu }^{+}$ (or $N_{\lambda ,\mu }^{-}$) if and only if $J_{u,v}^{\prime }(t)>0(or<0)$.

Lemma 9. 

Suppose $(u,v)\in W\setminus \{(0,0)\}$, then for any $\lambda ,\mu$ satisfying ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$, where ${\mathsf{\Lambda }}_1$ given in Lemma 3, there exist unique $t_1,t_2$ such that $0<t_1<t_{max}(u,v)<t_2$ and

$$\begin{array}{c}\hfill (t_{1}u,t_{1}v)\in N_{\lambda ,\mu }^{+},(t_{2}u,t_{2}v)\in N_{\lambda ,\mu }^{-}.\end{array}$$

Also,

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(t_{1}u,t_{1}v)=\underset{0\le t\le t_{max}(u,v)}{inf}{\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv),{\mathsf{\Phi }}_{\lambda ,\mu }(t_{2}u,t_{2}v)=\underset{t\ge 0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv).\end{array}$$

□

Proof. 

Since ${\int }_{{\mathbb{R}}^N}F(u,v)dx>0$, we see that (29) has no solution if and only if $\lambda$ and $\mu$ satisfy the following condition:

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}F(u,v)dx>J_{u,v}(t_{max}(u,v)).\end{array}$$

By Lemma 7, we obtain

$$\begin{array}{cc}\hfill J_{u,v}(t_{max}(u,v))& =\left[{\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p-q}}}-{({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}})}^{{\displaystyle \frac{q-p_s^{*}}{p-q}}}\right]{\displaystyle \frac{{\left({\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx\right)}^{\frac{p-p_s^{*}}{p-q}}}{{||(u,v)||}^{\frac{p(q-p_s^{*})}{p-q}}}}\hfill \\ \hfill & ={\displaystyle \frac{p-q}{p_s^{*}-q}}{\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p-q}}}{\displaystyle \frac{{\left({\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx\right)}^{\frac{p-p_s^{*}}{p-q}}}{{||(u,v)||}^{\frac{p(q-p_s^{*})}{p-q}}}}.\hfill \end{array}$$

Because of $1<q<p<p_s^{*}$ and (19), we have

$$\begin{array}{cc}\hfill J_{u,v}(t_{max}(u,v))& \ge {\displaystyle \frac{p-q}{p_s^{*}-q}}{\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p-q}}}{\displaystyle \frac{{\left[S^{-\frac{q}{p}}{\left({(\lambda |f|}_{q^{*}}{)}^{\frac{p}{p-q}}+{(\mu |f|}_{q^{*}}{)}^{\frac{p}{p-q}}\right)}^{\frac{p-q}{p}}{||(u,v)||}^q\right]}^{\frac{p-p_s^{*}}{p-q}}}{{||(u,v)||}^{\frac{p(q-p_s^{*})}{p-q}}}}\hfill \\ \hfill & ={\displaystyle \frac{p-q}{p_s^{*}-q}}{\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p-q}}}{S}^{-{\displaystyle \frac{q(p-p_s^{*})}{p(p-q)}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p}}}{||(u,v)||}^{p_s^{*}}.\hfill \end{array}$$

(32)

On the other hand, from (21), we obtain

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}F(u,v)dx\le K{S}^{-{\displaystyle \frac{p_s^{*}}{p}}}{||(u,v)||}^{p_s^{*}}.\end{array}$$

For any $\lambda$ and $\mu$ satisfying ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$, with ${\mathsf{\Lambda }}_1$ given in Lemma 3, we can obtain

$$\begin{array}{c}\hfill K{S}^{-{\displaystyle \frac{p_s^{*}}{p}}}\le {\displaystyle \frac{p-q}{p_s^{*}-q}}{\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p-q}}}{S}^{-{\displaystyle \frac{q(p-p_s^{*})}{p(p-q)}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p}}}.\end{array}$$

(33)

Hence, by (32) and (33), if $\lambda$ and $\mu$ satisfy ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$, we deduce that

$$\begin{array}{cc}\hfill 0<{\int }_{{\mathbb{R}}^N}F(u,v)dx& \le K{S}^{-{\displaystyle \frac{p_s^{*}}{p}}}{||(u,v)||}^{p_s^{*}}\hfill \\ \hfill & \le {\displaystyle \frac{p-q}{p_s^{*}-q}}{\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p-q}}}{S}^{-{\displaystyle \frac{q(p-p_s^{*})}{p(p-q)}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-p_s^{*}}{p}}}{||(u,v)||}^{p_s^{*}}\hfill \\ \hfill & <J_{u,v}(t_{max}(u,v)).\hfill \end{array}$$

Then, there are unique $t_1>0$ and $t_2>$, with $0<t_1<t_{max}(u,v)<t_2$, such that

$$\begin{array}{c}\hfill J_{u,v}(t_1)=J_{u,v}(t_2)={\int }_{{\mathbb{R}}^N}F(u,v)dx,J_{u,v}^{\prime }(t_1)>0>J_{u,v}^{\prime }(t_2).\end{array}$$

In addition, from (15) and (29), we can find that ${\mathsf{\Psi }}_{u,v}^{\prime }(t_1)={\mathsf{\Psi }}_{u,v}^{\prime }(t_2)=0$. Due to (31), we have that ${\mathsf{\Psi }}_{\lambda ,\mu }^{″}(t_1)>0$ and ${\mathsf{\Psi }}_{\lambda ,\mu }^{″}(t_2)<0$. Thus, it is not hard to find that ${\mathsf{\Psi }}_{u,v}$ has a local minimum at $t_1$ and a local maximum at $t_2$ such that $(t_{1}u,t_{1}v)\in N_{\lambda ,\mu }^{+}$ and $(t_{2}u,t_{2}v)\in N_{\lambda ,\mu }^{-}$. Because of ${\mathsf{\Psi }}_{u,v}(t)={\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv)$, we can find that ${\mathsf{\Phi }}_{\lambda ,\mu }(t_{2}u,t_{2}v)\ge {\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv)\ge {\mathsf{\Phi }}_{\lambda ,\mu }(t_{1}u,t_{1}v)$ for each $t\in [t_1,t_2]$ and ${\mathsf{\Phi }}_{\lambda ,\mu }(t_{1}u,t_{1}v)\le {\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv)$ for each $t\in [0,t_1]$. Therefore,

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(t_{1}u,t_{1}v)=\underset{0\le t\le t_{max}(u,v)}{inf}{\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv),{\mathsf{\Phi }}_{\lambda ,\mu }(t_{2}u,t_{2}v)=\underset{t\ge 0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv).\end{array}$$

We complete the proof.

4. Proof of Theorem 1

In this section, we establish the existence of a local minimum for ${\mathsf{\Phi }}_{\lambda ,\mu }$ on $N_{\lambda ,\mu }^{+}$. First, we state some preliminary results.

We assert that a sequence $\{(u_k,v_k)\}\subset W$ is a ${(PS)}_c$ sequence at level c, if ${\mathsf{\Phi }}_{\lambda }(u_k,v_k)\to c$ and ${\mathsf{\Phi }}_{\lambda }^{\prime }(u_k,v_k)\to 0$ as $k\to \infty$. ${\mathsf{\Phi }}_{\lambda }$ is said to satisfy the ${(PS)}_c$ condition if any ${(PS)}_c$ sequence contains a convergent subsequence.

Lemma 10. 

(i) If ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$, then the functional ${\mathsf{\Phi }}_{\lambda ,\mu }$ has a ${(PS)}_{c_{\lambda ,\mu }}$-sequence $\{(u_k,v_k)\}\subset N_{\lambda ,\mu }$,

(ii) If ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{({\displaystyle \frac{q}{p}})}^{{\displaystyle \frac{p}{p-q}}}{\mathsf{\Lambda }}_1$, then the functional ${\mathsf{\Phi }}_{\lambda ,\mu }$ has a ${(PS)}_{c_{\lambda ,\mu }^{-}}$-sequence $\{(u_k,v_k)\}\subset N_{\lambda ,\mu }^{-}$.

Proof. 

The proof is similar to that in [21,22]; we have omitted the details here. □

Lemma 11. 

If $\{(u_k,v_k)\}\subset W$ is a ${(PS)}_c$-sequence for ${\mathsf{\Phi }}_{\lambda ,\mu }$, then $\{(u_k,v_k)\}$ is bounded in W.

Proof. 

If $\{(u_k,v_k)\}\subset W$ is a ${(PS)}_c$-sequence for ${\mathsf{\Phi }}_{\lambda ,\mu }$, then we have

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u_k,v_k)\to c,{\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u_k,v_k)\to 0\mathrm{in}{W}^{-1}(\mathrm{the}\mathrm{dual}\mathrm{space}\mathrm{of}W)ask\to \infty .\end{array}$$

That is,

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{p}}||(u_k,v_k){||}^p-{\displaystyle \frac{1}{q}}{\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx-{\displaystyle \frac{1}{p_s^{*}}}{\int }_{{\mathbb{R}}^N}F(u_k,v_k)dx=c+o_k(1),\hfill \\ \hfill & ||(u_k,v_k){||}^p-{\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx-{\int }_{{\mathbb{R}}^N}F(u_k,v_k)dx=o_k(1)ask\to \infty .\hfill \end{array}$$

According to $(H_1)$, (7) the Hölder inequality and the $\epsilon$-Young inequality, we obtain

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx& \le \lambda {\left({\int }_{{\mathbb{R}}^N}{|f(x)|}^{q^{*}}dx\right)}^{{\displaystyle \frac{1}{q^{*}}}}{\left({\int }_{{\mathbb{R}}^N}{|u_k|}^{p_s^{*}}dx\right)}^{{\displaystyle \frac{q}{p_s^{*}}}}\hfill \\ \hfill & +\mu {\left({\int }_{{\mathbb{R}}^N}{|f(x)|}^{q^{*}}dx\right)}^{{\displaystyle \frac{1}{q^{*}}}}{\left({\int }_{{\mathbb{R}}^N}{|v_k|}^{p_s^{*}}dx\right)}^{{\displaystyle \frac{q}{p_s^{*}}}}\hfill \\ \hfill & \le {\lambda |f|}_{q^{*}}{S}^{-{\displaystyle \frac{q}{p}}}{\left[u_k\right]}_{s,p}^q+\mu {|f|}_{q^{*}}{S}^{-{\displaystyle \frac{q}{p}}}{\left[v_k\right]}_{s,p}^q\hfill \\ \hfill & \le S^{-{\displaystyle \frac{q}{p}}}{\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right)}^{{\displaystyle \frac{p-q}{p}}}||(u_k,v_k){||}^q\hfill \\ \hfill & \le \epsilon ||(u_k,v_k){||}^p+{\epsilon }^{-{\displaystyle \frac{q}{p-q}}}{S}^{-{\displaystyle \frac{q}{p-q}}}\left({(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}})\right).\hfill \end{array}$$

(34)

Therefore,

$$\begin{array}{cc}\hfill c+o_k(1)& ={\mathsf{\Phi }}_{\lambda ,\mu }(u_k,v_k)-{\displaystyle \frac{1}{p_s^{*}}}⟨{\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u_k,v_k),(u_k,v_k)⟩\hfill \\ \hfill & =({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u_k,v_k){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx\hfill \\ \hfill & \ge \left[({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}})\epsilon \right]||(u_k,v_k){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\epsilon }^{-{\displaystyle \frac{q}{p-q}}}{S}^{-{\displaystyle \frac{q}{p-q}}}{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}})).\hfill \end{array}$$

It follows that

$$\begin{array}{c}\hfill \left[({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}})\epsilon \right]||(u_k,v_k){||}^p\le c+({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\epsilon }^{-{\displaystyle \frac{q}{p-q}}}{S}^{-{\displaystyle \frac{q}{p-q}}}{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}))+o_k(1).\end{array}$$

Let $\epsilon <{\displaystyle \frac{q(p_s^{*}-p)}{p(p_s^{*}-q)}}$; we find that $\{(u_k,v_k)\}$ is bounded in W. □

Proof of Theorem 1. 

From Lemma 6-(i) and Lemma 10, there exists a minimizing sequence $\{(u_k,v_k)\}\subset N_{\lambda ,\mu }$ such that

$$\begin{array}{c}\hfill \underset{k\to \infty }{lim}{\mathsf{\Phi }}_{\lambda ,\mu }(u_k,v_k)=c_{\lambda ,\mu }\le c_{\lambda ,\mu }^{+}<0,{\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u_k,v_k)=o_k(1)\mathrm{in}{W}^{-1}.\end{array}$$

(35)

By Lemma 11, we know that $\{(u_k,v_k)\}$ is bounded in W. Then, there exists $(u_1,v_1)\in W$ and a subsequence, still denoted by $\{(u_k,v_k)\}\subset N_{\lambda ,\mu }$, such that

$$\begin{array}{cc}\hfill & u_k\rightharpoonup u_1,v_k\rightharpoonup v_1,{\dot{W}}^{s,p}({\mathbb{R}}^N),\hfill \\ \hfill & u_k\to u_1,v_k\to v_1,\mathrm{in}{L}_{loc}^r({\mathbb{R}}^N),p\le r<p_s^{*},\hfill \\ \hfill & u_k\to u_1,v_k\to v_1,a.e\mathrm{in}{\mathbb{R}}^N.\hfill \end{array}$$

(36)

Thus, according to the Dominated Convergence Theorem, we have

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx\to {\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_1{|}^q+\mu f(x)|v_1{|}^q)dx\mathrm{as}k\to \infty .\end{array}$$

From $(u_k,v_k)\in N_{\lambda ,\mu }$, we have

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u_k,v_k)=({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u_k,v_k){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx.\end{array}$$

That is,

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx={\displaystyle \frac{q(p_s^{*}-p)}{p(p_s^{*}-q)}}||(u_k,v_k){||}^p-{\displaystyle \frac{q{p}_s^{*}}{p_s^{*}-q}}{\mathsf{\Phi }}_{\lambda ,\mu }(u_k,v_k).\end{array}$$

(37)

Let $k\to \infty$; by Lemma 6-(i), we have

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_1{|}^q+\mu f(x)|v_1{|}^q)dx=\underset{k\to \infty }{lim}[{\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx]\ge -{\displaystyle \frac{q{p}_s^{*}}{p_s^{*}-q}}{c}_{\lambda ,\mu }>0.\end{array}$$

Thus, this indicates that $u_1$ and $v_1$ cannot both be zero.

Next, we prove that $(u_k,v_k)\to (u_1,v_1)$ strongly in W.

Indeed, by Fatou’s lemma and $(u_1,v_1)\in N_{\lambda ,\mu }$, we have

$$\begin{array}{cc}\hfill c_{\lambda ,\mu }\le {\mathsf{\Phi }}_{\lambda ,\mu }(u_1,v_1)& =({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u_1,v_1){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_1{|}^q+\mu f(x)|v_1{|}^q)dx\hfill \\ \hfill & \le \underset{k\to +\infty }{inflim}\left[({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u_k,v_k){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx\right]\hfill \\ \hfill & =\underset{k\to +\infty }{inflim}{\mathsf{\Phi }}_{\lambda ,\mu }(u_k,v_k)=c_{\lambda ,\mu },\hfill \end{array}$$

which implies that

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u_1,v_1)=c_{\lambda ,\mu },\underset{k\to \infty }{lim}||(u_k,v_k){||}^p=||(u_1,v_1){||}^p.\end{array}$$

Then according to the Brézis–Lieb lemma, one has

$$\begin{array}{c}\hfill ||(u_k-u_1),(v_k-v_1){||}^p=||(u_k,v_k){||}^p-||(u_1,v_1){||}^p+o_k(1).\end{array}$$

Hence, $(u_k,v_k)\to (u_1,v_1)$ strongly in W.

Finally, we claim that $(u_1,v_1)\in N_{\lambda ,\mu }^{+}$ and ${\mathsf{\Phi }}_{\lambda ,\mu }(u_1,v_1)=c_{\lambda ,\mu }^{+}<0$.

In fact, if $(u_1,v_1)\in N_{\lambda ,\mu }^{-}$, by Lemma 9, there are unique $t^{+}$ and $t^{-}$ such that

$$\begin{array}{c}\hfill (t^{+}{u}_1,t^{+}{v}_1)\in N_{\lambda ,\mu }^{+},(t^{-}{u}_1,t^{-}{v}_1)\in N_{\lambda ,\mu }^{-}\mathrm{and}{t}^{+}<t^{-}=1.\end{array}$$

Since

$$\begin{array}{c}\hfill {\displaystyle \frac{d}{dt}}{\mathsf{\Phi }}_{\lambda ,\mu }(t^{+}{u}_1,t^{+}{v}_1)=0\mathrm{and}{\displaystyle \frac{d^2}{d{t}^2}}{\mathsf{\Phi }}_{\lambda ,\mu }(t^{+}{u}_1,t^{+}{v}_1)>0,\end{array}$$

there exists a $t_0\in (t^{+},t^{-})$ such that ${\mathsf{\Phi }}_{\lambda ,\mu }(t^{+}{u}_1,t^{+}{v}_1)<{\mathsf{\Phi }}_{\lambda ,\mu }(t_0{u}_1,t_0{v}_1)$. By Lemma 9, we have

$$\begin{array}{c}\hfill c_{\lambda ,\mu }\le c_{\lambda ,\mu }^{+}\le {\mathsf{\Phi }}_{\lambda ,\mu }(t^{+}{u}_1,t^{+}{v}_1)<{\mathsf{\Phi }}_{\lambda ,\mu }(t_0{u}_1,t_0{v}_1)\le {\mathsf{\Phi }}_{\lambda ,\mu }(t^{-}{u}_1,t^{-}{v}_1)={\mathsf{\Phi }}_{\lambda ,\mu }(u_1,v_1)=c_{\lambda ,\mu },\end{array}$$

which is a contradiction. This shows that $(u_1,v_1)\in N_{\lambda ,\mu }^{+}$. By Lemma 6, we obtain that ${\mathsf{\Phi }}_{\lambda ,\mu }(u_1,v_1)=c_{\lambda ,\mu }^{+}<0$. Hence, from Lemma 5, we have that $(u_1,v_1)$ is a nontrivial solution of system (1). □

5. Proof of Theorem 2

In this section, we establish the existence of a local minimum for ${\mathsf{\Phi }}_{\lambda ,\mu }$ on $N_{\lambda ,\mu }^{-}$. For this, we give some necessary lemmas.

Lemma 12. 

If $\{(u_k,v_k)\}\subset W$ is a ${(PS)}_c$-sequence for ${\mathsf{\Phi }}_{\lambda ,\mu }$ with $(u_k,v_k)\rightharpoonup (u,v)$ in W, then ${\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u,v)=0$, and there exists a positive constant $C_0$ depending on $p,q,N,s$, such that

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)\ge -C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}).\end{array}$$

Proof. 

Let $\{(u_k,v_k)\}\subset W$ be a ${(PS)}_c$-sequence for ${\mathsf{\Phi }}_{\lambda ,\mu }$ with $(u_k,v_k)\rightharpoonup (u,v)$ in W, then we have

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u_k,v_k)\to 0\mathrm{strongly}\mathrm{in}{W}^{-1}ask\to \infty .\end{array}$$

For all $(\varphi ,\psi )\in W$, by standard argument [10], we obtain that

$$\begin{array}{c}\hfill ⟨{\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u_k,v_k)-{\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u,v),(\varphi ,\psi )⟩\to 0,\end{array}$$

which yields ${\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u,v)=0$. In particular, we have $⟨{\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u,v),(u,v)⟩=0$. That is,

$$\begin{array}{c}\hfill {||(u,v)||}^p={\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx+{\int }_{{\mathbb{R}}^N}F(u,v)dx.\end{array}$$

Then,

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)=({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x){|u|}^q+\mu f(x){|v|}^q)dx.\end{array}$$

By (11), the Hölder inequality and the Young inequality, we obtain

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,v)& =({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})||(u,v){||}^p-({\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{p_s^{*}}}){\int }_{{\mathbb{R}}^N}(\lambda f(x){|u|}^q+\mu f(x){|v|}^q)dx\hfill \\ \hfill & \ge {\displaystyle \frac{s}{N}}{||(u,v)||}^p-{\displaystyle \frac{p_s^{*}-q}{q{p}_s^{*}}}{(\lambda |f|}_{q^{*}}{|u|}_{p_s^{*}}^q+{\mu |f|}_{q^{*}}{|v|}_{p_s^{*}}^q)\hfill \\ \hfill & \ge {\displaystyle \frac{s}{N}}{||(u,v)||}^p-{\displaystyle \frac{p_s^{*}-q}{q{p}_s^{*}}}({\displaystyle \frac{s}{N}}{({\displaystyle \frac{p_s^{*}-q}{q{p}_s^{*}}})}^{-1})({\left[u\right]}_{s,p}^q+{\left[v\right]}_{s,p}^q)\hfill \\ \hfill & +{\displaystyle \frac{p-q}{p}}{\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{q}{p-q}}}{S}^{-{\displaystyle \frac{q}{p-q}}}{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}))\hfill \\ \hfill & ={\displaystyle \frac{s}{N}}{||(u,v)||}^p-{\displaystyle \frac{s}{N}}{||(u,v)||}^p-C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}})\hfill \\ \hfill & =-C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}),\hfill \end{array}$$

where $C_0={\displaystyle \frac{p-q}{p}}{({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}})}^{{\displaystyle \frac{q}{p-q}}}{S}^{-{\displaystyle \frac{q}{p-q}}}$. We complete the proof. □

Set

$$\begin{array}{c}\hfill S_F=\underset{(u,v)\in D\setminus \{(0,0)\}}{inf}\left\{{\displaystyle \frac{{||(u,v)||}^p}{{({\int }_{{\mathbb{R}}^N}F(u,v)dx)}^{\frac{p}{p_s^{*}}}}}\right\}.\end{array}$$

(38)

By (21), we have

$$\begin{array}{c}\hfill {({\int }_{{\mathbb{R}}^N}F(u,v)dx)}^{{\displaystyle \frac{p}{p_s^{*}}}}\le K^{{\displaystyle \frac{p}{p_s^{*}}}}{S}^{-1}{||(u,v)||}^p,\end{array}$$

which implies that

$$\begin{array}{c}\hfill 0<S{K}^{-{\displaystyle \frac{p}{p_s^{*}}}}\le S_F.\end{array}$$

(39)

Lemma 13. 

The functional ${\mathsf{\Phi }}_{\lambda ,\mu }$ satisfies the ${(PS)}_c$ condition with c satisfying

$$\begin{array}{c}\hfill c<c_{*}={\displaystyle \frac{s}{N}}{S}_F^{{\displaystyle \frac{N}{ps}}}-C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}),\end{array}$$

where $C_0$ is the positive constant given in Lemma 12.

Proof. 

Let $\{(u_k,v_k)\}\subset W$ be a ${(PS)}_c$-sequence for ${\mathsf{\Phi }}_{\lambda ,\mu }$ with $c<c_{*}$. From Lemma 11, we know that $\{(u_k,v_k)\}$ is bounded in W. Then, there exists a subsequence still denoted by $\{(u_k,v_k)\}\subset W$ and $(u,v)\in W$, such that

$$\begin{array}{cc}\hfill & u_k\rightharpoonup u,v_k\rightharpoonup v,{\dot{W}}^{s,p}({\mathbb{R}}^N),\hfill \\ \hfill & u_k\to u,v_k\to v,\mathrm{in}{L}_{loc}^r({\mathbb{R}}^N),p\le r<p_s^{*},\hfill \\ \hfill & u_k\to u,v_k\to v,a.e\mathrm{in}{\mathbb{R}}^N.\hfill \end{array}$$

Thus, we obtain that ${\mathsf{\Phi }}_{\lambda ,\mu }^{\prime }(u,v)=0$ and

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_k{|}^q+\mu f(x)|v_k{|}^q)dx={\int }_{{\mathbb{R}}^N}{(\lambda f(x)|u|}^q+{\mu f(x)|v|}^q)dx+o_k(1).\end{array}$$

(40)

Now, we set $({\overline{u}}_k,{\overline{v}}_k)=(u_k-u,v_k-v)$; then, according to the Brézis–Lieb lemma [35] and arguing as in [31], we obtain

$$\begin{array}{cc}\hfill ||({\overline{u}}_k,{\overline{v}}_k){||}^p& =||(u_k,v_k){||}^p-||(u,v){||}^p+o_k(1).\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{R}}^N}F({\overline{u}}_k,{\overline{v}}_k)dx& ={\int }_{{\mathbb{R}}^N}F(u_k,v_k)dx-{\int }_{{\mathbb{R}}^N}F(u,v)dx+o_k(1).\hfill \end{array}$$

(42)

From (40)–(42), we deduce that

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{p}}||({\overline{u}}_k,{\overline{v}}_k){||}^p-{\displaystyle \frac{1}{p_s^{*}}}{\int }_{{\mathbb{R}}^N}F({\overline{u}}_k,{\overline{v}}_k)dx=c-{\mathsf{\Phi }}_{\lambda ,\mu }(u,v)+o_k(1),\hfill \\ \hfill & ||({\overline{u}}_k,{\overline{v}}_k){||}^p-{\int }_{{\mathbb{R}}^N}F({\overline{u}}_k,{\overline{v}}_k)dx=o_k(1).\hfill \end{array}$$

(43)

Thus, we may assume that

$$\begin{array}{c}\hfill ||({\overline{u}}_k,{\overline{v}}_k){||}^p\to l,{\int }_{{\mathbb{R}}^N}F({\overline{u}}_k,{\overline{v}}_k)dx\to l\mathrm{as}k\to \infty .\end{array}$$

If $l=0$, the proof is completed. Assume that $\sigma >0$; then, from (38), we have

$$\begin{array}{c}\hfill S_F{l}^{{\displaystyle \frac{p}{p_s^{*}}}}=S_F\underset{k\to \infty }{lim}{({\int }_{{\mathbb{R}}^N}F({\overline{u}}_k,{\overline{v}}_k)dx)}^{{\displaystyle \frac{p}{p_s^{*}}}}\le \underset{k\to \infty }{lim}||({\overline{u}}_k,{\overline{v}}_k){||}^p=l,\end{array}$$

which implies that $l\ge S_F^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}}$. Therefore, by (43) and Lemma 12, we have

$$\begin{array}{c}\hfill c={\displaystyle \frac{1}{p}}l-{\displaystyle \frac{1}{p_s^{*}}}l+{\mathsf{\Phi }}_{\lambda ,\mu }(u,v)\ge {\displaystyle \frac{s}{N}}{S}_F^{{\displaystyle \frac{N}{ps}}}-C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}})=c^{*},\end{array}$$

which is a contradiction. Hence, $(u_k,v_k)\to (u,v)$ strongly in W. The proof of Lemma 15 is completed. □

Lemma 14. 

Let $1<{\displaystyle \frac{N(p-1)}{N-ps}}\le q<p$ for $N>s{p}^2$. Assume that $(H_1)$ and $(H_2)$ hold. Then there exist $(u,v)\in W\setminus \{(0,0)\}$ and ${\mathsf{\Lambda }}_2>0$, such that

$$\begin{array}{c}\hfill \underset{t\ge 0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv)<c^{*}={\displaystyle \frac{s}{N}}{S}_F^{{\displaystyle \frac{N}{ps}}}-C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}})\end{array}$$

for ${0<(\lambda |f(x)|}_{L^t}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f(x)|}_{L^t}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_2$.

Proof. 

We can follow the approach presented in [21] to complete the proof of Lemma 15. Now, we consider the functional $I:W\to \mathbb{R}$ defined by

$$\begin{array}{c}\hfill I(u,v)={\displaystyle \frac{1}{p}}{||(u,v)||}^p-{\displaystyle \frac{1}{p_s^{*}}}{\int }_{{\mathbb{R}}^N}F(u,v)dx\mathrm{for}\mathrm{all}(u,v)\in W.\end{array}$$

Set $u_0=e_1{u}_{\epsilon ,\delta },v_0=e_2{u}_{\epsilon ,\delta }$, where $u_{\epsilon ,\delta }$ is defined by (11), $(e_1,e_2)\in {({\mathbb{R}}^{+})}^2$, $|e_1{|}^p+{|e_2|}^p=1$, such that $F(e_1,e_2)=K$. Define

$$\begin{array}{c}\hfill h(t):=I(t{e}_1{u}_{\epsilon ,\delta },t{e}_2{u}_{\epsilon ,\delta })={\displaystyle \frac{t^p}{p}}\int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{|u_{\epsilon ,\delta }(x)-u_{\epsilon ,\delta }{(y)|}^p}{{|x-y|}^{N+ps}}}dxdy-{\displaystyle \frac{K{t}^{p_s^{*}}}{p_s^{*}}}{\int }_{{\mathbb{R}}^N}{|u_{\epsilon ,\delta }(x)|}^{p_s^{*}}dx\mathrm{for}\mathrm{all}t\ge 0.\end{array}$$

Using the following fact,

$$\begin{array}{cc}\hfill & \underset{t\ge 0}{sup}({\displaystyle \frac{A{t}^p}{p}}-{\displaystyle \frac{B{t}^{p_s^{*}}}{p_s^{*}}})=({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}}){({\displaystyle \frac{A}{B^{\frac{p}{p_s^{*}}}}})}^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}},A,B>0,\hfill \\ \hfill & {(a+b)}^r\le a^r+r{(a+b)}^{r-1}b,\forall a,b>0,r\ge 1.\hfill \end{array}$$

Combining (12) with (39), we deduce that

$$\begin{array}{cc}\hfill \underset{t\ge 0}{sup}h(t)& \le \left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}}\right){\left({\displaystyle \frac{\int {\int }_{{\mathbb{R}}^{2N}}\frac{|u_{\epsilon ,\delta }(x)-u_{\epsilon ,\delta }{(y)|}^p}{{|x-y|}^{N+ps}}dxdy}{(K{\int }_{{\mathbb{R}}^N}|u_{\epsilon ,\delta }{{|}^{p_s^{*}}dx)}^{\frac{p}{p_s^{*}}}}}\right)}^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}}\hfill \\ \hfill & \le \left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}}\right){({\displaystyle \frac{1}{K^{\frac{p}{p_s^{*}}}}})}^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}}{\left({\displaystyle \frac{S^{\frac{N}{ps}}+C({(\frac{\epsilon }{\delta })}^{\frac{N-ps}{p-1}})}{(S^{\frac{N}{ps}}-C{({(\frac{\epsilon }{\delta })}^{\frac{N}{p-1}})}^{\frac{p}{p_s^{*}}}}}\right)}^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}}\hfill \\ \hfill & \le {\displaystyle \frac{s}{N}}{S}_F^{{\displaystyle \frac{N}{ps}}}+O{({\displaystyle \frac{\epsilon }{\delta }})}^{{\displaystyle \frac{N-ps}{p-1}}}.\hfill \end{array}$$

(44)

We can choose ${\delta }_1>0$, such that for all

$$\begin{array}{c}\hfill {0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\delta }_1,\end{array}$$

we have

$$\begin{array}{c}\hfill c_{*}={\displaystyle \frac{s}{N}}{S}_F^{{\displaystyle \frac{N}{ps}}}-C_0{((\lambda |f(x)|}_{L^t}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f(x)|}_{L^t}{)}^{{\displaystyle \frac{p}{p-q}}})>0.\end{array}$$

Then,

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(t{u}_0,t{v}_0)\le {\displaystyle \frac{t^p}{p}}||(u_0,v_0){||}^p\le C{t}^p\mathrm{for}t\ge 0\mathrm{and}\lambda ,\mu >0,\end{array}$$

which implies that there exists $t_0\in (0,1)$ satisfying

$$\begin{array}{c}\hfill \underset{0\le t\le t_0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(t{u}_0,t{v}_0)<c_{*},\end{array}$$

for all ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\delta }_1.$ By $(H_1),(H_2)$, and (44), we have

$$\begin{array}{cc}\hfill \underset{t\ge t_0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(t{u}_0,t{v}_0)& =\underset{t\ge t_0}{sup}\left[I(t{u}_0,t{v}_0)-{\displaystyle \frac{t^q}{q}}{\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_0{|}^q+\mu f(x)|v_0{|}^q)dx\right]\hfill \\ \hfill & \le ({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})S_F^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}}+O{({\displaystyle \frac{\epsilon }{\delta }})}^{{\displaystyle \frac{N-ps}{p-1}}}-{\displaystyle \frac{t_0^q}{q}}m\kappa (\lambda +\mu ){\int }_{{\mathbb{R}}^N}{|u_{\epsilon ,\delta }|}^{q}dx,\hfill \end{array}$$

(45)

where $m=min\{|e_1{|}^q,|e_2{|}^q\}$. In view of (13) and (45), we have

$$\begin{array}{c}\hfill \underset{t\ge t_0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(t{u}_0,t{v}_0)\le ({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})S_F^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}}+C{({\displaystyle \frac{\epsilon }{\delta }})}^{{\displaystyle \frac{N-ps}{p-1}}}-C(\lambda +\mu )\left\{\begin{array}{cc}{\epsilon }^{N-{\displaystyle \frac{N-ps}{p}}q},\hfill & \mathrm{if}q>{\displaystyle \frac{N(p-1)}{N-ps}},\hfill \\ {\epsilon }^{N-{\displaystyle \frac{N-ps}{p}}q}|log\epsilon |,\hfill & \mathrm{if}q={\displaystyle \frac{N(p-1)}{N-ps}}.\hfill \end{array}\right.\end{array}$$

For $\epsilon ={\left[{(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}\right]}^{{\displaystyle \frac{p-1}{N-ps}}}\in (0,{\displaystyle \frac{\delta }{2}})$, we have

$$\begin{array}{cc}\hfill & \underset{t\ge t_0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(t{u}_0,t{v}_0)\hfill \\ \hfill & =\underset{t\ge t_0}{sup}\left[I(t{u}_0,t{v}_0)-{\displaystyle \frac{t^q}{q}}{\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_0{|}^q+\mu f(x)|v_0{|}^q)dx\right]\hfill \\ \hfill & \le ({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{p_s^{*}}})S_F^{{\displaystyle \frac{p_s^{*}}{p_s^{*}-p}}}+{C((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}})\hfill \\ \hfill & -C(\lambda +\mu )\left\{\begin{array}{cc}{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{{)}^{{\displaystyle \frac{p}{p-q}}})}^{{\displaystyle \frac{p-1}{N-ps}}(N-{\displaystyle \frac{N-ps}{p}})q},\hfill & \mathrm{if}q>{\displaystyle \frac{N(p-1)}{N-ps}},\hfill \\ {((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}{)}^{{\displaystyle \frac{N(p-1)}{p(N-ps)}}}{|log((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}})|,\hfill & \mathrm{if}q={\displaystyle \frac{N(p-1)}{N-ps}}.\hfill \end{array}\right.\hfill \end{array}$$

We observe that

$$\begin{array}{c}\hfill 1+{\displaystyle \frac{p}{p-q}}{\displaystyle \frac{p-1}{N-ps}}(N-{\displaystyle \frac{N-ps}{p}}q)<{\displaystyle \frac{p}{p-q}}\iff q>{\displaystyle \frac{N(p-1)}{N-ps}}.\end{array}$$

Thus, if $q>{\displaystyle \frac{N(p-1)}{N-ps}}$, we can choose ${\delta }_2>0$, such that $\lambda ,\mu$ satisfying ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\delta }_2$, we can obtain

$$\begin{array}{cc}\hfill & {C((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}{)-C(\lambda +\mu )((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{{)}^{{\displaystyle \frac{p}{p-q}}})}^{{\displaystyle \frac{p-1}{N-ps}}(N-{\displaystyle \frac{N-ps}{p}})q}\hfill \\ \hfill & <-C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}),\hfill \end{array}$$

where $C_0$ is the positive constant defined in Lemma 12.

In addition, if $q={\displaystyle \frac{N(p-1)}{N-ps}}$, we can choose ${\delta }_3>0$ such that $\lambda ,\mu$ satisfying ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\delta }_3$, we can obtain

$$\begin{array}{cc}\hfill & {C((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}{)-C(\lambda +\mu )((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}{)}^{{\displaystyle \frac{N(p-1)}{p(N-ps)}}}{|log((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}})|\hfill \\ \hfill & <-C_0{((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}),\hfill \end{array}$$

when ${|log((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}})|\to +\infty$ for $\lambda ,\mu \to 0$ and

$$\begin{array}{c}\hfill {(\lambda +\mu )((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}{)}^{{\displaystyle \frac{N(p-1)}{p(N-ps)}}}\simeq {((\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}).\end{array}$$

Therefore, we take ${\mathsf{\Lambda }}_2=min\{{\delta }_1,{\delta }_2,{\delta }_3,{({\displaystyle \frac{\delta }{2}})}^{{\displaystyle \frac{N-ps}{p-1}}}\}>0$, for all $\lambda ,\mu$ satisfying ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_2$, we obtain

$$\begin{array}{c}\hfill \underset{t\ge 0}{sup}{\mathsf{\Phi }}_{\lambda ,\mu }(tu,tv)<c^{*}.\end{array}$$

This completes the proof. □

Lemma 15. 

Let ${\mathsf{\Lambda }}_3=min\{{\mathsf{\Lambda }}_2,{({\displaystyle \frac{q}{p}})}^{{\displaystyle \frac{p}{p-q}}}{\mathsf{\Lambda }}_1\}$. Assume $(H_1)$ and $(H_2)$. Then, for all ${0<(\lambda |f(x)|}_{L^t}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f(x)|}_{L^t}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_3$, there exists a $(u_2,v_2)\in N_{\lambda ,\mu }^{-}$ such that

(i) ${\mathsf{\Phi }}_{\lambda ,\mu }(u_2,v_2)=c_{\lambda ,\mu }^{-}>0$.

(ii) $(u_2,v_2)$ is a nontrivial solution of (1).

Proof. 

If ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_3$, then by Lemma 10-(ii), there exists a minimizing sequence $\{({\overline{u}}_k,{\overline{v}}_k)\}\subset N_{\lambda ,\mu }^{-}$ for ${\mathsf{\Phi }}_{\lambda ,\mu }$. From Lemma 13, Lemma 14, and Lemma 10-(ii), we know that ${\mathsf{\Phi }}_{\lambda ,\mu }$ satisfies the ${(PS)}_{c_{\lambda ,\mu }^{-}}$ condition and $c_{\lambda ,\mu }^{-}>0$. By Lemma 11, we obtain that $\{({\overline{u}}_k,{\overline{v}}_k)\}$ is bounded in W. Therefore, there exists a subsequence, still denoted by $\{({\overline{u}}_k,{\overline{v}}_k)\}$ and $(u_2,v_2)\in N_{\lambda ,\mu }^{-}$, such that $({\overline{u}}_k,{\overline{v}}_k)\to (u_2,v_2)$ strongly in W and ${\mathsf{\Phi }}_{\lambda ,\mu }(u_2,v_2)=c_{\lambda ,\mu }^{-}>0$ for all ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_3$. Finally, by the same arguments as in the proof of Theorem 1, we obtain that $(u_2,v_2)$ is a nontrivial solution of system (1). □

Proof of Theorem 2. 

Applying Theorem 1, we know that system (1) has a nontrivial solution $(u_1,v_1)\in N_{\lambda ,\mu }^{+}$ for all ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_1$. On the other hand, taking ${\mathsf{\Lambda }}_{*}=min\{{\mathsf{\Lambda }}_1,{\mathsf{\Lambda }}_2,{\mathsf{\Lambda }}_3\}$, from Lemma 15, we obtain the second nontrivial solution $(u_2,v_2)\in N_{\lambda ,\mu }^{-}$ for ${0<(\lambda |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}+{(\mu |f|}_{q^{*}}{)}^{{\displaystyle \frac{p}{p-q}}}<{\mathsf{\Lambda }}_{*}$. Since $N_{\theta ,\eta }^{+}\bigcap N_{\theta ,\eta }^{-}=\varnothing$, then those two solutions are distinct.

Now we prove that $(u_1,v_1)$ and $(u_2,v_2)$ are not semi-trivial solutions. By Theorem 1 and Lemma 15, we have

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u_2,v_2)<0,{\mathsf{\Phi }}_{\lambda ,\mu }(u_2,v_2)>0.\end{array}$$

(46)

We know that if $(u,0)$ (or $(0,v)$) is a semi-trivial solution of problem (1), then (1) becomes

$$\begin{array}{c}\hfill {(-{\Delta }_p)}^{s}u=\lambda f(x){|u|}^{q-2}u,x\in {\mathbb{R}}^N.\end{array}$$

(47)

Then, the corresponding energy functional of Equation (47) is given by

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(u,0)={\displaystyle \frac{1}{p}}\int {\int }_{{\mathbb{R}}^{2N}}{\displaystyle \frac{{|u(x)-u(y)|}^p}{{|x-y|}^{N+ps}}}dxdy-{\displaystyle \frac{1}{q}}{\int }_{{\mathbb{R}}^N}\lambda f(x){|u|}^{q}dx=-{\displaystyle \frac{p-q}{pq}}{\left[u\right]}_{s,p}^p<0.\end{array}$$

(48)

By (46) and (48), we know that $(u_2,v_2)$ is not a semi-trivial solution.

Next, we show that $(u_1,v_1)$ is not a semi-trivial solution. Without loss of generality, we may assume that $v_1\equiv 0$. Then, $u_1$ is a nontrivial solution of Equation (47) and

$$\begin{array}{c}\hfill ||(u_1,0){||}^p={\left[u_1\right]}_{s,p}^p=\lambda {\int }_{{\mathbb{R}}^N}f(x){|u_1|}^{q}dx>0.\end{array}$$

Additionally, we may choose $u_3\in {\dot{W}}^{s,p}({\mathbb{R}}^N)\setminus \{0\}$, such that

$$\begin{array}{c}\hfill ||(0,u_3){||}^p={\left[u_3\right]}_{s,p}^p=\mu {\int }_{{\mathbb{R}}^N}f(x){|u_3|}^{q}dx>0.\end{array}$$

From Lemma 7, we know that there exists a unique $0<t_1<t_{max}(u_1,u_3)$, such that $(t_1{u}_1,t_1{u}_3)\in N_{\lambda ,\mu }^{+}$, where

$$\begin{array}{c}\hfill t_{max}(u_1,u_3)={\left({\displaystyle \frac{(p_s^{*}-q){\int }_{{\mathbb{R}}^N}(\lambda f(x)|u_1{|}^q+\mu f(x)|u_3{|}^q)dx}{(p_s^{*}-p)||(u_1,u_3){||}^p}}\right)}^{{\displaystyle \frac{1}{p-q}}}={\left({\displaystyle \frac{p_s^{*}-q}{p_s^{*}-p}}\right)}^{{\displaystyle \frac{1}{p-q}}}>1.\end{array}$$

Moreover,

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\lambda ,\mu }(t_1{u}_1,t_1{u}_3)=\underset{0\le t\le t_{max}(u_1,u_3)}{inf}{\mathsf{\Phi }}_{\lambda ,\mu }(t{u}_1,t{u}_3).\end{array}$$

By means of the fact that $(u_1,0)\in N_{\lambda ,\mu }^{+}$, we have that

$$\begin{array}{c}\hfill c_{\lambda ,\mu }^{+}\le {\mathsf{\Phi }}_{\lambda ,\mu }(t_1{u}_1,t_1{u}_3)\le {\mathsf{\Phi }}_{\lambda ,\mu }(u_1,u_3)<{\mathsf{\Phi }}_{\lambda ,\mu }(u_1,0)=c_{\lambda ,\mu }^{+},\end{array}$$

which implies a contradiction. Thus, $(u_1,v_1)$ is not a semi-trivial solution. We complete the proof. □

6. Conclusions

In this paper, we have obtained the existence and multiplicity results of nontrivial solutions for a class of fractional p-Laplacian systems with critical homogeneous nonlinearity. In the next work, we will consider the case of fractional p-Laplacian systems with electromagnetic fields.