On Critical Fractional p&q-Laplacian Equations with Potential Vanishing at Infinity

Abstract

The goal of the present paper is to investigate the critical Schrödinger-type fractional $p\&q$-Laplacian problems. By employing the mountain pass theorem, we prove the existence and asymptotic property of nontrivial solutions for the problem.

1. Introduction and Main Results

In the present work, we study the existence of solutions to the critical Schrödinger-type fractional p&q-Laplacian equations:

$${(-\Delta )}_p^{s}u+{(-\Delta )}_q^{s}u+V\left(x\right)\left({\left|u\right|}^{p-2}u+{\left|u\right|}^{q-2}u\right)={M\left(x\right)(\mu f(x,u)+|u|}^{q_s^{*}-2}u)\mathrm{in}{\mathbb{R}}^N,$$

(1)

where $s\in (0,1),1<p<q<N/s,q_s^{*}=Nq/(N-qs)$. The function f is a continuous function with suitable conditions and $M,V$ are nonnegative continuous functions with appropriate assumptions. $\mu >0$ is a real parameter. The main operator ${(-\Delta )}_{\lambda }^s$ with $\lambda \in \{p,q\}$ is the fractional $\lambda$-Laplace operator which, up to a normalizing constant, may be defined as

$${(-\Delta )}_{\lambda }^{s}u\left(x\right):=2\underset{\epsilon \to 0^{+}}{lim}{\int }_{{\mathbb{R}}^N\backslash B_{\epsilon }\left(x\right)}\frac{{|u\left(x\right)-u\left(y\right)|}^{\lambda -2}(u\left(x\right)-u\left(y\right))}{{|x-y|}^{N+s\lambda }}\mathrm{d}y,x\in {\mathbb{R}}^N,$$

for any $u\in {\mathcal{C}}_c^{\infty }\left({\mathbb{R}}^N\right)$, where $B_{\epsilon }\left(x\right)=\left\{y\in {\mathbb{R}}^N:|x-y|<\epsilon \right\}$.

Throughout the paper, we assume that $(V,M)\in \mathcal{M}$ if the following conditions are fulfilled:

• $\left(V{M}_1\right)$$V\left(x\right),M\left(x\right)>0$ for all $x\in {\mathbb{R}}^N$ and $M\in L^{\infty }\left({\mathbb{R}}^N\right).$
• $\left(V{M}_2\right)$ if ${\left\{A_n\right\}}_{n\in \mathbb{N}}$ is a sequence of Borel sets such that the Lebesgue measure $\left|A_n\right|\le R$
  • for some $R>0$, then

    $$\underset{\rho \to \infty }{lim}{\int }_{A_n\cap B_{\rho }^c\left(0\right)}M\left(x\right)\mathrm{d}x=0,\mathrm{uniformly}\mathrm{in}n\in \mathbb{N},$$

    where $B_{\rho }^c\left(0\right):={\mathbb{R}}^N\backslash B_{\rho }\left(0\right).$
  • Furthermore, one of the following hypotheses occurs:

• $\left(V{M}_3\right)$${\displaystyle \frac{M}{V}\in L^{\infty }\left({\mathbb{R}}^N\right)}$.
• $\left(V{M}_4\right)$ there exists $m\in \left(q,q_s^{*}\right)$ such that:

  $$\underset{\left|x\right|\to \infty }{lim}\frac{M\left(x\right)}{{\left(V\left(x\right)\right)}^{\left(q_s^{*}-m\right)/\left(q_s^{*}-q\right)}}=0,\mathrm{where}{q}_s^{*}=qN/(N-qs).$$

For $p=2,s=1$, the assumptions on $V\left(x\right)$ and $M\left(x\right)$ were initially presented in [1], while these assumptions can be found in [2] as $p\ne 2,s=1$.

As for the nonlinearity f, we suppose that $f\in \mathcal{C}({\mathbb{R}}^N\times \mathbb{R},\mathbb{R})$ satisfies the following growth assumptions in the origin and at infinity:

• $\left(f_1\right)$${\displaystyle \underset{t\to 0}{lim}\frac{f(x,t)}{{\left|t\right|}^{p-1}}=0}$.
• $\left(f_2\right)$ there exists $\nu \in (q,q_s^{*})$ such that

  $$\underset{\left|t\right|\to \infty }{lim}\frac{f(x,t)}{{\left|t\right|}^{\nu -1}}=0.$$
• ${\left(f_1\right)}^{\prime }$ there exists $C>0$ such that

  $$\left|f(x,t)\right|\le {C\left|t\right|}^{m-1},$$
  • where m is given in $\left(V{M}_4\right)$.
• $\left(f_3\right)$ there exists $\theta \in \left(q,q_s^{*}\right)$ such that

  $$0<\theta F(x,t):=\theta {\int }_0^{t}f(x,\tau )\mathrm{d}\tau \le f(x,t)t\mathrm{for}\mathrm{all}\left|t\right|>0.$$
• $\left(f_4\right)$$f(x,t)=0$ for all $t\le 0$.

Due to its interesting structure and wide range of applications in areas such as finance, anomalous diffusion, phase transition, optimization, quasi-geostrophic flows, material science, soft thin films, water waves, multiple scattering, obstacle problem and so forth, nonlinear problems involving nonlocal operators have attracted a lot of attention of mathematical community in recent years. For more information, see [3,4].

It was well known that when $p=q=2$, Equation (1) arises in the investigation of the standing wave solutions $\psi (x,t)=$$u\left(x\right)e^{-ı\omega t}$ for the fractional Schrödinger equation:

$$ı\hslash \frac{\partial \psi }{\partial t}={\hslash }^2{(-\Delta )}^s\psi +W\left(x\right)\psi -g\left(\right|\psi \left|\right)\mathrm{in}{\mathbb{R}}^N,$$

where ℏ is the Planck constant, $W:{\mathbb{R}}^N\to \mathbb{R}$ is an external potential and g is a suitable nonlinearity. Due to its appearance in issues involving condensed matter physics, plasma physics and nonlinear optics, one of the most significant objects in fractional quantum mechanics is the fractional Schrödinger equation. By extending the Feynman path integral from the Brownian-like to the Lévy-like quantum mechanical paths, Laskin [5] proposed this equation for the first time. The investigation of fractional Schrödinger equations has recently attracted the interest of many mathematicians, and several works about the multiplicity, existence, regularity, and asymptotic behavior of solutions to subcritical or critical fractional Schrödinger equations under various conditions on the potentials have been published, see [6,7,8,9,10]. For instance, in [11] the authors considered the case that $V\left(x\right)\to 0\mathrm{as}\left|x\right|\to \infty$ for the following problem:

$$-\Delta u+V\left(x\right)u={M\left(x\right)\left|u\right|}^{\gamma },1<\gamma <2^{*}-1,$$

where $V,M\in \mathcal{C}({\mathbb{R}}^N,\mathbb{R})$ and there exist constants $b_1,b_2,b_3,B,k_1>0$, such that:

$$\frac{b_3}{1+{\left|x\right|}^{b_1}}\le V\left(x\right)\le B,0<M\left(x\right)\le \frac{k_1}{1+{\left|x\right|}^{b_2}},\forall x\in {\mathbb{R}}^N.$$

After that, Alves and Souto [1] considered more general weighted functions V and $M,$ so that the weighted Sobolev embedding theorems could be applied. As result, they obtained a ground state solution using a Hardy-type inequality and variational method. Applying the approach in [1], do Ó et al. [12] also obtained the existence of solutions for the equation:

$${(-\Delta )}^{s}u+V\left(x\right)u=\lambda M\left(x\right)g\left(u\right)+{\left|u\right|}^{2_s^{*}-2}u\mathrm{in}{\mathbb{R}}^N.$$

However, we think that there are some gaps in their paper. To prove the energy functional satisfying the conditions of mountain pass theorem, we need use the continuous embedding from $W\hookrightarrow L_M^{2_s^{*}}\left({\mathbb{R}}^N\right)$, then the term ${\left|u\right|}^{2_s^{*}-2}u$ must be replaced by the form ${M\left(x\right)\left|u\right|}_s^{2_s^{*}-2}u$, since M may vanish at infinity.

When $p=q=2$, Equation (1) reduces to the following critical fractional p-Laplacian equations of Schrödinger-type:

$${(-\Delta )}_p^{s}u+V\left(x\right){\left|u\right|}^{p-2}u=M\left(x\right)\left(\mu f(x,u)+{\left|u\right|}^{p_s^{*}-2}u\right)\mathrm{in}{\mathbb{R}}^N.$$

Here we emphasize that the nonlocality of fractional p-Laplacian and the interaction of nonlinearity make the study of the related fractional problems very challenging. In fact, the lack of Hilbertian structure in $W^{s,p}\left({\mathbb{R}}^N\right)$ for $p\ne 2$ makes it appear that standard tools used to analyze the linear situation $p=2$ are not trivially adaptable in the situation of $p\ne 2$. Due to these reasons, the related models involving the fractional p-Laplacian operator have attracted a lot of attention in the context of nonlocality; for example, see [13,14,15,16,17] and the references therein.

The study of fractional $p\&q$-Laplacian problems, on the other hand, has recently received a lot of interest; we list [18,19,20,21,22,23] for some existence and multiplicity results, and [24] (see also [20] ) for some regularity results. Few articles, nevertheless, address fractional problems such (1). Isernia [25] obtained the existence of a positive and a negative ground state solution to the following equation:

$${(-\Delta )}_p^{s}u+{(-\Delta )}_q^{s}u+V\left(x\right)\left({\left|u\right|}^{p-2}u+{\left|u\right|}^{q-2}u\right)=M\left(x\right)f\left(u\right)\mathrm{in}{\mathbb{R}}^N.$$

Very recently, the authors in [26] studied the following Kirchhoff-type equations:

$$\begin{array}{cc}\hfill K& \left({\int \int }_{{\mathbb{R}}^{\mathrm{2N}}}\frac{{|u\left(x\right)-u\left(y\right)|}^p}{{|x-y|}^{N+sp}}\mathrm{d}x\mathrm{d}y+{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\left(x\right)\right|}^p\mathrm{d}x\right)\left({(-\Delta )}_p^{s}u\left(x\right)+V\left(x\right){\left|u\right|}^{p-2}u\right)\hfill \\ & =M\left(x\right)\left(\lambda f(x,u)+{\left|u\right|}^{p_s^{*}-2}u\right).\hfill \end{array}$$

where $K:[0,+\infty )\to [0,+\infty )$ is a continuous Kirchhoff function, f is a continuous function satisfying the Ambrosetti-Rabinowitz type condition, M may vanish at infinity. They used the mountain pass theorem to demonstrate the existence of solutions for the above equation.

In the current article, we are interested in the existence of nontrivial nonnegative solutions to a fractional Schrödinger type $p\&q$-Laplacian problem with potentials allowing for vanishing behavior at infinity in this study, which is motivated by the aforementioned studies.

First, we introduce some notations before launching into our findings. Let $u:{\mathbb{R}}^N\mapsto \mathbb{R}.$ For $0<s<1$ and $p>1$, let us define ${\mathcal{D}}^{s,p}\left({\mathbb{R}}^N\right)$ be the closure of ${\mathcal{C}}_c^{\infty }\left({\mathbb{R}}^N\right)$ with respect to

$${\left[u\right]}_{s,p}:={\left[{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{{|u\left(x\right)-u\left(y\right)|}^p}{{|x-y|}^{N+sp}}\mathrm{d}x\mathrm{d}y\right]}^{\frac{1}{p}}.$$

We denote $W^{s,p}\left({\mathbb{R}}^N\right)$ as the following fractional Sobolev space

$$W^{s,p}\left({\mathbb{R}}^N\right):=\left\{{u:\left|u\right|}_p<+\infty ,{\left[u\right]}_{s,p}<+\infty \right\}$$

equipped with the natural norm

$${\parallel u\parallel }_{Ws,p\left({\mathbb{R}}^N\right)}:={\left({\left[u\right]}_{s,p}^p+{\left|u\right|}_p^p\right)}^{\frac{1}{p}},$$

where

$${\left|u\right|}_p^p:={\int }_{{\mathbb{R}}^N}{\left|u\right|}^p\mathrm{d}x.$$

Now, let us recall the embedding property, $W^{s,p}\left({\mathbb{R}}^N\right)$ is continuously embedded in $L^r\left({\mathbb{R}}^N\right)$ for any $r\in \left[p,p_s^{*}\right]$ and compactly embedded in $L_{\mathrm{loc}}^r\left({\mathbb{R}}^N\right)$ for any $r\in \left[1,p_s^{*}\right)$. See the introductory paper or monograph [3,5] for more details.

Let $E^{s,p}$ be the completion of ${\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^N\right)$, with respect to the norm:

$${\parallel u\parallel }_{V,p}={\left({\left[u\right]}_{s,p}^p+{\left|u\right|}_{p,V}^p\right)}^{1/p}{,\left|u\right|}_{p,V}^p={\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\left(x\right)\right|}^p\mathrm{d}x.$$

Let $E^{s,q}$ denote by the completion of ${\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^N\right)$, with respect to the norm:

$${\parallel u\parallel }_{V,q}={\left({\left[u\right]}_{s,q}^q+{\left|u\right|}_{q,V}^q\right)}^{1/q}{,\left|u\right|}_{q,V}^q={\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\left(x\right)\right|}^q\mathrm{d}x.$$

Then, $E^{s,p}$ and $E^{s,q}$ are uniformly convex Banach spaces (see Lemma 10 in [27]), and hence, $E^{s,p}$ and $E^{s,q}$ are reflexive Banach spaces. Let us define the weighted Lebesgue space

$$L_M^r\left({\mathbb{R}}^N\right)=\left\{u:{\mathbb{R}}^N\to \mathbb{R}|u\mathrm{is}\mathrm{measurable}\mathrm{and}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^r\mathrm{d}x<+\infty \right\},$$

with its norm

$${\parallel u\parallel }_{L_M^r\left({\mathbb{R}}^N\right)}^r={\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^r\mathrm{d}x$$

and the space

$$X=\left\{u\in W^{s,p}\left({\mathbb{R}}^N\right)\cap W^{s,q}\left({\mathbb{R}}^N\right):{\int }_{{\mathbb{R}}^N}V\left(x\right)\left({\left|u\right|}^p+{\left|u\right|}^q\right)\mathrm{d}x<\infty \right\}$$

with its norm

$${\parallel u\parallel }_X:={\parallel u\parallel }_{V,p}+{\parallel u\parallel }_{V,q}.$$

Definition 1.1. 

We say that $u\in X$ is a weak solution of problem (1) if

$$\begin{array}{ccc}& & {\int \int }_{{\mathbb{R}}^{2N}}\frac{{|u\left(x\right)-u\left(y\right)|}^{p-2}(u\left(x\right)-u\left(y\right))(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+sp}}\mathrm{d}x\mathrm{d}y\hfill \\ & & +{\int \int }_{{\mathbb{R}}^{2N}}\frac{{|u\left(x\right)-u\left(y\right)|}^{q-2}(u\left(x\right)-u\left(y\right))(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+sq}}\mathrm{d}x\mathrm{d}y\hfill \\ & & +{\int }_{{\mathbb{R}}^N}{V\left(x\right)\left|u\right|}^{p-2}u\phi dx+{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\right|}^{q-2}u\phi \mathrm{d}x\hfill \\ & =& \mu {\int }_{{\mathbb{R}}^N}M\left(x\right)f(x,u)\phi \mathrm{d}x+{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^{q_s^{*}-2}u\phi \mathrm{d}x\hfill \end{array}$$

for any $\phi \in X$.

Our main result can be stated as follows:

Theorem 1. 

Suppose that f satisfies $\left(f_1\right)-\left(f_4\right).$ Let $\left(V{M}_1\right),\left(V{M}_2\right)$ and $\left(V{M}_3\right)$ hold. Then there exists ${\mu }_{*}>0$ such that for all $\mu \ge {\mu }_{*}$, problem (1) possesses a nontrivial nonnegative solution $u_{\mu }\in X$. Moreover, we obtain $\parallel u_{\mu }{\parallel }_X\to 0$ as $\mu \to +\infty .$

Remark 1. 

In the case of $1<q<p<N/s,p<q_s^{*},f$ satisfies $\left(f_3\right)$ for $\theta \in (p,q_s^{*})$, the conclusion is also hold.

When potentials $V,M$ satisfy the conditions $\left(V{M}_1\right),\left(V{M}_2\right)$ and $\left(V{M}_4\right)$, we consider the following problem:

$${(-\Delta )}_p^{s}u+{(-\Delta )}_q^{s}u+V\left(x\right)\left({\left|u\right|}^{p-2}u+{\left|u\right|}^{q-2}u\right)={M\left(x\right)(\mu f(x,u)+|u|}^{m-2}u)\mathrm{in}{\mathbb{R}}^N,$$

(2)

where $\mu >0$ is a real parameter. Consequently, we obtain the following second main result:

Theorem 2. 

Suppose that f satisfies ${\left(f_1\right)}^{\prime },\left(f_3\right)$ for $\theta \in (q,m),\left(f_4\right)$. Let $\left(V{M}_1\right),\left(V{M}_2\right)$ and $\left(V{M}_4\right)$ hold. Then there is ${\mu }_{*}>0$ such that for all $\mu \ge {\mu }_{*}$, problem (2) possesses a nontrivial nonnegative solution $u_{\mu }\in X$. Moreover, we obtain $\parallel u_{\mu }{\parallel }_X\to 0$ as $\mu \to +\infty .$

Remark 2. 

In the case of $1<q<p<N/s,p<q_s^{*},\left(V{M}_4\right)$ hold for $m\in (p,q_s^{*})$ and f satisfies $\left(f_3\right)$ for $\theta \in (p,m)$, the conclusion is also hold.

The plan of this paper is as follows. In Section 2, we give some technical lemmas. In Section 3, we deal with the compactness. Our main results are proved in the last section.

2. Preliminary Results

At the beginning of this section, we give the following continuous and compactness result.

Lemma 1. 

(Lemma 2.2 and Lemma 2.3 in [25]) Suppose $(V,M)\in \mathcal{M}$.

(i)

If $\left(V{M}_3\right)$ holds true, then the embedding $E^{s,q}\hookrightarrow L_M^r\left({\mathbb{R}}^N\right)$ is continuous for all $r\in \left[q,q_s^{*}\right]$, and compact for all $r\in \left(q,q_s^{*}\right)$.

(ii)

If $\left(V{M}_4\right)$ holds true, then the embedding $E^{s,q}\hookrightarrow L_M^m\left({\mathbb{R}}^N\right)$ is continuous and compact.

By Lemma 1, there exists a best constant:

$$S_r=\underset{u\in E^{s,q},u\ne 0}{sup}\frac{{\parallel u\parallel }_{L_M^r\left({\mathbb{R}}^N\right)}}{{\parallel u\parallel }_{V,q}}$$

(3)

for any $r\in \left[q,q_s^{*}\right]$ if $\left(V{M}_3\right)$ holds, and

$$S_m=\underset{u\in E^{s,q},u\ne 0}{sup}\frac{{\parallel u\parallel }_{L_M^m\left({\mathbb{R}}^N\right)}}{{\parallel u\parallel }_{V,q}}$$

(4)

if $\left(V{M}_4\right)$ holds. In the following, we will give a result from which we can obtain the functional of (1) is ${\mathcal{C}}^1(X,\mathbb{R})$.

Lemma 2. 

Let $\left(V{M}_1\right),\left(V{M}_2\right)$ hold. Suppose that f fulfills $\left(f_1\right)$ and $\left(f_2\right)$ if $\left(V{M}_3\right)$ hold or f fulfills ${\left(f_1\right)}^{\prime }$ if $\left(V{M}_4\right)$ hold. Let

$$\mathrm{\Phi }\left(u\right)={\int }_{{\mathbb{R}}^N}M\left(x\right)F(x,u)\mathrm{d}x,u\in X,$$

then $\mathrm{\Phi }\in {\mathcal{C}}^1(X,\mathbb{R})$. Moreover, we obtain

$$⟨{\mathrm{\Phi }}^{\prime }\left(u\right),\phi ⟩={\int }_{{\mathbb{R}}^N}M\left(x\right)f(x,u)\phi \mathrm{d}x$$

for all $\phi \in X$.

Proof. 

By $\left(f_1\right)$ and $\left(f_2\right)$, we see that for all $\sigma >0$, there exists $C_{\sigma }>0$ such that

$$\left|f(x,\tau )\right|\le {\sigma \left|\tau \right|}^{p-1}+C_{\sigma }{\left|\tau \right|}^{q_s^{*}-1}\mathrm{for}\mathrm{all}(x,\tau )\in {\mathbb{R}}^N\times \mathbb{R}.$$

(5)

Then

$$\left|F(x,\tau )\right|\le \frac{\sigma }{p}{\left|\tau \right|}^p+\frac{C_{\sigma }}{q_s^{*}}{\left|\tau \right|}^{q_s^{*}}\mathrm{for}\mathrm{all}(x,\tau )\in {\mathbb{R}}^N\times \mathbb{R}.$$

(6)

According to Lemma 1, there exists $S_{\alpha }>0$ such that ${\parallel u\parallel }_{L_M^{\alpha }\left({\mathbb{R}}^N\right)}\le$$S_{\alpha }{\parallel u\parallel }_{V,q}$ for all $\alpha \in \left[q,q_s^{*}\right]$. Recalling that $\frac{M}{V}\in L^{\infty }\left({\mathbb{R}}^N\right),$ for all $u\in X,$ we obtain

$$\begin{array}{ccc}\hfill {\int }_{{\mathbb{R}}^N}M\left(x\right)\left|F(x,u)\right|\mathrm{d}x& =& {\int }_{{\mathbb{R}}^N}\left(\frac{\sigma }{p}{M\left(x\right)\left|u\right|}^p+\frac{C_{\sigma }}{q_s^{*}}M\left(x\right){\left|u\right|}^{q_s^{*}}\right)\mathrm{d}x\hfill \\ & \le & \frac{\sigma }{p}{\left|\frac{M}{V}\right|}_{\infty }{\parallel u\parallel }_{V,p}^p+\frac{C_{\sigma }}{q_s^{*}}{S}_{q_s^{*}}^{q_s^{*}}{\parallel u\parallel }_{V,q}^{q_s^{*}}\hfill \\ & \le & \frac{\sigma }{p}{\left|\frac{M}{V}\right|}_{\infty }{\parallel u\parallel }_X^p+\frac{C_{\sigma }}{q_s^{*}}{S}_{q_s^{*}}^{q_s^{*}}{\parallel u\parallel }_X^{q_s^{*}}<\infty .\hfill \end{array}$$

(7)

So $\mathrm{\Phi }$ is well defined on X. For any $\left|\iota \right|\in [0,1],$ it follows from $u,\phi \in X,r\in \left[q,q_s^{*}\right]$ that

$${|u+\iota \phi |}^{r-1}\le 2^{r-1}\left({\left|u\right|}^{r-1}+{\left|\phi \right|}^{r-1}\right).$$

Then we obtain

$$\begin{array}{cc}\hfill {|u+\iota \phi |}^{r-1}\left|\phi \right|& \le 2^{r-1}\left({\left|u\right|}^{r-1}\left|\phi \right|+{\left|\phi \right|}^r\right)\hfill \\ & \le 2^{r-1}\left(\frac{{\left|u\right|}^r}{r/(r-1)}+\frac{{\left|\phi \right|}^r}{r}+{\left|\phi \right|}^r\right)\hfill \\ & \le D\left({\left|u\right|}^r+{\left|\phi \right|}^r\right)\hfill \end{array}$$

by employing Young’s inequality, where $D>0$ is a constant. It implies that

$$\left|f(x,u+\iota \phi )\phi \right|\le {\sigma D\left(\right|u|}^p+{\left|\phi \right|}^p)+C_{\sigma }{D\left(\right|u|}^{q_s^{*}}+{\left|\phi \right|}^{q_s^{*}}).$$

(8)

From (8) and $\left(V{M}_3\right)$, for all $u,\phi \in X,$ it follows that

$$\begin{array}{cc}& {\int }_{{\mathbb{R}}^N}M\left(x\right)\left|f(x,u+\iota \phi )\phi \right|\mathrm{d}x\hfill \\ & \le \sigma D{\left|\frac{M}{V}\right|}_{\infty }\left({\parallel u\parallel }_{V,p}^p+{\parallel \phi \parallel }_{V,p}^p\right)+C_{\sigma }D{S}_{q_s^{*}}^{q_s^{*}}\left({\parallel u\parallel }_{V,q}^{q_s^{*}}+{\parallel \phi \parallel }_{V,q}^{q_s^{*}}\right)<+\infty ,\hfill \end{array}$$

which implies $M\left(x\right)f(x,u+\iota \phi )\phi \in L^1\left({\mathbb{R}}^N\right)$. For any $ϵ>0$, there exists $\xi =ϵ/C$ such that $\left|M\right(x\left)f\right(x,u+\iota \phi \left)\phi \right|\le C$ a.e. in ${\mathbb{R}}^N.$ Consequently, for any measurable set $U\subset {\mathbb{R}}^N$ such that $\left|U\right|<\xi$, we have

$${\int }_U\left|M\left(x\right)f(x,u+\iota \phi )\phi \right|\mathrm{d}x\le C\left|U\right|<C\xi =ϵ.$$

(9)

Additionally, since $M\left(x\right)f(x,u+\iota \phi )\phi \in L^1\left({\mathbb{R}}^N\right)$, there exists $\varrho >0$ such that

$${\int }_{{\mathbb{R}}^N\backslash B_{\varrho }\left(0\right)}\left|M\left(x\right)f(x,u+\iota \phi )\phi \right|\mathrm{d}x<ϵ.$$

(10)

It follows from (9) and (10) that ${\displaystyle {\int }_{{\mathbb{R}}^N}M\left(x\right)\left|f(x,u+\iota \phi )\phi \right|\mathrm{d}x}$ is equi-integrable. Please note that

$$\begin{array}{cc}\hfill ⟨{\mathrm{\Phi }}^{\prime }\left(u\right),\phi ⟩& =\underset{\iota \to 0}{lim}\frac{\mathrm{\Phi }(u+\iota \phi )-\mathrm{\Phi }\left(u\right)}{\iota }\hfill \\ & =\underset{\iota \to 0}{lim}{\int }_{{\mathbb{R}}^N}\frac{M\left(x\right)\left(F\right(x,u+\iota \phi )-F(x,u\left)\right)}{\iota }\mathrm{d}x.\hfill \end{array}$$

By dominated convergence theorem, the above integrals and limits can be exchanged in order, and since F is continuous, we can use Lagrange type formulas for the second variable, then

$$\begin{array}{ccc}& & \underset{\iota \to 0}{lim}{\int }_{{\mathbb{R}}^N}\frac{M\left(x\right)\left(F\right(x,u+\iota \phi )-F(x,u\left)\right)}{\iota }\mathrm{d}x\hfill \\ & =& \underset{\iota \to 0}{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right)f(x,u+\kappa \iota \phi )\phi \mathrm{d}x.\hfill \end{array}$$

Since $\kappa \left|\iota \right|\in [0,1]$ and for all $x\in {\mathbb{R}}^N,M\left(x\right)f\left(x,u+\kappa \iota \phi \right)\phi \to M\left(x\right)f(x,u)\phi$ as $\iota \to 0$, we obtain

$$⟨{\mathrm{\Phi }}^{\prime }\left(u\right),\phi ⟩={\int }_{{\mathbb{R}}^N}M\left(x\right)f(x,u)\phi \mathrm{d}x.$$

Therefore, $\mathrm{\Phi }$ is Gâteaux differentiable. It follows from (5) and the Hölder’s inequality that

$$\begin{array}{cc}\hfill \left|⟨{\mathrm{\Phi }}^{\prime }\left(u\right),\phi ⟩\right|& \le {\int }_{{\mathbb{R}}^N}M\left(x\right)\left|f(x,u)\phi \right|\mathrm{d}x\hfill \\ \hfill & \le \sigma {\int }_{{\mathbb{R}}^N}{M\left(x\right)\left|u\right|}^{p-1}\left|\phi \right|\mathrm{d}x+C_{\sigma }{\int }_{{\mathbb{R}}^N}{M\left(x\right)\left|u\right|}^{q_s^{*}-1}\left|\phi \right|\mathrm{d}x\hfill \\ \hfill & \le \sigma {\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^p\mathrm{d}x\right)}^{(p-1)/p}{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|\phi \right|}^p\mathrm{d}x\right)}^{1/p}\hfill \\ \hfill & +C_{\sigma }{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^{q_s^{*}}\mathrm{d}x\right)}^{(q_s^{*}-1)/q_s^{*}}{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|\phi \right|}^{q_s^{*}}\mathrm{d}x\right)}^{1/q_s^{*}}.\hfill \end{array}$$

(11)

Combining (11) and the inequality ${\parallel \phi \parallel }_{L_M^{\alpha }\left({\mathbb{R}}^N\right)}\le S_{\alpha }{\parallel \phi \parallel }_{V,q},$ we obtain

$$\left|⟨{\mathrm{\Phi }}^{\prime }\left(u\right),\phi ⟩\right|\le \left(\sigma {\left|\frac{M}{V}\right|}_{\infty }{\parallel u\parallel }_X^{p-1}+C_{\sigma }{S}_{q_s^{*}}^{q_s^{*}}{\parallel u\parallel }_X^{q_s^{*}-1}\right){\parallel \phi \parallel }_X.$$

It means that ${\mathrm{\Phi }}^{\prime }\left(u\right)\in X^{*}$.

Next, we will show that ${\mathrm{\Phi }}^{\prime }:X\to X^{*}$ is continuous on $X.$ Assume that $u_n\to u$ in $X,$ then we obtain

$$\begin{array}{cc}& u_n\to u\mathrm{in}X,\hfill \\ & u_n\to u\mathrm{in}{L}_M^r\left({\mathbb{R}}^N\right),r\in \left[q,q_s^{*}\right],\hfill \\ & M^{1/r}{u}_n\to M^{1/r}u\mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^N.\hfill \end{array}$$

Since $M\left(x\right)>0$ on ${\mathbb{R}}^N$, $u_n\left(x\right)\to u\left(x\right)$ a.e. in ${\mathbb{R}}^N$. Be aware that

$$\begin{array}{cc}\hfill ∥{\mathrm{\Phi }}^{\prime }\left(u_n\right)-{\mathrm{\Phi }}^{\prime }\left(u\right)∥& =\underset{{\parallel \phi \parallel }_X=1}{sup}\left|⟨{\mathrm{\Phi }}^{\prime }\left(u_n\right)-{\mathrm{\Phi }}^{\prime }\left(u\right),\phi ⟩\right|\hfill \\ & =\underset{{\parallel \phi \parallel }_X=1}{sup}\left|{\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right)\phi -f(x,u)\phi \right)\mathrm{d}x\right|.\hfill \end{array}$$

Set

$$\alpha :=\underset{n\to \infty }{lim}\underset{{\parallel \phi \parallel }_{X=1}}{sup}\left|{\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right)\phi -f(x,u)\phi \right)\mathrm{d}x\right|\ge 0.$$

If $\alpha >0$, then there exists a sequence $\left\{{\phi }_n\right\}\subset X,{∥{\phi }_n∥}_X=1$, such that, for n large enough,

$$\left|{\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)\mathrm{d}x\right|>\frac{\alpha }{2}.$$

Because of the boundedness of $\left\{{\phi }_n\right\}$ in X, we have

$$\begin{array}{cc}& {\phi }_n\rightharpoonup \phi \mathrm{in}X,\hfill \\ & {\phi }_n\to \phi \mathrm{in}{L}_M^r\left({\mathbb{R}}^N\right),r\in \left(q,q_s^{*}\right),\hfill \\ & M^{1/r}{\phi }_n\to M^{1/r}\phi \mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^N.\hfill \end{array}$$

Since $M\left(x\right)>0$ on ${\mathbb{R}}^N$, then ${\phi }_n\left(x\right)\to \phi \left(x\right)$ a.e. in ${\mathbb{R}}^N$. Just as with the same arguments that in (11), we obtain

$$\begin{array}{cc}\hfill & {\displaystyle \left|{\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)\mathrm{d}x\right|}\hfill \\ \le \hfill & {\displaystyle {\int }_{{\mathbb{R}}^N}M\left(x\right)\left(\left|f\left(x,u_n\right){\phi }_n\right|+\left|f(x,u){\phi }_n\right|\right)\mathrm{d}x}\hfill \\ \le \hfill & {\displaystyle \sigma {\int }_{{\mathbb{R}}^N}M\left(x\right)\left({\left|u_n\right|}^{p-1}\left|{\phi }_n\right|+{\left|u\right|}^{p-1}\left|{\phi }_n\right|\right)\mathrm{d}x}\hfill \\ \hfill & {\displaystyle +C_{\sigma }{\int }_{{\mathbb{R}}^N}M\left(x\right)\left({\left|u_n\right|}^{q_s^{*}-1}\left|{\phi }_n\right|+{\left|u\right|}^{q_s^{*}-1}\left|{\phi }_n\right|\right)\mathrm{d}x}\hfill \\ \le \hfill & {\displaystyle \sigma {\left|\frac{M}{V}\right|}_{\infty }\left(\parallel u_n{\parallel }_{V,p}^{p-1}\parallel {\phi }_n{\parallel }_{V,p}+{\parallel u\parallel }_{V,p}^{p-1}{\parallel {\phi }_n\parallel }_{V,p}\right)}\hfill \\ & {\displaystyle +C_{\sigma }{S}_{q_s^{*}}^{q_s^{*}}\left({∥u_n∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}-1}{∥{\phi }_n∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}+{\parallel u\parallel }_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}-1}{\parallel {\phi }_n\parallel }_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}\right)}\hfill \\ <\hfill & +\infty .\hfill \end{array}$$

(12)

Consequently, $M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-\right.$$\left.f(x,u){\phi }_n\right)\in L^1\left({\mathbb{R}}^N\right)$, and there is a constant $T>0$, such that $\left|M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)\right|\le T$ a.e. in ${\mathbb{R}}^N$. For any $ϵ>0$, there is $\zeta =ϵ/T>0$, such that for all $E\subset {\mathbb{R}}^N,$$\left|E\right|<\zeta$, we obtain

$${\int }_E\left|M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)\right|\mathrm{d}x\le T\left|E\right|<T\zeta =ϵ.$$

It implies that $M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)$ is equi-integrable on ${\mathbb{R}}^N$. Since $u_n\to u$ in $L_M^q\left({\mathbb{R}}^N\right)$ and $L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)$, the Brézis-Lieb Lemma implies that there is $\varrho >0$ such that

$${\int }_{{\mathbb{R}}^N\backslash B_{\varrho }\left(0\right)}M\left(x\right){\left|u\right|}^{q_s^{*}}\mathrm{d}x<{ϵ}^{q_s^{*}}\mathrm{and}{\int }_{{\mathbb{R}}^N\backslash B_{\varrho }\left(0\right)}M\left(x\right){\left|u_n\right|}^{q_s^{*}}\mathrm{d}x<{\left(2ϵ\right)}^{q_s^{*}}.$$

(13)

Choosing $\sigma$ sufficiently small in (12), then combining (12) and (13), we obtain

$${\int }_{{\mathbb{R}}^N\backslash B_{\varrho }\left(0\right)}\left|M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)\right|\mathrm{d}x<T_{*}ϵ,$$

where $T_{*}>0$ is a constant. Since

$$M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)\to 0\mathrm{a}.\mathrm{e}.\mathrm{on}{\mathbb{R}}^N,$$

it follows from the Vitali’s theorem that

$${\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right){\phi }_n-f(x,u){\phi }_n\right)\mathrm{d}x\to 0\mathrm{as}n\to \infty ,$$

that is a contradiction. So $\alpha =0$, and hence

$$∥{\mathrm{\Phi }}^{\prime }\left(u_n\right)-{\mathrm{\Phi }}^{\prime }\left(u\right)∥=\underset{{\parallel \phi \parallel }_X=1}{sup}\left|{\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right)\phi -f(x,u)\phi \right)\mathrm{d}x\right|\to 0\mathrm{as}n\to \infty .$$

As a result, ${\mathrm{\Phi }}^{\prime }$ is continuous on X, and therefore $\mathrm{\Phi }\in {\mathcal{C}}^1(X,\mathbb{R})$. Similarly, it is simple to prove the case $\left(V{M}_4\right)$. In fact, by ${\left(f_1\right)}^{\prime }$, we see that there exists C such that

$$\left|f(x,\tau )\right|\le {C\left|\tau \right|}^{m-1}\mathrm{for}\mathrm{all}(x,\tau )\in {\mathbb{R}}^N\times \mathbb{R}.$$

Hence, one has

$$\left|F(x,\tau )\right|\le \frac{C}{m}{\left|\tau \right|}^m\mathrm{for}\mathrm{all}(x,\tau )\in {\mathbb{R}}^N\times \mathbb{R}.$$

Just as with the same arguments for the case $\left(V{M}_3\right)$, we obtain that this lemma is also true for the case $\left(V{M}_4\right).$ □

We take the following energy functional into consideration while we look for solutions to problem (1):

$$\mathcal{J}\left(u\right)=\frac{1}{p}{\parallel u\parallel }_{V,p}^p+\frac{1}{q}{\parallel u\parallel }_{V,q}^q-\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)F(x,u)\mathrm{d}x-\frac{1}{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u^{+}\right|}^{q_s^{*}}\mathrm{d}x.$$

From Lemma 2, it is simple to obtain that $\mathcal{J}\in {\mathcal{C}}^1(X,\mathbb{R})$ and

$$\begin{array}{ccc}\hfill ⟨{\mathcal{J}}^{\prime }\left(u\right),\phi ⟩& =& {\int \int }_{{\mathbb{R}}^{2N}}\frac{{|u\left(x\right)-u\left(y\right)|}^{p-2}(u\left(x\right)-u\left(y\right))(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+sp}}\mathrm{d}x\mathrm{d}y\hfill \\ & & +{\int \int }_{{\mathbb{R}}^{2N}}\frac{{|u\left(x\right)-u\left(y\right)|}^{q-2}(u\left(x\right)-u\left(y\right))(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+sq}}\mathrm{d}x\mathrm{d}y\hfill \\ & & +{\int }_{{\mathbb{R}}^N}{V\left(x\right)\left|u\right|}^{p-2}u\phi \mathrm{d}x+{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\right|}^{q-2}u\phi \mathrm{d}x\hfill \\ & & -\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)f(x,u)\phi \mathrm{d}x-{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u^{+}\right|}^{q_s^{*}-1}\phi \mathrm{d}x\hfill \end{array}$$

for all $\phi \in X$.

To find solution of problem (2), we similarly take into account the functional

$${\mathcal{J}}_m\left(u\right)=\frac{1}{p}{\parallel u\parallel }_{V,p}^p+\frac{1}{q}{\parallel u\parallel }_{V,q}^q-\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)F(x,u)\mathrm{d}x-\frac{1}{m}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u^{+}\right|}^m\mathrm{d}x$$

instead of $\mathcal{J}\left(u\right)$.

3. Compactness

The Palais-Smale condition provides the compactness assumption needed by the mountain pass theorem (see [28,29] and references therein), so we first give the definition of Palais-Smale condition.

Definition 3.1. 

Let $\mathcal{J}$ be a functional in ${\mathcal{C}}^1(X,\mathbb{R})$. We say $\mathcal{J}$ satisfies the ${\left(PS\right)}_d$ condition if any sequence $\left\{u_n\right\}$ in X, such that $\mathcal{J}\left(u_n\right)\to d$ and $\underset{{\parallel \varphi \parallel }_X=1}{sup}\mid ⟨{\mathcal{J}}^{\prime }\left(u_n\right),\varphi ⟩$$\mid \to 0$, possesses a convergent subsequence in X.

Here the sequence $\left\{u_n\right\}$ in X such that $\mathcal{J}\left(u_n\right)\to d$ and $\underset{{\parallel \varphi \parallel }_X=1}{sup}⟨{\mathcal{J}}^{\prime }\left(u_n\right),\varphi ⟩\to 0$ is called the $\left(PS\right)$ sequence at level $d\in \mathbb{R}$.

Lemma 3. 

Let $\left(f_1\right)-\left(f_3\right)$ and $\left(V{M}_1\right),\left(V{M}_2\right),\left(V{M}_3\right)$ hold. Then, for all $\mu >0$, the following properties are fulfilled for the functional $\mathcal{J}$:

(i)

there exist positive constants ${\rho }_0,{\delta }_0$, such that $\mathcal{J}\left(u\right)\ge {\delta }_0$ for all $u\in X$ with ${\parallel u\parallel }_X={\rho }_0$.

(ii)

there exists $u_0\in X$ with $\parallel {u_0\parallel }_X>{\rho }_0$ such that $\mathcal{J}\left(u_0\right)<0$, where ${\rho }_0>0$ is given in (i).

Proof. 

(i) Using $\left(f_1\right)-\left(f_2\right)$ for all $\sigma >0,$ we can take $C_{\sigma }>0$ such that

$$\begin{array}{ccc}\hfill \mathcal{J}\left(u\right)& =& \frac{1}{p}{\parallel u\parallel }_{V,p}^p+\frac{1}{q}{\parallel u\parallel }_{V,q}^q-\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)F(x,u)\mathrm{d}x-\frac{1}{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u^{+}\right|}^{q_s^{*}}\mathrm{d}x\hfill \\ & \ge & \frac{1}{p}{\parallel u\parallel }_{V,p}^p+\frac{1}{q}{\parallel u\parallel }_{V,q}^q-\frac{\mu \sigma }{p}{\int }_{{\mathbb{R}}^N}{M\left(x\right)\left|u\right|}^p\mathrm{d}x-\frac{\mu C_{\sigma }+1}{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^{q_s^{*}}\mathrm{d}x\hfill \\ & \ge & \frac{1}{p}{\parallel u\parallel }_{V,p}^p+\frac{1}{q}{\parallel u\parallel }_{V,q}^q-\frac{\mu \sigma }{p}{\left|\frac{M}{V}\right|}_{\infty }{\int }_{{\mathbb{R}}^N}{V\left(x\right)\left|u\right|}^p\mathrm{d}x-\frac{\mu C_{\sigma }+1}{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^{q_s^{*}}\mathrm{d}x.\hfill \end{array}$$

Taking $\sigma =\frac{q-p}{\mu {\left|\frac{K}{V}\right|}_{\infty }q},$ choosing ${\parallel u\parallel }_X={\rho }_0$ small, applying Lemma 1, we obtain

$$\begin{array}{cc}\hfill \mathcal{J}\left(u\right)& \ge \frac{1}{q}\left({\parallel u\parallel }_{V,p}^p+{\parallel u\parallel }_{V,q}^q\right)-\frac{\left(\mu C_{\sigma }+1\right)S_{q_s^{*}}^{q_s^{*}}}{q_s^{*}}{\parallel u\parallel }_{V,q}^{q_s^{*}}\hfill \\ & \ge \frac{1}{q}\left({\parallel u\parallel }_{V,p}^q+{\parallel u\parallel }_{V,q}^q\right)-\frac{\left(\mu C_{\sigma }+1\right)S_{q_s^{*}}^{q_s^{*}}}{q_s^{*}}{\parallel u\parallel }_X^{q_s^{*}}\hfill \\ & \ge \frac{1}{2^{q-1}q}{\parallel u\parallel }_X^q-\frac{\left(\mu C_{\sigma }+1\right)S_{q_s^{*}}^{q_s^{*}}}{q_s^{*}}{\parallel u\parallel }_X^{q_s^{*}}.\hfill \end{array}$$

Since $1<q<q_s^{*}$, (i) is fulfilled.

(ii) For any $u\in {\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^N\right)$ with $u\ge 0$ in ${\mathbb{R}}^N,$$u\not\equiv 0$, we obtain

$$\mathcal{J}\left(tu\right)\le \frac{t^p}{p}{\parallel u\parallel }_{V,p}^p+\frac{t^q}{q}{\parallel u\parallel }_{V,q}^q-\frac{t^{q_s^{*}}}{q_s^{*}}{\int }_{suppu}M\left(x\right){\left|u^{+}\right|}^{q_s^{*}}\mathrm{d}x$$

for any $t>0$. Since $p<q<q_s^{*}$, we obtain $\mathcal{J}\left(tu\right)\to -\infty$ as $t\to +\infty$. Therefore, property (ii) also holds true. □

Fix $\mu >0$ and set

$$c_{\mu }=\underset{\chi \in \mathsf{\Gamma }}{inf}\underset{\tau \in [0,1]}{max}\mathcal{J}\left(\chi \left(\tau \right)\right),$$

(14)

where

$$\mathsf{\Gamma }=\{\chi \in \mathcal{C}([0,1],X):\chi (0)=0,\mathcal{J}(\chi \left(1\right))<0\}.$$

Undoubtedly, $c_{\mu }>0$ according to Lemma 3. Furthermore, we obtain the following lemma:

Lemma 4. 

Let $\left(f_1\right)-\left(f_3\right)$, and $\left(V{M}_1\right),\left(V{M}_2\right),\left(V{M}_3\right)$ hold. Then, $c_{\mu }\to 0$ as $\mu \to \infty .$

Proof. 

From (ii) in Lemma 3, we obtain $\mathcal{J}\left(t{u}_0\right)=-\infty$ as $t\to +\infty ,$ then, there exists $t_{\mu }>0$ such that $\mathcal{J}\left(t_{\mu }{u}_0\right)=\underset{t\ge 0}{max}\mathcal{J}\left(t{u}_0\right).$ Hence, $⟨{\mathcal{J}}^{\prime }\left(t_{\mu }{u}_0\right),t_{\mu }{u}_0⟩=0$. It implies that

$${∥t_{\mu }{u}_0∥}_{V,p}^p+{∥t_{\mu }{u}_0∥}_{V,q}^q=\mu t_{\mu }{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,t_{\mu }{u}_0\right)u_0\mathrm{d}x+t_{\mu }^{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^{q_s^{*}}\mathrm{d}x.$$

(15)

We now prove the boundedness of the sequence $\left\{t_{\mu }\right\}$. From (15) and $\left(f_3\right)$, we have

$$\begin{array}{cc}\hfill {∥t_{\mu }{u}_0∥}_X^p+{∥t_{\mu }{u}_0∥}_X^q& \ge {∥t_{\mu }{u}_0∥}_{V,p}^p+{∥t_{\mu }{u}_0∥}_{V,q}^q\hfill \\ \hfill & =\mu t_{\mu }{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,t_{\mu }{u}_0\right)v\mathrm{d}x+t_{\mu }^{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^{q_s^{*}}\mathrm{d}x\hfill \\ \hfill & \ge t_{\mu }^{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^{q_s^{*}}\mathrm{d}x.\hfill \end{array}$$

(16)

Due to $p<q<q_s^{*}$ and ${\displaystyle 0<{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^{q_s^{*}}\mathrm{d}x<+\infty }$, we can infer that $\left\{t_{\mu }\right\}$ is bounded. Fix any sequence $\left\{{\mu }_n\right\}$ such that ${\mu }_n\to \infty$. Then, up to a subsequence, there exists $t_0\ge 0$ such that $t_{{\mu }_n}\to t_0.$ We claim that $t_0=0.$ If $t_0>0$, the dominated convergence theorem leads to

$${\mu }_n{t}_{{\mu }_n}{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,t_{{\mu }_n}{u}_0\right)u_0\mathrm{d}x+t_{{\mu }_n}^{q_s^{*}}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^{q_s^{*}}\mathrm{d}x\to +\infty \mathrm{as}n\to \infty ,$$

which contradicts (16). Hence, $t_0=0$. That is to say, $t_{\mu }\to 0$ as $\mu \to \infty$. Put $\overline{\chi }\left(t\right)=t{u}_0$, we have $\overline{\chi }\in \mathsf{\Gamma }$, and thus

$$0<c_{\mu }\le \underset{t\ge 0}{max}\mathcal{J}\left(\overline{\chi }\left(t\right)\right)=\mathcal{J}\left(t_{\mu }{u}_0\right)\le \frac{1}{p}{∥t_{\mu }{u}_0∥}_{V,p}^p+\frac{1}{q}{∥t_{\mu }{u}_0∥}_{V,q}^q.$$

Letting $\mu \to +\infty ,$ we obtain $c_{\mu }\to 0$. □

Lemma 5. 

For each $\mu >0$. The $\left(PS\right)$ sequence $\left\{u_n\right\}\subset X$ for $\mathcal{J}$ at the level $c\in \mathbb{R}$ is bounded.

Proof. 

By a simple computation, for $n\in \mathbb{N}$ large enough we observe that

$$\begin{array}{ccc}\hfill C\left(1+{∥u_n∥}_X\right)& \ge & \mathcal{J}\left(u_n\right)-\frac{1}{\theta }⟨{\mathcal{J}}^{\prime }\left(u_n\right),u_n⟩\hfill \\ & =& \left(\frac{1}{p}-\frac{1}{\theta }\right){∥u_n∥}_{V,p}^p+\left(\frac{1}{q}-\frac{1}{\theta }\right){∥u_n∥}_{V,q}^q\hfill \\ & & +\frac{\mu }{\theta }{\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right)u_n-\theta F\left(x,u_n\right)\right)\mathrm{d}x\hfill \\ & & +\left(\frac{1}{\theta }-\frac{1}{q_s^{*}}\right){\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n^{+}\right|}^{q_s^{*}}\mathrm{d}x\hfill \\ & \ge & \left(\frac{1}{q}-\frac{1}{\theta }\right)\left({∥u_n∥}_{V,p}^p+{∥u_n∥}_{V,q}^q\right),\hfill \end{array}$$

(17)

thanks to $\mu >0,p<q<\theta <q_s^{*},\left(f_3\right)$ and $\left(f_4\right)$. With this in mind, we deduce that the sequence $\left\{u_n\right\}\subset X$ is bounded, and we omit the details here. Thus, the proof is completed. □

Lemma 6. 

Fix $u\in X$, define, for all $\phi \in X,$

$$G_u\left(\phi \right):={\int \int }_{{\mathbb{R}}^{2N}}\frac{{|u\left(x\right)-u\left(y\right)|}^{p-2}(u\left(x\right)-u\left(y\right))(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+sp}}\mathrm{d}x\mathrm{d}y+{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\right|}^{p-2}u\phi \mathrm{d}x$$

(18)

and

$${\tilde{G}}_u\left(\phi \right)={\int \int }_{{\mathbb{R}}^{2N}}\frac{{|u\left(x\right)-u\left(y\right)|}^{q-2}(u\left(x\right)-u\left(y\right))(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+sq}}\mathrm{d}x\mathrm{d}y+{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\right|}^{q-2}u\phi \mathrm{d}x.$$

(19)

If $u_n\rightharpoonup u_{\mu }$ in X, then,

(i)

$\underset{n\to \infty }{lim}{G}_{u_{\mu }}\left(u_n-u_{\mu }\right)=0\mathrm{and}\underset{n\to \infty }{lim}{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)=0.$

(ii)

$\underset{n\to \infty }{lim}{G}_{u_n}\left(\phi \right)=G_{u_{\mu }}\left(\phi \right)\mathrm{and}\underset{n\to \infty }{lim}{\tilde{G}}_{u_n}\left(\phi \right)={\tilde{G}}_{u_{\mu }}\left(\phi \right)\mathrm{for}\mathrm{all}\phi \in X.$

Proof. 

(i) By the Hölder inequality, it is obvious that $G_u$ is continuous and linear, and

$$\left|G_u\left(\phi \right)\right|\le 2{\parallel u\parallel }_{V,p}^{p-1}{\parallel \phi \parallel }_{V,p}\le 2{\parallel u\parallel }_X^{p-1}{\parallel \phi \parallel }_X\mathrm{for}\mathrm{all}\phi \in X.$$

Similarly, ${\tilde{G}}_u$ is also a continuous linear mapping on X. From $u_n\rightharpoonup u_{\mu }$ in X, then we have

$$\underset{n\to \infty }{lim}{G}_{u_{\mu }}\left(u_n-u_{\mu }\right)=0\mathrm{and}\underset{n\to \infty }{lim}{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)=0.$$

(ii) Set $t\in \{p,q\}$. Since the sequence

$${\left\{\frac{{\left|u_n\left(x\right)-u_n\left(y\right)\right|}^{t-2}\left(u_n\left(x\right)-u_n\left(y\right)\right)}{{|x-y|}^{(N+st)\left(1-\frac{1}{t}\right)}}\right\}}_{n\in \mathbb{N}}\mathrm{is}\mathrm{bounded}\mathrm{in}{L}^{\frac{t}{t-1}}\left({\mathbb{R}}^{2N}\right)$$

and

$$\frac{{\left|u_n\left(x\right)-u_n\left(y\right)\right|}^{t-2}\left(u_n\left(x\right)-u_n\left(y\right)\right)}{{|x-y|}^{(N+st)\left(1-\frac{1}{t}\right)}}\to \frac{|u_{\mu }\left(x\right)-u_{\mu }{\left(y\right)|}^{t-2}(u_{\mu }\left(x\right)-u_{\mu }\left(y\right))}{{|x-y|}^{(N+st)\left(1-\frac{1}{t}\right)}}\mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^{2N},$$

up to a subsequence, we may suppose that for any $h\in L^t\left({\mathbb{R}}^{2N}\right)$ it holds

$$\begin{array}{ccc}\hfill & & {\int \int }_{{\mathbb{R}}^{2N}}\frac{{\left|u_n\left(x\right)-u_n\left(y\right)\right|}^{t-2}\left(u_n\left(x\right)-u_n\left(y\right)\right)}{{|x-y|}^{(N+st)\left(1-\frac{1}{t}\right)}}h(x,y)\mathrm{d}x\mathrm{d}y\hfill \\ & \to & {\int \int }_{{\mathbb{R}}^{2N}}\frac{|u_{\mu }\left(x\right)-u_{\mu }{\left(y\right)|}^{t-2}(u_{\mu }\left(x\right)-u_{\mu }\left(y\right))}{{|x-y|}^{(N+st)\left(1-\frac{1}{t}\right)}}h(x,y)\mathrm{d}x\mathrm{d}y.\hfill \end{array}$$

(20)

Let $\phi \in X$ and

$$h(x,y):=\frac{\phi \left(x\right)-\phi \left(y\right)}{{|x-y|}^{\frac{N+st}{t}}}.$$

(21)

Thus, $h\in L^t\left({\mathbb{R}}^{2N}\right)$, and using (21) in (20) we obtain that

$$\begin{array}{c}{\int \int }_{{\mathbb{R}}^{2N}}\frac{{\left|u_n\left(x\right)-u_n\left(y\right)\right|}^{t-2}\left(u_n\left(x\right)-u_n\left(y\right)\right)(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+st}}\mathrm{d}x\mathrm{d}y\\ \to {\int \int }_{{\mathbb{R}}^{2N}}\frac{|u_{\mu }\left(x\right)-u_{\mu }{\left(y\right)|}^{t-2}(u_{\mu }\left(x\right)-u_{\mu }\left(y\right))(\phi \left(x\right)-\phi \left(y\right))}{{|x-y|}^{N+st}}\mathrm{d}x\mathrm{d}y.\end{array}$$

Please note that

$${\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_n\right|}^{t-2}{u}_n\phi \mathrm{d}x\to {\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_{\mu }\right|}^{t-2}{u}_{\mu }\phi \mathrm{d}x,$$

thus, we obtain

$$\underset{n\to \infty }{lim}{G}_{u_n}\left(\phi \right)=G_{u_{\mu }}\left(\phi \right)\mathrm{and}\underset{n\to \infty }{lim}{\tilde{G}}_{u_n}\left(\phi \right)={\tilde{G}}_{u_{\mu }}\left(\phi \right)\mathrm{for}\mathrm{all}\phi \in X.$$

□

Lemma 7. 

Let $\left(f_1\right)-\left(f_3\right)$, and $\left(V{M}_1\right),\left(V{M}_2\right),\left(V{M}_3\right)$ hold. Then, there exists ${\mu }_{*}>0$, for all $\mu \ge {\mu }_{*}$, $\mathcal{J}$ satisfies the ${\left(PS\right)}_{c_{\mu }}$ condition on $X.$

Proof. 

Let $\left\{u_n\right\}\subset X$ be the ${\left(PS\right)}_{c_{\mu }}$ sequence for $\mathcal{J}$, then there exists $C>0$, such that $\left|⟨{\mathcal{J}}^{\prime }\left(u_n\right),u_n⟩\right|\le C{∥u_n∥}_X$ and $\left|\mathcal{J}\left(u_n\right)\right|\le C$. From Lemma 5 it follows that $\left\{u_n\right\}$ is bounded. Thus, up to a subsequence (still denoted by itself), there exists $u_{\mu }\in X$ and ${\delta }_{\mu }\ge 0,{\xi }_{\mu }\ge 0,{\zeta }_{\mu }\ge 0$ such that

$$\begin{array}{cc}\hfill & u_n\rightharpoonup u_{\mu }\mathrm{in}X\cap L_M^{q_s^{*}}\left({\mathbb{R}}^N\right),\hfill \\ \hfill & u_n\to u_{\mu }\mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^N,\hfill \\ \hfill & {∥u_n∥}_{V,p}\to {\xi }_{\mu },{∥u_n∥}_{V,q}\to {\zeta }_{\mu },{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}\to {\delta }_{\mu },\hfill \\ \hfill & {\left|u_n\right|}^{q_s^{*}-2}{u}_n\rightharpoonup {\left|u_{\mu }\right|}^{q_s^{*}-2}{u}_{\mu }\mathrm{in}{L}_M^{{\left(q_s^{*}\right)}^{\prime }}\left({\mathbb{R}}^N\right).\hfill \end{array}$$

(22)

Notice that

$$\parallel {u_n\parallel }_{V,p}^p+{\parallel u_n\parallel }_{V,q}^q\to {\xi }_{\mu }^p+{\zeta }_{\mu }^q:={\beta }_{\mu }\ge 0.$$

Next, we prove that

$$\underset{\mu \to +\infty }{lim}{\beta }_{\mu }=0.$$

(23)

If there is a sequence ${\lambda }_k\to \infty$ such that ${\beta }_{{\lambda }_k}\to \beta >0$ as $k\to \infty$. Like (17), we have

$$c_{{\mu }_k}\ge \left(\frac{1}{q}-\frac{1}{\theta }\right)\left({∥u_n∥}_{V,p}^p+{∥u_n∥}_{V,q}^q\right).$$

When we use Lemma 4 and take $k\to \infty$ into consideration on both sides of the inequality above, there is a contradiction. So (23) is proved. It is easy to see that

$$\underset{\mu \to +\infty }{lim}{\xi }_{\mu }=0\mathrm{and}\underset{\mu \to +\infty }{lim}{\zeta }_{\mu }=0.$$

Moreover, we can deduce that ${∥u_{\mu }∥}_X\le \underset{n\to \infty }{lim}$${∥u_n∥}_X={\xi }_{\mu }+{\zeta }_{\mu }$ since $u_n\rightharpoonup u_{\mu }$ in X. From ${∥u_n∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}\le S_{q_s^{*}}{∥u_n∥}_{V,q},$ we obtain

$$\underset{\mu \to \infty }{lim}{∥u_{\mu }∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}=\underset{\mu \to \infty }{lim}{∥u_{\mu }∥}_X=0.$$

(24)

Combining Lemma 6 and (22), we obtain

$$\begin{array}{ccc}\hfill o_n\left(1\right)& =& ⟨{\mathcal{J}}^{\prime }\left(u_n\right)-{\mathcal{J}}^{\prime }\left(u_{\mu }\right),u_n-u_{\mu }⟩\hfill \\ & =& G_{u_n}\left(u_n-u_{\mu }\right)+{\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-G_{u_{\mu }}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & & -\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)\left[f\left(x,u_n\right)-f\left(x,u_{\mu }\right)\right]\left(u_n-u_{\mu }\right)\mathrm{d}x\hfill \\ & & -{\int }_{{\mathbb{R}}^N}M\left(x\right)\left({\left|u_n^{+}\right|}^{q_s^{*}-1}-{\left|u_{\mu }^{+}\right|}^{q_s^{*}-1}\right)\left(u_n-u_{\mu }\right)\mathrm{d}x\hfill \\ & =& G_{u_n}\left(u_n-u_{\mu }\right)-G_{u_{\mu }}\left(u_n-u_{\mu }\right)+{\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & & -{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}+o_n\left(1\right).\hfill \end{array}$$

(25)

Here, we make use of the following estimations:

(i)

${\displaystyle \underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^{q_s^{*}-2}{u}_n{u}_{\mu }\mathrm{d}x={\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }\right|}^{q_s^{*}}\mathrm{d}x}$ since ${\left|u_n\right|}^{q_s^{*}-2}{u}_n\rightharpoonup$${\left|u_{\mu }\right|}^{q_s^{*}-2}{u}_{\mu }$ in $L_M^{{\left(q_s^{*}\right)}^{\prime }}\left({\mathbb{R}}^N\right)$;

(ii)

${\displaystyle \underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }\right|}^{q_s^{*}-2}{u}_{\mu }{u}_n\mathrm{d}x={\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }\right|}^{q_s^{*}}\mathrm{d}x}$, since $u_n\rightharpoonup u_{\mu }$ in $L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)$

and ${\left|u_{\mu }\right|}^{q_s^{*}-2}{u}_{\mu }\in L_M^{{\left(q_s^{*}\right)}^{\prime }}\left({\mathbb{R}}^N\right)$;

(iii)

By the Brézis-Lieb Lemma, we obtain:

$$\begin{array}{cc}& {∥u_n-u_{\mu }∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}={∥u_n∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}-{∥u_{\mu }∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}+o\left(1\right),\hfill \\ & {∥u_n-u_{\mu }∥}_{V,p}^p={∥u_n∥}_{V,p}^p-{\parallel u_{\mu }\parallel }_{V,p}^p+o\left(1\right),\hfill \\ & {∥u_n-u_{\mu }∥}_{V,q}^q={∥u_n∥}_{V,q}^q-{\parallel u_{\mu }\parallel }_{V,q}^q+o\left(1\right).\hfill \end{array}$$

(iv)

${\displaystyle \underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right)\left[f\left(x,u_n\right)-f\left(x,u_{\mu }\right)\right]\left(u_n-u_{\mu }\right)\mathrm{d}x=0.}$

Now, we show (iv). It follows from the Hölder’s inequality that

$$\begin{array}{ccc}\hfill & & \left|{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_n\right)\left(u_n-u_{\mu }\right)\mathrm{d}x\right|\hfill \\ & \le & \sigma {\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^{p-1}\left|u_n-u_{\mu }\right|\mathrm{d}x+C_{\sigma }{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^{\nu -1}\left|u_n-u_{\mu }\right|\mathrm{d}x\hfill \\ & \le & \sigma {\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^p\mathrm{d}x\right)}^{\frac{p-1}{p}}{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x\right)}^{\frac{1}{p}}\hfill \\ & & +C_{\sigma }{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^{\nu }\mathrm{d}x\right)}^{\frac{\nu -1}{\nu }}{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^{\nu }\mathrm{d}x\right)}^{\frac{1}{\nu }}.\hfill \end{array}$$

(26)

By means of $\left(V{M}_3\right)$, we have

$$\begin{array}{ccc}\hfill {\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^p\mathrm{d}x& \le & {\left|\frac{M}{V}\right|}_{\infty }{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_n\right|}^p\mathrm{d}x<\infty .\hfill \end{array}$$

(27)

Since $\nu \in (q,q_s^{*})$, from (3), we have

$$\parallel u_n{\parallel }_{L_M^{\nu }\left({\mathbb{R}}^N\right)}\le S_{\nu }\parallel u_n{\parallel }_{V,q}\le S_{\nu }{\parallel u_n\parallel }_X<\infty .$$

(28)

According to Lemma 1, the embedding $E^{s,q}\hookrightarrow L_M^{\nu }\left({\mathbb{R}}^N\right)$ is compact. Then we have

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^{\nu }\mathrm{d}x=0.$$

(29)

Next, we claim that

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x=0.$$

(30)

By $\left(V{M}_3\right)$, we obtain

$${\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x\le {\left|\frac{M}{V}\right|}_{\infty }{\int }_{{\mathbb{R}}^N}V\left(x\right){|u_n-u_{\mu }|}^p\mathrm{d}x<\infty ,$$

that is to say, $u_n-u_{\mu }\in L_M^p\left({\mathbb{R}}^N\right),$ then for any $ϵ>0$, there exists $R_0>0,$ such that

$${\int }_{{\mathbb{R}}^N\backslash B_R}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x<ϵ\mathrm{for}\mathrm{all}R\ge R_0.$$

(31)

According to the embedding theorem on bounded domain $B_R$, up to a subsequence, we may assume that $u_n\to u_{\mu }$ in $L^p\left(B_R\right)$. Since $M\left(x\right)\in L^{\infty }\left({\mathbb{R}}^N\right),$ we have

$${\int }_{B_R}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x\le {\left|M\left(x\right)\right|}_{\infty }{\int }_{B_R}{\left|u_n-u_{\mu }\right|}^p\mathrm{d}x=0,$$

then,

$$\begin{array}{cc}\hfill & \left|{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x\right|\hfill \\ \hfill & \le \left|{\int }_{B_R}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x\right|+\left|{\int }_{{\mathbb{R}}^N\backslash B_R}M\left(x\right){\left|u_n-u_{\mu }\right|}^p\mathrm{d}x\right|\to 0.\hfill \end{array}$$

(32)

Therefore, we obtain (30). According to (26)–(30), we obtain

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_n\right)\left(u_n-u_{\mu }\right)\mathrm{d}x=0.$$

Similarly,

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_{\mu }\right)\left(u_n-u_{\mu }\right)\mathrm{d}x=0.$$

Hence, (iv) follows. From (25) it follows that

$$\begin{array}{ccc}\hfill & & \underset{n\to \infty }{lim}{G}_{u_n}\left(u_n-u_{\mu }\right)-G_{u_{\mu }}\left(u_n-u_{\mu }\right)+{\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & =& \underset{n\to \infty }{lim}{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}.\hfill \end{array}$$

(33)

Using the Brézis-Lieb Lemma and (22), we obtain

$$\begin{array}{ccc}\hfill c_{\mu }+o_n\left(1\right)& =& \mathcal{J}\left(u_n\right)-\frac{1}{\theta }⟨{\mathcal{J}}^{\prime }\left(u_n\right),u_n⟩\hfill \\ & \ge & \left(\frac{1}{\theta }-\frac{1}{q_s^{*}}\right){∥u_n^{+}∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}\hfill \\ & =& \left(\frac{1}{\theta }-\frac{1}{q_s^{*}}\right)\left({\delta }_{\mu }^{q_s^{*}}+{∥u_{\mu }^{+}∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}\right)+o\left(1\right).\hfill \end{array}$$

(34)

Combining Lemma 4 and (24), we have

$$\underset{\mu \to +\infty }{lim}{\delta }_{\mu }=0.$$

(35)

Next, we divide the proof into two cases.

First, we study the case $q\ge 2$. Using the well-known Simon inequality:

$${|a-b|}^t\le c_t\left({\left|a\right|}^{t-2}a-{\left|b\right|}^{t-2}b\right)(a-b),\mathrm{for}t\ge 2,$$

we obtain

$$\begin{array}{ccc}\hfill & & G_{u_n}\left(u_n-u_{\mu }\right)-G_{u_{\mu }}\left(u_n-u_{\mu }\right)+{\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & & \ge c_q^{-1}\parallel u_n-u_{\mu }{\parallel }_{V,q}^q\ge c_q^{-1}{S}_{q_s^{*}}^{-q}\parallel u_n-u_{\mu }{\parallel }_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^q\ge c_q^{-1}{S}_{q_s^{*}}^{-q}{\parallel {(u_n-u_{\mu })}^{+}\parallel }_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^q.\hfill \end{array}$$

(36)

Combining (22), (33), and (36), we obtain

$${\delta }_{\mu }^{q_s^{*}}\ge S_{q_s^{*}}^{-q}{c}_q^{-1}{\delta }_{\mu }^q.$$

(37)

If ${\delta }_{{\mu }_k}>0$ for some sequence $\left\{{\mu }_k\right\}:{\mu }_k\to +\infty$ as $k\to \infty ,$ then from (33), and note that $G_{u_{\mu }}\left(u_n-u_{\mu }\right)\to 0,{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\to 0$ as $n\to \infty ,G_{u_n}\left(u_n\right)={∥u_n∥}_{V,p}^p\to {\xi }_{\mu }^p,{\tilde{G}}_{u_n}\left(u_n\right)={∥u_n∥}_{V,q}^q\to {\zeta }_{\mu }^q$ as $n\to \infty$ and $G_{u_n}\left(u_{\mu }\right)\to G_{u_{\mu }}\left(u_{\mu }\right)\ge 0,{\tilde{G}}_{u_n}\left(u_{\mu }\right)\to G_{u_{\mu }}\left(u_{\mu }\right)\ge 0$, we have

$${\beta }_{{\mu }_k}-G_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)-{\tilde{G}}_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)={\delta }_{{\mu }_k}^{q_s^{*}}.$$

(38)

By (37) and (38), we obtain

$${\left({\delta }_{{\mu }_k}^{q_s^{*}}\right)}^{\left(q_s^{*}-q\right)/q_s^{*}}={\left({\beta }_{{\mu }_k}-G_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)-{\tilde{G}}_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)\right)}^{\left(q_s^{*}-q\right)/q_s^{*}}\ge S_{q_s^{*}}^{-q}{c}_q^{-1}.$$

This implies that

$${\beta }_{{\mu }_k}^{\left(q_s^{*}-q\right)/q_s^{*}}\ge {\left({\beta }_{{\mu }_k}-G_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)-{\tilde{G}}_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)\right)}^{\left(q_s^{*}-q\right)/q_s^{*}}\ge S_{q_s^{*}}^{-q}{c}_q^{-1}.$$

Thus, we obtain

$${\beta }_{{\mu }_k}^{q_s^{*}-q}\ge {\left(S_{q_s^{*}}^{-q}{c}_q^{-1}\right)}^{q_s^{*}}.$$

(39)

Combining (23) and (39), we obtain a contradiction. So ${\delta }_{\mu }=0$ for some ${\mu }_{*}>0$ if $\mu \ge {\mu }_{*}.$ That is

$$\underset{n\to \infty }{lim}{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^{q_s^{*}}=0$$

(40)

for all $\mu \ge {\mu }_{*}$. Appealing to the Brézis-Lieb Lemma and combining (25), (33), (40), we obtain

$$\underset{n\to \infty }{lim}\left({∥u_n-u_{\mu }∥}_{V,p}^p+{∥u_n-u_{\mu }∥}_{V,q}^q\right)=0.$$

Thus,

$$\underset{n\to \infty }{lim}{∥u_n-u_{\mu }∥}_X=0,$$

which implies $u_n\to u_{\mu }$ in X for all $\mu \ge {\mu }_{*}$.

For the case $1<q<2$. Since $u_n$ is bounded in X and $u_n\rightharpoonup u$ in X, then $\parallel {u_{\mu }\parallel }_X\le L$ for some $L>0.$ Thus

$$\begin{array}{ccc}\hfill & & {∥u_n-u∥}_{V,q}^q\hfill \\ & \le & {\left[u_n-u_{\mu }\right]}_{s,q}^q+{\left|u_n-u_{\mu }\right|}_{q,V}^q\hfill \\ & =& {\int \int }_{{\mathbb{R}}^{2N}}{\left|u_n\left(x\right)-u_n\left(y\right)-\left(u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right)\right|}^q{\left|x-y\right|}^{-\left(N+sq\right)}\mathrm{d}x\mathrm{d}y\hfill \end{array}$$

(41)

$$\begin{array}{ccc}& & +{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_n-u_{\mu }\right|}^q\mathrm{d}x.\hfill \end{array}$$

(42)

The Simon’s inequality:

$${|a-b|}^t\le C_t{\left[\left({\left|a\right|}^{t-2}a-{\left|b\right|}^{t-2}b\right)(a-b)\right]}^{t/2}{\left({\left|a\right|}^t+{\left|b\right|}^t\right)}^{(2-t)/2},\mathrm{for}1<t<2,$$

implies that

$$\begin{array}{ccc}\hfill {\left[u_n-u_{\mu }\right]}_{s,q}^q& \le & C_q{\int \int }_{{\mathbb{R}}^{2N}}[\left({\left|u_n\left(x\right)-u_n\left(y\right)\right|}^{q-2}\left(u_n\left(x\right)-u_n\left(y\right)\right)\right.\hfill \\ & & \left.-{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^{q-2}\left(u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right)\right)\left(u_n\left(x\right)-u_n\left(y\right)\right.\hfill \\ & & \left.-u_{\mu }\left(x\right)+u_{\mu }\left(y\right)\right){|x-y|}^{-(N+sq)}{]}^{q/2}\hfill \\ & & \times {\left[\left({\left|u_n\left(x\right)-u_n\left(y\right)\right|}^q+{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^q\right){|x-y|}^{-(N+sq)}\right]}^{(2-q)/2}\mathrm{d}x\mathrm{d}y\hfill \\ & \le & C_q\left({\int \int }_{{\mathbb{R}}^{2N}}\right[{\left|u_n\left(x\right)-u_n\left(y\right)\right|}^{q-2}\left(u_n\left(x\right)-u_n\left(y\right)\right)\hfill \\ & & -{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^{q-2}(u_{\mu }\left(x\right)\hfill \\ & & -u_{\mu }\left(y\right)\left)\right]\left(u_n\left(x\right)-u_n\left(y\right)-u_{\mu }\left(x\right)+u_{\mu }\left(y\right)\right){\left|x-y\right|}^{-\left(N+sq\right)}\mathrm{d}x\mathrm{d}y{)}^{q/2}\hfill \\ & & \times \left({\int \int }_{{\mathbb{R}}^{2N}}\left({\left|u_n\left(x\right)-u_n\left(y\right)\right|}^q\right.\right.\hfill \\ & & \left.+{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^q\right){\left|x-y\right|}^{-\left(N+sq\right)}\mathrm{d}x\mathrm{d}y{)}^{(2-q)/2}.\hfill \end{array}$$

(43)

Similarly, we have

$$\begin{array}{ccc}\hfill & & {\left|u_n-u_{\mu }\right|}_{q,V}^q\hfill \\ & =& {\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_n\left(x\right)-u_{\mu }\left(x\right)\right|}^q\mathrm{d}x\hfill \\ & \le & C_q{\int }_{{\mathbb{R}}^N}{\left[V\left(x\right)\left({\left|u_n\left(x\right)\right|}^{q-2}{u}_n\left(x\right)-{\left|u_{\mu }\left(x\right)\right|}^{q-2}{u}_{\mu }\left(x\right)\right)\left(u_n\left(x\right)-u_{\mu }\left(x\right)\right)\right]}^{q/2}\hfill \\ & & \times {\left[V\left(x\right)\left({\left|u_n\left(x\right)\right|}^q+{\left|u_{\mu }\left(x\right)\right|}^q\right)\right]}^{(2-q)/2}\mathrm{d}x\hfill \\ & \le & {\left({\int }_{{\mathbb{R}}^N}\left[V\left(x\right)\left({\left|u_n\left(x\right)\right|}^{q-2}{u}_n\left(x\right)-{\left|u_{\mu }\left(x\right)\right|}^{q-2}{u}_{\mu }\left(x\right)\right)\right]\left(u_n\left(x\right)-u_{\mu }\left(x\right)\right)\mathrm{d}x\right)}^{q/2}\hfill \end{array}$$

(44)

$$\begin{array}{ccc}& & \times {\left({\int }_{{\mathbb{R}}^N}V\left(x\right)\left({\left|u_n\left(x\right)\right|}^q+{\left|u_{\mu }\left(x\right)\right|}^q\right)\mathrm{d}x\right)}^{(2-q)/2}.\hfill \end{array}$$

(45)

By (41)–(44), we get

$${∥u_n-u∥}_{V,q}^q\le 2C_q{\left(2L\right)}^{(2-q)/2}{\left({\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\right)}^{q/2}.$$

It implies that

$$\begin{array}{cc}\hfill & {\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & \ge {\left(\frac{{\left(2L\right)}^{(q-2)/2}}{2C_q}\right)}^{2/q}{∥u_n-u_{\mu }∥}_{V,q}^2\hfill \\ \hfill & \ge S_{q_s^{*}}^{-2}{\left(\frac{{\left(2L\right)}^{(q-2)/2}}{2C_q}\right)}^{2/q}{∥u_n-u_{\mu }∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^2\hfill \\ \hfill & \ge S_{q_s^{*}}^{-2}{\left(\frac{{\left(2L\right)}^{(q-2)/2}}{2C_q}\right)}^{2/q}{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^{q_s^{*}}\left({\mathbb{R}}^N\right)}^2.\hfill \end{array}$$

(46)

Combining (22), (33) and (46), we obtain

$${\delta }_{\mu }^{q_s^{*}}\ge S_{q_s^{*}}^{-2}{\left(\frac{{\left(2L\right)}^{(q-2)/2}}{2C_q}\right)}^{2/q}{\delta }_{\mu }^2.$$

(47)

If ${\delta }_{{\mu }_k}>0$ for some sequence $\left\{{\mu }_k\right\}:{\mu }_k\to +\infty$ as $k\to \infty ,$ then from (33), note that $G_{u_{\mu }}\left(u_n-u_{\mu }\right)\to 0,{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\to 0$ as $n\to \infty ,G_{u_n}\left(u_n\right)={∥u_n∥}_{V,p}^p\to {\xi }_{\mu }^p,{\tilde{G}}_{u_n}\left(u_n\right)={∥u_n∥}_{V,q}^q\to {\zeta }_{\mu }^q$ as $n\to \infty$ and $G_{u_n}\left(u_{\mu }\right)\to G_{u_{\mu }}\left(u_{\mu }\right)\ge 0,{\tilde{G}}_{u_n}\left(u_{\mu }\right)\to G_{u_{\mu }}\left(u_{\mu }\right)\ge 0$, we have

$${\beta }_{{\mu }_k}-G_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)-{\tilde{G}}_{u_{{\mu }_k}}\left(u_{{\mu }_k}\right)={\delta }_{{\mu }_k}^{q_s^{*}}.$$

(48)

For the similar case $q\ge 2,$ from (47) and (48), we obtain

$${\beta }_{{\mu }_k}^{q_s^{*}-2}\ge {\left(\frac{S_{q_s^{*}}^{-q}}{2C_q}{\left(2L\right)}^{(q-2)/2}\right)}^{2q_s^{*}/q}.$$

(49)

It contradicts (23). Then, ${\delta }_{\mu }=0$ for ${\mu }_{*}>0$ if $\mu \ge {\mu }_{*}.$ □

4. Proof of Theorem 1

Proof. 

From Lemmas 3–5, Lemma 7 and the mountain pass theorem, there exists ${\mu }_{*}>0$ such that for all $\mu \ge {\mu }_{*}$, problem (1) possesses a solution $u_{\mu }\in X$. Indeed, $\mathcal{J}\left(u_{\mu }\right)=c_{\mu }$ and ${\mathcal{J}}^{\prime }\left(u_{\mu }\right)=0$ in $X^{*}$.

Let $u_{\mu }^{-}:=min\{u_{\mu },0\}$. Since

$${|x-y|}^{t-2}(x-y)\left(x^{-}-y^{-}\right)\ge {\left|x^{-}-y^{-}\right|}^t,x,y\in \mathbb{R}\mathrm{and}t>1$$

and $⟨{\mathcal{J}}^{\prime }\left(u_{\mu }\right),u_{\mu }^{-}⟩=0,$ we obtain

$$\begin{array}{cc}\hfill {∥u_{\mu }^{-}∥}_{V,p}^p+{∥u_{\mu }^{-}∥}_{V,q}^q\le & {\int \int }_{{\mathbb{R}}^6}\frac{{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^{p-2}\left(u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right)\left(u_{\mu }^{-}\left(x\right)-u_{\mu }^{-}\left(y\right)\right)}{{|x-y|}^{N+sp}}\mathrm{d}x\mathrm{d}y\hfill \\ \hfill & +{\int \int }_{{\mathbb{R}}^6}\frac{{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^{q-2}\left(u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right)\left(u_{\mu }^{-}\left(x\right)-u_{\mu }^{-}\left(y\right)\right)}{{|x-y|}^{N+sq}}\mathrm{d}x\mathrm{d}y\hfill \\ \hfill & +{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_{\mu }\right|}^{p-2}{u}_{\mu }{u}_{\mu }^{-}\mathrm{d}x+{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_{\mu }\right|}^{q-2}{u}_{\mu }{u}_{\mu }^{-}\mathrm{d}x\hfill \\ \hfill & -\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_{\mu }\right)u_{\mu }^{-}\mathrm{d}x-{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }^{+}\right|}^{q_s^{*}-1}{u}_{\mu }^{-}\mathrm{d}x\hfill \\ \hfill =& 0,\hfill \end{array}$$

which implies that $u_{\mu }^{-}=0.$ So $u_{\mu }\ge 0$ in ${\mathbb{R}}^N$ and $u_{\mu }\not\equiv 0$.

Next, we claim that $\parallel u_{\mu }{\parallel }_X\to 0$ as $\mu \to +\infty .$ From $\mathcal{J}\left(u_{\mu }\right)=c_{\mu }$ and ${\mathcal{J}}^{\prime }\left(u_{\mu }\right)=0$ in $X^{*}$, it follows that

$$\begin{array}{cc}\hfill c_{\mu }& =\mathcal{J}\left(u_{\mu }\right)-\frac{1}{\theta }⟨{\mathcal{J}}^{\prime }\left(u_{\mu }\right),u_{\mu }⟩\hfill \\ \hfill & \ge \left(\frac{1}{q}-\frac{1}{\theta }\right)\left({∥u_{\mu }∥}_{V,p}^p+{∥u_{\mu }∥}_{V,q}^q\right).\hfill \end{array}$$

(50)

If $\underset{\mu \to +\infty }{lim}\left({∥u_{\mu }∥}_{V,p}^p+{∥u_{\mu }∥}_{V,q}^q\right)=a_0>0$, let $\mu \to +\infty$ in both sides of (50), we deduce

$$0\ge \left(\frac{1}{q}-\frac{1}{\theta }\right)a_0>0.$$

This is a contradiction. Hence, we obtain $\parallel {u_{\mu }\parallel }_X\to 0$ as $\mu \to +\infty .$ This ends the proof. □

5. Proof of Theorem 2

Similar to the proof of Theorem 1, we can obtain the proof of Theorem 2. In fact, we take into account the energy functional

$${\mathcal{J}}_m\left(u\right)=\frac{1}{p}{\parallel u\parallel }_{V,p}^p+\frac{1}{q}{\parallel u\parallel }_{V,q}^q-\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)F(x,u)\mathrm{d}x-\frac{1}{m}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^m\mathrm{d}x$$

instead of $\mathcal{J}\left(u\right)$. Now, we proof lemmas 3–5 and lemma 7 under corresponding conditions for ${\mathcal{J}}_m\left(u\right)$.

Lemma 8. 

Let ${\left(f_1\right)}^{\prime },\left(f_3\right)$ for $\theta \in (q,m),\left(f_4\right)$ and $\left(V{M}_1\right),\left(V{M}_2\right),\left(V{M}_4\right)$ hold. Then, for all $\mu >0$, the following properties are fulfilled for the functional ${\mathcal{J}}_m\left(u\right)$:

(i)

there exist positive constants ${\rho }_0,{\delta }_0$, such that ${\mathcal{J}}_m\left(u\right)\ge {\delta }_0$ for all $u\in X$ with ${\parallel u\parallel }_X={\rho }_0$.

(ii)

there exists $u_0\in X$ with $\parallel {u_0\parallel }_X>{\rho }_0$ such that ${\mathcal{J}}_m\left(u_0\right)<0$, where ${\rho }_0>0$ is given in (i).

Proof. 

(i) Using ${\left(f_1\right)}^{\prime }$, we have

$${\mathcal{J}}_m\left(u\right)\ge \frac{1}{p}{\parallel u\parallel }_{V,p}^p+\frac{1}{q}{\parallel u\parallel }_{V,q}^q-\frac{\mu C+1}{m}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u\right|}^m\mathrm{d}x.$$

Choosing ${\parallel u\parallel }_X={\rho }_0$ small, applying Lemma 1, we get

$$\begin{array}{cc}\hfill {\mathcal{J}}_m\left(u\right)& \ge \frac{1}{q}\left({\parallel u\parallel }_{V,p}^p+{\parallel u\parallel }_{V,q}^q\right)-\frac{\left(\mu C+1\right)S_m^m}{m}{\parallel u\parallel }_{V,q}^m\hfill \\ & \ge \frac{1}{q}\left({\parallel u\parallel }_{V,p}^q+{\parallel u\parallel }_{V,q}^q\right)-\frac{\left(\mu C+1\right)S_m^m}{m}{\parallel u\parallel }_X^m\hfill \\ & \ge \frac{1}{2^{q-1}q}{\parallel u\parallel }_X^q-\frac{\left(\mu C+1\right)S_m^m}{m}{\parallel u\parallel }_X^m.\hfill \end{array}$$

Since $1<q<m$, (i) is fulfilled.

(ii) For any $u\in {\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^N\right)$ with $u\ge 0$ in ${\mathbb{R}}^N,$$u\not\equiv 0$, we obtain

$${\mathcal{J}}_m\left(tu\right)\le \frac{t^p}{p}{\parallel u\parallel }_{V,p}^p+\frac{t^q}{q}{\parallel u\parallel }_{V,q}^q-\frac{t^m}{m}{\int }_{suppu}M\left(x\right){\left|u^{+}\right|}^m\mathrm{d}x$$

for any $t>0$. Since $p<q<m$, we obtain ${\mathcal{J}}_m\left(tu\right)\to -\infty$ as $t\to +\infty$. Therefore, property (ii) also holds true. □

Fix $\mu >0$ and set

$$c_{\mu }=\underset{\chi \in \mathsf{\Gamma }}{inf}\underset{\tau \in [0,1]}{max}{\mathcal{J}}_m\left(\chi \left(\tau \right)\right),$$

(51)

where

$$\mathsf{\Gamma }=\{\chi \in \mathcal{C}([0,1],X):\chi \left(0\right)=0,{\mathcal{J}}_m\left(\chi \left(1\right)\right)<0\}.$$

Undoubtedly, $c_{\mu }>0$ according to Lemma 8. Furthermore, we also obtain the following lemma:

Lemma 9. 

Let ${\left(f_1\right)}^{\prime },\left(f_3\right)$ for $\theta \in (q,m),\left(f_4\right)$, and $\left(V{M}_1\right),\left(V{M}_2\right),\left(V{M}_4\right)$ hold. Then, $c_{\mu }\to 0$ as $\mu \to \infty .$

Proof. 

From (ii) in Lemma 8, we obtain ${\mathcal{J}}_m\left(t{u}_0\right)=-\infty$ as $t\to +\infty ,$ then, there exists $t_{\mu }>0$ such that ${\mathcal{J}}_m\left(t_{\mu }{u}_0\right)=\underset{t\ge 0}{max}\mathcal{J}\left(t{u}_0\right).$ Hence, $⟨{\mathcal{J}}_m^{\prime }\left(t_{\mu }{u}_0\right),t_{\mu }{u}_0⟩=0$. It implies that

$${∥t_{\mu }{u}_0∥}_{V,p}^p+{∥t_{\mu }{u}_0∥}_{V,q}^q=\mu t_{\mu }{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,t_{\mu }{u}_0\right)u_0\mathrm{d}x+t_{\mu }^m{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^m\mathrm{d}x.$$

(52)

We now prove the boundedness of the sequence $\left\{t_{\mu }\right\}$. From (52) and $\left(f_3\right)$, we have

$$\begin{array}{cc}\hfill {∥t_{\mu }{u}_0∥}_X^p+{∥t_{\mu }{u}_0∥}_X^q& \ge {∥t_{\mu }{u}_0∥}_{V,p}^p+{∥t_{\mu }{u}_0∥}_{V,q}^q\hfill \\ \hfill & =\mu t_{\mu }{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,t_{\mu }{u}_0\right)v\mathrm{d}x+t_{\mu }^m{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^m\mathrm{d}x\hfill \\ \hfill & \ge t_{\mu }^m{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^m\mathrm{d}x.\hfill \end{array}$$

(53)

Due to $p<q<m$ and ${\displaystyle 0<{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^m\mathrm{d}x<+\infty }$, we can infer that $\left\{t_{\mu }\right\}$ is bounded. Fix any sequence $\left\{{\mu }_n\right\}$ such that ${\mu }_n\to \infty$. Then, up to a subsequence, there exists $t_0\ge 0$ such that $t_{{\mu }_n}\to t_0.$ We claim that $t_0=0.$ If $t_0>0$, the dominated convergence theorem leads to

$${\mu }_n{t}_{{\mu }_n}{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,t_{{\mu }_n}{u}_0\right)u_0\mathrm{d}x+t_{{\mu }_n}^m{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_0^{+}\right|}^m\mathrm{d}x\to +\infty \mathrm{as}n\to \infty ,$$

which contradicts (53). Hence, $t_0=0$. That is to say, $t_{\mu }\to 0$ as $\mu \to \infty$. Put $\overline{\chi }\left(t\right)=t{u}_0$, we have $\overline{\chi }\in \mathsf{\Gamma }$, and thus

$$0<c_{\mu }\le \underset{t\ge 0}{max}{\mathcal{J}}_m\left(\overline{\chi }\left(t\right)\right)={\mathcal{J}}_m\left(t_{\mu }{u}_0\right)\le \frac{1}{p}{∥t_{\mu }{u}_0∥}_{V,p}^p+\frac{1}{q}{∥t_{\mu }{u}_0∥}_{V,q}^q.$$

Letting $\mu \to +\infty ,$ we obtain $c_{\mu }\to 0$. □

Lemma 10. 

For each $\mu >0$. The $\left(PS\right)$ sequence $\left\{u_n\right\}\subset X$ for ${\mathcal{J}}_m$ at the level $c\in \mathbb{R}$ is bounded.

Proof. 

By a simple computation, for $n\in \mathbb{N}$ large enough we observe that

$$\begin{array}{ccc}\hfill C\left(1+{∥u_n∥}_X\right)& \ge & {\mathcal{J}}_m\left(u_n\right)-\frac{1}{\theta }⟨{\mathcal{J}}_m^{\prime }\left(u_n\right),u_n⟩\hfill \\ & =& \left(\frac{1}{p}-\frac{1}{\theta }\right){∥u_n∥}_{V,p}^p+\left(\frac{1}{q}-\frac{1}{\theta }\right){∥u_n∥}_{V,q}^q\hfill \\ & & +\frac{\mu }{\theta }{\int }_{{\mathbb{R}}^N}M\left(x\right)\left(f\left(x,u_n\right)u_n-\theta F\left(x,u_n\right)\right)\mathrm{d}x\hfill \\ & & +\left(\frac{1}{\theta }-\frac{1}{m}\right){\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n^{+}\right|}^m\mathrm{d}x\hfill \\ & \ge & \left(\frac{1}{q}-\frac{1}{\theta }\right)\left({∥u_n∥}_{V,p}^p+{∥u_n∥}_{V,q}^q\right),\hfill \end{array}$$

(54)

thanks to $\mu >0,p<q<\theta ,\left(f_3\right)$ for $\theta \in (q,m)$ and $\left(f_4\right)$. With this in mind, we deduce that the sequence $\left\{u_n\right\}\subset X$ is bounded, and we omit the details here. Thus, the proof is completed. □

Lemma 11. 

Let ${\left(f_1\right)}^{\prime },\left(f_3\right)$ for $\theta \in (q,m),\left(f_4\right)$, and $\left(V{M}_1\right),\left(V{M}_2\right),\left(V{M}_4\right)$ hold. Then, there exists ${\mu }_{*}>0$, for all $\mu \ge {\mu }_{*}$, ${\mathcal{J}}_m$ satisfies the ${\left(PS\right)}_{c_{\mu }}$ condition on $X.$

Proof. 

Let $\left\{u_n\right\}\subset X$ be the ${\left(PS\right)}_{c_{\mu }}$ sequence for ${\mathcal{J}}_m$, then there exists $C>0$, such that $\left|⟨{\mathcal{J}}_m^{\prime }\left(u_n\right),u_n⟩\right|\le C{∥u_n∥}_X$ and $\left|{\mathcal{J}}_m\left(u_n\right)\right|\le C$. From Lemma 10 it follows that $\left\{u_n\right\}$ is bounded. Thus, up to a subsequence (still denoted by itself), there exists $u_{\mu }\in X$ and ${\delta }_{\mu }\ge 0,{\xi }_{\mu }\ge 0,{\zeta }_{\mu }\ge 0$ such that

$$\begin{array}{cc}\hfill & u_n\rightharpoonup u_{\mu }\mathrm{in}X\cap L_M^m\left({\mathbb{R}}^N\right),\hfill \\ \hfill & u_n\to u_{\mu }\mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^N,\hfill \\ \hfill & {∥u_n∥}_{V,p}\to {\xi }_{\mu },{∥u_n∥}_{V,q}\to {\zeta }_{\mu },{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^m\left({\mathbb{R}}^N\right)}\to {\delta }_{\mu },\hfill \\ \hfill & {\left|u_n\right|}^{m-2}{u}_n\rightharpoonup {\left|u_{\mu }\right|}^{m-2}{u}_{\mu }\mathrm{in}{L}_M^{{\left(m\right)}^{\prime }}\left({\mathbb{R}}^N\right).\hfill \end{array}$$

(55)

Notice that

$$\parallel u_n{\parallel }_{V,p}^p+{\parallel u_n\parallel }_{V,q}^q\to {\xi }_{\mu }^p+{\zeta }_{\mu }^q:={\beta }_{\mu }\ge 0.$$

Next, we prove that

$$\underset{\mu \to +\infty }{lim}{\beta }_{\mu }=0.$$

(56)

If there is a sequence ${\lambda }_k\to \infty$ such that ${\beta }_{{\lambda }_k}\to \beta >0$ as $k\to \infty$. Like (54), we have

$$c_{{\mu }_k}\ge \left(\frac{1}{q}-\frac{1}{\theta }\right)\left({∥u_n∥}_{V,p}^p+{∥u_n∥}_{V,q}^q\right).$$

When we use Lemma 9 and take $k\to \infty$ into consideration on both sides of the inequality above, there is a contradiction. So (56) is proved. It is easy to see that

$$\underset{\mu \to +\infty }{lim}{\xi }_{\mu }=0\mathrm{and}\underset{\mu \to +\infty }{lim}{\zeta }_{\mu }=0.$$

Moreover, we can deduce that ${∥u_{\mu }∥}_X\le \underset{n\to \infty }{lim}$${∥u_n∥}_X={\xi }_{\mu }+{\zeta }_{\mu }$ since $u_n\rightharpoonup u_{\mu }$ in X. From ${∥u_n∥}_{L_M^m\left({\mathbb{R}}^N\right)}\le S_m{∥u_n∥}_{V,q},$ we obtain

$$\underset{\mu \to \infty }{lim}{∥u_{\mu }∥}_{L_M^m\left({\mathbb{R}}^N\right)}=\underset{\mu \to \infty }{lim}{∥u_{\mu }∥}_X=0.$$

(57)

Like (25), we have

$$\begin{array}{ccc}\hfill o_n\left(1\right)& =& ⟨{\mathcal{J}}_m^{\prime }\left(u_n\right)-{\mathcal{J}}_m^{\prime }\left(u_{\mu }\right),u_n-u_{\mu }⟩\hfill \\ & =& G_{u_n}\left(u_n-u_{\mu }\right)+{\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-G_{u_{\mu }}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & & -\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)\left[f\left(x,u_n\right)-f\left(x,u_{\mu }\right)\right]\left(u_n-u_{\mu }\right)\mathrm{d}x\hfill \\ & & -{\int }_{{\mathbb{R}}^N}M\left(x\right)\left({\left|u_n^{+}\right|}^{m-1}-{\left|u_{\mu }^{+}\right|}^{m-1}\right)\left(u_n-u_{\mu }\right)\mathrm{d}x\hfill \\ & =& G_{u_n}\left(u_n-u_{\mu }\right)-G_{u_{\mu }}\left(u_n-u_{\mu }\right)+{\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & & -{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^m\left({\mathbb{R}}^N\right)}^m+o_n\left(1\right).\hfill \end{array}$$

(58)

where G and $\tilde{G}$ are defined by (18) and (19), respectively. Here, we also make use of the following estimations:

(i)

${\displaystyle \underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^{m-2}{u}_n{u}_{\mu }\mathrm{d}x={\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }\right|}^m\mathrm{d}x}$ since ${\left|u_n\right|}^{m-2}{u}_n\rightharpoonup$${\left|u_{\mu }\right|}^{m-2}{u}_{\mu }$ in $L_M^{{\left(m\right)}^{\prime }}\left({\mathbb{R}}^N\right)$;

(ii)

${\displaystyle \underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }\right|}^{m-2}{u}_{\mu }{u}_n\mathrm{d}x={\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }\right|}^m\mathrm{d}x}$, since $u_n\rightharpoonup u_{\mu }$ in $L_M^m\left({\mathbb{R}}^N\right)$ and ${\left|u_{\mu }\right|}^{m-2}{u}_{\mu }\in L_M^{{\left(m\right)}^{\prime }}\left({\mathbb{R}}^N\right)$;

(iii)

By the Brézis-Lieb Lemma, we obtain:

$$\begin{array}{cc}& {∥u_n-u_{\mu }∥}_{L_M^m\left({\mathbb{R}}^N\right)}^m={∥u_n∥}_{L_M^m\left({\mathbb{R}}^N\right)}^m-{∥u_{\mu }∥}_{L_M^m\left({\mathbb{R}}^N\right)}^m+o\left(1\right),\hfill \\ & {∥u_n-u_{\mu }∥}_{V,p}^p={∥u_n∥}_{V,p}^p-{\parallel u_{\mu }\parallel }_{V,p}^p+o\left(1\right),\hfill \\ & {∥u_n-u_{\mu }∥}_{V,q}^q={∥u_n∥}_{V,q}^q-{\parallel u_{\mu }\parallel }_{V,q}^q+o\left(1\right).\hfill \end{array}$$

(iv)

${\displaystyle \underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right)\left[f\left(x,u_n\right)-f\left(x,u_{\mu }\right)\right]\left(u_n-u_{\mu }\right)\mathrm{d}x=0.}$

Now, we show (iv). It follows from the Hölder’s inequality that

$$\begin{array}{ccc}\hfill & & \left|{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_n\right)\left(u_n-u_{\mu }\right)\mathrm{d}x\right|\hfill \\ & \le & C{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^{m-1}\left|u_n-u_{\mu }\right|\mathrm{d}x\hfill \\ & \le & C{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n\right|}^m\mathrm{d}x\right)}^{\frac{m-1}{m}}{\left({\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^m\mathrm{d}x\right)}^{\frac{1}{m}}\hfill \end{array}$$

(59)

From (4), we have

$$\parallel u_n{\parallel }_{L_M^m\left({\mathbb{R}}^N\right)}\le S_m\parallel u_n{\parallel }_{V,q}\le S_m{\parallel u_n\parallel }_X<\infty .$$

(60)

According to Lemma 1, the embedding $E^{s,q}\hookrightarrow L_M^m\left({\mathbb{R}}^N\right)$ is compact. Then we have

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_n-u_{\mu }\right|}^m\mathrm{d}x=0.$$

(61)

According to (59)–(61), we obtain

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_n\right)\left(u_n-u_{\mu }\right)\mathrm{d}x=0.$$

Similarly,

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_{\mu }\right)\left(u_n-u_{\mu }\right)\mathrm{d}x=0.$$

Hence, (iv) follows. From (58) it follows that

$$\begin{array}{ccc}\hfill & & \underset{n\to \infty }{lim}{G}_{u_n}\left(u_n-u_{\mu }\right)-G_{u_{\mu }}\left(u_n-u_{\mu }\right)+{\tilde{G}}_{u_n}\left(u_n-u_{\mu }\right)-{\tilde{G}}_{u_{\mu }}\left(u_n-u_{\mu }\right)\hfill \\ & =& \underset{n\to \infty }{lim}{∥{(u_n-u_{\mu })}^{+}∥}_{L_M^m\left({\mathbb{R}}^N\right)}^m.\hfill \end{array}$$

(62)

Using the Brézis-Lieb Lemma and (55), we obtain

$$\begin{array}{ccc}\hfill c_{\mu }+o_n\left(1\right)& =& {\mathcal{J}}_m\left(u_n\right)-\frac{1}{\theta }⟨{\mathcal{J}}_m^{\prime }\left(u_n\right),u_n⟩\hfill \\ & \ge & \left(\frac{1}{\theta }-\frac{1}{m}\right){∥u_n^{+}∥}_{L_M^m\left({\mathbb{R}}^N\right)}^m\hfill \\ & =& \left(\frac{1}{\theta }-\frac{1}{m}\right)\left({\delta }_{\mu }^m+{∥u_{\mu }^{+}∥}_{L_M^m\left({\mathbb{R}}^N\right)}^m\right)+o\left(1\right).\hfill \end{array}$$

(63)

Combining Lemma 9 and (57), we have

$$\underset{\mu \to +\infty }{lim}{\delta }_{\mu }=0.$$

(64)

As the proof of Lemma 7, we may deduce that there exists ${\mu }_{*}>0$ such that, for all $\mu \ge {\mu }_{*},u_n\to u_{\mu }$ in X.

□

Proof of Theorem 2. 

From Lemmas 8–11 and the mountain pass theorem, there exists ${\mu }_{*}>0$ such that for all $\mu \ge {\mu }_{*}$, problem (2) possesses a solution $u_{\mu }\in X$. Indeed, ${\mathcal{J}}_m\left(u_{\mu }\right)=c_{\mu }$ and ${\mathcal{J}}_m^{\prime }\left(u_{\mu }\right)=0$ in $X^{*}$.

Let $u_{\mu }^{-}:=min\{u_{\mu },0\}$. Since

$${|x-y|}^{t-2}(x-y)\left(x^{-}-y^{-}\right)\ge {\left|x^{-}-y^{-}\right|}^t,x,y\in \mathbb{R}\mathrm{and}t>1$$

and $⟨{\mathcal{J}}^{\prime }\left(u_{\mu }\right),u_{\mu }^{-}⟩=0,$ we obtain

$$\begin{array}{cc}\hfill {∥u_{\mu }^{-}∥}_{V,p}^p+{∥u_{\mu }^{-}∥}_{V,q}^q\le & {\int \int }_{{\mathbb{R}}^6}\frac{{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^{p-2}\left(u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right)\left(u_{\mu }^{-}\left(x\right)-u_{\mu }^{-}\left(y\right)\right)}{{|x-y|}^{N+sp}}\mathrm{d}x\mathrm{d}y\hfill \\ \hfill & +{\int \int }_{{\mathbb{R}}^6}\frac{{\left|u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right|}^{q-2}\left(u_{\mu }\left(x\right)-u_{\mu }\left(y\right)\right)\left(u_{\mu }^{-}\left(x\right)-u_{\mu }^{-}\left(y\right)\right)}{{|x-y|}^{N+sq}}\mathrm{d}x\mathrm{d}y\hfill \\ \hfill & +{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_{\mu }\right|}^{p-2}{u}_{\mu }{u}_{\mu }^{-}\mathrm{d}x+{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u_{\mu }\right|}^{q-2}{u}_{\mu }{u}_{\mu }^{-}\mathrm{d}x\hfill \\ \hfill & -\mu {\int }_{{\mathbb{R}}^N}M\left(x\right)f\left(x,u_{\mu }\right)u_{\mu }^{-}\mathrm{d}x-{\int }_{{\mathbb{R}}^N}M\left(x\right){\left|u_{\mu }^{+}\right|}^{m-1}{u}_{\mu }^{-}\mathrm{d}x\hfill \\ \hfill =& 0,\hfill \end{array}$$

which implies that $u_{\mu }^{-}=0.$ So $u_{\mu }\ge 0$ in ${\mathbb{R}}^N$ and $u_{\mu }\not\equiv 0$.

Next, we claim that $\parallel u_{\mu }{\parallel }_X\to 0$ as $\mu \to +\infty .$ From ${\mathcal{J}}_m\left(u_{\mu }\right)=c_{\mu }$ and ${\mathcal{J}}_m^{\prime }\left(u_{\mu }\right)=0$ in $X^{*}$, it follows that

$$\begin{array}{cc}\hfill c_{\mu }& ={\mathcal{J}}_m\left(u_{\mu }\right)-\frac{1}{\theta }⟨{\mathcal{J}}_m^{\prime }\left(u_{\mu }\right),u_{\mu }⟩\hfill \\ \hfill & \ge \left(\frac{1}{q}-\frac{1}{\theta }\right)\left({∥u_{\mu }∥}_{V,p}^p+{∥u_{\mu }∥}_{V,q}^q\right).\hfill \end{array}$$

(65)

If $\underset{\mu \to +\infty }{lim}\left({∥u_{\mu }∥}_{V,p}^p+{∥u_{\mu }∥}_{V,q}^q\right)=a_0>0$, let $\mu \to +\infty$ in both sides of (65), we deduce

$$0\ge \left(\frac{1}{q}-\frac{1}{\theta }\right)a_0>0.$$

This is a contradiction. Hence, we obtain $\parallel {u_{\mu }\parallel }_X\to 0$ as $\mu \to +\infty .$ This ends the proof.

□