Critical Fractional Choquard–Kirchhoff Equation with p-Laplacian and Perturbation Terms on the Heisenberg Group

Abstract

In this paper, we are interested in a class of critical fractional Choquard–Kirchhoff equations with p-Laplacian on the Heisenberg group. By employing several critical point theorems, we obtain the existence and multiplicity of nontrivial solutions under different perturbation terms. Due to the critical convolution term, the compactness condition may fail. To overcome this, we apply the concentration-compactness principle. The results in this paper can be viewed as complementary to the previous results under the conditions of $s=1$, $p=2$, and in the subcritical case.

1. Introduction

This article investigates the following critical fractional Choquard–Kirchhoff equation with p-Laplacian operator on the Heisenberg group:

$$(a+b{\left[u\right]}_{H,s,p}^p){(-\Delta )}_{H,p}^{s}u={\lambda h\left(\xi \right)|u|}^{\kappa -2}u+\left({\int }_{{\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta \right){|u|}^{p_{\mu ,s}^{*}-2}uin{\mathbb{H}}^N,$$

(1)

where positive parameters $a,b$ are, respectively, represented as the overall stiffness of the system and the non-local coefficient in the Kirchhoff-type equation; ${(-\Delta )}_{H,p}^s$ is represented as the fractional p-Laplacian in ${\mathbb{H}}^N$; $s\in (0,1)$ represents the order of the fractional-order operator; $0<\mu <2sp$, $1<p,\kappa <p_{\mu ,s}^{*}$, $\lambda >0$ is a real parameter; and the homogeneous dimension Q of ${\mathbb{H}}^N$ is equal to $2N+2$, and the critical exponent $p_{\mu ,s}^{*}={\displaystyle \frac{p(2Q-\mu )}{2(Q-sp)}}$ according to the Hardy–Littlewood–Sobolev inequality. For any $u\in {\mathcal{C}}_c^{\infty }\left({\mathbb{H}}^N\right)$, one has

$${\left[u\right]}_{H,s,p}^p:={\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)-u\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta ,$$

where r is the Korányi norm. More details about the Heisenberg group can be found in [1,2]. Moreover, the weight function h needs to satisfy the following conditions:

($\mathcal{C}$)

$h\left(\xi \right)\ge 0$ for $\xi \in {\mathbb{H}}^N$, and $\Lambda =\{\xi \in {\mathbb{H}}^N:h>0\}$ with $0<\mathcal{L}(\Lambda )<\infty$, where $\mathcal{L}$ is the Lebesgue measure in ${\mathbb{H}}^N$, and $h\in L^{\theta }\left({\mathbb{H}}^N\right)$, with $\theta =p_s^{*}/(p_s^{*}-\kappa )$.

At the beginning of this paper, we first point out the features of the paper:

(i)

since the Heisenberg group possesses both the topological structure of Euclidean space and the analytical properties of non-Euclidean space, it is necessary to redefine the basic analytical concepts;

(ii)

the emergence of the p-Laplacian operator and perturbation terms makes Equation (1) more complex and interesting, which enables us to observe more meaningful phenomena;

(iii)

due to the unboundedness of the domain, the critical nonlinearity brings about an absence of compactness, which causes the failure of standard critical point theory;

(iv)

the proofs combine some refined estimates and some analysis techniques, including extension, topological and variational tools.

In recent years, the Heisenberg group has demonstrated its unique value and broad application prospects in the field of quantum theory, theta function theory, uncertainty principle, number theory, and commutation relations. For further physical insights, see Cherfils and Ilýasov [3]. On the other hand, the p-Laplacian is widely used in nonlinear problems. For example, such an operator can effectively characterize the state of fluid dynamics, where the fluid exhibits dilatant, pseudoplastic, or Newtonian behavior when the parameter p satisfies $p>2$, $p<2$, or $p=2$, respectively.

Meanwhile, we have noticed that an increasing number of researchers are turning their attention to the Choquard equation in Euclidean space. The general form of the Choquard equation is

$$-\Delta u+V\left(x\right)u=(I_{\alpha }\ast {|u|}^p{)|u|}^{p-2}u\mathrm{in}{\mathbb{R}}^N,$$

where $V\left(x\right)$ is a potential function. This equation was first proposed by Pekar [4] to describe the ground state of polarons in 1954. In quantum mechanics, it is used to describe the behavior of quantum particles in non-local potential fields, and it has applications in various fields such as plasma physics, condensed matter physics, and astrophysics. Lieb [5] and Lions [6] pioneered the research on the existence and symmetry of solutions to the Choquard equation. Some related results can be referred to in [7,8,9,10].

For $p=2$, Liang and Pucci [11] were interested in the following Choquard–Kirchhoff type equations:

$$-\left(a+b{\int }_{{\mathbb{R}}^N}{|\nabla u|}^{2}dx\right)\Delta u={\alpha k\left(x\right)|u|}^{q-2}u+\beta \left({\int }_{{\mathbb{R}}^N}{\displaystyle \frac{{|u\left(y\right)|}^{2_{\mu }^{*}}}{{|x-y|}^{\mu }}}\right){|u|}^{2_{\mu }^{*}-2}u,x\in {\mathbb{R}}^N,$$

where $a>0$, $b\ge 0$, $0<\mu <N$ with $N\ge 3$, the real parameters $\alpha ,\beta >0$, and $2_{\mu }^{*}={\displaystyle \frac{2N-\mu }{N-2}}$ is the critical exponent in the sense of the Hardy–Littlewood–Sobolev inequality, $k\in L^r\left({\mathbb{R}}^N\right)$, with $r=2^{*}/(2^{*}-q)$ if $1<q<2^{*}$ and $r=\infty$ if $q\ge 2^{*}$. They obtained multiplicity solutions for the problem through variational methods. Moreover, Goel [12] conducted an in-depth investigation into Brézis–Nirenberg type results pertaining to the Dirichlet problem involving Choquard type nonlinearity, employing the mountain pass theorem, the linking theorem, and related methodologies. Sun et al. [13], based on the limit index theory and the concentration-compactness principle, systematically explored the noncooperative Choquard–Kirchhoff problem.

As $p\ne 2$, Pucci and Xiang [14] made a contribution to resolving the following Schrödinger–Choquard–Kirchhoff type with the fractional p-Laplacian:

$${(a+b\parallel u\parallel }_{s,p}^{p(\theta -1)}{\left)\right[(-\Delta u)}_p^{s}u+{V\left(x\right)|u|}^{p-2}u]=\lambda f(x,u)+\left({\int }_{{\mathbb{R}}^n}{\displaystyle \frac{{|u|}^{p_{\mu ,s}^{*}}}{{|x-y|}^{\mu }}}dy\right){|u|}^{p_{\mu ,s}^{*}-2}u\mathrm{in}{\mathbb{R}}^N.$$

Under appropriate assumptions, by applying the Ekeland variational principle and the mountain pass theorem, the existence of nonnegative solutions for the above equation in both sublinear and superlinear cases were obtained. Subsequently, Wang et al. [15] proved that there exist infinitely many solutions for the fractional p-Laplacian equations by applying the concentration-compactness principle and Kajikiya’s symmetric mountain pass lemma in the fractional case. Later, Liang et al. [16] established the concentration-compactness principle for the Choquard equation involving p-Laplacian on the Heisenberg group, together with the mountain pass theorem and the Krasnoselskii genus theory, they obtained the existence of multiple solutions for a class of critical Choquard–Kirchhoff equations involving the p-sub-Laplacian equation in both the nondegenerate and degenerate cases. Other results for this topic can be found in [17,18,19]. Differently from the methods used in the present literature, Liu and Perera [20] obtained the existence and a multiplicity of solutions for $(p,q)$-Laplacian equations involving subcritical and critical growth using an abstract critical point theorem proposed by Perera [21]. For some other related results, please refer to [22,23,24,25].

In Euclidean space, the study of nonlinear equations with critical growth terms typically employs embedding theorems to address compactness issues. However, the introduction of the Heisenberg group structure fundamentally alters the geometric and measure-theoretic properties of the space, rendering the compactness compensation techniques effective in Euclidean space inapplicable. The inherent non-locality within the Heisenberg group further exacerbates these challenges, as the presence of critical nonlocal nonlinear terms may lead to the loss of compactness in the energy functional across the entire space, thereby presenting substantial obstacles to the identification of critical points. Moreover, the Heisenberg group possesses distinctive algebraic structures and geometric characteristics, notably its distance function and measure, which differ significantly from their Euclidean counterparts. Consequently, when investigating equations involving the fractional p-Laplacian operator on the Heisenberg group, it becomes imperative to reexamine fundamental aspects such as the definition of function spaces, the validity of embedding theorems, and the efficacy of the variational framework. These inherent distinctions render the techniques and results commonly utilized in Euclidean space potentially ineffective within the Heisenberg group context, thereby introducing novel complexities that are unattainable in the Euclidean setting.

Inspired by the aforementioned literature, this paper studies the existence and multiple solutions of nontrivial solutions for the critical fractional Choquard–Kirchhoff equation with p-Laplacian on the Heisenberg group. It is worth noting that the nonlinearity parameters $\mu$ and p play a crucial role in determining the behavior of solutions to Equation (1). As $\mu \to 0$, the critical exponent $p_{\mu ,s}^{*}\to {\displaystyle \frac{pQ}{Q-sp}}$, which is the critical Sobolev exponent on the Heisenberg group without the convolution term. In this case, the equation reduces to a fractional p-Laplacian equation with a critical Sobolev nonlinearity. This scenario has been extensively studied in [14], and our results can be seen as an extension of these studies to include the Choquard-type convolution term. When $p\to 2$, the fractional p-Laplacian operator becomes the fractional Laplacian, and the Equation (1) reduces to a Choquard–Kirchhoff-type equation with a critical nonlinearity in the sense of the Hardy–Littlewood–Sobolev inequality. This case was considered in [11] for the Euclidean setting. Our work connects to these existing results by providing a more general framework that includes the Heisenberg group geometry and the fractional p-Laplacian operator. Moreover, in different cases of perturbation terms, i.e., $1<\kappa <p$, $\kappa =p$ and $1<p<\kappa$, we will systematically discuss their influence on nontrivial solutions of Equation (1).

Next, we present the main results of this paper. Firstly, we present the results in the case of $1<\kappa <p$, which are given as follows:

Theorem 1. 

Let $1<\kappa <p<p_{\mu ,s}^{*}$. If $\left(\mathcal{C}\right)$ is satisfied, then there exists ${\lambda }_{*}>0$ such that if $\lambda \in (0,{\lambda }_{*})$ problem (1) has a sequence of nontrivial solutions ${\left\{u_n\right\}}_n$, with ${\mathcal{I}}_{\lambda }\left(u_n\right)\le 0$ and $u_n\to 0$ as $n\to \infty$.

Next, for the case of $\kappa =p$, we obtain the following result.

Theorem 2. 

Let $\kappa =p$ and $\left(\mathcal{C}\right)$ be satisfied, then there exists a positive constant $a^{\prime }$ such that for all $a>a^{\prime }$ and $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$, problem (1) has at least n pairs of nontrivial solutions.

Finally, for the case $p<\kappa <p_{\mu ,s}^{*}$, the main results of this paper are as follows:

Theorem 3. 

Assume $1<p<\kappa <p_{\mu ,s}^{*}$. Let $\left(\mathcal{C}\right)$ be satisfied and for every $m\in \mathbb{N}$, then there exists ${\lambda }_m>0$ such that problem (1) has m pairs of positive energy solutions for all $\lambda >{\lambda }_m$.

In this paper, the proof of our results is based on variational and topological methods. Compared with previous literature works, the exploration of Equation (1) is more interesting and challenging, since the combined effect of the p-Lapalcian, critical nonlinearity, and the non-local term occurs simultaneously. The main contributions are focused on the following three aspects:

First, the present paper uses different critical point theorems to prove the existence and multiplicity of nontrivial solutions of problem (1) involving p-Laplacian in the general case $p\ge 2.$ Our inspiration mainly stemmed from [11,13,16,26,27]. However, we are not simply repeating the method presented in the papers mentioned above. Indeed, for the cases $s\in (0,1)$ and $p\ne 2$, the operator ${(-\Delta )}_{H,p}^s$ is nonlocal and nonlinear, which gives it some distinctive properties that are different from $-{\Delta }_H$. In addition, the solution space we are studying is not a Hilbert space. Therefore, some advantageous properties of Hilbert spaces cannot be utilized. In certain estimations, we are forced to adopt a distinct approach, and some properties that are valid for the classical Laplacian, specifically when $p=2$, do not necessarily hold in the general case where $p\ge 2$. Hence, a precise estimation is required.

Next, in this paper, the essential step is to verify that the ${\left(PS\right)}_c$ condition is true. The ${\left(PS\right)}_c$ condition means that for sequences with energy level c, there exists a convergent subsequence. This is a crucial tool in nonlinear functional analysis for verifying the minimax principle. Due to the interaction between the nonlocal term and the critical nonlinearity, a loss of compactness occurs in the entire Heisenberg group. To solve this difficulty, we decided to employ the concentration compactness principle to recover the compactness.

Finally, this paper reveals the influence on nontrivial solutions of problem (1) under different perturbation terms. In particular, it seem to be the first time that Perera’s critical point theorem (Theorem 2.1 of [21]) has been applied to the critical fractional Choquard–Kirchhoff equation on the Heisenberg group. It should be noted that even though this idea has been employed in other problems, adapting it to this paper is not an easy task. Due to the presence of the nonlocal term, we have to re-evaluate this problem and require more precise estimates.

The structure of this paper is as follows: In Section 2, we introduce some useful notions and preparatory knowledge. In Section 3, we determine the ${\left(PS\right)}_c$ condition under certain specific energy levels. In Section 4, we establish the proof Theorem 1 by employing a new version of the symmetric mountain pass theorem proposed by Kajikiya. In Section 5, we study the existence and multiplicity results of the solutions to problem (1) when $\kappa =p$. In Section 6, we complete the proof of Theorem 3.

Notation 1. 

For the convenience of the readers, we adopt the following notations:

• $o_n\left(1\right)$ denotes a real sequence, such that $o_n\left(1\right)\to 0$ as $n\to 0$.
• ⇀ means weak convergence and → means strong convergence.
• According to the content of the paper, C is denoted by different normal numbers.
• S denotes the best Sobolev constant given in Section 2.
• $S_{HG,p}$ denotes the best constant of Hardy–Littlehood–Sobolev inequality.

2. Some Basic Lemmas and Compactness Conditions

In this section, we will prove a crucial factor of the variational problem, i.e., the compactness condition. To this end, we initially review some of the basics of the Heisenberg group. The Heisenberg group is represented by ${\mathbb{H}}^N$. If $\xi =(x,y,t)\in {\mathbb{H}}^N$, then the operation of this group is defined as

$$\xi \circ {\xi }^{\prime }=(x+x^{\prime },y+y^{\prime },t+t^{\prime }+2(x^{\prime }y-y^{\prime }x)),\forall \xi ,{\xi }^{\prime }\in {\mathbb{H}}^N,$$

the inverse is given by ${\xi }^{-1}=-\xi$, so that ${(\xi \circ {\xi }^{\prime })}^{-1}={\left({\xi }^{\prime }\right)}^{-1}\circ {\xi }^{-1}$.

We denote by r the Korányi norm, defined as

$$r\left(\xi \right)=r(z,t)={(|z|}^4+t^2{)}^{{\displaystyle \frac{1}{4}}},$$

where $\xi =(z,t),z(x,y)\in {\mathbb{R}}^N\times {\mathbb{R}}^N,t\in \mathbb{R},$ and $|z|$ is the the Euclidean norm in ${\mathbb{R}}^{2N}.$

The natural group of dilations on ${\mathbb{H}}^N$ is defined as

$${\delta }_s\left(\xi \right)=(sx,sy,s^{2}t),\forall \xi \in {\mathbb{H}}^N,$$

for any positive number. Hence, ${\delta }_s({\xi }_0\circ \xi )={\delta }_s\left({\xi }_0\right)\circ {\delta }_s\left(\xi \right)$. For all $\xi =(x,y,t)\in {\mathbb{H}}^N$, it is readily demonstrable that the determinant of the Jacobian matrix for the dilatations ${\delta }_s:{\mathbb{H}}^N\to {\mathbb{H}}^N$ is constant and is equivalent to $s^Q$, where the natural number $Q=2N+2$ is designated as the homogeneous dimension within the scope of ${\mathbb{H}}^N$. The gauge norm can be defined as

$${|\xi |}_H={[{(x^2+y^2)}^2+t^2]}^{{\displaystyle \frac{1}{4}}},\forall \xi \in {\mathbb{H}}^N.$$

Hence, the homogeneous degree of the gauge norm is equal to 1, in terms of dilations

$${\delta }_s:(x,y,t)\mapsto (sx,sy,s^{2}t),\forall s>0.$$

Define the left-invariant distance $d_H$ on ${\mathbb{H}}^N$ as

$$d_H({\xi }_0,\xi )={|{\xi }^{-1}\circ {\xi }_0|}_H.$$

The Heisenberg open ball of radius R centered at ${\xi }_0$ is defined as follows:

$$B_H({\xi }_0,R)=\{\xi \in {\mathbb{H}}^N:d_H({\xi }_0,\xi )<R\}.$$

To put it simply, we shall denote $B_R$ as the open ball with radius R centered at 0, where $O=(0,0)$ is the natural origin of ${\mathbb{H}}^N$. Consequently, we obtain

$$|B_H({\xi }_0,R)|=|B_1|R^Q.$$

Let ${\{X_j,Y_j\}}_{j=1}^N$ be the basis of the Heisenberg orthogonal transformation with respect to ${\mathbb{H}}^N$, which is constructed from the real Lie algebra associated with the horizontal left-invariant vector field:

$$T={\displaystyle \frac{\partial }{\partial t}},X_j={\displaystyle \frac{\partial }{\partial x_j}}+2y_j{\displaystyle \frac{\partial }{\partial t}},Y_j={\displaystyle \frac{\partial }{\partial y_j}}-2x_j{\displaystyle \frac{\partial }{\partial t}},forj=1,\dots N.$$

Therefore, we obtain that

$$[X_j,Y_j]=-4{\delta }_{jk}T,$$

$$[Y_j,Y_k]=[X_j,X_k]=[Y_j,T]=[X_j,T]=0.$$

A vector field in the span of ${[X_j,Y_j]}_{j=1}^N$ is called horizontal. The horizontal gradient of a $C^1$ function $u:{\mathbb{H}}^N\to R$ is defined by

$$D_{H}u=\sum _{j=1}^N[\left(X_{j}u\right)X_j+(Y_{j}u)Y_j].$$

Obviously, $D_{H}u\in {\{X_j,Y_j\}}_{j=1}^N$. In span ${\{X_j,Y_j\}}_{j=1}^N\cong {\mathbb{R}}^{2N}$, we define the natural inner product by

$${(X,Y)}_H=\sum _{j=1}^N(x^j{y}^j+{\tilde{x}}^j{\tilde{y}}^j),$$

where $X={\{x^j{X}^j+{\tilde{x}}^j{Y}^j\}}_{j=1}^N$ and $Y={\{y^j{X}^j+{\tilde{y}}^j{Y}^j\}}_{j=1}^N$. The inner product ${(·,·)}_H$ yields the Hilbertian norm

$$|D_H{u|}_H=\sqrt{{(D_{H}u,D_{H}u)}_H},$$

for the horizontal vector field $D_{H}u$. The Heisenberg gradient on ${\mathbb{H}}^N$ is

$${\nabla }_H=(X_1,X_2,···X_N,Y_1,Y_2,···Y_N).$$

For any horizontal vector field function $X=X\left(\xi \right)$, $X={\{x^j{X}_j+{\tilde{x}}^j{Y}_j\}}_{j=1}^N$ of class $C^1({\mathbb{H}}^N,{\mathbb{R}}^{2N})$, we take into account the horizontal divergence of X given by

$$di{v}_{H}X=\sum _{j=1}^N[X_j\left(x^j\right)+Y_j\left({\tilde{x}}^j\right)].$$

Similarly, on the $C^2\left({\mathbb{H}}^N\right)$, it is coequal to the horizontal Laplacian or the Kohn Laplacian, expressed as

$${\Delta }_H=\sum _{j=1}^N{X}_j^2+Y_j^2=\underset{j=1}{\overset{N}{\Sigma }}\left({\displaystyle \frac{{\partial }^2}{\partial x_j^2}}+{\displaystyle \frac{{\partial }^2}{\partial y_j^2}}+4y_j{\displaystyle \frac{{\partial }^2}{\partial x_j\partial t}}-4x_j{\displaystyle \frac{{\partial }^2}{\partial y_j\partial t}}+4(x_j^2+y_j^2){\displaystyle \frac{{\partial }^2}{\partial t^2}}\right).$$

By Hörmander’s Theorem in (Theorem 1.1 of [28]), we obtain that operator ${\Delta }_H$ is hypoelliptic, and

$${\Delta }_{H}u=di{v}_H\left({\nabla }_{H}u\right)\mathrm{for} \mathrm{all}u\in C^2\left({\mathbb{H}}^N\right).$$

A well-known generalization of the Kohn–Spencer Laplacian is the horizontal p-Laplacian on the Heisenberg group, where $p\in (1,\infty )$, denoted as

$${\Delta }_{H,p}u=di{v}_H(|D_{H}u{|}_H^{p-2}{D}_{H}u)\mathrm{for} \mathrm{all}u\in C_c^{\infty }\left({\mathbb{H}}^N\right).$$

Then, we denote the usual $L^p$-norm by

$${\parallel u\parallel }_p^p={\int }_{{\mathbb{H}}^N}{|u|}^{p}d\xi \mathrm{for} \mathrm{all}p\in [1,+\infty ).$$

Let $s\in (0,1)$ and $1<p<\infty ,$ and define the fractional Sobolev space $H{D}^{s,p}\left({\mathbb{H}}^N\right)$ as the completion of ${\mathcal{C}}_c^{\infty }\left({\mathbb{H}}^N\right)$ endowed the norm: $\parallel u\parallel :={\left[u\right]}_{H,s,p}.$

Meanwhile, the horizontal gradient of the fractional $(s,p)$ of every $u\in H{D}^{s,p}\left({\mathbb{H}}^N\right)$ is expressed as

$$|D_H^s{u|}^p\left(\xi \right)={\int }_{{\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)-u\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\eta ={\int }_{{\mathbb{H}}^N}{\displaystyle \frac{{|u(\xi \circ h)-u\left(\xi \right)|}^p}{r{\left(h\right)}^{Q+sp}}}dh.$$

(2)

In addition, we may define the horizontal gradient of a function $u\in H{D}^{s,p}\left({\mathbb{H}}^N\right)$ a.e. in ${\mathbb{H}}^N$ and $|D_H^s{u|}_p\in L^1\left({\mathbb{H}}^N\right)$ by Tonelli’s theorem.

Moreover, if $1<p<Q$, by Folland and Stein [29], there exists the continuous embedding $H{D}^{s,p}\left({\mathbb{H}}^N\right)\hookrightarrow L^{p_s^{*}}\left({\mathbb{H}}^N\right)$. Moreover, there exists a constant $C_{p_s^{*}}>0$ such that

$${\parallel u\parallel }_{p_s^{*}}\le C_{p_s^{*}}{\parallel D_H^{s}u\parallel }_p.$$

(3)

Furthermore, from Jerison and Lee [30], based on the $C^{\infty }$ function $\mathcal{U}(x,y,t)={\displaystyle \frac{c_0}{\sqrt{{(1+x^2+y^2)}^2+t^2}}}$, the best Sobolev constant can be obtained and expressed as

$$S=\underset{u\in S^{1,p}\left({\mathbb{H}}^N\right),u\ne 0}{inf}{\displaystyle \frac{|D_H^s{u|}_p^p}{{|u|}_{p_s^{*}}^p}}.$$

(4)

Since Equation (1) contains a convolutional term, the following famous Hardy–Littlewood–Sobolev inequality is often needed.

Proposition 1 

(see Folland and Stein [29]). Let $\zeta ,s>1$ and $0<\mu <Q$ with ${\displaystyle \frac{1}{\zeta }}+{\displaystyle \frac{\mu }{Q}}+{\displaystyle \frac{1}{s}}=2.$ There exists a sharp constant $C(\zeta ,s,\mu ,Q)$, such that

$$\int {\int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u(\xi \left)v\right(\eta )|}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi \le {C(\zeta ,s,\mu ,Q)\parallel u\parallel }_{\zeta }{\parallel v\parallel }_s,$$

(5)

where $u\in L^{\zeta }\left({\mathbb{H}}^N\right)$ and $v\in L^s\left({\mathbb{H}}^N\right)$.

If $\zeta =s={\displaystyle \frac{2Q}{2Q-\mu }}$, then

$$C(\mu ,Q)=C(\zeta ,s,\mu ,Q)={\left({\displaystyle \frac{{\pi }^{N+1}}{2^{N-1}N!}}\right)}^{{\displaystyle \frac{\mu }{Q}}}{\displaystyle \frac{N!\Gamma \left(\right(Q-\mu )/2)}{{\Gamma }^2((2Q-\mu )/2)}},$$

where $\Gamma$ is the usual Gamma function. Equality is achieved in (5) if and only if either $u\equiv 0$ or v is identically equal to a constant, and

$$v\left(\xi \right)=c\mathcal{U}\left({\delta }_R\left({\xi }_0^{-1}\xi \right)\right),\xi \in {\mathbb{H}}^N$$

for some $c\in C$, $R>0$, ${\xi }_0\in {\mathbb{H}}^N$ and for all $\xi =(x,y,t)\in {\mathbb{H}}^N$, $\mathcal{U}$ can be defined by

$$\mathcal{U}\left(\xi \right)=\mathcal{U}(x,y,t)=(t^2{+(1+|x|}^2+{|y|}^2{{)}^2)}^{-(2Q-\mu )/4}.$$

Proposition 1 guarantees that the integral

$${\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)|}^{\beta }{|u\left(\eta \right)|}^{\beta }}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi$$

(6)

is well defined in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$ if ${|u|}^{\beta }\in L^{\zeta }\left({\mathbb{H}}^N\right)$ for some $\zeta >1$ such that ${\displaystyle \frac{2}{\zeta }}+{\displaystyle \frac{\mu }{Q}}=2$, that is $\zeta ={\displaystyle \frac{2Q}{2Q-\mu }}$. From the Folland–Stein inequality (3), it follows that only under the condition $p\le \zeta \beta \le {\displaystyle \frac{Qp}{Q-p}}$, (6) is meaningful for $u\in S^{1,p}\left({\mathbb{H}}^N\right)$; that is, $\beta$ has to satisfy $p_{\mu ,s}^{♯}\le \beta \le p_{\mu ,s}^{*}$, where

$$p_{\mu ,s}^{♯}={\displaystyle \frac{p(2Q-\mu )}{2Q}}\mathrm{and}{p}_{\mu ,s}^{*}={\displaystyle \frac{p(2Q-\mu )}{2(Q-sp)}}.$$

Consequently, it is rather intuitive to refer to $p_{\mu ,s}^{♯}$ as the lower critical exponent and $p_{\mu ,s}^{*}$ as the upper critical exponent on the Heisenberg group. Hence, Proposition 1 indicates that

$${\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}}{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi \le C(\mu ,Q){|u|}_{p_s^{*}}^{2p_{\mu ,s}^{*}}$$

being $p_{\mu ,s}^{*}·{\displaystyle \frac{2Q}{2Q-\mu }}=p_s^{*}.$ Now, we define the best constant of the Hardy–Littlewood–Sobolev inequality as

$$S_{HG,p}=\underset{u\in S^{1,p}\left({\mathbb{H}}^N\right),u\ne 0}{inf}{\displaystyle \frac{|D_H^s{u|}_p^p}{{\parallel u\parallel }_{FL,p}^p}},$$

where

$${\parallel u\parallel }_{FL,p}={\left({\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u|}^{p_{\mu ,s}^{*}}{|u|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi \right)}^{1/2p_{\mu ,s}^{*}}.$$

From Proposition 1, we have

$$S_{HG,p}\ge \underset{u\in S^{1,p}\left({\mathbb{H}}^N\right),u\ne 0}{inf}{\displaystyle \frac{|D_H^s{u|}_p^p}{C{(\mu ,Q)}^{\frac{p}{2p_{\mu ,s}^{*}}}{|u|}_{p_s^{*}}^p}}=SC{(\mu ,Q)}^{-{\displaystyle \frac{p}{2p_{\mu ,s}^{*}}}},$$

(7)

where S represents the best Sobolev constant mentioned in (4).

In what follows, we define the Euler–Lagrange functional ${\mathcal{I}}_{\lambda }:H{D}^{s,p}\left({\mathbb{H}}^N\right)\to \mathbb{R}$ associated with problem (1) as follows:

$${\mathcal{I}}_{\lambda }\left(u\right)={\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p+{\displaystyle \frac{b}{2p}}{\left[u\right]}_{H,s,p}^{2p}-{\displaystyle \frac{\lambda }{\kappa }}{\int }_{{\mathbb{H}}^N}h{|u|}^{\kappa }d\xi -{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}}{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi .$$

(8)

Furthermore, ${\mathcal{I}}_{\lambda }\in C^1(H{D}^{s,p}\left({\mathbb{H}}^N\right),\mathbb{R})$ and for all $u,v\in H{D}^{s,p}\left({\mathbb{H}}^N\right)$, the Fréchet derivative of ${\mathcal{I}}_{\lambda }$ shows that

$$\begin{array}{cc}\hfill \langle {\mathcal{I}}_{\lambda }^{\prime }\left(u\right),v\rangle & =\left(a+b{\left[u\right]}_{H,s,p}^p\right){\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)-u\left(\eta \right)|}^{p-2}(u\left(\xi \right)-u\left(\eta \right))(v\left(\xi \right)-v\left(\eta \right))}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta \hfill \\ \hfill & -\lambda {\int }_{{\mathbb{H}}^N}h{|u|}^{\kappa -2}uvd\xi -{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}-2}u\left(\xi \right)v\left(\xi \right)}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi .\hfill \end{array}$$

(9)

In the following, we shall prove that the energy functional ${\mathcal{I}}_{\lambda }$ corresponding to problem (1) satisfies the ${\left(PS\right)}_c$ condition.

Lemma 1. 

Let $0<\mu <2sp$, $1<\kappa <p<p_{\mu ,s}^{*}$ and $c<0$. Then, there exists ${\lambda }_{*}>0$ small enough such that for any $\lambda \in (0,{\lambda }_{*})$ the functional ${\mathcal{I}}_{\lambda }$ satisfies the ${\left(PS\right)}_c$ condition.

Proof. 

Let ${\left\{u_n\right\}}_n\subset H{D}^{s,p}\left({\mathbb{H}}^N\right)$ be a ${\left(PS\right)}_c$ sequence of the functional ${\mathcal{I}}_{\lambda }$, i.e.,

$${\mathcal{I}}_{\lambda }\left(u_n\right)\to c\mathrm{and}{\mathcal{I}}_{\lambda }^{\prime }\left(u_n\right)\to 0\mathrm{as}n\to \infty .$$

(10)

By the Hölder inequality and the Sobolev embedding theorem, we have

$${\parallel u\parallel }_{h,\kappa }^{\kappa }:={\int }_{{\mathbb{H}}^N}{h\left(x\right)|u|}^{\kappa }d\xi \le S^{-{\displaystyle \frac{\kappa }{p}}}{\parallel h\parallel }_{\theta }{\left[u\right]}_{H,s,p}^{\kappa }.$$

(11)

To this end, we shall verify that three claims hold.

Claim 1. 

We claim that ${\left\{u_n\right\}}_n$ is bounded in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$.

Since $1<\kappa <p<p_{\mu ,s}^{*},$ by (11), we have that

$$\begin{array}{cc}\hfill c+o\left(1\right)\parallel u_n\parallel & ={\mathcal{I}}_{\lambda }\left(u_n\right)-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\langle {\mathcal{I}}_{\lambda }^{\prime }\left(u_n\right),u_n\rangle \hfill \\ \hfill & =({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}})a{\left[u_n\right]}_{H,s,p}^p+({\displaystyle \frac{1}{2p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}})b{\left[u_n\right]}_{H,s,p}^{2p}-({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}})\lambda {\int }_{{\mathbb{H}}^N}h\left(x\right){|u|}^{\kappa }d\xi \hfill \\ \hfill & \ge ({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}})a{\left[u_n\right]}_{H,s,p}^p+({\displaystyle \frac{1}{2p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}})b{\left[u_n\right]}_{H,s,p}^{2p}-({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}})\lambda S^{-{\displaystyle \frac{\kappa }{p}}}{\parallel h\parallel }_{\theta }{\left[u\right]}_{H,s,p}^{\kappa }\hfill \end{array}$$

(12)

which implies that ${\left\{u_n\right\}}_n$ is bounded in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$. Take a subsequence, according to the concentration-compactness principle (Theorem 1.3 of [27]), we assume that for some $u\in H{D}^{s,p}\left({\mathbb{H}}^N\right),$

$$\begin{array}{c}{u}_n\rightharpoonup uinH{D}^{s,p}\left({\mathbb{H}}^N\right),\hfill \\ u_n\rightharpoonup uin{L}^{\wp }\left({\mathbb{H}}^N\right)for all\wp \in [1,p_{\mu ,s}^{*}),\hfill \\ u_n\to ua.e. in{\mathbb{H}}^N,\hfill \end{array}$$

(13)

$$\left({\int }_{{\mathbb{H}}^N}{\displaystyle \frac{|u_n{\left(\eta \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta \right)|u_n{\left(\xi \right)|}^{p_{\mu ,s}^{*}}d\xi \stackrel{*}{\rightharpoonup }\nu =\left({\int }_{{\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta \right){|u\left(\xi \right)|}^{p_{\mu ,s}^{*}}d\xi +\sum _{j\in J}{\nu }_j{\delta }_{{\xi }_j},$$

$$|D_H^s{u}_n{|}^{p}d\xi \stackrel{*}{\rightharpoonup }\omega \ge {|D_H^{s}u|}^{p}d\xi +\sum _{j\in J}{\omega }_j{\delta }_{{\xi }_j}$$

which are in the sense of measures of $H{D}^{s,p}\left({\mathbb{H}}^N\right)$, and

$$S_{HG,p}{\nu }_j^{{\displaystyle \frac{p}{2p_{\mu ,s}^{*}}}}\le {\omega }_{j}for allj\in J.$$

(14)

Furthermore, taking the same arguments as (Theorem 3.1 of [13]), we can obtain the concentration-compactness principle at infinity. Thus,

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{limsup}{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n{\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u_n\left(\xi \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi =\nu \left({\mathbb{H}}^N\right)+{\nu }_{\infty },\hfill \\ \hfill & \underset{n\to \infty }{limsup}{\int }_{{\mathbb{H}}^N}{|D_H^s{u}_n|}^{p}d\xi =\omega \left({\mathbb{H}}^N\right)+{\omega }_{\infty },\hfill \\ \hfill & S_{HG,p}{\nu }_{\infty }^{{\displaystyle \frac{p}{2p_{\mu ,s}^{*}}}}\le {\omega }_{\infty }.\hfill \end{array}$$

Claim 2. 

We claim that

$$J=\varnothing and{\nu }_{\infty }=0.$$

(15)

In fact, by contradiction, we assume that $J\ne \varnothing .$ Let $j\in J$ and take a smooth cut-off function $\psi .$ Fix $\psi \in {\mathcal{C}}_c^{\infty }\left({\mathbb{H}}^N\right)$ such that $0\le \psi \le 1,$ $\psi \left(O\right)=1$ and supp$\left(\psi \right)={\overline{B}}_1$. Choosing $\epsilon >0$ and putting ${\psi }_{\epsilon }\left(\xi \right)=\psi \left({\delta }_{1/\epsilon }\left(\xi \right)\right),\xi \in {\mathbb{H}}^N.$ Clearly, $\left\{u_n{\psi }_{\epsilon }\right\}$ is bounded in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$. Therefore, by (10), we have that $\langle {{\mathcal{I}}_{\lambda }}^{\prime }\left(u_n\right),u_n{\psi }_{\epsilon }\rangle =0$ as $n\to \infty$; that is,

$$\begin{array}{cc}\hfill & \left(a+b{\left[u_n\right]}_{H,s,p}^p\right)\left({\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^p{\psi }_{\epsilon }\left(\eta \right)}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta +\langle {\mathcal{A}}_p\left(u_n\right),{\psi }_{\epsilon }\rangle \right)\hfill \\ \hfill & =\lambda {\int }_{{\mathbb{H}}^N}h{|u_n|}^{\kappa }{\psi }_{\epsilon }d\xi +{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n{\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u_n\left(\xi \right)|}^{p_{\mu ,s}^{*}}{\psi }_{\epsilon }}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi +o\left(1\right),\hfill \end{array}$$

(16)

where

$$\langle {\mathcal{A}}_p\left(u_n\right),{\psi }_{\epsilon }\rangle :={\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^{p-2}(u_n\left(\xi \right)-u_n\left(\eta \right))u_n\left(\eta \right)({\psi }_{\epsilon }\left(\xi \right)-{\psi }_{\epsilon }\left(\eta \right))}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta .$$

Furthermore, let $\delta >0$ be an arbitrary and fixed value. Utilizing the boundedness of ${\left\{u_n\right\}}_n$ in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$ and the Young inequality, we obtain that

$$\begin{array}{cc}\hfill |\langle {\mathcal{A}}_p\left(u_n\right),{\psi }_{\epsilon }\rangle |& \le \delta {\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta +C{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{|u_n\left(\eta \right)|}^p{\displaystyle \frac{|{\psi }_{\epsilon }\left(\xi \right)-{\psi }_{\epsilon }{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta \hfill \\ \hfill & \le C\delta +C{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{|u_n\left(\eta \right)|}^p{\displaystyle \frac{|{\psi }_{\epsilon }\left(\xi \right)-{\psi }_{\epsilon }{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta .\hfill \end{array}$$

(17)

Taking the same arguments as (Lemma 4.4 of [31]), we have that

$$\underset{\epsilon \to 0}{lim}\underset{n\to \infty }{lim}{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{|u_n\left(\eta \right)|}^p{\displaystyle \frac{|{\psi }_{\epsilon }\left(\xi \right)-{\psi }_{\epsilon }{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta =0.$$

(18)

Therefore, passing the limit as $\epsilon \to 0$ and $n\to \infty$ in (17), we have

$$\underset{\epsilon \to 0}{lim\; sup}\underset{n\to \infty }{lim\; sup}\langle {\mathcal{A}}_p\left(u_n\right),{\psi }_{\epsilon }\rangle =0,$$

(19)

since $\delta >0$ is arbitrary. Note that

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^p{\psi }_{\epsilon }\left(\eta \right)}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta & \ge {\int }_{{\mathbb{H}}^N}{\psi }_{\epsilon }d\omega \hfill \\ \hfill & \ge {\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)-u\left(\eta \right)|}^p{\psi }_{\epsilon }\left(\eta \right)}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta +{\omega }_j\hfill \end{array}$$

(20)

and

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n{\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u_n\left(\xi \right)|}^{p_{\mu ,s}^{*}}{\psi }_{\epsilon }}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi & ={\int }_{{\mathbb{H}}^N}{\psi }_{\epsilon }d\nu \hfill \\ \hfill & ={\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}}{\psi }_{\epsilon }}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi +{\nu }_j.\hfill \end{array}$$

(21)

Moreover, by (13) and $1<\kappa <p<p_{\mu ,s}^{*},$ we have that

$$\underset{\epsilon \to 0}{lim}\underset{n\to \infty }{lim}{\int }_{{\mathbb{H}}^N}h{|u_n|}^{\kappa }{\psi }_{\epsilon }d\xi =0.$$

(22)

Thus, from (16)–(22), we obtain that ${\nu }_j\ge a{\omega }_j$. Together with (14), we have that

$${\omega }_j=0or{\omega }_j\ge {\left(a{S}_{HG,p}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{p}}}\right)}^{{\displaystyle \frac{p}{2p_{\mu ,s}^{*}-p}}}.$$

Using (11) and the Young inequality, we have

$$\begin{array}{cc}\hfill \lambda {\int }_{{\mathbb{H}}^N}h\left(\xi \right){|u|}^{\kappa }d\xi \le & \lambda S^{-{\displaystyle \frac{\kappa }{p}}}{\parallel h\parallel }_{\theta }{\left[u\right]}_{H,s,p}^{\kappa }\hfill \\ \hfill =& \left({\left[\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{a}{\kappa }}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}\right]}^{{\displaystyle \frac{\kappa }{p}}}{\left[u\right]}_{H,s,p}^{\kappa }\right)\hfill \\ \hfill & \times \left({\left[\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{a}{\kappa }}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}\right]}^{-{\displaystyle \frac{\kappa }{p}}}\lambda {\parallel h\parallel }_{\theta }{S}^{-{\displaystyle \frac{\kappa }{p}}}\right)\hfill \\ \hfill \le & \left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{a}{p}}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}{\left[u\right]}_{H,s,p}^p\hfill \\ \hfill & +{\displaystyle \frac{p-\kappa }{p}}{\left[\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{aS}{\kappa }}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}\right]}^{{\displaystyle \frac{\kappa }{\kappa -p}}}{\left({\lambda \parallel h\parallel }_{\theta }\right)}^{{\displaystyle \frac{p}{p-\kappa }}}.\hfill \end{array}$$

(23)

If the latter holds, we obtain

$$\begin{array}{cc}\hfill 0& >c=\underset{n\to \infty }{lim}\left({\mathcal{I}}_{\lambda }\left(u_n\right)-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\langle {{\mathcal{I}}_{\lambda }}^{\prime }\left(u_n\right),u_n\rangle \right)\hfill \\ \hfill & =\underset{n\to \infty }{lim}\left\{\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)a{\left[u_n\right]}_{H,s,p}^p+\left({\displaystyle \frac{1}{2p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)b{\left[u_n\right]}_{H,s,p}^{2p}-\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)\lambda {\int }_{{\mathbb{H}}^N}h{|u_n|}^{\kappa }d\xi \right\}\hfill \\ \hfill & \ge \left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)a{\left[u\right]}_{H,s,p}^p-\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)\lambda {\int }_{{\mathbb{H}}^N}h{|u|}^{\kappa }d\xi \hfill \\ \hfill & \ge \left({\displaystyle \frac{1}{p^2}}-{\displaystyle \frac{1}{2p·p_{\mu ,s}^{*}}}\right)(p-1)a{\left[u\right]}_{H,s,p}^p-{\displaystyle \frac{p-\kappa }{p}}{\left[\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{aS}{\kappa }}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}\right]}^{{\displaystyle \frac{\kappa }{\kappa -p}}}{\left({\lambda \parallel h\parallel }_{\theta }\right)}^{{\displaystyle \frac{p}{p-\kappa }}}\hfill \\ \hfill & \ge \left({\displaystyle \frac{1}{p^2}}-{\displaystyle \frac{1}{2p·p_{\mu ,s}^{*}}}\right)(p-1)a{\omega }_j-{\displaystyle \frac{p-\kappa }{p}}{\left[\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{aS}{\kappa }}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}\right]}^{{\displaystyle \frac{\kappa }{\kappa -p}}}{\left({\lambda \parallel h\parallel }_{\theta }\right)}^{{\displaystyle \frac{p}{p-\kappa }}}\hfill \\ \hfill & \ge \left({\displaystyle \frac{1}{p^2}}-{\displaystyle \frac{1}{2p·p_{\mu ,s}^{*}}}\right)(p-1){\left(a{S}_{HG,p}\right)}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{2p_{\mu ,s}^{*}-p}}}-{\displaystyle \frac{p-\kappa }{p}}{\left[\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{aS}{\kappa }}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}\right]}^{{\displaystyle \frac{\kappa }{\kappa -p}}}{\left({\lambda \parallel h\parallel }_{\theta }\right)}^{{\displaystyle \frac{p}{p-\kappa }}}.\hfill \end{array}$$

(24)

Hence, we can find ${\lambda }_1$ small enough such that the right-hand side of (24) is greater than zero for each $\lambda \in (0,{\lambda }_1)$, which is impossible. Therefore, we obtain that $J=\varnothing$.

Then, we verify that ${\nu }_{\infty }=0$. By contradiction, we assume that ${\mu }_{\infty }>0$. Similarly, in order to ascertain the mass concentration at infinity, we define a cut-off function ${\varphi }_R\in {\mathcal{C}}_c^{\infty }\left({\mathbb{H}}^N\right)$ and $R>0$ such that $0\le {\varphi }_R\le 1$ and

$$\left\{\begin{array}{cc}{\varphi }_R=1\hfill & in{B}_2^c,\hfill \\ {\varphi }_R=0\hfill & in{B}_1.\hfill \end{array}\right.$$

(25)

Take $R>0$ and set ${\varphi }_R=\varphi \left({\delta }_{1/R}\left(\xi \right)\right),\xi \in {\mathbb{H}}^N.$ Clearly, $\left\{u_n{\varphi }_R\right\}$ is bounded in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$. Therefore, from (10), we have that $\langle {\mathcal{I}}^{\prime }\left(u_n\right),u_n{\varphi }_R\rangle =0$ as $n\to \infty$; that is,

$$\begin{array}{cc}\hfill & \left(a+b{\left[u_n\right]}_{H,s,p}^p\right){\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^p{\varphi }_R\left(\eta \right)}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta +\langle {\mathcal{A}}_p\left(u_n\right),{\varphi }_R\rangle \hfill \\ \hfill & =\lambda {\int }_{{\mathbb{H}}^N}h{|u_n|}^{\kappa }{\varphi }_{R}d\xi +{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n{\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u_n\left(\xi \right)|}^{p_{\mu ,s}^{*}}{\varphi }_R}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi +o\left(1\right),\hfill \end{array}$$

(26)

where

$$\langle {\mathcal{A}}_p\left(u_n\right),{\varphi }_R\rangle :={\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^{p-2}(u_n\left(\xi \right)-u_n\left(\eta \right))u_n\left(\eta \right)({\varphi }_R\left(\xi \right)-{\varphi }_R\left(\eta \right))}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta .$$

Furthermore, let $\delta >0$ be an arbitrary and fixed value. Utilizing the boundedness of ${\left\{u_n\right\}}_n$ in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$ and the Young inequality, we also obtain that

$$\begin{array}{cc}\hfill |\langle {\mathcal{A}}_p\left(u_n\right),{\varphi }_R\rangle |& \le \delta {\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta +C{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{|u_n\left(\eta \right)|}^p{\displaystyle \frac{|{\varphi }_R\left(\xi \right)-{\varphi }_R{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta \hfill \\ \hfill & \le C\delta +C{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{|u_n\left(\eta \right)|}^p{\displaystyle \frac{|{\varphi }_R\left(\xi \right)-{\varphi }_R{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta .\hfill \end{array}$$

(27)

Taking the same proof as (18), we obtain that

$$\underset{R\to \infty }{lim}\underset{n\to \infty }{lim}{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{|u_n\left(\eta \right)|}^p{\displaystyle \frac{|{\varphi }_R\left(\xi \right)-{\varphi }_R{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta =0.$$

(28)

Therefore, passing the limit as $R\to \infty$ and $n\to \infty$ in (28), we obtain that

$$\underset{R\to \infty }{lim\; sup}\underset{n\to \infty }{lim\; sup}\langle {\mathcal{A}}_p\left(u_n\right),{\varphi }_R\rangle =0,$$

(29)

because of the arbitrariness of $\delta >0$. Additionally, we obtain that ${\nu }_{\infty }\ge a{\omega }_{\infty }$ and

$${\omega }_{\infty }=0or{\omega }_{\infty }\ge {\left(a{S}_{HG,p}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{p}}}\right)}^{{\displaystyle \frac{p}{2p_{\mu ,s}^{*}-p}}}.$$

Similarly to the analysis of (24), if ${\omega }_{\infty }\ge {\left(a{S}_{HG,p}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{p}}}\right)}^{{\displaystyle \frac{p}{2p_{\mu ,s}^{*}-p}}}$, we can find ${\lambda }_2$ small enough such that this situation is impossible. Therefore, we obtain that ${\nu }_{\infty }=0$.

Based on the above discussion, for any $c<0$, $\lambda \in (0,{\lambda }_{*})$ where ${\lambda }_{*}=min\{{\lambda }_1,{\lambda }_2\}$, then we have

$$J=\varnothing and{\nu }_{\infty }=0.$$

Claim 3. 

We claim

$${\left[u_n\right]}_{H,s,p}^p\to {\left[u\right]}_{H,s,p}^p.$$

(30)

Together with $J=\varnothing$ and ${\nu }_{\infty }=0$, we deduce that

$${\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n{\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u_n\left(\xi \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi \to {\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi .$$

(31)

According to Vitali convergence theorem, we draw the conclusion that

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{H}}^N}h\left(x\right)|u_n{|}^{\kappa }dx={\int }_{{\mathbb{H}}^N}h\left(x\right){|u|}^{\kappa }dx.$$

(32)

Thus, we obtain that

$${\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{|u_n\left(\xi \right)-u_n{\left(\eta \right)|}^p}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta =\lambda {\int }_{{\mathbb{H}}^N}h\left(x\right){|u|}^{\kappa }d\xi +{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi .$$

(33)

Since $\langle {\mathcal{I}}_{\lambda }^{\prime }\left(u\right),v\rangle =o\left(1\right)$ for any $H{D}^{s,p}\left({\mathbb{H}}^N\right),$ by (13), we obtain that

$$\begin{array}{cc}\hfill & {\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\xi \right)-u\left(\eta \right)|}^{p-2}(u\left(\xi \right)-u\left(\eta \right))(v\left(\xi \right)-v\left(\eta \right))}{r{({\eta }^{-1}\circ \xi )}^{Q+sp}}}d\xi d\eta \hfill \\ \hfill & =\lambda {\int }_{{\mathbb{H}}^N}h{|u|}^{\kappa -2}uvd\xi +{\int \int }_{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}-2}u\left(\xi \right)v\left(\xi \right)}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi .\hfill \end{array}$$

(34)

Therefore, by (33) and (34) with $v=u$, we obtain that ${\left[u_n\right]}_{H,s,p}^p\to {\left[u\right]}_{H,s,p}^p.$ This implies that $u_n\to u$ in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$. Therefore, we complete the proof of Lemma 1.

□

3. Proof of Theorem 1

The main content of this section is to verify that Equation (1) has infinitely many solutions approaching zero when $1<\kappa <p$. Therefore, we adopted the new version of the symmetric mountain pass theorem proposed by Kajikiya in (Theorem 1 of [32]) to address this issue.

Lemma 2. 

Let $\mathcal{W}$ be an infinite-dimensional Banach space and ${\mathcal{I}}_{\lambda }\in C^1\left(E\right)$. If the following conditions hold:

($U_1$)

${\mathcal{I}}_{\lambda }$ is even, has a lower bound in $\mathcal{W}$, ${\mathcal{I}}_{\lambda }\left(0\right)=0$, and satisfies the local ${\left(PS\right)}_c$ condition;

($U_2$)

there exist ${\mathcal{E}}_n\in {\Sigma }_n$ for any $n\in \mathbb{N}$ such that ${sup}_{u\in E_n}{\mathcal{I}}_{\lambda }\left(u\right)<0$ and $\gamma \left(\mathcal{E}\right)$ is a genus of $\mathcal{E}$, where

$${\Sigma }_n:=\{\mathcal{E}:\mathcal{E}\subset \mathcal{W}isclosedsymmetric,0\notin \mathcal{E},\gamma \left(\mathcal{E}\right)\ge n\}.$$

Then, there exists a sequence of critical points ${\left\{u_n\right\}}_n$ determined by the function ${\mathcal{I}}_{\lambda }$ such that for any n, ${\mathcal{I}}_{\lambda }\left(u_n\right)\le 0$, $u_n\ne 0$, and $u_n\to 0$ as $n\to \infty$.

Let ${\mathcal{I}}_{\lambda }$ be the functional defined in (8). Then, based on (11) and the Hardy–Littlewood–Sobolev inequality, we can obtain

$$\begin{array}{cc}\hfill {\mathcal{I}}_{\lambda }\left(u\right)& \ge {\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p-\lambda S^{-{\displaystyle \frac{\kappa }{2}}}{\parallel h\parallel }_{\theta }{\parallel u\parallel }^{\kappa }-{\displaystyle \frac{S_{HG,p}^{-1}}{2p_{\mu ,s}^{*}}}{\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}}\hfill \\ & =r_1{\parallel u\parallel }^p-\lambda r_2{\parallel u\parallel }^{\kappa }-r_3{\parallel u\parallel }^{2p_{\mu ,s}^{*}},\hfill \end{array}$$

where $r_1,r_2,r_3$ are some positive constants. Suppose that

$$z\left(t\right)=r_1{t}^p-\lambda r_2{t}^{\kappa }-r_3{t}^{2p_{\mu ,s}^{*}},t\in {\mathbb{R}}_0^{+}.$$

Then, for any $\lambda \in (0,{\lambda }_{*})$, we choose ${\lambda }_{*}>0$ small enough such that there exist $t_0$, $t_1$, with $0<t_0<t_1$ and

$$\left\{\begin{array}{cc}z\left(t\right)<0\hfill & \mathrm{in}(0,t_0),\hfill \\ z\left(t\right)>0\hfill & \mathrm{in}(t_0,t_1),\hfill \\ z\left(t\right)=0\hfill & \mathrm{in}(t_1,+\infty ).\hfill \end{array}\right.$$

Obviously, $z\left(t_0\right)=0=z\left(t_1\right)$. By using the same method as in reference [33] to consider the truncation function $\widetilde{{\mathcal{I}}_{\lambda }}$ of ${\mathcal{I}}_{\lambda }$ and defining for all $u\in H{D}^{s,p}\left({\mathbb{H}}^N\right)$ by

$$\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right):={\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p+{\displaystyle \frac{b}{2p}}{\left[u\right]}_{H,s,p}^{2p}-{\displaystyle \frac{\lambda }{\kappa }}{\psi \left(u\right)\parallel u\parallel }_{h,\kappa }^{\kappa }-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\psi \left(u\right){\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}},$$

(35)

where $\psi \left(u\right)=\sigma (\parallel u\parallel )$ and $\sigma :{\mathbb{R}}_0^{+}\to [0,1]$ is a non-increasing $C^{\infty }$ function such that $\sigma \left(t\right)=1$ when $t\in [0,t_0]$ and $\sigma \left(t\right)=0$ when $t\ge t_1$. Clearly, $\widetilde{{\mathcal{I}}_{\lambda }}\in C^1\left(H{D}^{s,p}\left({\mathbb{H}}^N\right)\right)$ and $\widetilde{{\mathcal{I}}_{\lambda }}$ is bounded from below in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$.

Based on the above argument and in accordance with all the assumptions stated in Theorem 1, we can draw the following conclusion:

Lemma 3. 

Let $\widetilde{{\mathcal{I}}_{\lambda }}$ be the functional introduced in (35). Then, the following properties hold:

(i)

Suppose that $\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)<0$, then $\parallel u\parallel \le t_0$ and $\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)={\mathcal{I}}_{\lambda }\left(u\right)$.

(ii)

Let $c<0$. Then, there exists ${\lambda }_{*}>0$ small enough such that for all $\lambda \in (0,{\lambda }_{*})$, $\widetilde{{\mathcal{I}}_{\lambda }}$ satisfies the ${\left(PS\right)}_c$ condition.

Next, we will once again review the content of Theorem 1 and begin the proof.

• Theorem 1: Let $1<\kappa <p<p_{\mu ,s}^{*}$. If $\left(\mathcal{C}\right)$ is satisfied, then there exists ${\lambda }_{*}>0$ such that if $\lambda \in (0,{\lambda }_{*})$, problem (1) has a sequence of nontrivial solutions ${\left\{u_n\right\}}_n$, with ${\mathcal{I}}_{\lambda }\left(u_n\right)\le 0$ and $u_n\to 0$ as $n\to \infty$.

Proof. 

Obviously, $\widetilde{{\mathcal{I}}_{\lambda }}\left(0\right)=0$, and $\widetilde{{\mathcal{I}}_{\lambda }}$ is a function belonging to $C^1\left(H{D}^{s,p}\left({\mathbb{H}}^N\right)\right)$, which is even, coercive, and has a lower bound.

Select n mutually disjoint open sets $T_j$, let ${\cup }_{j=1}^n{T}_j\subset \Omega$ hold, where $n\in \mathbb{N}$ and $\Omega$ is the nonempty open set mentioned in Theorem 1. Now, we take $u_j\in (H{D}^{s,p}\left({\mathbb{H}}^N\right)\cap C_0^{\infty }\left(T_j\right))\setminus \left\{0\right\}$ for all $j=1,2,\dots ,n$, with $\parallel u_j\parallel =1$. Suppose that ${\mathcal{W}}_n=span\{u_1,u_2,\dots ,u_n\}$.

Hence, for each $u\in {\mathcal{W}}_n$, let $\parallel u\parallel =m$, and we obtain

$$\begin{array}{cc}\hfill \widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)& \le {\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p+{\displaystyle \frac{b}{2p}}{\left[u\right]}_{H,s,p}^{2p}-{\displaystyle \frac{\lambda }{\kappa }}{\int }_{{\mathbb{H}}^N}{h\left(\xi \right)|u|}^{\kappa }d\xi -{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}{\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}}\hfill \\ & \le {\displaystyle \frac{a}{p}}{m}^p+{\displaystyle \frac{b}{2p}}{m}^{2p}-c_1{m}^{\kappa }-c_2{m}^{2p_{\mu ,s}^{*}},\hfill \end{array}$$

where the constants $c_1,c_2$ are positive. This is because in the finite dimensional space ${\mathcal{W}}_n$, every norm is equivalent. If $m>0$ is small enough and $1<\kappa <p$, then $\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)<0$. Thus, there is such a conclusion:

$$\{u\in {\mathcal{W}}_n:\parallel u\parallel =m\}\subset \{u\in {\mathcal{W}}_n:\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)<0\}.$$

By Chang [34], we have $\gamma \left(\right\{u\in {\mathcal{W}}_n:\parallel u\parallel =m\left\}\right)=n.$ Meanwhile, by referring to Krasnoselskii’s discussion in [35] about the monotonicity of the genus $\gamma$, we obtain

$$\gamma \left(\{u\in {\mathcal{W}}_n:\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)<0\}\right)\ge n.$$

Assuming ${\mathcal{E}}_n=\{u\in {\mathcal{W}}_n:\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)<0\}$, then we have ${\mathcal{E}}_n\in {\Sigma }_n$ and ${sup}_{u\in {\mathcal{E}}_n}\widetilde{{\mathcal{I}}_{\lambda }}\left(u\right)<0$. Moreover, given that $H{D}^{s,p}\left({\mathbb{H}}^N\right)$ constitutes a real infinite dimensional Sobolev space, all the prerequisites stipulated in Lemma 2 are fulfilled. Consequently, there exists a sequence ${\left\{u_n\right\}}_n$ in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$ such that $\widetilde{{\mathcal{I}}_{\lambda }}\left(u_n\right)\le 0$ when $u_n\ne 0$ and for any n, $\widetilde{{\mathcal{I}}_{\lambda }^{\prime }}\left(u_n\right)=0$. At the same time, we also obtain $\parallel u_n\parallel \to 0$ as $n\to 0$.

By Lemma 3, choose a sufficiently large n such that $\parallel u_n\parallel \le m$ is small; thus, the infinite number of nontrivial functions $u_n$ are solutions to Equation (1). □

4. Proof of Theorem 2

In this part, we want to prove the multiplicity of solutions for Equation (1) by means of the mountain pass theorem for even functionals. Let $\kappa =p$, and Equation (1) becomes

$$(a+b{\left[u\right]}_{H,s,p}^p){(-\Delta )}_{H,p}^{s}u={\lambda h\left(\xi \right)|u|}^{p-2}u+\left({\int }_{{\mathbb{H}}^N}{\displaystyle \frac{{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta \right){|u|}^{p_{\mu ,s}^{*}-2}u\mathrm{in}{\mathbb{H}}^N.$$

(36)

The functional relevant to problem (36) is

$${\mathcal{I}}_{\lambda }\left(u\right)={\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p+{\displaystyle \frac{b}{2p}}{\left[u\right]}_{H,s,p}^{2p}-{\displaystyle \frac{\lambda }{p}}{\parallel u\parallel }_{h,p}^p-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}{\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}}\mathrm{for} \mathrm{all}u\in H{D}^{s,p}\left({\mathbb{H}}^N\right).$$

Lemma 4. 

Let $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$ and ${\left\{u_n\right\}}_n$ be a ${\left(PS\right)}_c$ sequence for ${\mathcal{I}}_{\lambda }\left(u\right)$ in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$, with

$$c<c^{\prime }:={\displaystyle \frac{1}{2p}}{\left(a{S}_{HG,p}\right)}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{2p_{\mu ,s}^{*}-p}}}.$$

Then, ${\left\{u_n\right\}}_n$ includes a strongly convergent subsequence.

Proof. 

By Sobolev embedding theorem and the Hölder inequality, we have

$${\parallel u\parallel }_{h,p}^p\le S^{-{\displaystyle \frac{p}{2}}}{\parallel h\parallel }_{\theta }{\parallel u\parallel }^p\mathrm{for} \mathrm{all}u\in H{D}^{s,p}\left({\mathbb{H}}^N\right).$$

(37)

Given a ${\left(PS\right)}_c$ sequence of ${\mathcal{I}}_{\lambda }$ in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$ with respect to the level $c<c^{\prime }$. On account of $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$, $0<\mu <2sp$ and by (37), similarly to (24), we obtain

$$\begin{array}{cc}\hfill c^{\prime }>c& =\underset{n\to \infty }{lim}\left({\mathcal{I}}_{\lambda }\left(u_n\right)-{\displaystyle \frac{1}{2p}}\langle {\mathcal{I}}_{\lambda }^{\prime }\left(u_n\right),u_n\rangle \right)\hfill \\ \hfill & \ge {\displaystyle \frac{1}{2p}}a{\left[u_n\right]}_{H,s,p}^p-{\displaystyle \frac{\lambda }{2p}}{\parallel u_n\parallel }_{h,p}^p\hfill \\ \hfill & \ge {\displaystyle \frac{a}{2p}}{w}_{j_0}+{\displaystyle \frac{1}{2p}}(a-\lambda S^{-{\displaystyle \frac{p}{2}}}{\parallel h\parallel }_{\theta }{)\parallel u\parallel }^p\hfill \\ \hfill & \ge {\displaystyle \frac{1}{2p}}a{w}_{j_0}\ge {\displaystyle \frac{1}{2p}}{\left(a{S}_{HG,p}\right)}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{2p_{\mu ,s}^{*}-p}}}=c^{\prime }\hfill \end{array}$$

(38)

which is invalid. Hence, the compactness of the $\left(PS\right)$ sequence can be obtained. □

The main method of Theorem 2 used is the mountain pass theorem for even functions, and the proof of the theorem can be found in Rabinowitz [36]. The specific applications in this section are as follows:

Proposition 2. 

Let X be an infinite dimensional Banach space, and $X=V\oplus Y$, where V is finite dimensional. Let even functional $\mathcal{I}\in C^1\left(X\right)$ satisfy $\mathcal{I}\left(0\right)=0$ and the following conditions:

($I_1$)

For all $u\in \partial B_{\rho }\left(0\right)\cap Y$, there exist positive constants $\tau ,\rho$ such that $\mathcal{I}\left(u\right)\ge \tau$.

($I_2$)

There exists $c^{\prime }>0$ such that $\mathcal{I}$ satisfies the ${\left(PS\right)}_c$ condition for all $c\in (0,c^{\prime })$.

($I_3$)

There exists $R=R\left(\widehat{X}\right)$ for each finite dimensional subspace $\widehat{X}\subset X$ such that $\mathcal{I}\left(u\right)\le 0$ for all $u\in \widehat{X}\setminus B_R\left(0\right)$.

If V is h dimensional and $V=span\{e_1,e_2,\dots ,e_h\}$. Choose $e_{n+1}\notin X_n:=span\{e_1,e_2,\dots ,e_n\}$ for $n\ge h$. Let $R_n=R\left(X_n\right)$, $D_n=B_{R_n}\left(0\right)\cap X_n$ and define

$$\begin{array}{cc}\hfill G_n& :=\{g\in C(D_n,X):gisoddandg\left(u\right)=uforallu\in \partial B_{R_n}\left(0\right)\cap X_n\},\hfill \\ \hfill {\Gamma }_i& :=\{g\left(\overline{D_n\setminus E}\right):g\in G_n,n\ge i,E\in {\Sigma }_{n-i}and\gamma \left(E\right)\le n-i\},\hfill \\ \hfill {\Sigma }_n& :=\{E:E\in Xisclosedsymmetric,0\notin E,\gamma (E)\ge n\}.\hfill \end{array}$$

(39)

Let

$$c_i:=\underset{\mathcal{H}\in {\Gamma }_i}{inf}\underset{u\in \mathcal{H}}{max}\mathcal{I}\left(u\right)for eachi\in \mathbb{N}.$$

When $0<\tau \le c_i\le c_{i+1}$ for all $i>h$, $c_i$ is a critical value of $\mathcal{I}$ if $i>h$ and $c_i<c^{\prime }$. Furthermore, $\gamma \left({\mathcal{H}}_c\right)\ge l+1$ holds when $c_i=c_{i+1}=\dots =c_{i+l}=c<c^{\prime }$ for all $i>h$, where

$${\mathcal{H}}_c:=\{u\in E:\mathcal{I}\left(u\right)=cand{\mathcal{I}}^{\prime }\left(u\right)=0\}.$$

Lemma 5. 

For any $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$, the functional ${\mathcal{I}}_{\lambda }$ satisfies conditions $\left(I_1\right)$–$\left(I_3\right)$.

Proof. 

Since $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$, according to the definitions of S and $S_{HG,p}$, the following can be obtained:

$${\mathcal{I}}_{\lambda }\left(u\right)\ge {\displaystyle \frac{1}{p}}(a-\lambda S^{-{\displaystyle \frac{p}{2}}}{\parallel h\parallel }_{\theta }{)\parallel u\parallel }^p-{\displaystyle \frac{S_{HG,p}^{-1}}{2p_{\mu ,s}^{*}}}{\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}}.$$

Since $p<2p_{\mu ,s}^{*}$, for every $u\in H{D}^{s,p}\left({\mathbb{H}}^N\right)$, there exists $\tau >0$ such that ${\mathcal{I}}_{\lambda }\left(u\right)\ge \tau$, where $\parallel u\parallel =\rho$ and $\rho$ chosen is small enough. Hence, ${\mathcal{I}}_{\lambda }$ satisfies condition $\left(I_1\right)$.

Since $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$, according to Lemma 4 that ${\mathcal{I}}_{\lambda }$ satisfies $\left(I_2\right)$.

Now, let $\mathcal{E}$ be a finite dimensional subspace of $H{D}^{s,p}\left({\mathbb{H}}^N\right)$. For all $u\in \mathcal{E}$, when $\parallel u\parallel$ is large enough, we can obtain the following content:

$$\begin{array}{cc}\hfill {\mathcal{I}}_{\lambda }\left(u\right)& \le {\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p+{\displaystyle \frac{b}{2p}}{\left[u\right]}_{H,s,p}^{2p}+{\displaystyle \frac{\lambda }{p}}{\parallel u\parallel }_{h,p}^p-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}{\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}}\hfill \\ \hfill & \le {\displaystyle \frac{a}{p}}{\parallel u\parallel }^p+{\displaystyle \frac{b}{2p}}{\parallel u\parallel }^{2p}+{\displaystyle \frac{\lambda }{p}}{c}_1{\parallel u\parallel }^p-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}{c}_2{\parallel u\parallel }^{2p_{\mu ,s}^{*}},\hfill \end{array}$$

(40)

where the constants $c_1,c_2$ are positive. This is because in the finite dimensional space, all norms are equivalent. We conclude that for any $u\in \mathcal{E}$, when $p<2p<2p_{\mu ,s}^{*}$, there is ${\mathcal{I}}_{\lambda }\left(u\right)<0$, where $\parallel u\parallel \ge R$ and R is sufficiently large. Therefore, ${\mathcal{I}}_{\lambda }$ satisfies $\left(I_3\right)$. □

Lemma 6. 

There exists a sequence ${\left\{M_n\right\}}_n$ that is independent of λ and for all n and $\lambda >0$, it holds that $M_n\le M_{n+1}$, and

$$c_n^{\lambda }:=\underset{H\in {\Gamma }_n}{inf}\underset{u\in H}{max}{\mathcal{I}}_{\lambda }\left(u\right)<M_n.$$

Proof. 

Similarly to the proof process of (Lemma 5 of [37]), using condition $\left(\mathcal{C}\right)$, we obtain that

$$\begin{array}{cc}\hfill c_n^{\lambda }& =\underset{H\in {\Gamma }_n}{inf}\underset{u\in H}{max}\{{\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p+{\displaystyle \frac{b}{2p}}{\left[u\right]}_{H,s,p}^{2p}-{\displaystyle \frac{\lambda }{p}}{\parallel u\parallel }_{h,p}^p-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}{\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}}\}\hfill \\ \hfill & <\underset{H\in {\Gamma }_n}{inf}\underset{u\in H}{max}\{{\displaystyle \frac{a}{p}}{\parallel u\parallel }^p+{\displaystyle \frac{b}{2p}}{\parallel u\parallel }^{2p}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}{\parallel u\parallel }_{FL,p}^{2p_{\mu ,s}^{*}}\}:=M_n.\hfill \end{array}$$

(41)

Thus, by the definition of ${\Gamma }_n$, $M_n<\infty$ and $M_n\le M_{n+1}$. □

Next, we will once again review the content of Theorem 2 and begin the proof.

• Theorem 2: Let $\kappa =p$ and $\left(\mathcal{C}\right)$ be satisfied, then there exists a positive constant $a^{\prime }$ such that for all $a>a^{\prime }$ and $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$ problem (1) has at least n pairs of nontrivial solutions.

Proof. 

Choose a large enough $a^{\prime }>0$ such that for any $a>a^{\prime }$, using Lemma 6, we obtain

$$\underset{n}{sup}{M}_n<{\displaystyle \frac{1}{2p}}{\left(a{S}_{HG,p}\right)}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{2p_{\mu ,s}^{*}-p}}}.$$

Hence,

$$c_n^{\lambda }<M_n<{\displaystyle \frac{1}{2p}}{\left(a{S}_{HG,p}\right)}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{2p_{\mu ,s}^{*}-p}}}.$$

Therefore, for any $\lambda \in (0,a{S}^{{\displaystyle \frac{p}{2}}}\parallel h{\parallel }_{\theta }^{-1})$ and $a>a^{\prime }$, there exists

$$0<c_1^{\lambda }\le c_2^{\lambda }\le \dots \le c_n^{\lambda }<M_n<c^{\prime }.$$

According to Proposition 2, the levels $c_1^{\lambda }\le c_2^{\lambda }\le \dots \le c_n^{\lambda }$ constitute the critical value of ${\mathcal{I}}_{\lambda }$. Consequently, if $c_1^{\lambda }<c_2^{\lambda }<\dots <c_n^{\lambda }$ holds, then the function ${\mathcal{I}}_{\lambda }$ possesses at least n critical points. In addition, if $c_i^{\lambda }=c_{i+1}^{\lambda }$ is satisfied, by reapplying Proposition 2, it follows that $H_{c_i^{\lambda }}$ represents an infinite set, as detailed in (Chapter 7 of [36]), and under this scenario, (36) has infinitely many solutions. Therefore, (36) has at least n pairs of solutions in $H{D}^{s,p}\left({\mathbb{H}}^N\right)$. □

5. Proof of Theorem 3

Consider a Banach space X. Let A denote a symmetric subset in $X\setminus \left\{0\right\}$. Subsequently, the cohomological index of A is formally defined as $i\left(A\right)$, please refer to [21,38] for more details. Furthermore, if A is homeomorphic to the unit sphere $S^{m-1}$ in ${\mathbb{H}}^m$, then $i\left(A\right)=m$.

Theorem 4 

(Theorem 2.1 of [21]). Let X be a Banach space, and $\mathcal{J}:X\to \mathbb{R}$ be an even $C^1$-functional satisfying the Palais–Smale condition at level c, where $c\in (0,c^{*})$ and

$$c^{*}=\left({\displaystyle \frac{1}{p^2}}-{\displaystyle \frac{1}{2p·p_{\mu ,s}^{*}}}\right)(p-1){\left(a{S}_{HG,p}\right)}^{{\displaystyle \frac{2p_{\mu ,s}^{*}}{2p_{\mu ,s}^{*}-p}}}-{\displaystyle \frac{p-\kappa }{p}}{\left[\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right){\displaystyle \frac{aS}{\kappa }}{\left({\displaystyle \frac{1}{\kappa }}-{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\right)}^{-1}\right]}^{{\displaystyle \frac{\kappa }{\kappa -p}}}{\left({\lambda \parallel h\parallel }_{\theta }\right)}^{{\displaystyle \frac{p}{p-\kappa }}}.$$

Assuming that 0 is a strict local minimizer of $\mathcal{J}$ and there exists $R>0$ and a compact symmetric set $A\subset \partial B_R$, where $B_R=\{u\in X|\parallel u\parallel <R\}$, such that $i\left(A\right)=m$,

$$\underset{A}{max}\mathcal{J}\le 0,\underset{B}{max}\mathcal{J}\le c^{*},$$

where $B=\{tu|t\in [0,1],u\in A\}$, then the functional $\mathcal{J}$ possesses m pairs of nonzero critical points, all of which have positive critical values.

Next, we will once again review the content of Theorem 3 and begin the proof.

• Theorem 3: Assume $1<p<\kappa <p_{\mu ,s}^{*}$. Let $\left(\mathcal{C}\right)$ be satisfied and for every $m\in \mathbb{N}$, then there exists ${\lambda }_m>0$ such that problem (1) has m pairs of positive energy solutions for all $\lambda >{\lambda }_m$.

Proof. 

From Lemma 1, we obtain that ${\mathcal{I}}_{\lambda }$ satisfies the ${\left(PS\right)}_c$ condition for any $c\in (0,c^{*})$. Obviously, since $1<p<\kappa <p_s^{*}$ and $c^{*}>0$, $u=0$ is a strict local minimum of ${\mathcal{I}}_{\lambda }$. Let $X=\{u\in H{D}^{s,p}\left({\mathbb{H}}^N\right)\mid \mathrm{supp}u\subset \Lambda \}$, $\Lambda$ is given in the condition $\left(\mathcal{C}\right)$, so X is an infinite dimensional subspace on $H{D}^{s,p}\left({\mathbb{H}}^N\right)$. Assume that $X_{\ell }$ is a ℓ-dimensional subspace of $H{D}^{s,p}\left({\mathbb{H}}^N\right).$ Let ${\left[u\right]}_h={\left({\int h|u|}^{\kappa }d\xi \right)}^{{\displaystyle \frac{1}{\kappa }}}$ be a norm for all $u\in X_{\ell }$. Since $X_{\ell }$ is finite dimensional, all norms of $X_{\ell }$ are equivalent. Hence, we have

$$\begin{array}{cc}\hfill {\mathcal{I}}_{\lambda }\left(u\right)& ={\displaystyle \frac{a}{p}}{\left[u\right]}_{H,s,p}^p+{\displaystyle \frac{b}{2p}}{\left[u\right]}_{H,s,p}^{2p}-{\displaystyle \frac{\lambda }{\kappa }}{\int }_{{\mathbb{H}}^N}h{|u|}^{\kappa }d\xi \hfill \\ \hfill & -{\displaystyle \frac{1}{2p_{\mu ,s}^{*}}}\underset{{\mathbb{H}}^N\times {\mathbb{H}}^N}{\int \int }{\displaystyle \frac{{|u\left(\xi \right)|}^{p_{\mu ,s}^{*}}{|u\left(\eta \right)|}^{p_{\mu ,s}^{*}}}{|{\eta }^{-1}{\xi |}_H^{\mu }}}d\eta d\xi \hfill \\ \hfill & \le c_1{\parallel u\parallel }^p+c_2{\parallel u\parallel }^{2p}-\lambda c_3{\parallel u\parallel }^{\kappa }-c_4{\parallel u\parallel }^{2p_{\mu ,s}^{*}}\hfill \end{array}$$

(42)

for all $u\in X_{\ell }$.

Let $R>0$ such that

$$f\left(R\right):=c_1{R}^p+c_2{R}^{2p}-c_4{R}^{2p_{\mu ,s}^{*}}<0.$$

(43)

Taking $A:=X_{\ell }\cap \partial B_R$, we have $i\left(A\right)=\ell$. If $u\in A,$ for any $\lambda >0$, we have ${\mathcal{I}}_{\lambda }\left(u\right)\le f\left(R\right)<0$ by (42). Therefore, we obtain that ${max}_A{\mathcal{I}}_{\lambda }\le 0.$ Since the continuous function f given in (43) and $f\left(0\right)=0$, we can choose $\alpha \in (0,R)$ such that $f\left(t\right)<c^{*}$ for all $t\in [0,\alpha ]$. Suppose that

$${\lambda }_{\ell }=1+\underset{t\in [\alpha ,R]}{max}\left|{\displaystyle \frac{f\left(t\right)-c^{*}}{c_3{t}^{\kappa }}}\right|.$$

If $\lambda >{\lambda }_{\ell }$, we obtain

$$c^{*}>f\left(t\right)-\lambda c_3{t}^{\kappa },\forall t\in [\alpha ,R].$$

Therefore, for $u\in A$, the following properties are satisfied:

(a)

For $t\in [0,{\displaystyle \frac{\alpha }{R}}]$, then $\parallel \gamma u\parallel \le \alpha$, and

$${\mathcal{I}}_{\lambda }\left(\gamma u\right)\le f(\parallel \gamma u\parallel )<c^{*}.$$

(b)

For $\gamma \in [{\displaystyle \frac{\alpha }{R}},1]$, then $\parallel \gamma u\parallel \in [\alpha ,R]$, and

$${\mathcal{I}}_{\lambda }\left(\gamma u\right)\le f(\parallel \gamma u\parallel )-\lambda c_2{\parallel \gamma u\parallel }^{\kappa }<c^{*}.$$

Therefore, we obtain that

$$\underset{B}{max}{\mathcal{I}}_{\lambda }<c^{*}\mathrm{for}B=\{\gamma u\mid \gamma \in [0,1],u\in A\}.$$

By Theorem 3, we obtain that ${\mathcal{I}}_{\lambda }$ possesses m pairs of nonzero critical points. □

The application of Perera’s theorem to the fractional Choquard–Kirchhoff equation within the Heisenberg group framework not only extends the applicability of this theorem but also establishes a novel methodology for investigating nonlinear equations with critical exponents in this context. This innovative approach potentially offers valuable insights for addressing analogous problems in related research domains.

6. Conclusions

In this study, we systematically investigated a class of critical fractional Choquard–Kirchhoff equation with p-Laplacian on the Heisenberg group. Through the application of variational methods, the concentration-compactness principle on the Heisenberg group, and Perera’s critical point theorem, we rigorously established the existence and multiplicity of solutions for the problem across various parameter ranges of $\kappa$. Our findings underscore the inherent complexity and rich mathematical structure of this problem, primarily attributed to the interplay of the p-Laplacian operator, the critical nonlinear term governed by the Hardy–Littlewood–Sobolev inequality, and the inherent lack of compactness. This research not only advances the theoretical understanding of nonlinear equations on the Heisenberg group but also validates the efficacy of variational methods and critical point theory in addressing such complex problems. The obtained results establish a robust mathematical framework for modeling nonlocal interactions in quantum mechanical systems, thereby facilitating enhanced comprehension and predictive capabilities in quantum phenomena. Furthermore, as a representative nonlinear field theory model, this work holds significant theoretical implications for elucidating fundamental physical laws and developing more comprehensive field theory models.