Properties of Heavy Higgs Bosons and Dark Matter Under Current Experimental Limits in the μNMSSM

Abstract

Searches for new particles beyond the Standard Model (SM) are an important task for the Large Hadron Collider (LHC). In this paper, we investigate the properties of the heavy non-SM Higgs bosons in the $\mu$-term extended Next-to-Minimal Supersymmetric Standard Model ($\mu$NMSSM). We scan the parameter space of the $\mu$NMSSM considering the basic constraints from Higgs data, dark matter (DM) relic density, and LHC searches for sparticles. And we also consider the constraints from the LZ2022 experiment and the muon anomaly constraint at the 2$\sigma$ level. We find that the LZ2022 experiment has a strict constraint on the parameter space of the $\mu$NMSSM, and the limits from the DM-nucleon spin-independent (SI) and spin-dependent (SD) cross-sections are complementary. Then, we discuss the exotic decay modes of heavy Higgs bosons decaying into SM-like Higgs bosons. We find that for doublet-dominated Higgs $h_3$ and $A_2$, the main exotic decay channels are $h_3\to Z{A}_1$, $h_3\to h_1{h}_2$, $A_2\to A_1{h}_1$, and $A_2\to Z{h}_2$, and the branching ratio can reach to about 23%, 10%, 35%, and 10% respectively.

1. Introduction

In July 2012, both the ATLAS and CMS collaborations at the Large Hadron Collider (LHC) announced a scalar with mass near 125 GeV [1,2,3], and recently, the combined measurement of the muon anomalous magnetic moment by the Fermi National Accelerator Laboratory (FNAL) [4] and the Brookhaven National Laboratory (BNL) [5] showed a 4.2$\sigma$ discrepancy from the prediction in the Standard Model (SM). The continuously updated experimental results provide rich information about supersymmetry (SUSY). As an economic realization of SUSY, the Next-to-Minimal Supersymmetric Standard Model (NMSSM) [6,7,8,9,10] has attracted more attention. However, considering the recent experimental constraints, the parameter space of the NMSSM with a discrete $Z_3$-symmetry ($Z_3$-NMSSM) has been strictly constrained [11,12,13,14]. In order to obtain a broad parameter space that agrees with the recent experimental results, we extend the $Z_3$-NMSSM by adding an explicit $\mu$-term, which is called the $\mu$-term extended NMSSM ($\mu$NMSSM) [14,15]. Compared with $Z_3$-NMSSM, the $\mu$NMSSM can easily explain the discrepancy of the muon anomalous magnetic moment in a broad parameter space while also coinciding with the experimental results in Dark Matter (DM) and Higgs physics, as well as the LHC searches for sparticles [12,16]. In addition, the $\mu$NMSSM is free from the tadpole problem and domain-wall problem in the $Z_3$-NMSSM. An extension of the MSSM called the $\mu \nu$SSM [17] is a model similar to the $Z_3$-NMSSM except that the singlet whose vacuum expectation value (VEV) gives rise to the $\mu$ term also serves the role of a right-handed neutrino, thereby violating R-parity. Therefore, compared to the $\mu \nu$SSM, the $\mu$NMSSM can give a stable Lightest Supersymmetric Particle (LSP).

Since the discovery of a 125 GeV Standard Model (SM)-like Higgs boson at the LHC, the search for non-SM Higgs bosons has become even more pressing. In the $\mu$NMSSM, the lightest or next-to-lightest CP-even Higgs boson can be regarded as the SM-like Higgs boson. In addition to the SM-like Higgs boson ($h_1$ or $h_2$), the $\mu$NMSSM predicts another two CP-even neutral Higgs bosons ($h_1$/$h_2$ and $h_3$), two CP-odd neutral Higgs bosons ($A_1$ and $A_2$), and a pair of charged Higgs bosons ($H^{\pm }$). In this paper, we explore the discovery potential for the non-SM heavier Higgs bosons $h_3$ and $A_2$ in the $\mu$NMSSM at the LHC.

At present, besides the conventional search channels for heavy Higgs focusing on the decay modes into pairs of SM particles, the heavy Higgs exotic decay modes in the $\mu$NMSSM are kinematically open. The heavy neutral Higgs bosons can have a sizable branching ratio into two lighter neutral Higgs bosons, or into a lighter neutral Higgs boson and one Z boson. The relevant searches have been carried out at the LHC [18,19,20,21,22,23,24,25]. Ref. [26] has presented benchmark planes with cross-sections via gluon fusion for the exotic decay channels of heavy Higgs bosons in the NMSSM. And some discussions about the heavy Higgs exotic decays have also been conducted in the Two-Higgs-Doublet Model (2HDM) [27,28]. However, there have been no relevant discussions regarding the properties of the heavy Higgs boson in the $\mu$NMSSM. Therefore, our study aims to investigate the properties of the heavier CP-even Higgs boson $h_3$ and CP-odd Higgs boson $A_2$ in the $\mu$NMSSM. We focus on the searches for heavy Higgs bosons in final states with two lighter scalars, or one light scalar and a Z boson.

The outline of this paper is as follows: in Section 2, we briefly describe the relevant theoretical preliminaries of $\mu$NMSSM including the Higgs sector, the neutralino sector, and the DM-nucleon scattering cross-section. In Section 3, we give the numerical results considering the constraints of DM from the LZ experiment and investigate the properties of heavy Higgs bosons. In Section 4, we list the summary of this paper.

2. Theoretical Preliminaries

2.1. The Basics of the $\mu$NMSSM

To solve the problem in the Minimal Supersymmetric Standard Model (MSSM), such as the $\mu$ problem, the NMSSM is introduced. The NMSSM consisits of two Higgs doublet superfields, ${\widehat{H}}_u$ and ${\widehat{H}}_d$, and one singlet chiral superfield, $\widehat{S}$. After the electroweak symmetry breaking, the Higgs fields acquire the vacuum expected values (vevs); i.e., $<H_u>=v_u$, $<H_d>=v_d$, $<S>=v_s$, and $v=\sqrt{{v_u}^2+{v_d}^2}$, $tan\beta ={\displaystyle \frac{v_u}{v_d}}$. The Higgs fields in the NMSSM can be written as follows [10]:

$$\begin{array}{c}{\widehat{H}}_u=\left(\begin{array}{c}{H}_u^{+}\\ v_u+{\displaystyle \frac{1}{\sqrt{2}}}\left({\varphi }_u+i{\phi }_u\right)\end{array}\right),{\widehat{H}}_d=\left(\begin{array}{c}{v}_d+{\displaystyle \frac{1}{\sqrt{2}}}\left({\varphi }_d+i{\phi }_d\right)\\ H_d^{-}\end{array}\right),\widehat{S}=v_s+{\displaystyle \frac{1}{\sqrt{2}}}\left({\varphi }_s+i{\phi }_s\right)\end{array}$$

(1)

where ${\varphi }_u$, ${\varphi }_d$, and ${\varphi }_s$ denote the neutral CP-even Higgs fields; ${\phi }_u$, ${\phi }_d$, and ${\phi }_s$ denote the neutral CP-odd Higgs fields; and $H_u^{+}$ and $H_d^{-}$ denote the charged Higgs fields.

The general form of the superpotential in the NMSSM can be given by [10,29,30]

$$W_{\mathrm{NMSSM}}=W_{\mathrm{Yukawa}}+(\mu +\lambda \widehat{S}){\widehat{H}}_u·{\widehat{H}}_d+{\xi }_F\widehat{S}+{\displaystyle \frac{1}{2}}{\mu }^{\prime }{\widehat{S}}^2+{\displaystyle \frac{\kappa }{3}}{\widehat{S}}^3$$

(2)

where the term $W_{\mathrm{Yukawa}}$ is the same as that of the MSSM; $\mu$ and ${\mu }^{\prime }$ are bilinear mass coefficients; $\lambda$ and $\kappa$ are dimensionless coupling coefficients; ${\xi }_F$ is the supersymmetric tadpole term of mass square dimension; and the parameters $\mu$, ${\mu }^{\prime }$, and ${\xi }_F$ can be used to solve the tadpole problem and domain-wall problem in the $Z_3$-symmetry NMSSM [31,32,33,34,35].

In this work, we consider a specific scenario in which the parameters ${\mu }^{\prime }$ and ${\xi }_F$ in Equation (2) are equal to 0. This special scenario can be called the $\mu$-term extended NMSSM ($\mu$NMSSM), which is more economical than GNMSSM in explaining the SM-like Higgs mass and the properties of DM. The superpotential and the corresponding soft breaking Lagrangian can be written as follows [33,34]:

$$W_{\mu \mathrm{NMSSM}}=W_{\mathrm{Yukawa}}+(\mu +\lambda \widehat{S}){\widehat{H}}_u·{\widehat{H}}_d+{\displaystyle \frac{\kappa }{3}}{\widehat{S}}^3,$$

(3)

$$\begin{array}{cc}\hfill -{\mathcal{L}}_{\mathrm{soft}}& =\left[A_{\lambda }\lambda S{H}_u·H_d+{\displaystyle \frac{1}{3}}{A}_{\kappa }\kappa S^3+B_{\mu }\mu H_u·H_d+h.c.\right]+m_{H_u}^2{\left|H_u\right|}^2+m_{H_d}^2{\left|H_d\right|}^2+m_S^2{\left|S\right|}^2,\hfill \end{array}$$

(4)

where $H_u$, $H_d$, and S are the scalar parts of the superfields ${\widehat{H}}_u$, ${\widehat{H}}_d$, and $\widehat{S}$, respectively. By solving the minimization equation of the scalar potential, the soft breaking mass parameters $m_{H_u}^2$, $m_{H_d}^2$, and $m_S^2$ can be expressed in terms of the vacuum expected values of the scalar field. To simplify the calculation, we set $B_{\mu }$ to be 0. Therefore, the Higgs sector is partially determined by the following parameters:

$$tan\beta ,{\mu }_{\mathrm{eff}}=\lambda v_s/\sqrt{2},\lambda ,\kappa ,A_{\lambda },A_{\kappa },\mu .$$

(5)

For convenience, we define $H_{\mathrm{SM}}\equiv sin\beta \mathrm{Re}\left(H_u^0\right)+cos\beta \mathrm{Re}\left(H_d^0\right)$, $H_{\mathrm{NSM}}\equiv cos\beta \mathrm{Re}\left(H_u^0\right)-sin\beta \mathrm{Re}\left(H_d^0\right)$, and $A_{\mathrm{NSM}}\equiv cos\beta \mathrm{Im}\left(H_u^0\right)-sin\beta \mathrm{Im}\left(H_d^0\right)$, where $H_{\mathrm{SM}}$ is the SM Higgs field and its vev is $v/\sqrt{2}$, $H_{\mathrm{NSM}}$ is the other CP-even doublet Higgs field and its vev is zero, and $A_{\mathrm{NSM}}$ corresponds to the CP-odd Higgs boson in the MSSM [36,37]. In the basis $(H_{\mathrm{NSM}},H_{\mathrm{SM}},\mathrm{Re}\left(S\right))$, the elements of the CP-even Higgs mass symmetric matrix $M_S^2$ can be written as

$$\begin{array}{cc}\hfill M_{S,11}^2& ={\displaystyle \frac{2{\mu }_{eff}(\lambda A_{\lambda }+\kappa {\mu }_{eff})}{\lambda sin2\beta }}+{\displaystyle \frac{1}{2}}(2m_Z^2-{\lambda }^2{v}^2){sin}^{2}2\beta ,\hfill \\ \hfill M_{S,12}^2& =-{\displaystyle \frac{1}{4}}(2m_Z^2-{\lambda }^2{v}^2)sin4\beta ,\hfill \\ \hfill M_{S,13}^2& =-{\displaystyle \frac{1}{\sqrt{2}}}(\lambda A_{\lambda }+2\kappa {\mu }_{eff})vcos2\beta ,\hfill \\ \hfill M_{S,22}^2& =m_Z^2{cos}^{2}2\beta +{\displaystyle \frac{1}{2}}{\lambda }^2{v}^2{sin}^{2}2\beta ,\hfill \\ \hfill M_{S,23}^2& ={\displaystyle \frac{v}{\sqrt{2}}}[2\lambda ({\mu }_{eff}+\mu )-(\lambda A_{\lambda }+2\kappa {\mu }_{eff})sin2\beta ],\hfill \\ \hfill M_{S,33}^2& ={\displaystyle \frac{\lambda A_{\lambda }sin2\beta }{4{\mu }_{eff}}}\lambda v^2+{\displaystyle \frac{{\mu }_{eff}}{\lambda }}(\kappa A_{\kappa }+{\displaystyle \frac{4{\kappa }^2{\mu }_{eff}}{\lambda }})-{\displaystyle \frac{\mu }{2{\mu }_{eff}}}{\lambda }^2{v}^2.\hfill \end{array}$$

(6)

And the elements of the CP-odd Higgs mass symmetric matrix $M_P^2$ under the basis $(A_{NSM},\mathrm{Im}\left(S\right))$ is given by

$$\begin{array}{cc}\hfill & M_{P,11}^2={\displaystyle \frac{2{\mu }_{eff}(\lambda A_{\lambda }+\kappa {\mu }_{eff})}{\lambda sin2\beta }},\hfill \\ \hfill & M_{P,22}^2={\displaystyle \frac{(\lambda A_{\lambda }+4\kappa {\mu }_{eff})sin2\beta }{4{\mu }_{eff}}}\lambda v^2-{\displaystyle \frac{3\kappa A_{\kappa }{\mu }_{eff}}{\lambda }}-{\displaystyle \frac{\mu }{2{\mu }_{eff}}}{\lambda }^2{v}^2,\hfill \\ \hfill & M_{P,12}^2={\displaystyle \frac{v}{\sqrt{2}}}(\lambda A_{\lambda }-2\kappa {\mu }_{eff}).\hfill \end{array}$$

(7)

By diagonalizing $M_S^2$ and $M_P^2$ using the unitary matrix V and U, we can obtain the CP-even Higgs mass eigenstate $h_i(i=1,2,3)$ with $m_{h_1}<m_{h_2}<m_{h_3}$, and CP-odd Higgs mass eigenstate $A_i(i=1,2)$ with $m_{A_1}<m_{A_2}$, respectively [34,38,39].

$$\begin{array}{cc}\hfill & h_i=V_{h_i}^{\mathrm{NSM}}{H}_{\mathrm{NSM}}+V_{h_i}^{\mathrm{SM}}{H}_{\mathrm{SM}}+V_{h_i}^S\mathrm{Re}\left(S\right),\hfill \\ \hfill & A_i=U_{A_i}^{\mathrm{NSM}}{A}_{\mathrm{NSM}}+U_{A_i}^S\mathrm{Im}\left(S\right).\hfill \end{array}$$

(8)

Each of the three CP-even Higgs bosons $h_i(i=1,2,3)$ can be either SM-like (h), or $H_{\mathrm{NSM}}$ dominant (H), or singlet dominant ($h_s$). Likewise, each of the two CP-odd Higgs bosons $A_i$ can be either singlet dominant ($A_s$), or $H_{\mathrm{NSM}}$ dominant ($A_H$).

The mass eigenstate of charged Higgs bosons is $H^{\pm }=cos\beta H_u^{\pm }+sin\beta H_d^{\pm }$, and their masses can be written as

$$\begin{array}{cc}\hfill & m_{H^{\pm }}^2={\displaystyle \frac{2{\mu }_{eff}(\lambda A_{\lambda }+\kappa {\mu }_{eff})}{\lambda sin2\beta }}+m_W^2-{\lambda }^2{v}^2.\hfill \end{array}$$

(9)

For neutralino sector, the neutralino mass eigenstate in the basis of ${\psi }^0=\left(-i{\tilde{B}}^0,-i{\tilde{W}}^0,{\tilde{H}}_d^0,{\tilde{H}}_u^0,{\tilde{S}}^0\right)$ is

$$M_{\tilde{N}}=\left(\begin{array}{ccccc}{M}_1& 0& -m_{Z}sin{\theta }_{\mathrm{w}}cos\beta & m_{Z}sin{\theta }_{\mathrm{w}}sin\beta & 0\\ 0& M_2& m_{Z}cos{\theta }_{\mathrm{w}}cos\beta & -m_{Z}cos{\theta }_{\mathrm{w}}sin\beta & 0\\ -m_{Z}sin{\theta }_{\mathrm{w}}cos\beta & m_{Z}cos{\theta }_{\mathrm{w}}cos\beta & 0& -{\mu }_{\mathrm{tot}}& -{\displaystyle \frac{1}{\sqrt{2}}}\lambda vsin\beta \\ m_{Z}sin{\theta }_{\mathrm{w}}sin\beta & -m_{Z}cos{\theta }_{\mathrm{w}}sin\beta & -{\mu }_{\mathrm{tot}}& 0& -{\displaystyle \frac{1}{\sqrt{2}}}\lambda vcos\beta \\ 0& 0& -{\displaystyle \frac{1}{\sqrt{2}}}\lambda vsin\beta & -{\displaystyle \frac{1}{\sqrt{2}}}\lambda vcos\beta & 2{\displaystyle \frac{\kappa }{\lambda }}{\mu }_{\mathrm{eff}}\end{array}\right),$$

(10)

where ${\mu }_{\mathrm{tot}}\equiv \mu +{\mu }_{\mathrm{eff}}$, and $M_1$ and $M_2$ are Bino and Wino soft breaking masses, respectively. After diagonalizing the mass matrix $M_{\tilde{N}}$ by rotation matrix N, we can obtain the neutralino mass eigenstate ${\tilde{\chi }}_i^0$ (i = 1, 2, 3, 4, 5) labeled in mass-ascending order, which can be expressed as

$${\tilde{\chi }}_i^0=N_{ij}{\psi }_j^0(j=1,2,3,4,5).$$

(11)

Assuming the lightest neutralino ${\tilde{\chi }}_1^0$ is the LSP, which can be considered as an ideal candidate for DM. Evidently, $N_{11}^2$, $N_{12}^2$, $N_{13}^2+N_{14}^2$ and $N_{15}^2$ denote the Bino, Wino, Higgsino, and Singlino fractions in ${\tilde{\chi }}_1^0$, respectively. Different from the case in the $Z_3$-NMSSM, 2|$\kappa$| may be much larger than $\lambda$ in obtaining Singlino-dominated DM.

2.2. The Heavy Higgs Bosons

In this work we require the lightest CP-even Higgs boson $h_1$ is SM-like, and investigate the properties of the heavy Higgs bosons $h_3$ and $A_2$. At the LHC, the heavy Higgs boson H ($h_3$ or $A_2$) is mainly produced through gluon-gluon fusion (ggF), and the production cross-section can be obtained by

$${\displaystyle \frac{\sigma (ggF\to H)}{\sigma (ggF\to h_{SM})}}={\left|C_{ggF}^H\right|}^2,$$

(12)

where $h_{\mathrm{SM}}$ denotes the SM-like Higgs boson, and $C_{ggF}^H$ is the reduced coupling coefficient relative to the prediction in the SM. In the $\mu$NMSSM, the exotic decay modes of heavy Higgs bosons are open and heavy Higgs bosons $h_3$ and $A_2$ can have sizable branching ratio into two lighter Higgs bosons, e.g., $h_3\to h_1{h}_2$, $A_2\to A_1{h}_1$, which can be called Higgs-to-Higgs decays. In addition, heavy Higgs bosons $h_3$ and $A_2$ may decay into one light Higgs boson and a Z boson, e.g., $h_3\to A_{1}Z$, $A_2\to h_{1}Z$. The branching ratio of the Higgs-to-Higgs decays depends on trilinear Higgs couplings. For the typical case with $v_s$, $A_{\lambda }\gg v_u,v_d\approx M_Z$, the relevant trilinear Higgs couplings relative to Higgs-to-Higgs decays can be expressed by (neglecting contributions of $\mathcal{O}\left(M_Z\right)$) [10,26] the following:

$$\begin{array}{cc}\hfill & \left(1\right)H_{\mathrm{NSM}}{H}_{\mathrm{SM}}\mathrm{Re}\left(S\right):-{\displaystyle \frac{\lambda }{\sqrt{2}}}(2\kappa v_s+A_{\lambda }),\hfill \\ \hfill & \left(2\right)A_{\mathrm{NSM}}{H}_{\mathrm{SM}}\mathrm{Im}\left(S\right):\sqrt{2}\lambda sin\beta cos\beta (-2\kappa v_s+A_{\lambda }).\hfill \end{array}$$

(13)

For the decays $h_i\to A_j+Z$ and $A_j\to h_i+Z$, the relevant couplings are

$$h_i\left(p\right)A_j\left(p^{\prime }\right)Z_{\mu }:-ig{V}_{h_i}^{\mathrm{NSM}}{U}_{A_i}^{\mathrm{NSM}}{(p-p^{\prime })}_{\mu }$$

(14)

where $V_{h_i}^{\mathrm{NSM}}$ is the $H_{\mathrm{NSM}}$ component of the physical state $h_i$; and $U_{A_i}^{\mathrm{NSM}}$ is the $A_{\mathrm{NSM}}$ component of the physical state $A_j$.

2.3. The Anomalous Magnetic Moment of the Muon in the $\mu$NMSSM

The recent measurement of the muon anomalous magnetic moment $a_{\mu }^{exp}$ by the FNAL has been updated, and its value is [4]

$$a_{\mu }^{exp}\left(\mathrm{FNAL}\right)=116592040\left(54\right)\times {10}^{-11}.$$

(15)

The result $a_{\mu }^{exp}\left(\mathrm{FNAL}\right)$ is in full agreement with the BNL E821 result $a_{\mu }^{exp}\left(\mathrm{BNL}\right)$ [5]:

$$a_{\mu }^{exp}\left(\mathrm{BNL}\right)=116592080\left(63\right)\times {10}^{-11}.$$

(16)

And the combined experimental average $a_{\mu }^{exp}$ is [40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57]

$$a_{\mu }^{exp}=116592061\left(41\right)\times {10}^{-11}.$$

(17)

The latest lattice calculations have led to a prediction that differs from the experimental result of $a_{\mu }$ by only $0.9\sigma$ [58]. While the current value for the anomaly, combining the latest Standard Model prediction [40] and the improved experimental result [59], is $\mathrm{\Delta }{a}_{\mu }^{\exp }=(249\pm 48)\times {10}^{-11}$. In our analysis, we use the same value for the discrepancy between theory and experiment as used in ref. [60], namely, $\mathrm{\Delta }{a}_{\mu }^{\exp }=(251\pm 59)\times {10}^{-11}$ [4], in order to make the comparison systematic.

In SUSY, the contributions to $a_{\mu }^{\mathrm{SUSY}}$ mainly originate from the loops mediated by a smuon and a neutralino or a chargino and a muon-type sneutrino [61,62,63,64,65,66,67,68,69,70,71]. In the $\mu$NMSSM, the one-loop contributions to $a_{\mu }^{\mathrm{SUSY}}$ can be written as [72]

$$\begin{array}{c}{a}_{\mu }^{SUSY}=a_{\mu }^{{\tilde{\chi }}^0\tilde{\mu }}+a_{\mu }^{{\tilde{\chi }}^{\pm }\tilde{\nu }},\\ a_{\mu }^{{\tilde{\chi }}^0\tilde{\mu }}={\displaystyle \frac{m_{\mu }}{16{\pi }^2}}{\displaystyle \sum _{i,l}}\left\{-{\displaystyle \frac{m_{\mu }}{12m_{{\tilde{\mu }}_l}^2}}\left({\left|n_{il}^{\mathrm{L}}\right|}^2+{\left|n_{il}^{\mathrm{R}}\right|}^2\right)F_1^{\mathrm{N}}\left(x_{il}\right)+{\displaystyle \frac{m_{{\tilde{\chi }}_i^0}}{3m_{{\tilde{\mu }}_l}^2}}Re\left(n_{il}^{\mathrm{L}}{n}_{il}^{\mathrm{R}}\right)F_2^{\mathrm{N}}\left(x_{il}\right)\right\},\\ a_{\mu }^{{\tilde{\chi }}^{\pm }\tilde{\nu }}={\displaystyle \frac{m_{\mu }}{16{\pi }^2}}{\displaystyle \sum _k}\left\{{\displaystyle \frac{m_{\mu }}{12m_{{\tilde{\nu }}_{\mu }}^2}}\left({\left|c_k^{\mathrm{L}}\right|}^2+{\left|c_k^{\mathrm{R}}\right|}^2\right)F_1^{\mathrm{C}}\left(x_k\right)+{\displaystyle \frac{2m_{{\tilde{\chi }}_k^{\pm }}}{3m_{{\tilde{\nu }}_{\mu }}^2}}Re\left(c_k^{\mathrm{L}}{c}_k^{\mathrm{R}}\right)F_2^{\mathrm{C}}\left(x_k\right)\right\},\end{array}$$

(18)

where $i=1,2,3,4,5$, $j=1,2$, $l=1,2$ represent the neutralino, chargino, and smuon index, respectively. And

$$\begin{array}{c}{n}_{il}^{\mathrm{L}}={\displaystyle \frac{1}{\sqrt{2}}}\left(g_2{N}_{i2}+g_1{N}_{i1}\right)X_{l1}^{\ast }-y_{\mu }{N}_{i3}{X}_{l2}^{\ast },n_{il}^{\mathrm{R}}=\sqrt{2}{g}_1{N}_{i1}{X}_{l2}+y_{\mu }{N}_{i3}{X}_{l1},\\ c_k^{\mathrm{L}}=-g_2{V}_{k1}^{\mathrm{c}},c_k^{\mathrm{R}}=y_{\mu }{U}_{k2}^{\mathrm{c}},x_{il}=m_{{\tilde{\chi }}_i^0}^2/m_{{\tilde{\mu }}_l}^2,x_k=m_{{\tilde{\chi }}_k^{\pm }}^2/m_{{\tilde{\nu }}_{\mu }}^2,\end{array}$$

(19)

where X denotes the smuon mass rotation matrices; and $U^C$ and $V^C$ denote the chargino mass rotation matrix. $F_1\left(x\right)$ and $F_2\left(x\right)$ are loop functions of the kinematic variables $x_{il}$ and $x_k$, and their expressions are written as [73,74] follows:

$$\begin{array}{cc}\hfill & F_1^N\left(x\right)={\displaystyle \frac{2}{{(1-x)}^4}}\left[1-6x+3x^2+2x^3-6x^{2}lnx\right],\hfill \\ \hfill & F_2^N\left(x\right)={\displaystyle \frac{3}{{(1-x)}^3}}\left[1-x^2+2xlnx\right],\hfill \\ \hfill & F_1^C\left(x\right)={\displaystyle \frac{2}{{(1-x)}^4}}\left[2+3x-6x^2+x^3+6xlnx\right],\hfill \\ \hfill & F_2^C\left(x\right)=-{\displaystyle \frac{3}{2{(1-x)}^3}}\left[3-4x+x^2+2lnx\right].\hfill \end{array}$$

(20)

For the scenario with mass-degenerate sparticles, the relationship $F_1^N\left(1\right)=F_2^N\left(1\right)=F_1^C\left(1\right)=F_2^C\left(1\right)=1$ holds.

2.4. DM-Nucleon Scattering Cross-Section

In the $\mu$NMSSM, the lightest neutralino ${\tilde{\chi }}_1^0$ as the LSP can be considered as a DM candidate [75,76]. The Higgsino fraction of ${\tilde{\chi }}_1^0$ plays an important role in elastic scattering between ${\tilde{\chi }}_1^0$ and nucleon. In the scenario with massive squarks, the Spin-Dependent (SD) scattering of ${\tilde{\chi }}_1^0$ with nucleons is mediated by exchanging a Z boson, and the scattering cross-section is approximated given by [77,78]

$${\sigma }_{{\tilde{\chi }}_1^0-N}^{\mathrm{SD}}\simeq C_N\times {\left({\displaystyle \frac{N_{13}^2-N_{14}^2}{0.1}}\right)}^2,$$

(21)

where $N=p\left(n\right)$, denoting protons (neutrons), $C_p\simeq 4\times {10}^{-4}$ pb, $C_n\simeq 3.1$ pb, and

$$N_{13}^2-N_{14}^2={\left({\displaystyle \frac{\lambda v}{\sqrt{2}{\mu }_{tot}}}\right)}^2{\displaystyle \frac{N_{15}^{2}cos2\beta }{1-{(m_{{\tilde{\chi }}_1^0}/{\mu }_{tot})}^2}}.$$

(22)

The spin-independent (SI) scattering of ${\tilde{\chi }}_1^0$ with nucleons is mainly produced by exchanging CP-even Higgs bosons through the t-channel, and the cross-section is as follows [15,79,80]:

$${\sigma }_{{\tilde{\chi }}_1^0-N}^{SI}={\displaystyle \frac{4{\tilde{\mu }}_R^2}{\pi }}{\left|f^{\left(N\right)}\right|}^2,$$

(23)

where ${\tilde{\mu }}_R\equiv m_N{m}_{{\tilde{\chi }}_1^0}/(m_N+m_{{\tilde{\chi }}_1^0})$, denoting the reduced mass of the DM-nucleon system. The expressions of the effective couplings $f^{\left(N\right)}$ are

$$f^{\left(N\right)}=\sum _{h_i=h,H,h_S}{f}_{h_i}^{\left(N\right)}=\sum _{h_i=h,H,h_S}{\displaystyle \frac{C_{{\tilde{\chi }}_1^0{\tilde{\chi }}_1^0{h}_i}{C}_{NN{h}_i}}{2m_{h_i}^2}},$$

(24)

with $C_{NN{h}_i}$ being the coupling coefficient between the CP-even Higgs bosons and nucleon,

$$C_{NN{h}_i}=-{\displaystyle \frac{m_N}{v}}\left[F_d^{\left(N\right)}(V_{h_i}^{\mathrm{SM}}-tan\beta V_{h_i}^{\mathrm{NSM}})+F_u^{\left(N\right)}\left(V_{h_i}^{\mathrm{SM}}+{\displaystyle \frac{1}{tan\beta }}{V}_{h_i}^{\mathrm{NSM}}\right)\right],$$

(25)

where $F_d^{\left(N\right)}=f_d^{\left(N\right)}+f_s^{\left(N\right)}+{\displaystyle \frac{2}{27}}{f}_G^{\left(N\right)}$ and $F_u^{\left(N\right)}=f_u^{\left(N\right)}+{\displaystyle \frac{4}{27}}{f}_G^{\left(N\right)}$. The form factors $f_q^{\left(N\right)}=m_N^{-1}〈N|m_{q}q\overline{q}|N〉$ ($q=u,d,s$) denote the normalized light quarks contribution to the nucleon mass, and $f_G^{\left(N\right)}=1-{\sum }_{q=u,d,s}{f}_q^{\left(N\right)}$ represents other heavy quarks mass fraction in the nucleon.

3. Numerical Results

The parameter space of the $\mu$NMSSM has been scanned by EasyScan_HEP [81] with the Metropolis–Hastings algorithm:

$$\begin{array}{c}\left|M_1\right|\le 1.5\mathrm{TeV},100\mathrm{GeV}\le M_2\le 1.5\mathrm{TeV},\\ 0\le \lambda \le 0.75,\left|\kappa \right|\le 0.75,1\le tan\beta \le 60,2\mathrm{TeV}\le \left|A_t\right|\le 5\mathrm{TeV},\\ 10\mathrm{GeV}\le \mu \le 1\mathrm{TeV},100\mathrm{GeV}\le {\mu }_{\mathrm{tot}}\le 1\mathrm{TeV},\left|A_{\kappa }\right|\le 700\mathrm{GeV},\\ 100\mathrm{GeV}\le m_{{\tilde{\mu }}_L}\le 1\mathrm{TeV},100\mathrm{GeV}\le m_{{\tilde{\mu }}_E}\le 1\mathrm{TeV}.\end{array}$$

(26)

For other supersymmetric parameters, we fix them at 2 TeV. We use the package SARAH-4.14.3 [82,83,84,85] to generate the model files in the $\mu$NMSSM, use the program SPheno-4.0.4 [86,87] to obtain the particle spectrum, and use the package MicrOMEGAs-5.2.13 [88,89,90,91,92,93,94,95,96,97] to calculate the DM observables.

To be specific, we require the samples to satisfy the following basic constraints:

1. The lightest CP-even Higgs boson $h_1$ should be SM-like, and its mass should be between 121 GeV and 129 GeV. We utilize the code HiggsSignals-2.2.3 [98] to fit the properties of the SM-like Higgs boson to LHC Higgs data and utilize the code HiggsBounds-5.3.2 [99] to implement the constraints from the direct search for extra Higgs bosons at the LEP and Tevatron.

2. We assume the lightest neutralino is one of the DM candidates, so when comparing the dark matter scattering cross-section below with the experimental limit, we need the DM relic density $\mathrm{\Omega }{h}^2<0.120$ [100]. The SI and SD DM cross-sections should be scaled by a factor $\mathrm{\Omega }{h}^2/0.120$.

3. We consider the constraints from the direct detection experiments for sparticles at the LHC, and use SModelS-2.3.2 [101,102,103,104] to decompose the spectrum including these processes:

$$\begin{array}{c}\hfill pp\to {\tilde{\chi }}_1^{+}{\tilde{\chi }}_1^{-},pp\to {\tilde{\chi }}_1^{\pm }{\tilde{\chi }}_{1,2,3}^0,pp\to {\tilde{\mu }}_L{\tilde{\mu }}_L^{\ast },pp\to {\tilde{\mu }}_R{\tilde{\mu }}_R^{\ast }.\end{array}$$

(27)

The Next-to-Leading Order (NLO) cross-sections of these processes at $\sqrt{s}$ = 13 TeV are calculated by Prospino2.1 [105]. In the following discussions, all the surviving samples satisfy these basic constraints.

3.1. Properties of Dark Matter

We project all the surviving samples from the scan onto a two-dimensional diagram, as shown below. The surviving samples are divided into three categories by three different colors: the purple samples satisfy the basic constraints mentioned above; the yellow samples satisfy not only the basic constraints but also the muon anomaly constraint within the 2$\sigma$ level; and the red samples satisfy the basic constraints, the muon anomaly constraint within the 2$\sigma$ level, and also the LZ experiment constraint in the year 2022 (LZ2022) [106].

Figure 1 shows the surviving samples on the plane $m_{{\tilde{\chi }}_1^0}-{\sigma }_p^{SI}$ and $m_{{\tilde{\chi }}_1^0}-{\sigma }_n^{SD}$. The green line on the left (right) plot is the upper limit of SI (SD) nucleon-DM cross-sections, which comes from the results of the recent LZ2022 experiment. The samples above the green line are excluded by the LZ2022 experiment. From this figure, we conclude that the results of recent nucleon-DM experiments impose strong constraints on the parameter space in the $\mu$NMSSM, and the SD limit is complementary to the SI limit in limiting the parameter space in the $\mu$NMSSM. The figure also reveals 100 GeV $\lesssim m_{{\tilde{\chi }}_1^0}\lesssim$ 400 GeV considering the constraints from the recent LZ2022 experiment.

We display the characteristics of DM components in Figure 2. The upper left plot exhibits the Bino-component $N_{11}^2$, the upper right plot exhibits the Wino-component $N_{12}^2$, the lower left plot shows the Higgsino-component $N_{13}^2+N_{14}^2$, and the lower right plot shows the Singlino-component $N_{15}^2$. Considering the constraints from the anomaly of ${(g-2)}_{\mu }$ and the recent LZ2022 experiment, the dark matter are mainly Wino-dominated or Higgsino-dominated. A few samples are Bino-dominated, but no samples are Singlino-dominated. The mass of Wino-dominated DM is less than 300 GeV, the mass of Higgsino-dominated DM is less than 350 GeV, and the mass of Bino-dominated DM is less than 400 GeV.

To investigate the properties of the surviving parameter space, we pick out the red samples in Figure 1 and Figure 2 and project them onto $|{\mu }_{tot}/{\mu }_{eff}|-2|\kappa |/\lambda$, $m_{{\tilde{\chi }}_1^0}-tan\beta$, $M_2-{\mu }_{tot}$ planes in Figure 3. For the Higgsino-dominated DM, 2$\left|\kappa \right|/\lambda$ is much larger than $|{\mu }_{tot}/{\mu }_{eff}|$ as can be seen from the left plot, which is significantly different from the scenario in the $Z_3$-NMSSM. In the $Z_3$-NMSSM, Higgsino-dominated DM only requires 2$\left|\kappa \right|/\lambda$ are larger than 1. The middle plot shows that $tan\beta$ is greater than 20 for the Wino-dominated or Higgsino-dominated DM. From the right plot, we can see that the survival samples mainly tend to be 850 GeV $\lesssim M_2\lesssim$ 1500 GeV and 100 GeV $\lesssim {\mu }_{\mathrm{tot}}\lesssim$ 300 GeV for Higgsino-dominated DM, and 100 GeV $\lesssim M_2\lesssim$ 300 GeV and 800 GeV $\lesssim {\mu }_{\mathrm{tot}}\lesssim$ 1000 GeV for the Wino-dominated DM.

3.2. Properties of Heavy Higgs Bosons

We pick out the survival samples satisfying the basic constraints mentioned above, and also the constraints from the anomaly of ${(g-2)}_{\mu }$ and the limit of SI (SD) nucleon-DM cross-sections to investigate the properties of heavy Higgs bosons $h_3$ and $A_2$. In Figure 4, we show the singlet component of the non-SM CP-even and CP-odd Higgs bosons. From the upper plots, we can see that for most of the survival samples, the next-to-lightest CP-even Higgs boson $h_2$ can be mostly singlet-dominated or doublet-dominated. Correspondingly, the heaviest CP-even Higgs boson $h_3$ can be mostly doublet-dominated or singlet-dominated. But for a portion of the samples, singlet-doublet mixing can be large. The lower plots show that for most of the surviving samples, the lightest CP-odd Higgs boson $A_1$ is mostly singlet-dominated and the heaviest CP-odd Higgs boson $A_2$ is mostly doublet-dominated. However, for a part of the samples, singlet-doublet mixing can be large. And for a small fraction of the samples, $A_2$ can be mostly singlet-dominated.

As discussed above, the exotic decay channels of heavy Higgs bosons $h_3$ and $A_2$ are open. In Figure 5 and Figure 6 we show the exotic decay channels of $h_3$ and $A_2$, and we only consider the decay channels of heavy Higgs boson decaying into lighter Higgs boson. The left (right) plot of Figure 5 shows that $h_3$ is doublet-dominated (singlet-dominated), and the left (right) plot of Figure 6 shows $A_2$ is doublet-dominated (singlet-dominated). For the doublet-dominated Higgs boson $h_3$, the main decay channels are $h_3\to Z{A}_1$ and $h_3\to h_1{h}_2$, and the branching ratio can reach about 23% and 10%, respectively. The decay $h_3\to Z{A}_1$ is proportional to the $A_{\mathrm{NSM}}$ component of the physical state $A_1$. The large branching ratio of $h_3\to Z{A}_1$ just corresponds to the scenario that the doublet component of $A_1$ is relatively large. The decay $h_3\to h_1{h}_2$ is proportional to the Higgs trilinear coupling shown in the first equation of Equation (13), which is usually relatively large when the mixing between doublet and singlet scalar fields is large. The singlet-dominated Higgs boson $h_3$ mainly decays into $A_1{A}_1$, and the branching ratio of $h_3\to A_1{A}_1$ can reach to about 1.

Figure 6 shows that for the doublet-dominated Higgs boson $A_2$, the main decay channels of the Higgs boson $A_2$ are $A_2\to A_1{h}_1$ and $A_2\to Z{h}_2$, and the branching ratio can reach to about 35% and 10%, respectively. The decay $A_2\to A_1{h}_1$ is proportional to the Higgs trilinear coupling shown in the second equation of Equation (13), which is usually relatively large when the mixing between doublet and singlet pseudoscalar fields is large, as the off-diagonal element $M_{P,12}^2$ shown. The decay $A_2\to Z{h}_2$ is proportional to the $H_{\mathrm{NSM}}$ component of the physical state $h_2$. The large branching ratio of $A_2\to Z{h}_2$ just corresponds to the scenario that the doublet component of $h_2$ is relatively large. The branching ratio of the decay $A_2\to Z{h}_1$ approaches 0 because the $H_{\mathrm{NSM}}$ component of the SM-like $h_1$ is much lower. The main decay channels of the singlet-dominated Higgs boson $A_2$ are $A_2\to A_1{h}_1$ and $A_2\to Z{h}_2$.

Since the production cross-section of singlet-dominated Higgs bosons at the LHC is very small, we only consider the production of doublet-dominated Higgs bosons $h_3$ and $A_2$. In Figure 7 and Figure 8, we show the production cross-section of the Higgs bosons $h_3$ and $A_2$ with $h_3$ and $A_2$ decaying into the SM-like Higgs at $\sqrt{s}=13$ TeV LHC. We find that the cross-sections $ggF\to h_3\to h_1{h}_2$ and $ggF\to A_2\to A_1{h}_1$ can reach to about ${10}^{-11}$ pb and ${10}^{-10}$ pb, respectively.

4. Summary

In this paper, we have performed phenomenological studies on the properties of dark matter and heavy Higgs bosons $h_3$ and $A_2$ in the $\mu$NMSSM. Considering the basic constraints from Higgs data, DM relic density, and LHC searches for sparticles, we have scanned the parameter space of the $\mu$NMSSM. We find that the LZ2022 experiment has a strict constraint on the parameter space of the $\mu$NMSSM, and the limits from the DM-nucleon SI and SD cross-sections are complementary. Samples surviving the LZ2022 experiment and the muon anomaly constraint at the 2$\sigma$ level are mainly featured by $tan\beta \lesssim 20$, 850 GeV $\lesssim M_2\lesssim$ 1500 GeV, and 100 GeV $\lesssim {\mu }_{\mathrm{tot}}\lesssim$ 300 GeV for Higgsino-dominated DM, or 100 GeV $\lesssim M_2\lesssim$ 300 GeV, and 800 GeV $\lesssim {\mu }_{\mathrm{tot}}\lesssim$ 1000 GeV for Wino-dominated DM.

The detections of heavy Higgs bosons through exotic decay modes into SM-like Higgs bosons are important for analyzing the Higgs signals. We find that for doublet-dominated Higgs $h_3$, and $A_2$, the main exotic decay channels are $h_3\to Z{A}_1$, $h_3\to h_1{h}_2$, $A_2\to A_1{h}_1$ and $A_2\to Z{h}_2$, and the branching ratio can reach about 23%, 10%, 35%, and 10%, respectively. At the 13 TeV LHC, the production cross-section of processes $ggF\to h_3\to h_1{h}_2$ and $ggF\to A_2\to A_1{h}_1$ can reach to about ${10}^{-11}$ pb and ${10}^{-10}$ pb, respectively. This spectrum is hardly tested, but it is free from current constraints from the LHC on exotic Higgs. It is unfortunate that these heavy Higgs still cannot be tested at the High-Luminosity LHC (HL-LHC) [107]; one has to wait for the next-generation hadron colliders (such as the SPPC [108] and FCC-hh [109]) in order to investigate this parameter space.